Audio interaction processing methods and system, server, client and electronic device

Abstract

Audio interaction processing methods and system, a server, a client and an electronic device. A method is executed by an audio processing server, and comprises: converting an input audio sent by a client into a text to be replied to (S102); sending to a text processing server the text to be replied to, so as to generate a reply text stream (S104); receiving a request of the client for a reply audio stream corresponding to the reply text stream (S106); on the basis of a message identifier, acquiring the reply text stream from the text processing server (S108); generating a reply audio stream on the basis of the reply text stream (S110); and sending the reply audio stream to the client for playing (S112).

Description

Method, system, server, client and electronic device for processing audio interaction TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular to a method, system, server, client and electronic device for processing audio interaction. BACKGROUND

[0002] With the development of Internet technology, human-computer voice interaction technology is applied in more and more scenarios. For example, smart home, intelligent customer service, intelligent assistant, etc.

[0003] In some human-computer voice interaction scenarios, the user inputs voice, not only gets the voice of the robot reply, but also generates the corresponding text based on the voice input by the user, and the text of the robot reply is also displayed accordingly, so that the user can receive the reply through the text and voice, better understand the content of the reply, and improve the user experience.

[0004] SUMMARY

[0005] This summary is provided to introduce a selection of concepts, which are further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in limiting the scope of the claimed subject matter.

[0006] According to some embodiments of the present disclosure, a method for processing audio interaction is provided, executed by an audio processing server, comprising: converting input audio sent by a client into a to-be-replied text; sending the to-be-replied text to a text processing server to generate a reply text stream; receiving a request of the client for a reply audio stream corresponding to the reply text stream, wherein the request comprises a message identifier corresponding to the reply text stream, and the message identifier is sent by the text processing server to the client; obtaining the reply text stream from the text processing server according to the message identifier; generating the reply audio stream according to the reply text stream; and sending the reply audio stream to the client for playing.

[0007] According to some other embodiments of the present disclosure, a method for processing audio interaction is provided, executed by a client, comprising: sending input audio to an audio processing server to convert into a to-be-replied text, wherein the to-be-replied text is sent to a text processing server to generate a reply text stream; receiving a message identifier corresponding to the reply text stream sent by the text processing server; sending a request of the client for a reply audio stream corresponding to the reply text stream to the audio processing server, wherein the request comprises the message identifier; receiving the reply audio stream sent by the audio processing server; and playing the reply audio stream.

[0008] According to still some embodiments of the present disclosure, an audio processing server is provided, comprising: a conversion module configured to convert input audio sent by a client into to-be-replied text; a first sending module configured to send the to-be-replied text to a text processing server to generate a reply text stream; a receiving module configured to receive a request of the client for a reply audio stream corresponding to the reply text stream, wherein the request comprises a message identifier corresponding to the reply text stream, and the message identifier is sent by the text processing server to the client; an obtaining module configured to obtain the reply text stream from the text processing server according to the message identifier; a generating module configured to generate the reply audio stream according to the reply text stream; and a second sending module configured to send the reply audio stream to the client for playing.

[0009] According to still some embodiments of the present disclosure, a client is provided, comprising: a first sending module configured to send input audio to an audio processing server to convert the input audio into to-be-replied text, wherein the to-be-replied text is sent to a text processing server to generate a reply text stream; a first receiving module configured to receive a message identifier corresponding to the reply text stream sent by the text processing server; a second sending module configured to send a request of the client for a reply audio stream corresponding to the reply text stream to the audio processing server, wherein the request comprises the message identifier; a second receiving module configured to receive the reply audio stream sent by the audio processing server; and a playing module configured to play the reply audio stream.

[0010] According to still some embodiments of the present disclosure, an audio interaction processing system is provided, comprising: the audio processing server of any of the embodiments of the present disclosure and the client of any of the embodiments of the present disclosure.

[0011] According to still some embodiments of the present disclosure, an electronic device is provided, comprising: a processor; and a memory coupled to the processor, configured to store instructions, the instructions being executed by the processor to cause the processor to perform the audio interaction processing method of any of the embodiments of the present disclosure.

[0012] According to still some embodiments of the present disclosure, a computer readable storage medium is provided, having stored thereon a computer program, the program being executed by a processor to perform the audio interaction processing method of any of the embodiments of the present disclosure.

[0013] According to still some embodiments of the present disclosure, a computer program product is provided, comprising: instructions, the instructions being executed by a processor to implement the audio interaction processing method of any of the embodiments of the present disclosure.

[0014] According to still some embodiments of the present disclosure, a computer program is provided, comprising: instructions, the instructions being executed by a processor to implement the audio interaction processing method of any of the embodiments of the present disclosure.

[0015] Other features, aspects, and advantages of the present disclosure will become apparent from the following detailed description of the exemplary embodiments of the present disclosure with reference made to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0016] The preferred embodiments of the present disclosure will be described below with reference to the accompanying drawings. The accompanying drawings are used to provide further understanding of the present disclosure, and together with the specific description below, form a part of the description of the present disclosure, and are used to explain the present disclosure. It should be understood that the accompanying drawings below only relate to some embodiments of the present disclosure, and do not constitute a limitation on the present disclosure. In the drawings:

[0017] FIG. 1 shows a flowchart of a processing method of audio interaction according to some embodiments of the present disclosure;

[0018] FIG. 2 shows a flowchart of a processing method of audio interaction according to some other embodiments of the present disclosure;

[0019] FIG. 3 shows a schematic diagram of audio data packet retransmission by an audio processing server according to some embodiments of the present disclosure;

[0020] FIG. 4 shows a flowchart of a processing method of audio interaction according to some other embodiments of the present disclosure;

[0021] FIG. 5 shows a schematic diagram of audio data packet retransmission by a client according to some embodiments of the present disclosure;

[0022] FIG. 6A shows a schematic diagram of information interaction flow in an ASR message link according to some embodiments of the present disclosure;

[0023] FIG. 6B shows a schematic diagram of information interaction flow in a TTS message link according to some embodiments of the present disclosure;

[0024] FIG. 6C shows a flowchart of information interaction of a processing method of audio interaction according to some embodiments of the present disclosure;

[0025] FIG. 7 shows a schematic diagram of a structure of an audio processing server according to some embodiments of the present disclosure;

[0026] FIG. 8 shows a schematic diagram of a structure of a client according to some embodiments of the present disclosure;

[0027] FIG. 9 shows a schematic diagram of a structure of an electronic device according to some embodiments of the present disclosure;

[0028] FIG. 10 shows a schematic diagram of a structure of a computer system according to some embodiments of the present disclosure;

[0029] FIG. 11 shows a schematic diagram of a structure of a processing system of audio interaction according to some embodiments of the present disclosure.

[0030] It should be understood that the dimensions of the various parts shown in the drawings are not necessarily to scale. Identical or similar components are identified throughout the various figures with identical or similar reference numerals. Therefore, when a component is identified in one figure, it can not be further discussed in subsequent figures. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only some of the embodiments of the present disclosure, but not all the embodiments. The description of the embodiments below is actually only illustrative, and should not be construed as any limitation on the present disclosure and its application or use. It should be understood that the present disclosure can be implemented in various forms, and should not be construed as being limited to the embodiments set forth herein.

[0032] It should be understood that the various steps recited in the method embodiments of the present disclosure can be performed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit performing the steps shown. The scope of the present disclosure is not limited in this respect. Unless specifically stated otherwise, the relative arrangement of components and steps set forth in these embodiments, numerical expressions, and numerical values should be interpreted as merely exemplary, not limiting the scope of the present disclosure.

[0033] The term "comprise" and variations thereof used in the present disclosure means an open term that includes at least the recited elements / features, but does not exclude other elements / features. In addition, the term "include" and variations thereof used in the present disclosure means an open term that includes at least the recited elements / features, but does not exclude other elements / features. Therefore, include and comprise are synonymous. The term "based on" means "at least partially based on".

[0034] Throughout the specification, the terms "one embodiment", "some embodiments" or "embodiments" mean that the specific features, structures or characteristics described in connection with the embodiments are included in at least one embodiment of the present disclosure. For example, the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; and the term "some embodiments" means "at least some embodiments". Moreover, the appearance of the phrase "in one embodiment", "in some embodiments" or "in embodiments" at various places in the specification does not necessarily all refer to the same embodiment, but can refer to different embodiments.

[0035] It should be noted that the terms "first", "second", and the like in the present disclosure are merely used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units. Unless otherwise specified, the terms "first", "second", and the like are not intended to imply a given order or any other manner of given order in time, space, ranking, or any other manner.

[0036] It should be noted that the terms "one", "multiple" in the present disclosure are illustrative and not limiting, and those skilled in the art should understand that unless otherwise explicitly specified in the context, it should be understood as "one or more".

[0037] The names of the messages or information exchanged between the devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0038] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings, but the present disclosure is not limited to these specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in some embodiments. In addition, in one or more embodiments, specific features, structures or characteristics can be combined by any suitable means from the present disclosure which is clear to those skilled in the art.

[0039] During the process of user interaction with the robot, the interaction can be based on text, voice or a combination of both. In some scenarios, the user input voice (audio) can be converted into text in real time for display, and the robot can reply to both text and voice at the same time, improving the user's interaction experience.

[0040] To achieve simultaneous text and voice reply by the robot, a text processing server capable of processing text and an audio processing server capable of processing audio need to be configured. The timeliness and synchronization of user robot text and voice reply directly affect the user's experience.

[0041] The present disclosure proposes an audio interaction processing method to improve the efficiency and timeliness of text and voice reply to input audio.

[0042] Some embodiments of the audio interaction processing method of the present disclosure will be described below with reference to FIGS. 1-6C.

[0043] FIG. 1 is a structural diagram of some embodiments of the audio interaction processing method of the present disclosure. As shown in FIG. 1, the method of this embodiment includes steps S102-S112. The method of this embodiment can be executed by an audio processing server. The audio processing server can be implemented by hardware or software, such as a server or application program capable of audio processing, etc.

[0044] In step S102, the input audio sent by the client is converted into a text to be replied.

[0045] The client can be implemented by hardware or software, for example, an application program that can be implemented in a terminal or set in a terminal, etc. In a human-computer interaction scenario, a user can input audio through the client, the client can transcode the audio file and divide it into multiple segments of audio, then compress the multiple segments of audio and store them locally using a queue, and then send them to an audio processing server in a streaming manner for audio recognition and conversion into a text to be replied. The audio processing server can be implemented based on a machine learning model, for example, the machine learning model can implement bidirectional conversion of speech and text. For example, the audio processing server is a server with ASR (Automatic Speech Recognition) and TTS (Text To Speech) functions.

[0046] In step S104, the text to be replied is sent to a text processing server to generate a reply text stream.

[0047] The audio processing server directly sends the text to be replied to the text processing server, which improves the efficiency of the text processing server in obtaining the text to be replied, and thus improves the overall processing efficiency, compared with the method of sending the text to be replied to the client and then sending it to the text processing server by the client.

[0048] The text processing server is responsible for generating a reply text stream according to the text to be replied. The text processing server can be implemented based on a machine learning model, for example, based on a large language model (LLM) or a foundation model. For example, the text processing server can output a real-time generated reply text segment in a streaming manner, i.e., generate a reply text stream, which can improve the timeliness and efficiency of generating the reply text, without waiting for the entire content of the reply text to be generated before feeding back, thus improving the user experience.

[0049] In step S106, a request for a reply audio stream corresponding to the reply text stream is received from the client.

[0050] For example, the request includes a message identifier corresponding to the reply text stream, which is sent by the text processing server to the client. After receiving the text to be replied, the text processing server can generate a corresponding message identifier and send it to the client. The message identifier is used to identify the reply text stream generated for the text to be replied. After receiving the message identifier, the client generates a request for the reply audio stream, which is used to request the reply audio stream from the audio processing server.

[0051] In step S108, the reply text stream is obtained from the text processing server according to the message identifier.

[0052] The audio processing server can directly obtain the reply text stream from the text processing server according to the message identifier, which is more efficient than the text processing server sending the reply text stream to the client and then the client sending the reply text stream to the audio processing server, thereby improving the efficiency of the audio processing server in obtaining the reply text stream and generating the reply audio.

[0053] The audio processing server obtains the reply text stream from the text processing server, that is, the text processing server uses streaming to transmit the generated reply text segments to the audio processing server in real time, rather than generating the entire reply text and then sending it to the audio processing server, thereby improving the transmission efficiency and the subsequent processing efficiency of the audio processing server.

[0054] In step S110, the reply audio stream is generated according to the reply text stream.

[0055] The audio processing server converts each reply text segment in the received reply text stream into audio in real time and outputs the reply audio stream in a streaming manner.

[0056] In step S112, the reply audio stream is sent to the client for playback.

[0057] The audio processing server outputs the generated reply audio segments in a streaming manner to form the reply audio stream and transmits it to the client for real-time playback, without the need to generate the entire reply audio and then return it to the client for playback.

[0058] The method of the above embodiment provides an overall implementation scheme for audio interaction. The audio processing server converts the input audio from the client into a text to be replied, and then directly interacts with the text processing server. On one hand, the text to be replied is sent directly to the text processing server to generate a reply text stream, and on the other hand, the reply text stream is directly obtained from the text processing server to generate a reply audio stream and return it to the client for playback. The audio processing server directly communicates with the text processing server, which is more efficient than the method of forwarding the text to be replied and the reply text stream through the client, thereby improving the transmission efficiency of the text to be replied and the reply text stream, enabling the text processing server to obtain the text to be replied in time and the audio processing server to obtain the reply audio stream in time, thereby improving the generation efficiency of the reply text stream and the reply audio stream. Moreover, the reply text and the reply audio are generated and transmitted in a streaming manner throughout the process, which can provide timely feedback to the input audio, thereby improving the efficiency and timeliness of text and voice reply to the input audio.

[0059] In addition, the audio processing server obtains the reply audio stream from the text processing server in real time, and generates the reply audio stream, so that the synchronization effect of text and voice reply on the client can be improved, and the user experience can be improved.

[0060] The processing method of audio interaction of the present disclosure will be further described below in combination with some specific embodiments.

[0061] In some embodiments, the reply text stream includes a plurality of text data packets, and the audio processing server periodically pulls a current text data packet generated by the text processing server according to a message identifier, or receives a current text data packet generated by the text processing server pushed by the text processing server, wherein each text data packet in the plurality of text data packets includes a message identifier.

[0062] The text processing server generates text data packets in a streaming manner to form the reply text stream. For example, the entire reply text generated by the text processing server is “Hello, glad to chat with you”, and a plurality of data packets can be generated for this sentence in a streaming manner, for example, “Hello”, “glad”, “with you”, and “chat” as the content of different data packets. The streaming generation and transmission manner makes the reply more timely, more efficient, and the user experience better. Each text data packet includes a message identifier (message_id), so that the audio processing server can accurately identify each text data packet that replies to the input audio. The audio processing server can poll the message identifier to obtain each text data packet, or the text processing server can push each text data packet to the audio processing server after generating the text data packet.

[0063] Pulling the reply audio stream by the audio processing server can reduce the invasion of the message link and improve the stability of the overall link. The audio processing server is relatively isolated from the ASR and TTS two links. If the reply text stream is pushed by the text processing server, the active push capability provided by the audio processing server needs to be coordinated, and the modification cost of the existing link is high.

[0064] In some embodiments, before the audio processing server obtains the reply text stream from the text processing server, the audio processing server generates a current audio segment corresponding to the current text data packet according to a text segment in the current text data packet, a text segment in one or more text data packets before the current text data packet, and a first preset number of texts, and generates a current audio data packet in the reply audio stream.

[0065] Each round of interaction process includes client input audio, conversion to reply text, generation of reply text stream, and generation of reply audio stream. For each round of interaction process, the audio processing server can obtain the first preset number of texts of the previous one or more rounds of dialogue, combine the text segment in the current text data packet and the text segment before the current text data packet, and generate the current audio segment corresponding to the current text data packet. The audio processing server can generate the current audio segment in combination with the context of the multi-round dialogue, better understand the semantics and context, improve the accuracy of audio generation, and improve the accuracy of reading.

[0066] Since the reply text is generally stored in the text processing server, the text of the previous one or more rounds of interaction can be obtained from the text processing server. If the audio processing server can store the reply text and the reply text in the previous one or more rounds of interaction, it can be stored after obtaining the reply text stream each time, without the need to repeatedly obtain from the text processing server. By opening up the communication between the text processing server and the audio processing server, the acquisition delay of the reply text stream and the text of the previous one or more rounds of interaction can be reduced, the efficiency can be improved, and the efficiency of the audio processing server in generating the reply audio stream can be improved.

[0067] In some embodiments, the audio processing server determines whether to perform punctuation after the text segment in the current text data packet according to the text segment in the current text data packet and the text segment in one or more text data packets before the current text data packet; and in response to performing punctuation after the text segment in the current text data packet, generates the current audio segment corresponding to the current text data packet according to the text segment in the current text data packet, the text segment in the text data packet before the current text data packet, and the first preset number of texts.

[0068] The audio processing server performs punctuation judgment according to the text segment in the current text data packet and the text segment in one or more text data packets before the current text data packet, and if punctuation is needed, generates the current audio segment, and if punctuation is not needed, continues to receive the text data packet, and generates the audio segment when punctuation is needed. The audio processing server performs punctuation judgment in combination with the context, improves the fluency of the generated reply audio and the fluency of reading. In addition, compared with the method of sending the reply text stream to the client by the text processing server, performing punctuation by the client, and then sending to the audio processing server to generate audio, the efficiency of generating audio is improved.

[0069] The text processing server generates a reply text stream in a streaming manner and sends the reply text stream to the client for display. The audio processing server generates a reply audio stream in a streaming manner and sends the reply audio stream to the client for playing. In addition, after the client inputs audio, the audio processing server generates a to-be-replied text in a streaming manner and sends the to-be-replied text to the client for display. During the interaction, the content that needs to be sent to the client for display or playing is generated and transmitted in a streaming manner, which reduces the time delay of display and playing on the client and improves the timeliness and efficiency of display and playing on the client.

[0070] In some embodiments, the audio processing server performs speech recognition on the input audio and converts the input audio into a text stream of the to-be-replied text.

[0071] The audio processing server uses the ASR technology to generate a text stream of the to-be-replied text in a streaming manner and outputs the text stream to the client for display, which improves the timeliness of displaying the to-be-replied text and enhances the user experience.

[0072] In some embodiments, the audio processing server obtains a second preset number of texts before the to-be-replied text from the text processing server, performs speech recognition on the input audio and the second preset number of texts, and converts the input audio into a text stream of the to-be-replied text.

[0073] For each round of interaction, the audio processing server can obtain a second preset number of texts of the previous one or more rounds of dialogue, combine the input audio, and generate a text stream of the to-be-replied text. The audio processing server can generate a text stream of the to-be-replied text in combination with the context of multiple rounds of dialogue, which can better understand the semantics and context and improve the accuracy of generating the to-be-replied text.

[0074] If the audio processing server can store the to-be-replied text and the reply text in each round of interaction history, it is not necessary to repeatedly obtain the texts from the text processing server. By breaking through the communication between the text processing server and the audio processing server, the time delay of obtaining the historical interaction texts can be reduced, the efficiency can be improved, and thus the efficiency of the audio processing server in generating the to-be-replied text can be improved.

[0075] In some embodiments, the audio processing server and the text processing server perform data transmission through a remote procedure call manner.

[0076] For example, the audio processing server and the text processing server can communicate through RPC (google Remote Procedure Call) or gRPC to realize the transmission of the to-be-replied text and the reply text stream, improve the transmission efficiency, and improve the efficiency of subsequent replies to the client.

[0077] In some embodiments, the audio processor transmits data with the client through an audio service gateway, and a long connection is established between the audio service gateway and the client in response to the start of the audio interactive application. Correspondingly, the client establishes a long connection with the audio service gateway corresponding to the audio processing server in response to the start of the audio interactive application, and transmits data with the audio processor through the audio service gateway.

[0078] The audio interactive application starts to establish a long connection, for example, a Websocket connection. The long connection can be multiplexed to reduce the time delay of repeated connection establishment and improve the efficiency of subsequent transmission and audio interaction.

[0079] During the communication between the client and the audio server or the text server, network switching or network disconnection may occur, which may cause the audio data packet sent by the audio processing server to the client to be lost, and when the network is restored, the audio played by the client may be problematic.

[0080] Some other embodiments of the audio interactive processing method of the present disclosure will be described below in conjunction with FIG. 2.

[0081] FIG. 2 is a structural diagram of some other embodiments of the audio interactive processing method of the present disclosure. As shown in FIG. 2, the method of this embodiment includes steps S202-S206.

[0082] In step S202, the client sends a retransmission request to the audio processing server in response to network interruption, and the audio server receives the retransmission request sent by the client.

[0083] In some embodiments, the reply audio stream includes a plurality of audio data packets, and each audio data packet in the plurality of audio data packets includes a sequence number identifier of the audio data packet. By setting the sequence number identifier, the order between the audio data packets can be determined, so that in the case of missing audio data packets, retransmission, etc., the order of identifying each audio data packet and whether there is a missing can be determined.

[0084] For example, the retransmission request includes the sequence number identifier of the last audio data packet received by the client before the network interruption.

[0085] In step S204, the audio processing server determines whether there is a to-be-retransmitted audio data packet according to the sequence number identifier of the last audio data packet received by the client before the network interruption.

[0086] For example, the data packet to be retransmitted is an audio data packet sent to the client after the last audio data packet received by the client before the network interruption. The audio processing server can determine the sequence number of the last audio data packet received by the client according to the retransmission request, and then determine that the audio data packet after the sequence number has been lost if it has been sent, i.e., the audio data packet to be retransmitted.

[0087] In step S206, the audio processing server sends the audio data packet to be retransmitted to the client if there is an audio data packet to be retransmitted.

[0088] The client receives the retransmitted audio data packet sent by the audio processing server, and the retransmitted data packet is a data packet sent to the client after the last audio data packet received by the client before the network interruption.

[0089] As shown in FIG. 3, the text processing server sends each text data packet of the generated reply text stream, Text T1, Text T2, …, to the audio processing server through gRPC. The audio processing server generates a plurality of audio data packets (Stream block, stream data block) S1, S2, …, and transmits them to the client through the external network. If a network interruption occurs after S1, the client sends a retransmission request to the audio processing server. The retransmission request can include the sequence number of S1, and the audio processing server determines whether retransmission is needed and which audio data packets need to be retransmitted according to the sequence number.

[0090] For example, each audio data packet in the reply audio stream includes the reply text, audio data, and Markdown filtered reply text corresponding to the audio data packet. Each audio data packet can include a key field Seq_id indicating the sequence number. If a network interruption occurs, the client can send a retransmission request to the audio processing server to indicate that a network error has occurred and that all audio data packets after the Seq_id of the last audio data packet received by the client need to be retransmitted. The client can identify the Seq_id of each received audio data packet to avoid receiving duplicate audio data packets.

[0091] The method of the above embodiment, in the case of network interruption, the client only needs to inform the audio processing server of the sequence number identifier, and can pull the subsequent audio for playing. If the client receives the reply text stream of the text processing server, performs sentence segmentation, and then sends the reply text stream to the audio processing server to generate the reply audio stream, after network interruption, the client will continue to send the reply text data packet to the audio processing server, however, the audio processing server cannot distinguish whether the received data packet is used to continue synthesizing the audio. The audio processing server and the text processing server communicate through the intranet, the network is stable, and the recovery of network errors is easier to handle, and the cache of the reply text and the audio is increased, so that the retransmission and the continuous transmission are simpler and more efficient. The method of the above embodiment reduces the failure of playing the reply audio caused by the interruption of the reply audio stream, and improves the user experience.

[0092] Some other embodiments of the processing method of the audio interaction of the present disclosure will be described below in combination with FIG. 4. The method of the embodiment can be executed by the client.

[0093] FIG. 4 is a structural diagram of some embodiments of the processing method of the audio interaction of the present disclosure. As shown in FIG. 4, the processing method of the embodiment includes steps S402-S410. The method of the embodiment can be executed by the client.

[0094] In step S402, the input audio is sent to the audio processing server to be converted into the text to be replied.

[0095] The text to be replied is sent by the audio processing server to the text processing server to generate the reply text stream. For details, reference can be made to the foregoing embodiments, which will not be described herein again.

[0096] In step S404, the message identifier corresponding to the reply text stream sent by the text processing server is received.

[0097] After receiving the text to be replied, the text processing server can generate the message identifier and send it to the client. For details, reference can be made to the foregoing embodiments, which will not be described herein again.

[0098] In step S406, the request for the reply audio stream corresponding to the reply text stream is sent to the audio processing server.

[0099] The request sent by the client includes the message identifier.

[0100] In step S408, the reply audio stream sent by the audio processing server is received.

[0101] The reply audio stream is generated by the audio processing server according to the message identifier and the reply text stream obtained from the text processing server. For details of the process of the audio processing server obtaining the reply audio stream from the text processing server, reference can be made to the foregoing embodiments, which will not be described herein again.

[0102] In step S410, the reply audio stream is played.

[0103] The method of the above embodiment, the client sends the input audio to the audio processing server to convert into the text to be replied, and then directly interacts with the text processing server. On the one hand, the text to be replied is directly sent to the text processing server to generate a reply text stream, and on the other hand, the reply text stream is directly obtained from the text processing server to generate a reply audio stream and return to the client for playing. The audio processing server directly communicates with the text processing server, which improves the transmission efficiency of the text to be replied and the reply text stream, enables the text processing server to obtain the text to be replied in time, and enables the audio processing server to obtain the reply audio stream in time, thereby improving the generation efficiency of the reply text stream and the reply audio stream. In the entire process, the reply text and the reply audio are generated and transmitted in a streaming manner, which can timely feedback to the input audio and improve the efficiency and timeliness of the text and voice reply to the input audio.

[0104] In addition, since the audio processing server obtains the reply audio stream from the text processing server in real time and generates the reply audio stream, the synchronization effect of the text and voice reply on the client can be improved, and the user experience can be improved.

[0105] The processing method of audio interaction of the present disclosure will be further described below in combination with some embodiments.

[0106] In some embodiments, the client divides the input audio into a plurality of audio segments, compresses each audio segment in the plurality of audio segments, and stores the compressed audio segments in a first queue in order. The client takes the first audio segment from the first queue, sends it to the audio processing server for voice recognition, and in response to receiving the information sent by the audio processing server that the information has been received, repeatedly executes the taking of the first audio segment from the first queue and the sending of the first audio segment to the audio processing server for voice recognition until the first queue is empty.

[0107] The client stores the compressed plurality of audio segments in the first queue, which can be sent to the audio processing server in order for voice recognition and conversion into the text to be replied. The reply confirmation of the audio processing server can determine that each audio segment is sent to the audio processing server, reducing the loss of audio segments due to network interruption and the like, and improving the accuracy of voice interaction.

[0108] In some embodiments, the client re-sends the most recently sent audio segment in response to not receiving the information sent by the audio processing server that the information has been received for a predetermined time period.

[0109] The client waits for a preset time length without receiving the information that the audio processing server sends to confirm the receipt, which can be the loss of the corresponding audio segment, re-sending the corresponding audio segment, reducing the probability that the audio processing server cannot accurately identify the input audio, and improving the accuracy of subsequent voice interaction.

[0110] As shown in FIG. 5, the client sends each audio data packet of the input audio to the audio processing server. If the client re-sends the S1 data packet after the network interruption, the client waits for a preset time length without receiving the information that the audio processing server sends to confirm the receipt, and then re-sends the S1 data packet.

[0111] The client divides the input audio file into multiple audio segments after transcoding and stores them in the first queue locally, and transmits them to the audio processing server for audio recognition. When receiving the information that the audio processing server confirms the receipt, the subsequent audio segments are dequeued and transmitted. When detecting abnormal conditions such as network interruption, it is determined whether retransmission is needed and each time a preset time length is waited for retry. The head audio segment of the latest transmission is retried, which can ensure that the input audio streaming recognition can run normally and obtain correct recognition results in the case of network switching or poor network quality.

[0112] In some embodiments, the reply audio stream includes multiple audio data packets, each audio data packet in the multiple audio data packets includes a sequence number identifier of the audio data packet, and the client stores the current audio data packet in the second queue according to the sequence number identifier of the received current audio data packet. The audio data packets in the second queue are played in order.

[0113] Adding a sequence number identifier in the audio data packet can make the client play each audio data packet in order, and even if there is a network transmission error, the accurate playback of the reply audio stream can be realized.

[0114] In some embodiments, the client determines whether there is an audio data packet with the same sequence number identifier as the current audio data packet in the second queue according to the sequence number identifier of the received current audio data packet. If there is an audio data packet with the same sequence number identifier as the current audio data packet in the second queue, the current audio data packet is discarded. If there is no audio data packet with the same sequence number identifier as the current audio data packet in the second queue, the current audio data packet is stored in the queue.

[0115] The client can determine whether there is a duplicate audio data packet according to the sequence number identifier of the audio data packet. Especially in the case of network interruption, retransmission, and retransmission, there can be repeated transmission of audio data packets, which improves the accuracy of the transmission of the reply audio stream and improves the accuracy of the playback of the reply audio stream.

[0116] In some embodiments, the client receives a text stream of the text to be replied sent by the audio processing server, displays the text stream of the text to be replied, receives a reply text stream sent by the text processing server, and displays the reply text stream.

[0117] The reply text stream is generated by the text processing server after receiving the text to be replied sent by the audio processing server. The client receives the text stream of the text to be replied sent by the audio processing server, the reply text stream sent by the text processing server, and the reply audio stream sent by the audio processing server, and displays and plays them.

[0118] The method of the embodiments of the present disclosure can obtain the reply text stream in real time for the input audio, quickly convert it into a reply audio stream, and push it to the client in real time for playing. In the scenario of higher real-time output of streaming, the reply text to audio playing experience is greatly improved, and the generation of the reply audio stream in combination with the context can improve the accuracy. Through retrying and retransmitting the reply audio stream and the input audio stream, even if the network quality decreases, the network is disconnected, or the network is jittered during long text or long ASR recognition of the user, the ASR and TTS can still be accurately recognized or played.

[0119] The processing method of audio interaction of the present disclosure is described below in combination with some application examples.

[0120] Figure 6A is a flow of information interaction in the ASR message link. As shown in Figure 6A, (0) the audio module (APP Audio) in the audio interaction application of the client establishes a long connection with the audio service gateway (GateWay) when the audio interaction application is started. The user sends voice, starts the ASR message transmission, and inputs to the audio module in the audio interaction application of the client. (1) The user voice (input audio) is sent to the audio service gateway through the audio module. The audio service gateway belongs to the service side device. (2) The audio service gateway sends the user voice to the audio processing module. The audio processing module and the ASR module can form the audio processing server in the foregoing embodiments. The ASR module can also be deployed separately, and the audio processing module is separately used as the audio processing server in the foregoing embodiments. The audio processing module calls the ASR module through gRPC to realize user voice recognition, and obtains voice recognition text (to-be-replied text). (3) The audio processing module returns the voice recognition text to the audio service gateway. (4) The audio service gateway returns the voice device text to the audio module in the client. (5) The audio module in the client sends the voice recognition text to the message module (APP message) in the client for display. (6) After obtaining the voice recognition text, the audio processing module sends the voice recognition text to the interaction processing module. (7) The interaction processing module sends a text message to the LLM and requests the LLM to process the text message. The interaction processing module and the LLM can form the text processing server in the foregoing embodiments. The LLM can also be deployed separately, and the interaction processing module is separately used as the text processing server in the foregoing embodiments. (8) The interaction processing module sends the user message full text (voice recognition text) to the LLM, and the LLM returns a key for polling to the interaction processing module. (9) The interaction processing module polls the incremental reply of the LLM according to the key. The LLM generates a reply in a streaming manner, and the interaction processing module only needs to obtain the incremental reply text each time. (10) The interaction processing module sends the reply text to the Frotier (boundary) device. (11) The Frotier device sends the reply text to the message module in the client for display.

[0121] The audio service gateway and the Frotier device can be used as a WebSocket access layer to realize transmission between the external network and the internal network.

[0122] FIG. 6B is a flow of information interaction in the TTS message link. As shown in FIG. 6B, (1) the audio processing module sends the speech recognition text (text to be replied) to the interaction processing module, (2) the interaction processing module sends the speech recognition text to the LLM, (3) the interaction processing module sends the message id of the reply text stream to the Frotier device, (4) the Frotier device sends the message id of the reply text stream to the message module in the client, (5) the message module sends the message id of the reply text stream to the audio module in the client to start the TTS session, (6) the audio module sends a request to subscribe to the TTS audio stream (reply audio stream) to the audio service gateway, (7) the audio service gateway sends a request to subscribe to the TTS audio stream (reply audio stream) to the audio processing module, (8) the audio processing module requests the reply text from the interaction processing module, (9) the audio processing module generates the TTS audio stream (reply audio stream) according to the reply text and sends it to the audio service gateway, (10) the audio service gateway sends the TTS audio stream to the audio module in the client for playing.

[0123] FIG. 6C is a flow of information interaction of the processing method of audio interaction. As shown in FIG. 6C, to implement the scheme of the present disclosure, an audio SDK can be added in the client to interact with the audio processing server. The audio SDK can be located in the client or can be independently set. (1) The client sends the input audio (i.e., Audio uplink) to the audio SDK. (2) The audio SDK sends the input audio to the audio processing server. (3) The audio processing server requests the context content (i.e., the text of the historical interaction before the current interaction) from the text processing server. (4) The text processing server returns the context content to the audio processing server. (5) The audio processing server performs ASR recognition to obtain the ASR recognition text (text to be replied). The audio processing server performs ASR recognition in combination with the context content, which is more accurate. (6) The audio processing server sends the ASR recognition text to the audio SDK. (7) The audio SDK sends the ASR recognition text to the client. (8) The client displays the ASR recognition text.

[0124] (9) The audio processing server sends the ASR recognition text (text to be replied) to the text processing server. (10) The text processing server sends the message id (Message id) to the client. (11) The text processing server generates the reply text and returns the reply text stream to the client in a streaming manner. (12) The client displays the reply text.

[0125] (13) The client sends a request for TTS audio data (reply audio stream) to the audio SDK, which carries the Message id.(14) The audio SDK sends a request for TTS audio data to the audio processing server.(15) The audio processing server requests the reply text from the text processing server.(16) The text processing server sends the reply text stream to the audio processing server.(17) The audio processing server generates the TTS audio stream according to the reply text stream.(18) The audio processing server sends the TTS audio stream to the audio SDK.(19) The audio SDK sends the TTS audio stream to the client.(20) The client plays the TTS audio.

[0126] The above steps (1)-(8) are the process of generating and displaying the text to be replied, and (9)-(12) are the process of generating and displaying the reply text. (13)-(20) are the process of generating and playing the reply audio stream. For ease of understanding, the steps in the same process are described together, and the order of these steps in the actual processing process is not limited to the order and numbering shown in the figure. For example, the audio processing server can send the ASR recognized text to the text processing server after generating the ASR recognized text, that is, (6) and (9) can be synchronized or (9) can be after (6).

[0127] The present disclosure also provides an audio processing server, which is described below in conjunction with FIG. 7.

[0128] FIG. 7 is a structural diagram of some embodiments of the audio processing server of the present disclosure. As shown in FIG. 7, the audio processing server 70 of this embodiment includes a conversion module 710, a first sending module 720, a receiving module 730, an acquisition module 740, a generation module 750, and a second sending module 760.

[0129] The conversion module 710 is configured to convert the input audio sent by the client into the text to be replied.

[0130] The first sending module 720 is configured to send the text to be replied to the text processing server to generate a reply text stream.

[0131] The receiving module 730 is configured to receive a request for a reply audio stream corresponding to the reply text stream from the client, wherein the request includes a message identifier corresponding to the reply text stream, and the message identifier is sent to the client by the text processing server.

[0132] The acquisition module 740 is configured to acquire the reply text stream from the text processing server according to the message identifier.

[0133] The generation module 750 is configured to generate the reply audio stream according to the reply text stream.

[0134] The second sending module 760 is configured to send the reply audio stream to the client for playing.

[0135] In some embodiments, the reply audio stream includes a plurality of audio data packets, each of the plurality of audio data packets includes a sequence number identifier of the audio data packet, the audio processing server 70 further includes a determination module 770, and the receiving module 730 is further configured to receive a retransmission request sent by the client, wherein the retransmission request includes the sequence number identifier of the last audio data packet received by the client before the network interruption; the determination module 770 is configured to determine whether there is a to-be-retransmitted audio data packet according to the sequence number identifier of the last audio data packet, wherein the to-be-retransmitted audio data packet is an audio data packet sent to the client after the last audio data packet; and the second sending module 760 is configured to send the to-be-retransmitted audio data packet to the client in the case that there is a to-be-retransmitted audio data packet.

[0136] In some embodiments, the reply text stream includes a plurality of text data packets, and the obtaining module 740 is configured to periodically pull a current text data packet generated by the text processing server from the text processing server according to a message identifier, or receive a current text data packet generated by the text processing server pushed by the text processing server, wherein each of the plurality of text data packets includes a message identifier.

[0137] In some embodiments, the obtaining module 740 is further configured to obtain a first preset number of texts before the reply text stream from the text processing server, and the generating module 750 is configured to generate a current audio segment corresponding to the current text data packet according to the text segment in the current text data packet, the text segment in one or more text data packets before the current text data packet, and the first preset number of texts, and generate a current audio data packet in the reply audio stream.

[0138] In some embodiments, the generating module 750 is configured to determine whether to perform punctuation after the text segment in the current text data packet according to the text segment in the current text data packet and the text segment in one or more text data packets before the current text data packet, and in response to performing punctuation after the text segment in the current text data packet, generate a current audio segment corresponding to the current text data packet according to the text segment in the current text data packet, the text segment in the text data packet before the current text data packet, and the first preset number of texts.

[0139] In some embodiments, the conversion module 710 is configured to perform speech recognition on the input audio to convert the input audio into a text stream of to-be-replied text, and the second sending module 760 is further configured to send the text stream of to-be-replied text to the client for display.

[0140] In some embodiments, the obtaining module 740 is further configured to obtain, from the text processing server, a second preset number of texts before the text to be replied, and the converting module 710 is configured to perform speech recognition according to the input audio and the second preset number of texts, and convert the input audio into a text stream of the text to be replied.

[0141] In some embodiments, the audio processing server and the text processing server perform data transmission through a remote procedure call mode.

[0142] In some embodiments, the audio processor performs data transmission with the client through an audio service gateway, and a long connection is established between the audio service gateway and the client in response to starting of the audio interaction application.

[0143] The present disclosure also provides a client, which is described below in conjunction with FIG. 8.

[0144] FIG. 8 is a structural diagram of some embodiments of the client of the present disclosure. As shown in FIG. 8, the client 80 of the embodiment includes a first sending module 810, a first receiving module 820, a second sending module 830, a second receiving module 840, and a playing module 850.

[0145] The first sending module 810 is configured to send the input audio to the audio processing server for conversion into a text to be replied, wherein the text to be replied is sent to the text processing server to generate a reply text stream.

[0146] The first receiving module 820 is configured to receive a message identifier corresponding to the reply text stream sent by the text processing server.

[0147] The second sending module 830 is configured to send, to the audio processing server, a request for a reply audio stream corresponding to the reply text stream, wherein the request includes the message identifier.

[0148] The second receiving module 840 is configured to receive the reply audio stream sent by the audio processing server.

[0149] The playing module 850 is configured to play the reply audio stream.

[0150] In some embodiments, the first sending module 810 is configured to divide the input audio into a plurality of audio segments, compress each of the plurality of audio segments, and store the compressed audio segments in a first queue in order; take the first audio segment from the first queue, and send the first audio segment to the audio processing server for speech recognition; in response to receiving information indicating that the audio processing server has received the first audio segment, repeatedly perform the operations of taking the first audio segment from the first queue and sending the first audio segment to the audio processing server for speech recognition, until the first queue is empty.

[0151] In some embodiments, the first sending module 810 is configured to resend the last sent audio segment in response to not receiving the information indicating that the audio processing server end has received the audio segment within a preset time period.

[0152] In some embodiments, the reply audio stream includes a plurality of audio data packets, each of the plurality of audio data packets includes a sequence number identifier of the audio data packet, and the playing module 850 is configured to store the current audio data packet into the second queue according to the sequence number identifier of the received current audio data packet, and play the audio data packets in the second queue in sequence.

[0153] In some embodiments, the playing module 850 is configured to determine whether an audio data packet with the same sequence number identifier as the current audio data packet already exists in the second queue according to the sequence number identifier of the received current audio data packet, discard the current audio data packet if an audio data packet with the same sequence number identifier as the current audio data packet already exists in the second queue, and store the current audio data packet into the queue if an audio data packet with the same sequence number identifier as the current audio data packet does not exist in the second queue.

[0154] In some embodiments, the second sending module 830 is further configured to send a retransmission request to the audio processing server end in response to network interruption, wherein the retransmission request includes a sequence number identifier of the last audio data packet received by the client before the network interruption, and the second receiving module 840 is further configured to receive a retransmission audio data packet sent by the audio processing server end, wherein the retransmission data packet is a data packet that has been sent by the audio processing server end to the client after the last audio data packet.

[0155] In some embodiments, the client 80 further includes a third receiving module 860 configured to receive a text stream of the text to be replied sent by the audio processing server end, and a display module 870 configured to display the text stream of the text to be replied.

[0156] In some embodiments, the client 80 further includes a fourth receiving module 880 configured to receive a reply text stream sent by the text processing server end, wherein the reply text stream is generated by the text processing server end after receiving the text to be replied sent by the audio processing server end, and the display module 870 is further configured to display the reply text stream.

[0157] In some embodiments, in response to the start of the audio interaction application, the client 80 establishes a long connection with an audio service gateway corresponding to the audio processing server end, and the client transmits data to the audio processor through the audio service gateway.

[0158] It should be noted that each unit (module) described above is a logical division according to the specific function implemented by it, and is not intended to limit the specific implementation manner, for example, it can be implemented in software, hardware or a combination of software and hardware. In actual implementation, each unit described above can be implemented as an independent physical entity, or can also be implemented by a single entity (for example, a processor (CPU or DSP, etc.), an integrated circuit, etc.). In addition, each unit described above is indicated by a dashed line in the drawing, indicating that these units can not actually exist, and the operations / functions implemented by them can be implemented by the processing circuit itself.

[0159] In addition, although not shown, the device can also include a memory, which can store various information generated by the device, each unit included in the device in operation, programs and data for operation, data to be transmitted by the communication unit, etc. The memory can be a volatile memory and / or a non-volatile memory. For example, the memory can include, but is not limited to, a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a read-only memory (ROM), a flash memory. Of course, the memory can also be located outside the device. Alternatively, although not shown, the device can also include a communication unit, which can be used for communication with other devices. In one example, the communication unit can be implemented in a suitable manner known in the art, for example, including communication components such as an antenna array and / or a radio frequency link, various types of interfaces, communication units, etc. Here will not be described in detail. In addition, the device can also include other components not shown, such as a radio frequency link, a baseband processing unit, a network interface, a processor, a controller, etc. Here will not be described in detail.

[0160] Some embodiments of the present disclosure also provide an electronic device. FIG. 9 shows a block diagram of some embodiments of the electronic device of the present disclosure. For example, in some embodiments, the electronic device 90 can be various types of devices, for example, can include but not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablets), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc. and fixed terminals such as digital TVs, desktop computers, etc. For example, the electronic device 90 can include a display panel for displaying data and / or execution results utilized in the scheme according to the present disclosure. For example, the display panel can be various shapes, such as a rectangular panel, an oval panel or a polygonal panel, etc. In addition, the display panel can not only be a flat panel, but also a curved panel, or even a spherical panel.

[0161] As shown in FIG. 9, the electronic device 90 of this embodiment includes a memory 91 and a processor 92 coupled to the memory 91. It should be noted that the components of the electronic device 90 shown in FIG. 9 are merely exemplary and non-limiting, and the electronic device 90 can also have other components according to actual application needs. The processor 92 can control other components in the electronic device 90 to perform desired functions.

[0162] In some embodiments, the memory 91 is configured to store one or more computer readable instructions. When the processor 92 executes the computer readable instructions, the computer readable instructions are executed by the processor 92 to implement the method according to any of the above embodiments. For specific implementation of each step of the method and related explanations, please refer to the above embodiments, and the repeated parts will not be described here.

[0163] For example, the processor 92 and the memory 91 can directly or indirectly communicate with each other. For example, the processor 92 and the memory 91 can communicate through a network. The network can include a wireless network, a wired network, and / or any combination of a wireless network and a wired network. The processor 92 and the memory 91 can also communicate with each other through a system bus, and the present disclosure does not limit the processor 92 and the memory 91.

[0164] For example, the processor 92 can be embodied as various appropriate processors, processing devices, etc., such as a central processing unit (CPU), a graphics processing unit (GPU), a network processing unit (NP), etc.; and can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic component, a discrete hardware component. The central processing unit (CPU) can be X86 or ARM architecture, etc. For example, the memory 91 can include any combination of various forms of computer readable storage media, such as volatile memory and / or non-volatile memory. The memory 91 may, for example, include system memory, which stores, for example, an operating system, application programs, a boot loader, a database, and other programs, etc. Various application programs and various data, etc. can also be stored in the storage medium.

[0165] In addition, according to some embodiments of the present disclosure, various operations / processes according to the present disclosure, when implemented by software and / or firmware, can install programs constituting the software from a storage medium or a network to a computer system with a dedicated hardware structure, such as the computer system (or electronic device) 100 shown in FIG. 10, which is capable of performing various functions when various programs are installed. FIG. 10 is a block diagram showing an example structure of a computer system that can be employed according to embodiments of the present disclosure.

[0166] In FIG. 10, a central processing unit (CPU) 1001 performs various processing in accordance with a program stored in a read only memory (ROM) 1002 or a program loaded from a storage section 1008 to a random access memory (RAM) 1003. In the RAM 1003, data required when the CPU 1001 performs various processing and the like is also stored as necessary. The central processing unit is merely exemplary, and can also be other types of processors, such as the various processors described above. The ROM 1002, the RAM 1003, and the storage section 1008 can be various forms of computer readable storage media, as described below. Note that, although the ROM 1002, the RAM 1003, and the storage 1008 are shown separately in FIG. 10, one or more of them can be combined or located in the same or different memory or storage modules.

[0167] The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output interface 1005 is also connected to the bus 1004.

[0168] The following components are connected to the input / output interface 1005: an input section 1006, such as a touch panel, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, and the like; an output section 1007, including a display, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a speaker, a vibrator, and the like; a storage section 1008, including a hard disk, a magnetic tape, and the like; and a communication section 1009, including a network interface card, such as a LAN card, a modem, and the like. The communication section 1009 allows communication processing to be performed via a network, such as the Internet. It is readily understood that, although the various devices or modules in the computer system 100 are shown in FIG. 10 as communicating via the bus 1004, they can also communicate through a network or other means, where the network can include a wireless network, a wired network, and / or any combination of a wireless network and a wired network.

[0169] A drive 1010 is also connected to the input / output interface 1005 as necessary. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like, is attached to the drive 1010 as necessary, so that a computer program read therefrom is installed in the storage section 1008 as necessary.

[0170] In the case where the above-described series of processing is implemented by software, the program constituting the software can be installed from a network, such as the Internet, or a storage medium, such as the removable medium 1011.

[0171] According to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product including a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication device 1009, or installed from the storage device 1008, or installed from the ROM 1002. When the computer program is executed by the CPU 1001, the above-described functions defined in the methods of the embodiments of the present disclosure are executed.

[0172] Note that, in the context of the present disclosure, a computer-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, the computer-readable signal medium can include a data signal carried in a baseband or as part of a carrier wave, in which the computer-readable program code is carried. Such a propagated data signal can take any of a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium that is not a computer-readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wire, cable, fiber optic, RF (radio frequency), or any suitable combination of the above.

[0173] The above-described computer-readable medium can be included in the above-described electronic device; or can exist separately from the electronic device and not be assembled into the electronic device.

[0174] In some embodiments, the present disclosure also provides a computer program comprising instructions which, when executed by a processor, cause the processor to perform the method of any one of the above embodiments. For example, the instructions can be embodied as computer program code.

[0175] The present disclosure also provides a processing system of audio interaction, which is described below in connection with FIG. 11.

[0176] FIG. 11 is a structural diagram of some embodiments of the processing system of audio interaction of the present disclosure. As shown in FIG. 11, the processing system of audio interaction 11 of this embodiment comprises the audio processing server 70 of any embodiment of the present disclosure and the client 80 of any embodiment of the present disclosure.

[0177] In some embodiments, the system 11 further comprises a text processing server 110 configured to generate a reply text stream based on the input audio converted reply text, and send the reply text stream to the audio processing server 70 and the client 80.

[0178] In some embodiments, the text processing server 110 is further configured to receive the reply text sent by the audio processing server 70, and send the message identifier corresponding to the reply audio stream to the client 80.

[0179] In embodiments of the present disclosure, computer program code for carrying out operations of the present disclosure can be written in one or more programming languages or combinations of languages including object oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0180] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts and block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks noted in succession can in fact be executed substantially concurrently or can sometimes be executed in reverse order, depending on the functionality involved. It will also be noted that each block of the block diagrams and / or flowcharts, and combinations thereof, can be implemented by dedicated hardware-based systems which perform the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0181] The modules, components or units described in the embodiments of the present disclosure can be implemented by software, or by hardware. In some cases, the name of the module, component or unit does not constitute a limitation on the module, component or unit itself.

[0182] The functions described in the above description can be performed at least in part by one or more hardware logic components. For example, and without limitation, example hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0183] According to some embodiments of the present disclosure, a method for processing audio interaction is provided, which is executed by an audio processing server, and includes: converting input audio sent by a client into a text to be replied; sending the text to be replied to a text processing server to generate a reply text stream; receiving a request of the client for a reply audio stream corresponding to the reply text stream, wherein the request includes a message identifier corresponding to the reply text stream, and the message identifier is sent to the client by the text processing server; obtaining the reply text stream from the text processing server according to the message identifier; generating the reply audio stream according to the reply text stream; and sending the reply audio stream to the client for playing.

[0184] In some embodiments, the reply audio stream includes a plurality of audio data packets, each of the plurality of audio data packets includes a sequence number identifier of the audio data packet, and the processing method further includes: receiving a retransmission request sent by the client, wherein the retransmission request includes a sequence number identifier of a last audio data packet received by the client before the network interruption; determining whether there is a to-be-retransmitted audio data packet according to the sequence number identifier of the last audio data packet, wherein the to-be-retransmitted audio data packet is an audio data packet sent to the client after the last audio data packet; and in a case where there is a to-be-retransmitted audio data packet, sending the to-be-retransmitted audio data packet to the client.

[0185] In some embodiments, the reply text stream includes a plurality of text data packets, and the obtaining of the reply text stream from the text processing server according to the message identifier includes: periodically pulling a current text data packet generated by the text processing server from the text processing server according to the message identifier; or receiving a generated current text data packet pushed by the text processing server, wherein each of the plurality of text data packets includes the message identifier.

[0186] In some embodiments, the processing method further includes: obtaining a first preset number of texts before the reply text stream from the text processing server, and the generating of the reply audio stream according to the reply text stream includes: generating a current audio segment corresponding to the current text data packet according to a text segment in the current text data packet, text segments in one or more text data packets before the current text data packet, and the first preset number of texts, and generating a current audio data packet in the reply audio stream.

[0187] In some embodiments, the generating of the current audio segment corresponding to the current text data packet according to the text segment in the current text data packet, the text segments in the one or more text data packets before the current text data packet, and the first preset number of texts includes: determining whether to perform punctuation after the text segment in the current text data packet according to the text segment in the current text data packet and the text segments in the one or more text data packets before the current text data packet; and in response to performing punctuation after the text segment in the current text data packet, generating the current audio segment corresponding to the current text data packet according to the text segment in the current text data packet, the text segments in the one or more text data packets before the current text data packet, and the first preset number of texts.

[0188] In some embodiments, the converting of the input audio sent by the client into the to-be-replied text includes: performing speech recognition on the input audio to convert the input audio into a text stream of the to-be-replied text, and the processing method further includes: sending the text stream of the to-be-replied text to the client for display.

[0189] In some embodiments, the processing method further comprises: obtaining, from the text processing server, a second preset number of texts before the text to be replied; wherein the speech recognition on the input audio to convert into the text stream of the text to be replied comprises: performing speech recognition on the input audio and the second preset number of texts to convert the input audio into the text stream of the text to be replied.

[0190] In some embodiments, the audio processing server and the text processing server perform data transmission through a remote procedure call mode.

[0191] In some embodiments, the audio processor performs data transmission with the client through an audio service gateway, and a long connection is established between the audio service gateway and the client in response to the start of the audio interaction application.

[0192] According to some other embodiments of the present disclosure, an audio interaction processing method is provided, which is executed by a client and comprises: sending input audio to an audio processing server to convert into a text to be replied, wherein the text to be replied is sent to a text processing server to generate a reply text stream; receiving a message identifier corresponding to the reply text stream sent by the text processing server; sending a request for a reply audio stream corresponding to the reply text stream to the audio processing server, wherein the request comprises the message identifier; receiving the reply audio stream sent by the audio processing server; and playing the reply audio stream.

[0193] In some embodiments, sending the input audio to the audio processing server comprises: dividing the input audio into a plurality of audio segments; compressing each of the plurality of audio segments and storing them in a first queue in order; taking the first audio segment from the first queue and sending it to the audio processing server for speech recognition; and in response to receiving information confirming receipt sent by the audio processing server, repeatedly performing the taking of the first audio segment from the first queue and sending it to the audio processing server for speech recognition until the first queue is empty.

[0194] In some embodiments, sending the input audio to the audio processing server further comprises: in response to not receiving the information confirming receipt sent by the audio processing server for a preset time period, resending the most recently sent audio segment.

[0195] In some embodiments, the reply audio stream comprises a plurality of audio data packets, each of the plurality of audio data packets comprises a sequence number identifier of the audio data packet, and playing the reply audio stream comprises: storing a current audio data packet in a second queue according to the sequence number identifier of the received current audio data packet; and playing the audio data packets in the second queue in order.

[0196] In some embodiments, storing the current audio data packet into the second queue according to the sequence number identifier of the received current audio data packet comprises: determining whether an audio data packet with the same sequence number identifier as the current audio data packet already exists in the second queue according to the sequence number identifier of the received current audio data packet; discarding the current audio data packet if an audio data packet with the same sequence number identifier as the current audio data packet already exists in the second queue; and storing the current audio data packet into the second queue if an audio data packet with the same sequence number identifier as the current audio data packet does not exist in the second queue.

[0197] In some embodiments, the processing method further comprises: in response to a network interruption, sending a retransmission request to the audio processing server, wherein the retransmission request comprises a sequence number identifier of a last audio data packet received by the client before the network interruption; and receiving a retransmission audio data packet sent by the audio processing server, wherein the retransmission data packet is a data packet that has been sent by the audio processing server to the client after the last audio data packet.

[0198] In some embodiments, the processing method further comprises: receiving a text stream of the text to be replied sent by the audio processing server; displaying the text stream of the text to be replied; receiving a reply text stream sent by the text processing server, wherein the reply text stream is generated by the text processing server after receiving the text to be replied sent by the audio processing server; and displaying the reply text stream.

[0199] In some embodiments, the processing method further comprises: in response to starting of the audio interaction application, establishing a long connection with an audio service gateway corresponding to the audio processing server, wherein the client performs data transmission with the audio processor through the audio service gateway.

[0200] According to still other embodiments of the present disclosure, an audio processing server is provided, comprising: a conversion module configured to convert input audio sent by a client into text to be replied; a first sending module configured to send the text to be replied to a text processing server to generate a reply text stream; a receiving module configured to receive a request of the client for a reply audio stream corresponding to the reply text stream, wherein the request comprises a message identifier corresponding to the reply text stream, and the message identifier is sent by the text processing server to the client; an obtaining module configured to obtain the reply text stream from the text processing server according to the message identifier; a generating module configured to generate the reply audio stream according to the reply text stream; and a second sending module configured to send the reply audio stream to the client for playing.

[0201] According to still another embodiment of the present disclosure, a client is provided, comprising: a first sending module configured to send input audio to an audio processing server, and convert the input audio into a text to be replied, wherein the text to be replied is sent to a text processing server to generate a reply text stream; a first receiving module configured to receive a message identifier corresponding to the reply text stream sent by the text processing server; a second sending module configured to send a request for a reply audio stream corresponding to the reply text stream to the audio processing server, wherein the request comprises the message identifier; a second receiving module configured to receive the reply audio stream sent by the audio processing server; and a playing module configured to play the reply audio stream.

[0202] According to still another embodiment of the present disclosure, a processing system for audio interaction is provided, comprising: the audio processing server of any embodiment of the present disclosure, and the client of any embodiment of the present disclosure.

[0203] In some embodiments, the processing system further comprises: a text processing server configured to generate a reply text stream based on the text to be replied converted from the input audio, and send the reply text stream to the audio processing server and the client.

[0204] In some embodiments, the text processing server is further configured to receive the text to be replied sent by the audio processing server, and send a message identifier corresponding to the reply audio stream to the client.

[0205] According to still another embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory coupled to the processor, configured to store instructions, wherein the instructions are executed by the processor to cause the processor to perform the processing method for audio interaction of any embodiment of the present disclosure.

[0206] According to still another embodiment of the present disclosure, a computer readable storage medium is provided, having a computer program stored thereon, wherein the program is executed by a processor to perform the processing method for audio interaction of any embodiment of the present disclosure.

[0207] According to still another embodiment of the present disclosure, a computer program product is provided, comprising: instructions, wherein the instructions are executed by a processor to implement the processing method for audio interaction of any embodiment of the present disclosure.

[0208] According to still another embodiment of the present disclosure, a computer program is provided, comprising: instructions, wherein the instructions are executed by a processor to implement the processing method for audio interaction of any embodiment of the present disclosure.

[0209] The above description is only some embodiments of the present disclosure and an explanation of the principles of the technology used. Those skilled in the art should understand that the disclosed range of the present disclosure is not limited to the technical solutions formed by the specific combinations of the technical features described above, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with each other to form technical solutions with similar functions disclosed in the present disclosure (but not limited to).

[0210] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

[0211] In addition, while operations are depicted in a particular, chronological sequence, this should not be understood as requiring this particular sequence to be followed. One of ordinary skill in the art will appreciate that the operations of the disclosed processes can be performed in other orders and still function properly. Also, descriptions of functions and constructions that are obvious or that do not factor into the claimed disclosure can be omitted for the sake of brevity. Similarly, while several specific implementation details are discussed herein, this should not be construed as a limitation on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination.

[0212] While certain embodiments of the disclosure have been described herein, other embodiments will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure. Therefore, the disclosure is not limited to these embodiments, but instead has a scope as defined by the appended claims. Various modifications and changes can be made thereto by those skilled in the art which fall within the scope of the disclosure as defined by the following claims. Moreover, other implementations of the disclosure will be apparent from consideration of the specification and the practice of the disclosure as disclosed herein. Therefore, the disclosure is intended to cover any and all adaptations and modifications of the various embodiments of the disclosure and falls within the scope of the appended claims. Accordingly, the specification is to be regarded in an illustrative manner and references to specific structural features are to be regarded as being among numerous non-critical features included in the fabric of the disclosure.

Claims

1. A method for processing audio interaction, executed by an audio processing server, comprising: Convert the input audio sent by the client into text to be replied; Sending the text to be replied to to the text processing server to generate a reply text stream; receiving a request from the client for a reply audio stream corresponding to the reply text stream, wherein the request includes a message identifier corresponding to the reply text stream, and the message identifier is sent by the text processing server to the client; Obtaining the reply text stream from the text processing server according to the message identifier; generating the reply audio stream according to the reply text stream; The reply audio stream is sent to the client for playback.

2. The processing method according to claim 1, wherein The reply audio stream includes a plurality of audio data packets, each of the plurality of audio data packets includes a sequence number identifier of the audio data packet, and the processing method further includes: receiving a retransmission request sent by the client, wherein the retransmission request includes a sequence number identifier of the last audio data packet received by the client before the network interruption; determining, according to the sequence number identifier of the last audio data packet, whether there is an audio data packet to be retransmitted, wherein the data packet to be retransmitted is an audio data packet that has been sent to the client after the last audio data packet; In a case where the audio data packet to be retransmitted exists, the audio data packet to be retransmitted is sent to the client.

3. The processing method according to claim 1 or 2, wherein: The reply text stream includes a plurality of text data packets, and obtaining the reply text stream from the text processing server according to the message identifier includes: periodically pulling from the text processing server a current text data packet generated by the text processing server according to the message identifier; or Receive the generated current text data packet pushed by the text processing server, wherein each of the multiple text data packets includes the message identifier.

4. The processing method according to claim 3, further comprising: Obtaining a first preset amount of text before the reply text stream from the text processing server, The step of generating the reply audio stream according to the reply text stream includes: Based on the text segment in the current text data packet, the text segment in one or more text data packets before the current text data packet, and the first preset number of texts, a current audio segment corresponding to the current text data packet is generated, and a current audio data packet in the reply audio stream is generated.

5. The processing method according to claim 4, wherein: Generating the current audio segment corresponding to the current text data packet according to the text segment in the current text data packet, the text segment in the text data packet before the current text data packet, and the first preset number of texts includes: determining whether to perform sentence segmentation after the text segment in the current text data packet based on the text segment in the current text data packet and the text segments in one or more text data packets before the current text data packet; In response to segmentation after the text segment in the current text data packet, a current audio segment corresponding to the current text data packet is generated based on the text segment in the current text data packet, the text segment in the text data packet before the current text data packet, and the first preset number of texts.

6. The processing method according to any one of claims 1 to 5, wherein: Converting the input audio sent by the client into text to be replied includes: Performing speech recognition on the input audio to convert the audio into a text stream of the text to be replied; The processing method further comprises: The text stream of the text to be replied is sent to the client for display.

7. The processing method according to claim 6, further comprising: Obtaining a second preset number of texts before the text to be replied from the text processing server, The step of performing speech recognition on the input audio and converting the converted text into the text stream to be replied includes: Speech recognition is performed based on the input audio and the second preset amount of text, and the input audio is converted into a text stream of the text to be replied.

8. The processing method according to any one of claims 1 to 7, wherein: The audio processing server and the text processing server perform data transmission via remote procedure call.

9. The processing method according to any one of claims 1 to 8, wherein: The audio processor transmits data with the client through the audio service gateway. In response to the start of the audio interaction application, a long connection is established between the audio service gateway and the client.

10. A method for processing audio interaction, executed by a client, comprising: Sending the input audio to the audio processing server to convert it into a text to be replied, wherein the text to be replied is sent by the audio processing server to the text processing server to generate a reply text stream; receiving a message identifier corresponding to the reply text stream sent by the text processing server; Sending a request for a reply audio stream corresponding to the reply text stream to the audio processing server, wherein the request includes the message identifier; receiving the reply audio stream sent by the audio processing server, wherein the reply audio stream is generated by the audio processing server after obtaining the reply text stream from the text processing server according to the message identifier; The reply audio stream is played.

11. The processing method according to claim 10, wherein: The sending of the input audio to the audio processing server includes: Dividing the input audio into multiple audio segments; compressing each of the multiple audio clips and storing the compressed audio clips in a first queue in order; Taking the first audio clip from the first queue and sending it to the audio processing server for speech recognition; In response to receiving a confirmation message sent by the audio processing server, repeatedly taking out the first audio segment from the first queue and sending it to the audio processing server for speech recognition until the first queue is empty.

12. The processing method according to claim 11, wherein: The sending of the input audio to the audio processing server further includes: In response to not receiving a confirmation message sent by the audio processing server after waiting for a preset time period, resending a most recently sent audio segment.

13. The processing method according to claim 11 or 12, wherein: The reply audio stream includes a plurality of audio data packets, each of the plurality of audio data packets includes a sequence number identifier of the audio data packet, and playing the reply audio stream includes: storing the current audio data packet into a second queue according to the sequence number identifier of the received current audio data packet; The audio data packets in the second queue are played in the order in which they appear.

14. The processing method according to claim 13, wherein: The step of storing the current audio data packet in the second queue according to the sequence number of the received current audio data packet comprises: determining, based on the received sequence number identifier of the current audio data packet, whether an audio data packet having the same sequence number identifier as the current audio data packet already exists in the second queue; If an audio data packet with the same sequence number as the current audio data packet already exists in the second queue, discard the current audio data packet; If there is no audio data packet with the same sequence number as the current audio data packet in the second queue, the current audio data packet is stored in the queue.

15. The processing method according to any one of claims 10 to 14, further comprising: In response to a network interruption, sending a retransmission request to the audio processing server, wherein the retransmission request includes a sequence number identifier of the last audio data packet received by the client before the network interruption; Receive a retransmitted audio data packet sent by the audio processing server, wherein the retransmitted data packet is a data packet that has been sent by the audio processing server to the client after the last audio data packet.

16. The processing method according to any one of claims 10 to 15, further comprising: Receiving a text stream of the text to be replied sent by the audio processing server; Displaying the text stream of the text to be replied; Receiving the reply text stream sent by the text processing server, wherein the reply text stream is generated by the text processing server after receiving the text to be replied sent by the audio processing server; The reply text stream is displayed.

17. The processing method according to any one of claims 10 to 16, further comprising: In response to the start of the audio interaction application, a long connection is established with the audio service gateway corresponding to the audio processing server, wherein the client transmits data with the audio processor through the audio service gateway.

18. An audio processing server, comprising: The conversion module is configured to convert the input audio sent by the client into a text to be replied; A first sending module is configured to send the text to be replied to to a text processing server to generate a reply text stream; a receiving module configured to receive a request from the client for a reply audio stream corresponding to the reply text stream, wherein the request includes a message identifier corresponding to the reply text stream, and the message identifier is sent by the text processing server to the client; an acquisition module configured to acquire the reply text stream from the text processing server according to the message identifier; A generating module, configured to generate the reply audio stream according to the reply text stream; The second sending module is configured to send the reply audio stream to the client for playing.

19. A client, comprising: A first sending module is configured to send the input audio to the audio processing server to convert it into a text to be replied, wherein the text to be replied is sent to the text processing server to generate a reply text stream; A first receiving module is configured to receive a message identifier corresponding to the reply text stream sent by the text processing server; a second sending module configured to send a request for a reply audio stream corresponding to the reply text stream to the audio processing server, wherein the request includes the message identifier; A second receiving module is configured to receive the reply audio stream sent by the audio processing server; The playing module is configured to play the reply audio stream.

20. An audio interaction processing system, comprising: The audio processing server as described in claim 18 and the client as described in claim 19.

21. The processing system of claim 20, further comprising: The text processing server is configured to generate a reply text stream based on the text to be replied converted from the input audio, and send the reply text stream to the audio processing server and the client.

22. The processing system of claim 21, wherein: The text processing server is further configured to receive the text to be replied sent by the audio processing server, and send a message identifier corresponding to the reply audio stream to the client.

23. An electronic device comprising: processor; as well as A memory coupled to the processor, for storing instructions, wherein when the instructions are executed by the processor, the processor executes the audio interaction processing method according to any one of claims 1 to 9, or the audio interaction processing method according to any one of claims 10 to 17.

24. A computer-readable storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, it implements the audio interaction processing method described in any one of claims 1-9, or the audio interaction processing method described in any one of claims 10-17.

25. A computer program product comprising: An instruction, which, when executed by a processor, implements the audio interaction processing method described in any one of claims 1-9, or the audio interaction processing method described in any one of claims 10-17.

26. A computer program comprising: An instruction, which, when executed by a processor, implements the audio interaction processing method described in any one of claims 1-9, or the audio interaction processing method described in any one of claims 10-17.

Images