Tts audio processing method and device, electronic equipment and storage medium

Abstract

This invention provides a TTS audio processing method, apparatus, electronic device, and storage medium. In response to a TTS service message from the cloud indicating the start of a semantic segment for the current semantic segment, data in a preset buffer is deleted. In response to an audio arrival message from the TTS service indicating the current semantic segment, the audio of the current semantic segment is written into the preset buffer. When the preset buffer meets preset conditions, a portion of the audio in the preset buffer is output to the user terminal according to a preset rhythm based on the association information of the current semantic segment, until a semantic segment end message from the TTS service indicating the end of the current semantic segment is received, after which all remaining audio in the preset buffer is output to the user terminal. Data in the preset buffer is then deleted after a semantic segment start message from the TTS service indicating the start of the next semantic segment. Using this invention ensures that TTS audio from the cloud side is played stably at a certain rhythm on the user terminal side.

Description

Technical Field

[0001] This invention relates to the field of voice communication technology, and in particular to a TTS audio processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] In scenarios such as IVR (Interactive Voice Response), intelligent customer service, telephone voice assistant, and real-time voice translation, cloud-based TTS (Text-to-Speech) services typically output audio in units of sentences or semantic segments. However, telephone networks require audio to be output continuously at a fixed rhythm. This mismatch in audio output methods between the cloud-based TTS service and the telephone network can easily lead to problems such as fluctuating speech speed, stuttering, missing sentence endings, and even overlapping audio between adjacent sentences during audio playback. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a TTS audio processing method, apparatus, electronic device and storage medium to alleviate the above-mentioned problems existing in the related art.

[0004] In a first aspect, embodiments of the present invention provide a TTS audio processing method, comprising: deleting data in a preset buffer in response to a semantic segment start message from a TTS service provided in the cloud for the current semantic segment; writing audio of the current semantic segment into the preset buffer in response to an audio arrival message from the TTS service for the current semantic segment; wherein the audio arrival message represents the output order and length of the audio of the current semantic segment; when the preset buffer meets preset conditions, outputting a portion of the audio in the preset buffer to a user terminal according to a preset rhythm based on the association information of the current semantic segment; and deleting data in the preset buffer after responding to a semantic segment end message from the TTS service for the current semantic segment, based on the association information and according to the preset rhythm.

[0005] Secondly, embodiments of the present invention also provide a TTS audio processing device, comprising: a start module, configured to delete data in a preset buffer in response to a semantic segment start message from a TTS service provided in the cloud for the current semantic segment; a write module, configured to write the audio of the current semantic segment into the preset buffer in response to an audio arrival message from the TTS service for the current semantic segment; wherein the audio arrival message represents the output order and length of the audio of the current semantic segment; an output module, configured to output a portion of the audio in the preset buffer to a user terminal according to a preset rhythm based on the association information of the current semantic segment when the preset buffer meets preset conditions; and an end module, configured to output all remaining audio in the preset buffer to the user terminal according to the preset rhythm based on the association information in response to a semantic segment end message from the TTS service for the current semantic segment, so as to delete the data in the preset buffer after responding to a semantic segment start message from the TTS service for the next semantic segment.

[0006] Thirdly, embodiments of the present invention also provide an electronic device, including a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the method described in the first aspect above.

[0007] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the method described in the first aspect above.

[0008] This invention provides a TTS audio processing method, apparatus, electronic device, and storage medium. In response to a TTS service message from the cloud indicating the start of a semantic segment for the current semantic segment, data is deleted from a preset buffer. In response to an audio arrival message from the TTS service indicating the current semantic segment, the audio of the current semantic segment is written into the preset buffer. When the preset buffer meets preset conditions, a portion of the audio in the preset buffer is output to the user terminal according to a preset rhythm based on the association information of the current semantic segment. In response to a TTS service message indicating the end of a semantic segment for the current semantic segment, the remaining audio in the preset buffer is output to the user terminal according to a preset rhythm based on the association information, so that data in the preset buffer is deleted after responding to a TTS service message indicating the start of the next semantic segment. By introducing a message triggering mechanism for individual semantic segments to perform semantic segment-level buffering of cloud-side TTS audio and outputting it to the user terminal at a certain rhythm, it is possible to ensure that cloud-side TTS audio plays stably at a certain rhythm on the user terminal side, thereby avoiding problems such as fluctuating speech speed, stuttering, missing sentence endings, or even overlapping audio of adjacent sentences when cloud-side TTS audio is played on the user terminal side.

[0009] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0010] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0011] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating a TTS audio processing method according to an embodiment of the present invention; Figure 2 This is an example diagram illustrating the switching relationship between the three states in an embodiment of the present invention; Figure 3 This is an example diagram of the overall architecture of the TTS audio processing system in an embodiment of the present invention; Figure 4This is an example diagram of the three-state workflow and time control method in an embodiment of the present invention; Figure 5 This is an example diagram illustrating the audio buffer control and output process in an embodiment of the present invention; Figure 6 This is an example diagram illustrating the implementation process of the TTS audio processing method in an IVR dynamic broadcast scenario according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of a TTS audio processing device according to an embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] Currently, the audio output methods of the cloud-based TTS service and the telephone network are incompatible, which can easily lead to problems such as fluctuating speech speed, stuttering, missing sentence endings, and even overlapping audio between adjacent sentences during audio playback.

[0015] Based on this, the present invention provides a TTS audio processing method, apparatus, electronic device and storage medium that can alleviate the above-mentioned problems existing in related technologies.

[0016] To facilitate understanding of this embodiment, a TTS audio processing method disclosed in this invention will first be described in detail. (See [link to relevant documentation]). Figure 1 As shown, the method may include the following steps: Step S102: In response to the semantic segment start message for the current semantic segment provided by the TTS service in the cloud, delete the data in the preset buffer.

[0017] Step S104: In response to the arrival message of the audio for the current semantic segment from the TTS service, write the audio of the current semantic segment into a preset buffer.

[0018] Among them, the audio arrival message can represent the output order and length of the audio of the current semantic segment.

[0019] Step S106: When the preset buffer meets the preset conditions, based on the association information of the current semantic segment, output part of the audio in the preset buffer to the user terminal according to the preset rhythm.

[0020] In step S108, in response to the semantic segment end message of the TTS service for the current semantic segment, based on the association information, all remaining audio in the preset buffer is output to the user terminal according to the preset rhythm, so as to delete the data in the preset buffer after responding to the semantic segment start message of the TTS service for the next semantic segment.

[0021] This invention provides a TTS audio processing method that, in response to a TTS service message from the cloud providing a semantic segment start message for the current semantic segment, deletes data from a preset buffer; in response to an audio arrival message from the TTS service for the current semantic segment, writes the audio of the current semantic segment into a preset buffer; when the preset buffer meets preset conditions, outputs a portion of the audio in the preset buffer to the user terminal according to a preset rhythm based on the association information of the current semantic segment; in response to a TTS service message from the TTS service providing a semantic segment end message for the current semantic segment, outputs all remaining audio in the preset buffer to the user terminal according to a preset rhythm based on the association information, so as to delete data in the preset buffer after responding to a TTS service message from the TTS service providing a semantic segment start message for the next semantic segment. By introducing a message triggering mechanism for individual semantic segments to perform semantic segment-level buffering of cloud-side TTS audio and outputting it to the user terminal at a certain rhythm, this method ensures that cloud-side TTS audio plays stably at a certain rhythm on the user terminal, thereby avoiding problems such as fluctuating speech speed, stuttering, missing sentence endings, or even overlapping audio between adjacent sentences when cloud-side TTS audio is played on the user terminal.

[0022] As one possible implementation, after the step of deleting data in the preset buffer described above, the TTS audio processing method may further include: marking the preset buffer as an idle state; After writing the audio of the current semantic segment into the preset buffer in step S104 above, the TTS audio processing method may further include: marking the preset buffer as a pre-buffered state. Before outputting a portion of the audio in the preset buffer to the user terminal according to a preset rhythm based on the association information of the current semantic segment in step S106 above, the above TTS audio processing method may further include: marking the preset buffer as a playback state.

[0023] As one possible implementation, the above-described TTS audio processing method may further include: when a preset buffer is marked as a pre-buffered state, recording the time corresponding to when the preset buffer is marked as a pre-buffered state as the pre-buffered start time, and starting a timer from the pre-buffered start time until the preset buffer meets a preset condition and then stopping the timer. Based on this, the preset condition for the preset buffer to meet the preset condition may include: the amount of audio data in the preset buffer reaching a preset amount of data, or the timing duration reaching a preset duration.

[0024] For example, taking sentences as a semantic segment, a buffer control module can be added between the cloud-based TTS output terminal and the telephone audio playback terminal (i.e., the user terminal at this time). This buffer control module does not determine when to play based on the arrival speed of the audio data. Instead, it takes the sentence start event, sentence end event, the audio data (i.e., the audio block of the sentence) itself, as well as the sentence identifier, the sequence number of the audio block of the sentence, and other related information as input to uniformly control the buffering and playback process of the audio data (i.e., outputting the audio data from the cloud-based TTS output terminal to the telephone audio playback terminal for playback). The buffer control module adopts three working modes as a whole, as shown in Table 1.

[0025] Table 1. Meaning and main behaviors of the three states of the buffer control module

[0026] Taking semantic fragments as an example, Figure 2 This demonstrates the switching relationship between the three states of the buffer control module: When a sentence start message indicating the occurrence of a sentence start event is received, the system enters the preparation phase for the new sentence, is in an idle state (IDLE), has no data to play, and waits for the new sentence audio to arrive. When an audio arrival message indicating the arrival of the first frame of a new sentence is received, the audio entering the new sentence continuously enters the buffer stage, switching from the idle state to the pre-buffering state. The buffer continuously accumulates audio data and does not output audio to the telephone side (i.e., the telephone audio playback end). The time corresponding to entering the pre-buffering state is used as the pre-buffering start time to start timing, and the timing duration is used as the pre-buffering waiting time (i.e., the pre-buffering duration). Once the audio data continuously enters the buffer and meets the pre-buffering conditions, it enters the stable playback stage, switching from the pre-buffering state to the playback state. Every 20ms, the corresponding audio data is retrieved from the buffer to form a frame (320 bytes for PCM format audio) and output to the telephone side. The pre-buffering conditions are met when either of the following conditions is met simultaneously: the accumulated audio data in the buffer reaches a preset byte threshold, or the pre-buffering waiting time reaches a preset duration (i.e., a preset pre-buffering duration threshold). Once a sentence end message indicating the occurrence of a sentence end event is received and all remaining audio in the buffer has been sent to the telephone side, the playback state is switched to the idle state. Specifically, after receiving the sentence end message, all remaining data in the buffer is forcibly emptied and framed, and then output to the telephone side frame by frame. If the remaining data in the buffer is less than one frame, zeros are added to form a frame before it is output to the telephone side.

[0027] in accordance with Figure 2It can be seen that the buffer control module achieves audio playback by triggering switching actions between three states: the start of a new sentence, the arrival of the new sentence audio, the continuous accumulation of the new sentence audio in the buffer, the end of the new sentence, and the output of all audio in the buffer to the telephone side. It is not a simple continuous audio stream playback.

[0028] As one possible implementation, the audio of the current semantic segment may include several audio blocks of the current semantic segment; the semantic segment start message, audio arrival message, and semantic segment end message can all be generated by the TTS service based on the semantic boundary information of the current semantic segment; the semantic boundary information may include at least: unique identifier information representing the current semantic segment, first order information of the current semantic segment in the corresponding whole broadcast content, boundary type information representing the start or end of the current semantic segment, audio format information, and second order information of each audio block of the current semantic segment, as well as text segment information and / or text position information corresponding to the current semantic segment. Accordingly, the semantic segment start message, audio arrival message, semantic segment end message, and each audio block of the current semantic segment may carry the identifier information corresponding to the current semantic segment.

[0029] As one possible implementation, the above-mentioned TTS audio processing method may further include: when the preset buffer is marked as a playback state, recording the time corresponding to when the preset buffer is marked as a playback state as the playback start time.

[0030] As one possible implementation, the associated information may include identification information, first order information, audio format information, and second order information. Based on this, the step S106 above, which outputs a portion of the audio in the preset buffer to the user terminal according to a preset rhythm based on the associated information of the current semantic segment, may include: a) determining the preset rhythm based on the playback start time and the preset output period; b) based on the identification information, first order information, audio format information, and second order information, starting from the playback start time, framing the portion of the audio blocks in the preset buffer according to the preset rhythm and outputting them to the user terminal frame by frame; wherein, the time of each frame output to the user terminal is determined based on the playback start time, the preset output period, and the number of frames currently output to the user terminal.

[0031] As one possible implementation, step S108 above, which outputs all remaining audio in the preset buffer to the user terminal based on the association information and according to a preset rhythm, may include: based on the identification information, the first order information, the audio format information, and the second order information, continuing to frame all remaining audio blocks in the preset buffer and outputting them to the user terminal frame by frame according to a preset rhythm; wherein, when outputting the last frame to the user terminal, if the current remaining audio block in the preset buffer is less than one frame, the current remaining audio block is padded with zeros to form one frame and output to the user terminal.

[0032] Continuing from the previous example, the aforementioned semantic boundary information can come from any of the following sources: (1) Direct output of cloud TTS service: When the cloud TTS service streams back the audio converted from text, it also returns boundary events such as “sentence start”, “sentence end” or equivalent meaning as semantic boundary information. (2) Output of the semantic segmentation module upstream of the cloud TTS service: Before the text is sent into TTS, the upstream semantic segmentation module performs sentence segmentation, segmentation or semantic boundary recognition on the text to obtain the boundary result as semantic boundary information and synchronously transmits it to the local TTS adaptation layer (located between the cloud TTS service side and the telephone side). (3) Normalization generation of TTS adaptation layer: If the cloud TTS service returns lower-level callback, tag or fragment information, the local TTS adaptation layer will convert the content returned by the cloud TTS service into a unified format of boundary events (i.e. sentence start event, sentence end event, etc.) as semantic boundary information.

[0033] The specific data content of semantic boundary information may include: the identification information corresponding to the current sentence, the order information of the current sentence in the entire broadcast, the boundary type information (including at least the "start" information indicating the beginning of the sentence and the "end" information indicating the end of the sentence), the text segment or text position range corresponding to the current sentence, the order information of each audio block within the same sentence, the audio format information (such as audio encoding format, sampling rate, number of channels, number of bytes per frame, etc.), the total audio length prompt information of the sentence, and the source identifier indicating that the semantic boundary information comes from the cloud TTS service, the upstream semantic segmentation module, or the TTS adaptation layer, etc.

[0034] The start of a new sentence can be determined by receiving a sentence start event (i.e., a sentence start message) carrying a new sentence identifier. The specific determination process is as follows: when the TTS adaptation layer receives a sentence start callback or segment start marker from upstream, it generates the corresponding sentence start message. The segment start marker can be pre-marked by the semantic segmentation module upstream of the TTS service or directly marked using a custom marker; there is no limitation on this. Subsequent audio blocks of the current active sentence all carry the same sentence identifier. When a sentence start message corresponding to a new sentence identifier is received, it is determined that a new sentence has started. In other words, the start of a new sentence is determined by combining sentence boundary events (i.e., sentence start events) with sentence identifiers, rather than by inferring from audio frame timestamps.

[0035] Following the previous example, Figure 3The document demonstrates the link relationship of "Boundary Information Source - TTS Adaptation Layer Normalization - Buffer Control Module Processing". That is, the upstream first provides boundary information (i.e., sentence events) representing the start or end of a sentence. The TTS adaptation layer then organizes this boundary information into a unified format control message (such as sentence start message and sentence end message) and inputs it into the buffer control module. Finally, the buffer control module performs a clearing action (i.e., clearing the audio data in the buffer), a pre-buffering action (i.e., continuously accumulating the audio data in the buffer), a playback action (i.e., taking audio data from the buffer every 20ms to form a frame and outputting it to the telephone network side for playback), or a draining action (i.e. taking all the remaining audio data in the buffer to form a frame and outputting it to the telephone network side) according to the input control message.

[0036] Figure 3 The overall architecture of the TTS audio processing system was also demonstrated, including: a cloud-based TTS service, a TTS adaptation layer, a buffer control module, and a telephone sending module. The cloud-based TTS service converts text into PCM format audio using a TTS synthesis engine and returns streaming results to the TTS adaptation layer in segments (e.g., text + audio block sequence number + audio block, or audio block sequence number + audio block). The TTS adaptation layer normalizes the streaming results into the required data structure and then generates corresponding semantic events (e.g., sentence start event, sentence end event) and sends messages to the buffer control module (e.g., sentence start message, audio block arrival message, sentence end message). The cloud-based TTS service burstily returns audio data streams (i.e., PCM format audio blocks) to the TTS adaptation layer. The buffer control module updates its internal state based on the messages (i.e., switches between three states) to store the PCM format audio blocks in a buffer and outputs fixed-beat audio frames to the telephone sending module (i.e., timing is performed in an independent thread using a playback timer to retrieve audio blocks from the buffer every 20ms to assemble a frame and output that frame video to the telephone sending module). The messages sent by the TTS adaptation layer to the buffer control module are mainly divided into three categories: sentence start message, audio block arrival message, and sentence end message, as shown in Table 2.

[0037] Table 2. Sending Relationships, Main Contents, and Functions of the Three Types of Messages

[0038] When necessary, the messages sent by the TTS adaptation layer to the buffer control module may also include extended types such as exception messages indicating abnormalities, cancellation messages indicating cancellation of audio output, and retransmission messages indicating re-output of audio, without limitation.

[0039] In practical applications, an order guarantee mechanism can be introduced. Specifically, the three types of messages can be processed sequentially by the same message dispatch thread or the same event loop. The processing order of the three types of messages can be guaranteed through any of the following methods: (1) The TTS adaptation layer serially parses the callback results on the same streaming connection (i.e., the streaming results returned by the cloud TTS service to the TTS adaptation layer in segments according to sentences) and serially delivers the sentence start message, audio block arrival message and sentence end message to the buffer control module. (2) Use the same sentence identifier for messages with the same sentence, and increment the sequence number of the audio blocks of the same sentence according to the order of audio block reception; (3) The buffer control module only takes messages from the same ordered queue for processing (i.e., delivers them to the buffer control module). Specifically, the message at the beginning of a sentence corresponding to a certain sentence enters the queue first, followed by the arrival messages of several audio blocks corresponding to the sentence, and finally the message at the end of the sentence corresponding to the sentence enters the queue. The buffer control module processes the messages in the order they enter the queue without having to sort them further according to the timestamp of each audio frame.

[0040] Following the previous example, Figure 4 The sequence is displayed from left to right, showing the stages of receiving the sentence start message, entering the idle state, continuously writing audio blocks to the buffer, entering the pre-buffered state, entering the playback state, sending frames at fixed beats, receiving the sentence end message, and forcibly emptying the audio at the end of the sentence. For example... Figure 4 As shown, the three-state workflow and time control method can be described as follows: After receiving the sentence start message corresponding to sentence S1, the buffer with the buffer enters the idle state and waits for the first audio block of sentence S1 ( Figure 4 When the audio block corresponding to sentence S1 arrives (represented by a yellow rectangle labeled "Block 1"), the buffer enters a pre-buffering state and continuously writes the audio block corresponding to sentence S1 into the buffer until the buffer size reaches a preset byte threshold or the pre-buffering duration reaches a preset duration (e.g., 200ms). Then, the buffer enters a playback state, retrieving audio blocks from the buffer every 20ms to form a frame and outputting it to the phone. Upon receiving the sentence end message corresponding to sentence S1, the buffer forcibly empties the remaining audio blocks (i.e., sentence-ending audio) in the buffer at a rhythm of one frame output every 20ms, then frames them and outputs them to the phone. If the remaining audio blocks in the buffer are less than one frame, they are padded with zeros to form a frame and output to the phone. This process continues until the sentence start message corresponding to sentence S2, which is the new sentence, is received, after which the buffer enters an idle state and sequentially goes through the stages of continuously writing audio blocks into the buffer, entering the pre-buffering state, entering the playback state, sending frames at a fixed tempo, receiving the sentence end message, and forcibly emptying the sentence-ending audio. Figure 4It is known that the core of the three-state workflow and time control method is not to play the audio directly as soon as it arrives, but to first buffer and control the audio according to the boundary messages corresponding to each sentence before outputting it stably to the telephone side. This is different from the traditional TTS audio output method, which uses the processing logic of receiving the audio first and then playing it in order of timestamps.

[0041] As one possible implementation, the above-described TTS audio processing method may further include: Step A1: In the process of outputting part of the audio in the preset buffer to the user terminal according to the preset rhythm based on the association information of the current semantic segment in step S106 above, the framing state for framing the audio blocks of the current semantic segment is recorded, and the audio block processing progress, which represents the progress of the audio blocks of the current semantic segment being framed and output to the user terminal, is recorded.

[0042] Step A2: After responding to the TTS service's semantic segment end message for the current semantic segment, record the end flag that indicates the end of the current semantic segment.

[0043] Step A3: In step S108 above, after outputting all the remaining audio in the preset buffer to the user terminal according to the preset rhythm based on the association information, a blanking flag is recorded indicating that all audio blocks representing the current semantic segment have been framed and output to the user terminal.

[0044] Accordingly, before the step of marking the preset buffer as idle, the TTS audio processing method may further include: recording the identification information corresponding to the current semantic segment, and deleting the frame state, audio block processing progress, end flag, and empty flag corresponding to the previous semantic segment.

[0045] Continuing from the previous example, when the buffer control module receives a sentence start message, the buffer control module performs the following operations: clear all data in the current buffer; clear the residual frame assembly state of the previous sentence; record a new sentence identifier to uniquely identify the currently active sentence (i.e., the current sentence); clear the end flag, empty flag, and playback progress of the previous sentence; and restore the state to the idle state (the initial state for the first sentence, which is ready to receive audio from the new sentence). Among them, the residual frame assembly state of the previous sentence is recorded by the buffer control module when it retrieves audio blocks from the buffer to assemble frames for the previous sentence; the end flag of the previous sentence is recorded by the buffer control module after receiving the sentence end message corresponding to the previous sentence; the empty flag of the previous sentence is recorded by the buffer control module after emptying the remaining audio blocks corresponding to the previous sentence in the buffer; and the playback progress of the previous sentence is recorded by the buffer control module together with the frame assembly state of the previous sentence when it retrieves audio blocks from the buffer to assemble frames. The above frame assembly state may include: residual bytes that have not filled a frame, the number of bytes accumulated in the current frame, the sentence identifier of the current active sentence, the audio block processing progress of the current active sentence (used to indicate whether the audio block of the current active sentence has been played), the pre-buffering start time, the playback start time, and the sentence end empty flag. The action triggered by the sentence start event (i.e., receiving a sentence start message) ensures that the ending sound of the previous sentence does not mix into the next sentence, providing a clean buffer environment for the new sentence and improving the buffer control granularity from "continuous audio stream" to "sentence lifecycle".

[0046] Continuing from the previous example, when the buffer control module receives a sentence end message, it does not immediately interrupt audio playback. Instead, it performs the following operations: sets the end flag of the current sentence to true; maintains playback or enters the forced emptying process of the end-of-sentence audio, continuing to extract audio blocks from the buffer at a fixed rhythm (such as outputting one frame every 20ms) and outputting them frame by frame to the telephone side for playback; if the remaining audio blocks in the buffer are insufficient for a complete frame, the remaining audio blocks in the buffer are padded with zeros to form a complete frame and sent to the telephone side for playback; after the buffer control module has sent all audio blocks of the current sentence to the telephone side, it completes the emptying of the end-of-sentence audio blocks and restores the state of the buffer to the idle state.

[0047] The action triggered by the sentence end event (i.e., receiving a sentence end message) ensures that the audio at the end of the current sentence is not missed, and that the next sentence is only played after the entire sentence has finished playing. In other words, the role of the sentence end event is not just to record, but to participate in the buffer state transition as a sentence end control signal.

[0048] For ease of understanding, Figure 5 This further demonstrates the entire process of the buffer completing clearing, resetting, pre-buffering, fixed-beat frame sending, and sentence end emptying driven by the "sentence start event - audio block arrival - sentence end event" sequence. This process can convert the audio bursts from upstream sentence levels into a stable frame stream output from the telephone side. For example... Figure 5 As shown, the TTS audio processing system mainly includes an event input layer, a buffer control layer (including a buffer control module and a buffer with a buffer), and a telephone-side output layer. The audio buffer control and output process can mainly include the following steps: Step S1: Initially, the system is in an idle state.

[0049] After the TTS audio processing system starts, the buffer is in an idle state. At this time, there is no audio to be played, and it waits for the sentence start message and the arrival message of the subsequent audio block to arrive.

[0050] Step S2: After receiving the new sentence audio, enter the pre-buffering state.

[0051] When audio data belonging to the current sentence (carrying the sentence identifier corresponding to the current sentence) begins to arrive, the audio data first enters the buffer, the buffer changes from an idle state to a pre-buffered state, and the pre-buffering start time is recorded at the same time; the arrival time of the first audio block is not itself used as the frame sending time reference, and a fixed frame sending beat is only established when the playback state is entered.

[0052] The method for determining when new sentence audio data begins to arrive is as follows: if the sentence start message corresponding to the sentence has been received first, and then the arrival message of the first audio block carrying the same sentence identifier is received, then it is confirmed that the sentence audio has begun to enter the buffer (i.e., the sentence audio has begun to arrive).

[0053] Step S3: Once the pre-buffering conditions are met, playback begins.

[0054] In pre-buffered mode, the buffer transitions to playback mode when either of the following conditions is met: A) the accumulated audio data in the buffer reaches a preset byte threshold; B) the pre-buffering wait time reaches a preset duration. This condition-triggered mechanism prioritizes accumulating sufficient audio data in the buffer under normal circumstances to improve playback stability, and prevents indefinite waiting even if the audio arrives slowly, thus controlling startup latency.

[0055] Once playback begins, the local monotonic clock value is recorded as the start time of playback, and this is used as the starting point for timed frame transmission.

[0056] Step S4: Output frame by frame at a fixed period during playback.

[0057] During playback, the TTS audio processing system retrieves only one standard audio frame from the buffer at a time and sends it to the phone side at a fixed rhythm (e.g., sending an audio frame every 20ms).

[0058] For example, the time when the k-th audio frame is sent to the telephone side can be represented as: Playback start time + k × fixed transmission period The fixed transmission period is, for example, 20ms. The timing duration of the local timer can be set to a fixed transmission period, so that the buffer control module is triggered to send an audio frame to the telephone side every fixed transmission period under the timing action of the local timer.

[0059] Using the fixed-period frame-by-frame output method in step S4 above, instead of relying on each frame to have an independent timestamp to determine when to send an audio frame to the telephone side, the frame sending rhythm is determined by the local timer and the playback start time. The sentence boundary event is responsible for managing "when to start a round of sentence-level playback and when to end a round of sentence-level playback", rather than timestamping each audio frame.

[0060] Figure 4 The text indicates three key time points: "pre-buffering start point", "start of frame generation at fixed beat", and "end point of audio emptying at the end of sentence".

[0061] During the assembly of standard audio frames, the following operations are performed: If the remaining data of an audio block in the buffer is less than one frame, the next audio block in the buffer can be spliced; if there is already audio block data in the buffer but less than one frame, and the sentence end event has not yet been received, the system continues to wait for subsequent audio block data to enter the buffer before splicing the existing audio block data in the buffer with the subsequent audio block data; if the sentence end event has been received and the remaining audio block data in the buffer is still less than one frame, the remaining audio block data in the buffer is padded with zeros to form one frame and then sent to the telephone side; only one frame is output to the telephone side in each transmission cycle to match the telephone network's requirement for a fixed beat.

[0062] Step S5: Return to idle state after playback is complete.

[0063] The buffer returns to an idle state when the following conditions are met: the end-of-sentence message corresponding to the current sentence has been received, and all audio data associated with that sentence in the buffer has been sent to the telephone side; if a trailing frame exists (padding with zeros), its transmission to the telephone side has also been completed. Afterward, the buffer waits for the start-of-sentence message corresponding to the next sentence.

[0064] For ease of understanding, combined with Figures 2 to 4 The sentence boundary perception and control process of the above TTS audio processing method can be summarized as follows: Figure 3 In the process, the TTS adaptation layer first obtains sentence boundary information from the cloud TTS or the upstream semantic segmentation module. Then, the TTS adaptation layer converts the boundary information into unified sentence start messages and sentence end messages and generates audio block arrival messages to represent the arrival of sentence-level audio data. Figure 2The state machine of the buffer control module takes these messages (i.e., sentence start message, sentence end message, and audio block arrival message) as input to perform buffer control based on state switching: when a sentence start message is received, clearing and resetting are performed; when an audio block arrival message indicating the arrival of the first audio block of a sentence is received, the module enters the pre-buffering state; when the pre-buffering conditions are met (i.e., the accumulated audio data in the buffer reaches a preset byte threshold or the pre-buffering waiting time reaches a preset duration), the module enters the playback state; after receiving a sentence end message, the end-of-sentence audio is discharged. Figure 4 The temporal relationship is as follows: first, there is the sentence boundary event of the current sentence, then the audio block of the sentence is written, then frames are sent at regular intervals, and finally the end audio of the sentence is drained and finished after the sentence end event arrives.

[0065] Therefore, the above-mentioned "sentence boundary awareness" does not simply know the beginning and end of a sentence, but rather structures the boundary information of a sentence into corresponding boundary messages and makes them directly participate in buffer state transitions and transmission timing control.

[0066] In practical applications, a threaded processing mechanism can be introduced, where the same message dispatch thread sequentially processes the sentence start event, sentence end event, and TTS audio data writing. Introducing this threaded processing mechanism ensures that the sentence start event for the current sentence arrives before the first audio block of that sentence, and the sentence end event arrives after the last audio block of that sentence. Messages and audio blocks from different sentences will not be processed out of order at the buffer control layer.

[0067] Figure 3 In this system, the playback timer task (i.e., timing the playback via a playback timer so that audio plays at fixed 20ms intervals) can run in a separate thread. It coordinates with the write thread, which performs the buffer audio writing task, through necessary synchronization mechanisms to ensure data consistency. This separate thread primarily determines the timing of audio output to the telephone side based on the playback start time and the fixed sending cycle, and is not responsible for reconstructing sentence boundary order.

[0068] The aforementioned TTS audio processing method directly drives the buffer's audio buffering control through sentence boundary events. Sentence boundary events are not supplementary information, but rather control inputs that directly enter the buffer control link. Specifically, the sentence start event triggers buffer clearing and resetting, the arrival of the current sentence audio block triggers pre-buffering and playback state transition, and the sentence end event triggers remaining audio flushing and tail termination.

[0069] The aforementioned TTS audio processing method introduces a sentence-level lifecycle control mechanism. Instead of treating all audio as a single continuous stream, it treats each sentence as having an independent lifecycle. Specifically, at the beginning of a sentence, the context of the new sentence is established (e.g., sentence identifier, audio block processing progress, playback start time, end marker, emptying marker, playback progress, etc.) and the residual context from the previous sentence is cleared. In the middle of a sentence, pre-buffering is performed, and playback is stable and rhythmic. At the end of a sentence, the remaining audio is emptyed, and zeros are added if necessary to complete the last frame transmission. This sentence-level lifecycle control method helps reduce the probability of residual audio from the previous sentence interfering with the next sentence, ensures that the last audio of each sentence is output as completely as possible, and makes buffering control more aligned with the actual burst-back pattern of TTS audio.

[0070] The aforementioned TTS audio processing method employs a fixed-beat frame transmission method oriented towards telephone networks. Upon entering playback mode, instead of directly outputting audio based on the arrival speed of audio blocks, it records a uniform playback start time and outputs a standard audio frame at a fixed transmission cycle, with one standard audio frame output after each fixed transmission cycle. This fixed-beat frame transmission method can convert bursty audio blocks from upstream into a stable beat frame stream acceptable to the telephone side, which is more conducive to maintaining audio playback continuity under telephone network constraints.

[0071] The aforementioned TTS audio processing method also considers low latency, smooth playback, and sentence integrity. It employs a conditional triggering mechanism based on the accumulated audio data volume in the buffer or the pre-buffering wait time. This minimizes startup wait time while avoiding the instability caused by playing audio immediately upon arrival. Using this conditional triggering mechanism, a relatively stable playback experience can be achieved with lower latency, avoiding crosstalk between sentences, reducing sentence truncation, and resulting in a more regular audio frame sequence received by the telephone side.

[0072] The aforementioned TTS audio processing method has a clear and feasible engineering implementation path. By processing sentence start events, sentence end events, and audio write messages sequentially through the same message distribution thread, it ensures that: the sentence start event corresponding to the current sentence enters the control link before the first audio block of that sentence, and the sentence end event arrives after the last audio block of that sentence enters the control link. Control information and audio data from different sentences do not overlap disorderly in the buffer layer. Simultaneously, the playback thread only needs to send frames according to the playback start time and a fixed period, without introducing additional complex frame-by-frame timestamp alignment and out-of-order reordering logic. Therefore, the aforementioned TTS audio processing method is not only complete in its technical approach but also highly operable in its implementation path.

[0073] To facilitate understanding, the implementation process of the above TTS audio processing method is described below with an example in a specific application scenario.

[0074] Example 1: IVR dynamic broadcast scenario.

[0075] In the IVR system, after a user dials the customer service hotline, the IVR system dynamically generates a broadcast text based on the query results (such as querying balance, order status, ticket information, etc.), and then calls the cloud-based TTS service to generate the voice.

[0076] In Example 1, the upstream text is first segmented into several sentences, or the cloud-based TTS service directly outputs sentence start and sentence end events during the text-to-speech synthesis process. The TTS adaptation layer generates a unique sentence identifier for each sentence and performs the following operations: encapsulates the sentence start event into a sentence start message, encapsulates the audio blocks within the sentence into audio block arrival messages, and encapsulates the sentence end event into a sentence end message.

[0077] When the buffer control module receives a sentence start message, it clears the residual data from the previous sentence and resets the frame assembly state; upon receiving the arrival message of the first audio block of the sentence, it enters the pre-buffering state; upon reaching the threshold or timeout, it enters the playback state and sends one frame every 20ms starting from the playback start time; upon receiving a sentence end message, it continues to send the remaining audio, padding the last frame with zeros if necessary. This makes the playback sound smoother and more natural to the user.

[0078] Figure 6 The demonstration showcased the complete implementation chain in IVR dynamic broadcast scenarios, from the business system generating broadcast text, the TTS adaptation layer outputting sentence boundary events and audio blocks, to the buffer control module executing sentence beginning clearing, pre-buffering, fixed-beat frame sending, and sentence end clearing, and finally to smooth playback on the telephone side.

[0079] Figure 6In the process, the business system generates dynamically broadcast text (such as balance, order, and ticket information) and sends the text to the cloud-based TTS service. The cloud-based TTS service sends a text request to the TTS adaptation layer. Upon responding to the text request, the TTS adaptation layer receives the sentence start event from the cloud-based TTS service for the current sentence and sends a sentence start message corresponding to the current sentence to the buffer control module. Upon receiving the sentence start message, the buffer control module clears the old data in the buffer and resets the frame assembly state, entering an idle state. The TTS adaptation layer receives the audio block of the current sentence and forwards it to the buffer control module. When the buffer control module receives the first audio block of the current sentence, the TTS adaptation layer sends an audio block arrival message corresponding to the current sentence to the buffer control module. Upon receiving the audio block arrival message, the buffer control module writes the audio block corresponding to the current sentence into the buffer, and the audio block enters the buffer, beginning pre-buffering (i.e., entering the pre-buffering state). When the pre-buffering conditions are met, playback begins. The TTS adaptation layer retrieves audio blocks from the buffer every 20ms to form a standard audio frame and sends it to the phone side. The phone side receives a standard audio frame every 20ms and plays it. The TTS adaptation layer receives the sentence end event from the cloud TTS service for the current sentence and sends the corresponding sentence end message to the buffer control module. After receiving the sentence end message, the buffer control module continues to retrieve the remaining audio blocks from the buffer every 20ms and empty the remaining audio blocks in the buffer. If the remaining audio blocks in the buffer are less than one frame, they are padded with zeros to form a frame before being sent to the phone side. This process continues until the buffer control module has framed all the remaining audio blocks in the buffer and sent them to the phone side. Then, a new round of TTS audio processing begins for the next sentence (i.e., the implementation chain from the TTS adaptation layer outputting sentence boundary events and audio blocks, to the buffer control module performing sentence beginning clearing, pre-buffering, fixed-beat frame sending, and sentence end emptying, and finally to smooth playback on the phone side). This process enables smooth and continuous playback of TTS audio on the phone side, without overlapping or crosstalk between different sentence audios, and without sentence truncation.

[0080] Example 2: Real-time speech translation scenario.

[0081] In real-time telephone translation scenarios, the speaker's voice is sent to the cloud for recognition, translation, and TTS synthesis, and then the translated audio is played back to the listener.

[0082] Since translation and TTS are typically processed sentence by sentence, the returned audio exhibits segmented, bursty characteristics. The aforementioned TTS audio processing method, through a sentence boundary event management buffer, can stably convert bursty translated audio into a fixed-rhythm telephone audio stream of 20ms, thereby improving the listener's experience and especially avoiding crosstalk between different sentences in the translated text or later sentences interrupting the end of earlier sentences.

[0083] Example 3: Telephone voice assistant scenario.

[0084] In telephone voice assistant scenarios, the large model first generates the reply text, and then the speech is synthesized sentence by sentence by streaming TTS.

[0085] Because the generation speed of large models can fluctuate, the speech output by TTS is prone to sentence-level bursts. The TTS audio processing method described above can smoothly handle these sentence-level bursts, preventing changes in the generation speed of large models from directly affecting the playback effect. Especially when the reply content contains multiple short sentences, sentence identifiers and sentence boundary events can be used to allow each sentence to independently undergo the lifecycle of "clear-pre-buffering-playback-drain".

[0086] In an alternative approach, the pre-buffering duration or pre-buffering threshold does not necessarily have to be a fixed value and can be dynamically adjusted based on historical audio arrival speeds. However, in most practical scenarios, a fixed pre-buffering duration or a fixed pre-buffering threshold can already balance stability and implementation complexity; therefore, a fixed value is preferred for the pre-buffering duration or pre-buffering threshold.

[0087] If the TTS service itself does not directly provide sentence start / end events, sentence start / end events can be approximated in the following ways: inferred from text segmentation results; inferred from silence detection results; inferred from upstream semantic segmentation results.

[0088] However, the above approximate inference method is an alternative method. The preferred approach is to directly obtain sentence boundary events through TTS service or generate sentence boundary events through TTS adaptation layer normalization, and use sentence boundary events to drive playback-side buffer control.

[0089] If complete silence affects the listening experience while waiting for the next audio line, attenuated background or ambient sounds can be sent during idle periods to reduce jarring. This is an optional enhancement solution.

[0090] In another alternative, the semantic boundary is not necessarily limited to a complete sentence; it can also be a phrase, a sense group, or other semantic fragment, as long as the semantic boundary information is used to trigger the control process of clearing, pre-buffering, emptying, and sending frames.

[0091] The meanings, typical values, and explanations of the key parameters involved in the above TTS audio processing methods are shown in Table 3.

[0092] Table 3. Meaning, Typical Values ​​and Explanations of Key Parameters

[0093] A shorter pre-buffering time results in faster startup but relatively weaker playback stability; conversely, a longer pre-buffering time leads to more stable playback but also increases startup latency. Therefore, the value of the pre-buffering time or pre-buffering time threshold should strike a balance between playback smoothness and startup latency.

[0094] In the aforementioned TTS audio processing method, the buffer is cleared and reset at the beginning of a sentence, and the remaining audio is forcibly drained at the end of a sentence. It not only identifies sentence boundaries but also uses information such as sentence identifiers, boundary types, and audio block order to control the state machine. It introduces switching methods between idle, pre-buffered, and playback states, as well as a dual-condition triggering mechanism for switching from the pre-buffered state to the playback state. It utilizes sentence boundary information to convert bursty audio into a uniform, continuous frame stream, ensuring the correct order of sentence boundary events and audio data without relying on additional complex synchronization mechanisms, and completing fixed-beat audio output without frame-by-frame timestamps. It also introduces a complete output mechanism that supports assembling standard frames across audio blocks and zero-padding the last frame before transmission.

[0095] In summary, the beneficial effects of the above-mentioned TTS audio processing methods are mainly reflected in the following aspects: (i) Achieve smoother playback with lower latency: Stable playback can be achieved without relying on excessively deep buffering. For example, a pre-buffering duration of around 200ms can effectively convert burst output of TTS audio into continuous and smooth playback acceptable to the telephone side.

[0096] (ii) Ensure the complete playback of sentences and avoid crosstalk between sentences: By clearing the beginning of a sentence and emptying the end of a sentence, the last sound of the previous sentence can be prevented from mixing into the next sentence, and the end of a sentence can be played completely without being truncated.

[0097] (iii) More suitable for cloud-based TTS scenarios: Instead of treating TTS audio as a traditional network voice stream, it takes into account the essential characteristic of TTS audio "sudden return by sentence", and is therefore more suitable for cloud-based TTS audio playback links.

[0098] (iv) It does not rely on the timestamp of each frame for sorting, making engineering implementation more direct: Since sentence boundary events and audio data can be processed sequentially within the same message thread, the playback thread only needs to send frames according to the playback start time and fixed period. Therefore, it is not necessary to introduce complex frame-by-frame timestamp alignment, disordered reordering or multi-threaded event merging logic.

[0099] Based on the above-described TTS audio processing method, this invention also provides a TTS audio processing device, see [link to related document]. Figure 7 As shown, the device may include the following modules: The start module 702 is used to delete data in the preset buffer in response to the semantic segment start message for the current semantic segment provided by the TTS service in the cloud.

[0100] The writing module 704 is used to write the audio of the current semantic segment into the preset buffer in response to the audio arrival message of the TTS service for the current semantic segment; wherein the audio arrival message represents the output order and length of the audio of the current semantic segment.

[0101] The output module 706 is used to output a portion of the audio in the preset buffer to the user terminal according to a preset rhythm, based on the association information of the current semantic segment, when the preset buffer meets the preset conditions.

[0102] The termination module 708 is used to respond to the semantic segment termination message of the TTS service for the current semantic segment, and output all the remaining audio in the preset buffer to the user terminal according to the preset rhythm based on the association information, so as to delete the data in the preset buffer after responding to the semantic segment start message of the TTS service for the next semantic segment.

[0103] By using the aforementioned TTS audio processing device, a message triggering mechanism for individual semantic segments is introduced to buffer the cloud-side TTS audio at the semantic segment level and output it to the user terminal at a certain rhythm. This ensures that the cloud-side TTS audio plays stably at a certain rhythm on the user terminal side, thereby avoiding problems such as fluctuating speech speed, stuttering, missing sentence ends, or even overlapping audio of adjacent sentences when the cloud-side TTS audio is played on the user terminal side.

[0104] The TTS audio processing device provided in this embodiment of the invention has the same implementation principle and technical effects as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0105] This invention also provides an electronic device, such as... Figure 8 The diagram shows the structure of the electronic device, which includes a processor 81 and a memory 80. The memory 80 stores computer-executable instructions that can be executed by the processor 81. The processor 81 executes the computer-executable instructions to implement the TTS audio processing method described above.

[0106] exist Figure 8 In the illustrated embodiment, the electronic device further includes a bus 82 and a communication interface 83, wherein the processor 81, the communication interface 83, and the memory 80 are connected via the bus 82.

[0107] The memory 80 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 83 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 82 may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 82 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0108] The processor 81 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the aforementioned TTS audio processing method can be completed through the integrated logic circuitry in the hardware of the processor 81 or through software instructions. The processor 81 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the TTS audio processing method disclosed in the embodiments of this invention can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory, and the processor 81 reads the information in the memory and, in conjunction with its hardware, completes the steps of the TTS audio processing method of the aforementioned embodiment.

[0109] This invention also provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are called and executed by a processor, they cause the processor to implement the aforementioned TTS audio processing method. For specific implementation details, please refer to the foregoing method embodiments, which will not be repeated here.

[0110] The computer program products of the TTS audio processing method, apparatus and electronic device provided in the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0111] Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0112] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0113] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0114] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method of TTS audio processing, the method comprising: include: In response to the TTS service provided by the cloud for the start of the semantic segment message for the current semantic segment, delete the data in the preset buffer; In response to the audio arrival message of the TTS service for the current semantic segment, the audio of the current semantic segment is written into the preset buffer; wherein, the audio arrival message represents the output order and length of the audio of the current semantic segment; When the preset buffer meets the preset conditions, based on the association information of the current semantic segment, a portion of the audio in the preset buffer is output to the user terminal according to a preset rhythm; In response to the TTS service's semantic segment end message for the current semantic segment, based on the association information, all remaining audio in the preset buffer is output to the user terminal according to the preset rhythm, so as to delete the data in the preset buffer after responding to the TTS service's semantic segment start message for the next semantic segment.

2. The method of claim 1, wherein, After deleting the data in the preset buffer, the process also includes: The preset buffer is marked as idle. After writing the audio of the current semantic segment to the preset buffer, the process also includes: The preset buffer is marked as a pre-buffered state; Before outputting a portion of the audio in the preset buffer to the user terminal according to a preset rhythm based on the association information of the current semantic segment, the process further includes: The preset buffer is marked as being in playback mode.

3. The method of claim 2, wherein, Also includes: When the preset buffer is marked as a pre-buffered state, the time corresponding to when the preset buffer is marked as a pre-buffered state is recorded as the pre-buffered start time, and the timer starts from the pre-buffered start time until the preset buffer meets the preset conditions and then the timer stops. The preset buffer meets the preset conditions including: the amount of audio data in the preset buffer reaches a preset amount of data, or the timing duration reaches a preset duration.

4. The method of claim 3, wherein, The audio of the current semantic segment includes several audio blocks of the current semantic segment; the semantic segment start message, the audio arrival message and the semantic segment end message are all generated by the TTS service based on the semantic boundary information of the current semantic segment; The semantic boundary information includes at least: unique identifier information representing the current semantic segment, first order information of the current semantic segment in the corresponding whole broadcast content, boundary type information representing the start or end of the current semantic segment, audio format information and second order information of each audio block of the current semantic segment, and text segment information and / or text position information corresponding to the current semantic segment. The semantic segment start message, the audio arrival message, the semantic segment end message, and each audio block of the current semantic segment all carry identification information corresponding to the current semantic segment.

5. The method according to claim 4, characterized in that, Also includes: When the preset buffer is marked as a playback state, the time corresponding to when the preset buffer is marked as a playback state is recorded as the playback start time; The associated information includes the identification information, the first order information, the audio format information, and the second order information; Based on the association information of the current semantic segment, a portion of the audio in the preset buffer is output to the user terminal according to a preset rhythm, including: The preset rhythm is determined based on the playback start time and the preset output period; Based on the identification information, the first sequence information, the audio format information, and the second sequence information, starting from the playback start time, a portion of the audio blocks in the preset buffer are framed according to the preset rhythm and output to the user terminal frame by frame; wherein, the time of each frame output to the user terminal is determined based on the playback start time, the preset output period, and the number of frames currently output to the user terminal.

6. The method according to claim 5, characterized in that, Based on the associated information, the remaining audio in the preset buffer is output to the user terminal according to the preset rhythm, including: Based on the identification information, the first order information, the audio format information, and the second order information, the remaining audio blocks in the preset buffer are framed and output to the user terminal frame by frame according to the preset rhythm; wherein, when outputting the last frame to the user terminal, if the current remaining audio block in the preset buffer is less than one frame, the current remaining audio block is padded with zeros to make one frame and output to the user terminal.

7. The method according to claim 6, characterized in that, Also includes: In the process of outputting part of the audio in the preset buffer to the user terminal according to the preset rhythm based on the association information of the current semantic segment, the framing state for framing the audio blocks of the current semantic segment is recorded, and the audio block processing progress, which represents the progress of the audio blocks of the current semantic segment being framed and output to the user terminal, is recorded. After responding to the semantic segment end message from the TTS service for the current semantic segment, an end flag indicating the end of the current semantic segment is recorded; After outputting all the remaining audio in the preset buffer to the user terminal according to the preset rhythm based on the association information, a flag indicating that the audio blocks representing the current semantic segment have all been framed and output to the user terminal is recorded. Before marking the preset buffer as idle, the method further includes: Record the identification information corresponding to the current semantic segment, and delete the frame state, audio block processing progress, end flag, and empty flag corresponding to the previous semantic segment.

8. A TTS audio processing device, characterized in that, include: The start module is used to respond to the semantic segment start message provided by the TTS service in the cloud for the current semantic segment and delete the data in the preset buffer; The writing module is used to write the audio of the current semantic segment into the preset buffer in response to the audio arrival message of the TTS service for the current semantic segment; wherein the audio arrival message represents the output order and length of the audio of the current semantic segment; The output module is used to output a portion of the audio in the preset buffer to the user terminal according to a preset rhythm, based on the association information of the current semantic segment, when the preset buffer meets the preset conditions. The termination module is used to respond to the semantic segment termination message of the TTS service for the current semantic segment, and based on the association information, output all the remaining audio in the preset buffer to the user terminal according to the preset rhythm, so as to delete the data in the preset buffer after responding to the semantic segment start message of the TTS service for the next semantic segment.

9. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

Images