Streaming audio synthesis method and device, storage medium and electronic device

Abstract

The invention discloses a streaming audio synthesis method and device, a storage medium and an electronic device, and relates to the technical field of smart home, and the streaming audio synthesis method comprises the steps: obtaining a first text fragment which is accumulatively outputted by a large language model and meets a first set condition through a preset character and / or a preset analysis model; sending the first text fragment to a speech synthesis model within a first time interval to obtain a target audio packet synthesized by the speech synthesis model according to the first text fragment; cutting the target audio packet according to a second set condition; and pushing each audio frame obtained by cutting to a broadcasting terminal within a second time interval so as to broadcast the target audio packet through the broadcasting terminal, the second set condition comprises a sampling rate and a frame length for cutting the target audio packet. Through the method, the problem that the user voice interaction experience is poor due to the fact that the time consumed for feeding back the first voice packet to the user is too long in a large model streaming text returning scene is solved.

Description

Technical Field

[0001] This application relates to the field of smart home technology, and more specifically, to a streaming audio synthesis method and apparatus, storage medium and electronic device. Background Technology

[0002] With the rapid development of large-scale model technology, streaming text generation has become the mainstream interaction mode. In scenarios such as intelligent dialogue, real-time news broadcasting, and voice assistants, large models (such as large language models) need to return the generated text segments to the terminal one by one, and then convert them into real-time speech broadcasts through text-to-speech (TTS) technology. This real-time interaction scenario has extremely high requirements for voice response speed, and users have a very low tolerance for waiting for the "first packet of speech". For example, a delay of more than 200ms in the first packet of speech will significantly affect the smoothness of the interaction, and a delay of more than 500ms may cause users to perceive lag or interruption of operation.

[0003] However, TTS technology in related technologies is based on one-time synthesis of complete text input, which cannot adapt to the fragmented generation characteristics of large-scale streaming text. Although streaming TTS can support segment-by-segment synthesis, it has obvious bottlenecks in the processing of the first packet: on the one hand, there is a "window period" between the generation of large-scale text and TTS synthesis. After the generation of paragraph-level text fragments, it is necessary to wait for processes such as synthesis queue scheduling and resource allocation. Moreover, the synthesis of audio from paragraph-level text fragments consumes a lot of resources and time. On the other hand, the encoding and push of synthesized audio lack fine-grained control, making it difficult to match the rhythm of real-time playback on the terminal. This results in an excessively long end-to-end time from text generation to speech output for the first packet, becoming a core pain point restricting the real-time voice interaction experience.

[0004] In the context of large-scale model-based text streaming, the excessively long processing time of the first voice packet sent to the user in related technologies leads to a poor user voice interaction experience. No effective solution has yet been proposed.

[0005] Therefore, it is necessary to improve the relevant technology to overcome the aforementioned defects. Summary of the Invention

[0006] This application provides a streaming audio synthesis method, apparatus, storage medium, and electronic device to at least solve the problem in related technologies where the first voice packet sent to the user takes too long in large-model streaming text return scenarios, resulting in a poor user voice interaction experience.

[0007] According to one aspect of the embodiments of this application, a streaming audio synthesis method is provided, comprising: obtaining a first text segment that meets a first set condition by means of preset characters and / or preset analysis models, which is the cumulative output of a large language model; wherein the first set condition is that the first text segment has semantics; sending the first text segment to a speech synthesis model within a first time interval to obtain a target audio package synthesized by the speech synthesis model based on the first text segment; segmenting the target audio package according to a second set condition; and pushing each segmented audio frame to a broadcasting terminal within a second time interval to broadcast the target audio package through the broadcasting terminal; wherein both the first time interval and the second time interval are less than a time threshold; the second set condition includes: the sampling rate and frame length of the segmented target audio package.

[0008] In an exemplary embodiment, obtaining a first text segment satisfying a first set condition from the cumulative output of a large language model using a preset character and / or a preset analysis model includes: a determination step: determining whether the preset character exists in the latest second text segment output by the large language model; an update step: if the preset character does not exist in the second text segment, updating the second text segment to a preset buffer, and analyzing whether a third text segment has semantics using the preset analysis model, wherein the third text segment is the text segment in the preset buffer after updating the second text segment that does not contain the preset character; repeatedly executing the determination step and the update step until the determination step determines that the preset character exists in the second text segment, or the update step determines that the third text segment has semantics; and obtaining the first text segment using the preset character or the semantically meaningful third text segment.

[0009] In an exemplary embodiment, obtaining the first text segment through the preset character or a semantically meaningful third text segment includes one of the following: extracting a semantically meaningful third text segment from the preset buffer and using the semantically meaningful third text segment as the first text segment; updating the preset buffer with a second text segment containing the preset character; and extracting the first text segment from a fourth text segment according to the preset character, wherein the fourth text segment is a text segment in the preset buffer after updating the second text segment containing the preset character.

[0010] In an exemplary embodiment, sending the first text segment to a speech synthesis model within a first time interval to obtain a target audio package synthesized by the speech synthesis model based on the first text segment includes: sending a speech synthesis request to the speech synthesis model within the first time interval to request the speech synthesis model to synthesize the audio of the first text segment, wherein the speech synthesis request includes the first text segment; obtaining the target audio package based on header information of a preset length output by the speech synthesis model, wherein the header information is used to indicate the length of the audio content of the target audio package, and the target audio package includes: the header information and the audio content, wherein the audio of the first text segment includes the target audio package.

[0011] In an exemplary embodiment, pushing each segmented audio frame to a broadcasting terminal within a second time interval includes: encoding each segmented audio frame and determining the segmentation order of each audio frame; if the segmentation order is less than or equal to an order threshold, pushing each encoded audio frame to the broadcasting terminal within the second time interval; if the segmentation order is greater than the order threshold, pushing each encoded audio frame to the broadcasting terminal according to the push delay of each encoded audio frame.

[0012] In an exemplary embodiment, pushing each encoded audio frame to the broadcast terminal according to the push delay of each encoded audio frame includes: determining the push time of the first audio frame obtained by cutting the target audio packet; determining the push delay of each encoded audio frame by the push time, the cutting order and the current time; and pushing each encoded audio frame to the broadcast terminal within the second time interval if the push delay is delayed based on the current time.

[0013] In an exemplary embodiment, before pushing each encoded audio frame to the broadcasting terminal according to the push delay of each encoded audio frame, the method further includes: obtaining the broadcasting speed fed back by the broadcasting terminal; if the broadcasting speed is higher than the speed threshold, reducing the push delay by a first preset ratio; if the broadcasting speed is lower than the speed threshold, increasing the push delay by a second preset ratio.

[0014] According to another aspect of the embodiments of this application, a streaming audio synthesis apparatus is also provided, comprising: a first acquisition module, configured to acquire a first text segment that satisfies a first set condition, which is the cumulative output of a large language model, through a preset character and / or a preset analysis model; wherein the first set condition is that the first text segment has semantics; the first acquisition module, configured to send the first text segment to a speech synthesis model within a first time interval to acquire a target audio package synthesized by the speech synthesis model based on the first text segment; and a push module, configured to cut the target audio package according to a second set condition; and push each audio frame obtained by cutting to a broadcast terminal within a second time interval to broadcast the target audio package through the broadcast terminal; wherein both the first time interval and the second time interval are less than a time threshold; the second set condition includes: the sampling rate and frame length of cutting the target audio package.

[0015] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, which is configured to execute the above-described streaming audio synthesis method at runtime.

[0016] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above-described streaming audio synthesis method through the computer program.

[0017] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the above-described streaming audio synthesis method.

[0018] This application obtains a first text segment that meets a first set condition from the cumulative output of a large language model by using preset characters and / or preset analysis models; wherein the first set condition is that the first text segment has semantics; the first text segment is sent to a speech synthesis model within a first time interval to obtain a target audio package synthesized by the speech synthesis model based on the first text segment; the target audio package is segmented according to a second set condition; and each segmented audio frame is pushed to a broadcasting terminal within a second time interval to broadcast the target audio package through the broadcasting terminal; wherein both the first time interval and the second time interval are less than a time threshold; the second set condition includes: the sampling rate and frame length of the segmented target audio package. In other words, by immediately sending the first text segment that meets the first set condition to the speech synthesis model, the waiting time for the first text segment is reduced; and by performing audio frame-level segmentation on the target audio package generated by the speech synthesis model, and immediately pushing each segmented audio frame to the broadcasting terminal for broadcasting, frame-level segmentation and pushing of the audio package are achieved, further reducing the waiting time for broadcasting the target audio package. Therefore, by adopting the above technical solution, the problem of excessively long processing time for the first voice packet returned to the user in the scenario of large-model streaming text return in related technologies is solved, resulting in a poor user voice interaction experience; thus improving the user's interaction experience. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the hardware environment for an optional streaming audio synthesis method according to an embodiment of this application;

[0022] Figure 2 This is a flowchart of an optional streaming audio synthesis method according to an embodiment of this application;

[0023] Figure 3 This is a timing diagram of an optional streaming audio synthesis method according to an embodiment of this application;

[0024] Figure 4 This is a structural block diagram of an optional streaming audio synthesis apparatus according to an embodiment of this application;

[0025] Figure 5 This is another structural block diagram of an optional streaming audio synthesis apparatus according to an embodiment of this application. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] According to one aspect of the embodiments of this application, a streaming audio synthesis method is provided. This streaming audio synthesis method is widely used in whole-house intelligent digital control application scenarios such as smart homes, smart home ecosystems, and intelligencehouse ecosystems. Optionally, in this embodiment, the above-mentioned streaming audio synthesis method can be applied to, for example... Figure 1 The hardware environment shown consists of multiple terminal devices 102 and a server 104. For example... Figure 1 As shown, server 104 is connected to multiple terminal devices 102 via a network and can be used to provide services (such as application services) to terminals or clients installed on terminals. A database can be set up on the server or independently of the server to provide data storage services for server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data processing services for server 104.

[0029] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. The terminal device 102 may not be limited to PC, mobile phone, tablet computer, smart air conditioner, smart range hood, smart refrigerator, smart oven, smart stove, smart washing machine, smart water heater, smart washing equipment, smart dishwasher, smart projector, smart TV, smart clothes rack, smart curtains, smart audio-visual equipment, smart socket, smart speaker, smart speaker box, smart fresh air equipment, smart kitchen and bathroom equipment, smart bathroom equipment, smart robot vacuum cleaner, smart window cleaning robot, smart mopping robot, smart air purifier, smart steam oven, smart microwave oven, smart water heater, smart air purifier, smart water dispenser, smart door lock, etc.

[0030] This embodiment provides a streaming audio synthesis method, including but not limited to applications on cloud servers. Figure 2 This is a flowchart of an optional streaming audio synthesis method according to an embodiment of this application, the process including the following steps:

[0031] Step S202: Obtain the first text segment that meets the first set condition from the cumulative output of the large language model through preset characters and / or preset analysis model; wherein, the first set condition is that the first text segment has semantics;

[0032] Optionally, before performing step S202, the cloud server receives audio packets sent by the terminal application layer (such as a smart speaker), performs speech recognition on the audio packets sent by the terminal application layer using ASR, and converts the audio packets sent by the terminal application layer into computer-readable text (e.g., Figure 3 The "Beijing Weather" data is input into the large model for processing. The large model responds to the text format corresponding to the audio packet sent by the terminal application layer, and cumulatively outputs the first text segment that meets the first set condition.

[0033] Step S204: Send the first text segment to the speech synthesis model within a first time interval to obtain the target audio package synthesized by the speech synthesis model based on the first text segment;

[0034] Step S206: Cut the target audio packet according to the second set conditions; and push each audio frame obtained by the cutting to the broadcasting terminal within the second time interval, so as to broadcast the target audio packet through the broadcasting terminal; wherein, the first time interval and the second time interval are both less than the time threshold; the second set conditions include: the sampling rate and frame length of the target audio packet.

[0035] Optionally, both the first and second time intervals are within 1-2 milliseconds, i.e., the time threshold is 2 milliseconds.

[0036] Through the above steps, a first text segment satisfying a first set condition is obtained by using preset characters and / or a preset analysis model, which is the cumulative output of a large language model. The first set condition is that the first text segment has semantic meaning. The first text segment is sent to a speech synthesis model within a first time interval to obtain a target audio package synthesized by the speech synthesis model based on the first text segment. The target audio package is segmented according to a second set condition. Each segmented audio frame is pushed to a broadcasting terminal within a second time interval to broadcast the target audio package. Both the first and second time intervals are less than a time threshold. The second set condition includes the sampling rate and frame length for segmenting the target audio package. In other words, by immediately sending the first text segment satisfying the first set condition to the speech synthesis model, the waiting time for the first text segment is reduced. Furthermore, by performing audio frame-level segmentation on the target audio package generated by the speech synthesis model and immediately pushing each segmented audio frame to the broadcasting terminal for broadcasting, frame-level segmentation and pushing of the audio package are achieved, further reducing the waiting time for broadcasting the target audio package. Therefore, by adopting the above technical solution, the problem of excessively long processing time for the first voice packet returned to the user in the scenario of large-model streaming text return in related technologies is solved, resulting in a poor user voice interaction experience; thus improving the user's interaction experience.

[0037] Regarding step S202 above, in an optional embodiment, obtaining the first text segment that meets the first set condition from the cumulative output of the large language model through a preset character and / or a preset analysis model includes: a determination step: determining whether the preset character exists in the latest second text segment output by the large language model; an update step: if the preset character does not exist in the second text segment, updating the second text segment to a preset buffer, and analyzing whether the third text segment has semantics through the preset analysis model, wherein the third text segment is the text segment in the preset buffer after updating the second text segment that does not contain the preset character; repeatedly executing the determination step and the update step until the determination step determines that the preset character exists in the second text segment, or the update step determines that the third text segment has semantics; obtaining the first text segment through the preset character or the third text segment with semantics.

[0038] Furthermore, obtaining the first text segment through the preset character or a semantically meaningful third text segment includes one of the following: extracting a semantically meaningful third text segment from the preset buffer and using the semantically meaningful third text segment as the first text segment; updating the preset buffer with a second text segment containing the preset character; and extracting the first text segment from a fourth text segment according to the preset character, wherein the fourth text segment is the text segment in the preset buffer after updating the second text segment containing the preset character.

[0039] In other words, upon receiving an audio packet from the terminal application layer, the large language model responds by streaming its feedback text to the audio packet, for example, such as... Figure 3 As shown, the large model responds to the computer-readable text form "Beijing Weather" corresponding to the audio packet sent by the terminal application layer, and streams the feedback text "Today's weather is sunny, temperature 23 degrees Celsius, sunny". The cloud server continuously receives the latest second text segment output by the large language model. Each time a second text segment is received, it is determined whether a preset character exists in the second text segment. The preset character can be a punctuation mark (period, question mark, exclamation mark, etc.). The presence of the preset character indicates that the text segment before the preset character should have a complete semantic structure and be semantically meaningful. If the preset character exists in the second text segment, the second text segment is updated to a preset buffer, and all text segments in the updated fourth text segment in the preset buffer that are before the preset character are extracted as the first text segment and immediately sent to the speech synthesis model to synthesize the target audio packet.

[0040] If the preset character is not present in the second text segment, the second text segment is updated in the preset buffer, and semantic analysis is performed on the updated third text segment in the preset buffer using a preset analysis model (which can be a semantic analysis model). If the semantic analysis result indicates that the third text segment has semantic meaning, it is extracted from the preset buffer and immediately sent to the speech synthesis model to synthesize the target audio packet. If the semantic analysis result indicates that the third text segment does not have semantic meaning, it waits to receive the next second text segment output by the large model and continues to execute the determination and update steps.

[0041] For example, if the complete feedback text that the large model needs to output is: It is sunny today, the temperature is 23 degrees Celsius, and it is sunny. If the second text segment received is "今", then it belongs to a text segment without both preset characters and specific semantics. If the large model continues to output "天", then it belongs to a segment without preset characters. However, after updating to the preset buffer, "今天" can be analyzed by the semantic analysis model as a text segment with semantics. Then, "今天" is used as the first text segment, and the speech synthesis model is immediately requested to perform speech synthesis on the first text "今天".

[0042] It should be noted that the semantic analysis model in the embodiments of the present application is a small model based on Qwen7B, which is trained to quickly perform semantic analysis, identify, and understand the semantic boundaries of text, and then detect whether the input text segment has semantics.

[0043] Compared with paragraph-level or sentence-level text synthesis in the related art, the embodiments of the present application can quickly identify text segments with semantics from the output of the large model in real time to initiate speech synthesis. The text segments only required to have semantics often have small units, which can reduce the latency of the first-packet speech generation and meet the requirement of "immediate start" of the first packet. Among them, by quickly determining the synthesis starting point through preset characters or semantic analysis, the time of waiting for a complete sentence or paragraph is reduced, and the overall immediate response speed is improved. The introduction of the semantic analysis model also enables effective identification of semantic boundaries even in complex or non-standard text structures, avoiding unnecessary resource waste.

[0044] In an exemplary embodiment, cutting the target audio packet according to the second set condition includes: determining the number of bytes included in each audio frame obtained by cutting the target audio packet according to the second set condition; wherein, the second set condition includes: the sampling rate and frame length for cutting the target audio packet; and cutting the target audio packet based on the first byte of the target audio packet according to the number of bytes.

[0045] Optionally, the sampling rate in the embodiments of the present application can be selected as 16000 Hz, and the frame length can be selected as 60 ms. Determining the number of bytes included in each audio frame obtained by cutting the target audio packet according to the second set condition includes: the number of bytes (i.e., the frame size) = sampling rate × frame length / 1000. Then, cut the target audio packet from the starting position of the target audio packet, such as the first byte, according to the calculated number of bytes.

[0046] In an exemplary embodiment, sending the first text segment to a speech synthesis model within a first time interval to obtain a target audio package synthesized by the speech synthesis model based on the first text segment includes: sending a speech synthesis request to the speech synthesis model within the first time interval to request the speech synthesis model to synthesize the audio of the first text segment, wherein the speech synthesis request includes the first text segment; obtaining the target audio package based on header information of a preset length output by the speech synthesis model, wherein the header information is used to indicate the length of the audio content of the target audio package, and the target audio package includes: the header information and the audio content, wherein the audio of the first text segment includes the target audio package.

[0047] Understandably, once the first text segment is determined, the cloud server sends a speech synthesis request to the speech synthesis model within a first time interval, instructing the model to synthesize the audio of the first text segment. The audio of the first text segment is output as streaming audio packets, each packet beginning with a pre-defined header. This header indicates the length of the subsequent audio content within each streaming audio packet, allowing for accurate reading and processing of the audio data. For example, it could be a 4-byte header, such as "000003E8," where the hexadecimal number (decimal 1000) represents the length of the subsequent audio data, letting the cloud server know how much data to read. Therefore, after reading the first header, the cloud server reads the corresponding length of audio content, as indicated by the header, to obtain the first streaming audio packet (equivalent to the target audio packet). It should be noted that, in this embodiment of the application, after obtaining the first streaming audio packet, the cloud server will continue to receive other streaming audio packets after the first streaming audio packet in the audio of the first text segment. The processing method for other streaming audio packets is the same as that for the first streaming audio packet, which is to cut them into audio frames and push them to the broadcasting terminal for broadcasting. This will not be described in detail here.

[0048] In an exemplary embodiment, pushing each segmented audio frame to a broadcasting terminal within a second time interval includes: encoding each segmented audio frame and determining the segmentation order of each audio frame; if the segmentation order is less than or equal to an order threshold, pushing each encoded audio frame to the broadcasting terminal within the second time interval; if the segmentation order is greater than the order threshold, pushing each encoded audio frame to the broadcasting terminal according to the push delay of each encoded audio frame.

[0049] Each audio frame obtained from the segmentation is encoded, that is, the segmented audio frames are converted from PCM (Pulse Code Modulation) format to Opus encoding format. Encoding methods include, for example, encoding using short-term prediction technology with an Opus encoder; encoding using fast Fourier transform, etc.

[0050] Optionally, pushing each encoded audio frame to the broadcasting terminal according to the push delay of each encoded audio frame includes: determining the push time of the first audio frame obtained by cutting the target audio packet; determining the push delay of each encoded audio frame by the push time, the cutting order and the current time; and pushing each encoded audio frame to the broadcasting terminal within the second time interval if the push delay is delayed based on the current time.

[0051] Specifically, pushing each encoded audio frame to the broadcast terminal according to the push delay of each encoded audio frame can be achieved using the following formula:

[0052] Delay time = Math.abs(start timestamp + 60) (index - order threshold) - current timestamp).

[0053] Here, the delay time is equivalent to the push latency in the above embodiment, the start timestamp is equivalent to the push time in the above embodiment, and the index is used to represent the cutting order. The order threshold can be set to 50.

[0054] In streaming trunk audio, frame-level encoding, and push coordination, related technologies often encode in batches or segments before pushing the entire package, ignoring frame-level processing, resulting in time-consuming and delayed first-packet playback. However, this application's embodiment, by segmenting audio packets into audio frames and immediately pushing each segmented audio frame to the playback terminal after encoding, focuses more on frame-level processing, reducing the time and delay of first-packet playback. Furthermore, compared to related technologies where the end-to-end process optimization lacks coordinated scheduling in text reception and synthesis initiation, leading to mismatched processing rhythms and "waiting backlog" of first-packet data, causing terminal playback delays and affecting the real-time interactive experience, this application's embodiment immediately synthesizes audio packets through a speech synthesis model after obtaining the first text segment. After obtaining each audio frame from the segmented audio packet, each audio frame is immediately encoded and pushed for playback. This ensures a smooth process from text reception and synthesis initiation to audio frame push playback, with a matched rhythm and no waiting time, avoiding "waiting backlog" of first-packet data and improving the real-time interactive experience.

[0055] Optionally, before pushing each encoded audio frame to the broadcasting terminal according to the push delay of each encoded audio frame, the following steps are included: obtaining the broadcasting speed fed back by the broadcasting terminal; if the broadcasting speed is higher than the speed threshold, reducing the push delay by a first preset ratio; if the broadcasting speed is lower than the speed threshold, increasing the push delay by a second preset ratio.

[0056] The first and second preset ratios are selected within a certain range (e.g., between 5% and 15%). These ratios can be fixed or vary within this range depending on the speaking speed. For example, they can change proportionally to the speaking speed; if the speaking speed increases by 6% compared to the previous feedback, the push delay can be reduced by 6%. If the change exceeds the boundary of this range, the boundary value is used as the first or second preset ratio.

[0057] Obviously, the embodiments described above are only some embodiments of this application, and not all embodiments. To better understand the above streaming audio synthesis method, the process is described below with reference to embodiments, but this is not intended to limit the technical solutions of the embodiments of this application. Specifically:

[0058] The first packet voice broadcasting scheme in related technologies requires waiting for the generation of a complete text segment (such as a whole sentence) before starting the synthesis, which directly leads to waiting time for the first text segment; and because the synthesized text is pushed out as a whole in units of segments, it is impossible to achieve frame-level real-time output, so the overall time consumption of the first packet is inevitably high, making it difficult to meet the low latency requirements of real-time interaction.

[0059] To address the shortcomings of related technologies, such as high initial packet latency, inability to meet low latency requirements for real-time interaction, difficulty in adapting to ultra-low latency scenarios, and poor voice continuity, the embodiments of this application aim to achieve the following objectives: In the scenario of large-model streaming text return, by optimizing the synthesis startup mechanism, the synthesis can be started immediately upon the generation of the first text segment; frame-level independent encoding is adopted to reduce the encoding time of the first frame; and precise sleep push is implemented in combination with frame broadcasting time to match the real-time broadcasting rhythm of the terminal, thereby shortening the overall time from text reception to terminal broadcasting of the first TTS packet, and improving the fluency and user experience of real-time voice interaction.

[0060] The timing diagram of the strategy and method for optimizing the first packet latency of streaming TTS provided in the embodiments of this application is as follows: Figure 3 As shown, this can be achieved through the following steps 1 to 3:

[0061] Step 1: Text Aggregation Processing. Upon receiving an audio packet transmitted from the terminal application layer, the text identified by ASR speech recognition from the audio packet (e.g., Beijing weather) is input into the large model. The large model then streams its response text to the audio packet, such as "Today's weather is sunny, temperature 23 degrees Celsius." It should be noted that the large model's response to the Beijing weather message can be as follows: the large model recognizes the intent of the input text as a weather query; it can then call a weather query tool based on this intent to obtain information such as "sunny weather" and "temperature 23 degrees Celsius," and finally output the response text. In this embodiment, the aggregation strategy for the streaming content returned by the large model is optimized. The aggregation strategy for the first text returned by the large model (equivalent to the first text segment in the above embodiment) is: matching the first punctuation mark or the first semantically meaningful phrase is sufficient, thus optimizing the time taken for the first text from 300ms to 200ms.

[0062] The key to Step 1 lies in building a novel synthesis initiation mechanism that can quickly bypass the traditional process of waiting for the complete text and immediately trigger TTS synthesis as soon as the large model generates the first text fragment. This involves deep collaborative optimization between the text receiving module and the synthesis engine. Optionally, a text fragment buffer can be set up. Once the buffer captures the first usable text fragment (which may be a few words or a short sentence), it is immediately transmitted to the synthesis engine. At the same time, resource pre-allocation and synthesis task scheduling are completed, which can greatly shorten the window period from text generation to synthesis initiation.

[0063] Step 2: First Packet Streaming Processing. After acquiring the first text, a TTS request is made to synthesize speech on that text. The output uses a streaming audio packet mode, with each trunk packet consisting of 4 bytes in length plus audio content. First, the 4 bytes are read, and then the first audio packet is obtained based on the length. The time taken to acquire the first packet can be optimized from 300ms to 150ms.

[0064] Step 3: First Frame Encoding Processing. After the first audio packet is acquired, it is segmented at the frame level (e.g., sampling rate 16000, mono, frame length 60ms, frame size = 16000). (60 / 10=960 bytes); then, encoding is performed frame by frame, discarding the entire audio packet encoding mode. Once the first frame is encoded, it can be directly pushed to the terminal for playback. To achieve smooth terminal playback, the first 50 frames can be pushed without delay, and subsequent frames are pushed with a delay time = Math.abs(start timestamp + 60). (index-50) - current timestamp); the first frame push time can be optimized from 100ms to 10ms.

[0065] Step 3 abandons the traditional inter-frame dependency encoding method and adopts a frame-level independent encoding strategy. Each audio frame does not need to wait for the complex calculation results of the preceding frame during encoding; instead, it independently completes encoding based on its own audio characteristics. Lightweight encoding algorithms can be used; for example, for the short-term stationary characteristics of speech, a fast encoding method based on local audio spectrum analysis can be used to reduce the computational load and time overhead required for encoding the first frame, improving overall encoding efficiency and ensuring that encoded audio frames can be output quickly after synthesis. Furthermore, Step 3 overcomes the limitations of fixed-delay sleep, dynamically adjusting the push interval based on the actual playback time of each frame. Further, the actual playback time of each frame can be accurately calculated through comprehensive analysis of speech rate, intonation, and audio frame duration. For example, frames with fast speech rates have shorter playback times, so the sleep time is shortened accordingly; frames with slow speech rates have longer sleep times. In the push phase, the timing of each frame's push is precisely controlled based on the calculation results, ensuring a natural transition between audio frames during terminal playback, avoiding stuttering or overlap, and thus achieving a perfect match with the real-time playback rhythm of the terminal.

[0066] Through steps 1 to 3 of the embodiments described in this application, the response speed of the first packet can be significantly improved. Because this embodiment employs an "instantaneous synthesis mechanism," TTS synthesis can be triggered as soon as the first text fragment is generated in the large model, without waiting for the complete text or a fixed fragment. This directly shortens the interval from text generation to synthesis initiation. Compared to the traditional "waiting for complete fragments + batch synthesis" mode, the first packet synthesis initiation time can be reduced by more than 50%. Simultaneously, frame-level independent encoding avoids the first frame encoding delay caused by inter-frame dependencies, further compressing the first packet processing cycle. Ultimately, this significantly reduces the time from text reception to terminal playback of the first packet, meeting the core requirement of low latency in real-time interactive scenarios.

[0067] Furthermore, this application embodiment also achieves optimized resource utilization efficiency. Frame-level independent encoding adopts a lightweight algorithm, reducing the computational resource consumption of single-frame encoding. At the same time, the "instant startup + frame-by-frame processing" mode avoids the idle waste of a large number of pre-allocated resources in traditional batch synthesis. In addition, the dynamic sleep push strategy reduces the frequency of invalid network requests and data backlog (compared to batch push), reducing network bandwidth occupation and terminal cache pressure. From the system level, the overall resource utilization efficiency is better, and it can support higher concurrency real-time voice interaction scenarios. In addition, the solution of this application embodiment has stronger adaptability and scalability. Since the synthesis startup mechanism does not depend on fixed text fragment formats (such as complete sentences), and the frame-level encoding and push strategy can be dynamically adjusted through algorithm parameters (such as adapting to different languages, speech rates, and timbre voice features), this application embodiment can flexibly adapt to diverse large-scale streaming text generation scenarios, including short sentence interaction and long text real-time broadcasting. At the same time, the frame-level processing architecture facilitates the subsequent integration of more refined voice optimization functions (such as real-time emotion adjustment and dialect adaptation), and its scalability is significantly better than traditional fragment-based processing solutions.

[0068] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0069] This embodiment also provides a streaming audio synthesis apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware implementations, or a combination of software and hardware, are also possible and contemplated.

[0070] Figure 4 This is a structural block diagram of an optional streaming audio synthesis apparatus according to an embodiment of this application, the apparatus comprising:

[0071] The first acquisition module 42 is used to acquire a first text fragment that meets a first set condition from the cumulative output of the large language model through preset characters and / or preset analysis models; wherein, the first set condition is that the first text fragment has semantics;

[0072] The second acquisition module 44 is used to send the first text segment to the speech synthesis model within a first time interval to obtain the target audio package synthesized by the speech synthesis model based on the first text segment.

[0073] The push module 46 is used to cut the target audio packet according to a second set condition; and push each audio frame obtained by the cutting to the broadcast terminal within a second time interval, so as to broadcast the target audio packet through the broadcast terminal; wherein, both the first time interval and the second time interval are less than a time threshold; the second set condition includes: the sampling rate and frame length of the target audio packet.

[0074] The above-described apparatus obtains a first text segment that meets a first set condition from the cumulative output of a large language model by using preset characters and / or a preset analysis model; wherein the first set condition is that the first text segment has semantic meaning; the first text segment is sent to a speech synthesis model within a first time interval to obtain a target audio package synthesized by the speech synthesis model based on the first text segment; the target audio package is segmented according to a second set condition; and each segmented audio frame is pushed to a broadcasting terminal within a second time interval to broadcast the target audio package through the broadcasting terminal; wherein both the first time interval and the second time interval are less than a time threshold; the second set condition includes: the sampling rate and frame length of the segmented target audio package. In other words, by immediately sending the first text segment that meets the first set condition to the speech synthesis model, the waiting time for the first text segment is reduced; and by performing audio frame-level segmentation on the target audio package generated by the speech synthesis model, and immediately pushing each segmented audio frame to the broadcasting terminal for broadcasting, frame-level segmentation and pushing of the audio package are achieved, further reducing the waiting time for broadcasting the target audio package. Therefore, by adopting the above technical solution, the problem of excessively long processing time for the first voice packet returned to the user in the scenario of large-model streaming text return in related technologies is solved, resulting in a poor user voice interaction experience; thus improving the user's interaction experience.

[0075] In an exemplary embodiment, the first acquisition module 42 is further configured to perform: a determination step: determining whether the preset character exists in the latest output second text segment of the large language model; an update step: if the preset character does not exist in the second text segment, updating the second text segment to a preset buffer, and analyzing whether the third text segment has semantics through the preset analysis model, wherein the third text segment is the text segment in the preset buffer after updating the second text segment that does not contain the preset character; repeatedly executing the determination step and the update step until the determination step determines that the preset character exists in the second text segment, or the update step determines that the third text segment has semantics; and acquiring the first text segment through the preset character or the semantically meaningful third text segment.

[0076] In an exemplary embodiment, the first acquisition module 42 is further configured to perform one of the following: extract a semantically meaningful third text fragment from the preset buffer and use the semantically meaningful third text fragment as the first text fragment; update the preset buffer with a second text fragment containing the preset character; extract the first text fragment from a fourth text fragment according to the preset character, wherein the fourth text fragment is a text fragment in the preset buffer after the second text fragment containing the preset character has been updated.

[0077] In an exemplary embodiment, the second acquisition module 44 is further configured to send a speech synthesis request to the speech synthesis model within the first time interval to request the speech synthesis model to synthesize the audio of the first text segment, wherein the speech synthesis request includes the first text segment; and to acquire the target audio package based on header information of a preset length output by the speech synthesis model, wherein the header information is used to indicate the length of the audio content of the target audio package, and the target audio package includes: the header information and the audio content, wherein the audio of the first text segment includes the target audio package.

[0078] In an exemplary embodiment, the push module 46 is further configured to encode each of the segmented audio frames and determine the segmentation order of each audio frame; if the segmentation order is less than or equal to an order threshold, push each encoded audio frame to the broadcast terminal within the second time interval; if the segmentation order is greater than the order threshold, push each encoded audio frame to the broadcast terminal according to the push delay of each encoded audio frame.

[0079] In an exemplary embodiment, the push module 46 is further configured to determine the push time of the first audio frame obtained by cutting the target audio packet; determine the push delay of each encoded audio frame by means of the push time, the cutting order and the current time; and push each encoded audio frame to the broadcast terminal within the second time interval if the push delay is delayed based on the current time.

[0080] In one exemplary embodiment, such as Figure 5 As shown, the device further includes a delay adjustment module 48, which is used to obtain the broadcasting speed fed back by the broadcasting terminal before pushing each encoded audio frame to the broadcasting terminal according to the push delay of each encoded audio frame; when the broadcasting speed is higher than the speed threshold, the push delay is reduced according to a first preset ratio; when the broadcasting speed is lower than the speed threshold, the push delay is increased according to a second preset ratio.

[0081] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.

[0082] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0083] S1, obtain the first text segment that meets the first set condition from the cumulative output of the large language model through preset characters and / or preset analysis model; wherein, the first set condition is that the first text segment has semantics;

[0084] S2, within a first time interval, the first text segment is sent to the speech synthesis model to obtain the target audio package synthesized by the speech synthesis model based on the first text segment;

[0085] S3, the target audio packet is segmented according to the second set conditions; and each segmented audio frame is pushed to the broadcasting terminal within the second time interval to broadcast the target audio packet through the broadcasting terminal; wherein, both the first time interval and the second time interval are less than the time threshold; the second set conditions include: the sampling rate and frame length of the segmented target audio packet.

[0086] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0087] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0088] Embodiments of this application also provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0089] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0090] S1, obtain the first text segment that meets the first set condition from the cumulative output of the large language model through preset characters and / or preset analysis model; wherein, the first set condition is that the first text segment has semantics;

[0091] S2, within a first time interval, the first text segment is sent to the speech synthesis model to obtain the target audio package synthesized by the speech synthesis model based on the first text segment;

[0092] S3, the target audio packet is segmented according to the second set conditions; and each segmented audio frame is pushed to the broadcasting terminal within the second time interval to broadcast the target audio packet through the broadcasting terminal; wherein, both the first time interval and the second time interval are less than the time threshold; the second set conditions include: the sampling rate and frame length of the segmented target audio packet.

[0093] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0094] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0095] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0096] Embodiments of this application also provide a computer program that includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in any of the above method embodiments.

[0097] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0098] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0099] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A streaming audio synthesis method, characterized in that, include: The first text fragment that meets the first set condition is obtained by accumulating the output of the large language model through preset characters and / or preset analysis models; wherein, the first set condition is that the first text fragment has semantics; Within a first time interval, the first text segment is sent to the speech synthesis model to obtain the target audio package synthesized by the speech synthesis model based on the first text segment; The target audio packet is segmented according to the second set conditions; and each segmented audio frame is pushed to the broadcasting terminal within the second time interval to broadcast the target audio packet through the broadcasting terminal; wherein, both the first time interval and the second time interval are less than the time threshold; the second set conditions include: the sampling rate and frame length of the segmented target audio packet.

2. The streaming audio synthesis method according to claim 1, characterized in that, The first text fragments that meet the first set conditions are obtained by using preset characters and / or preset analysis models and are accumulated output by the large language model. These fragments include: Determination Step: Determine whether the preset character exists in the second text segment output by the latest version of the large language model; Update steps: If the preset character is not present in the second text segment, update the second text segment to the preset buffer, and analyze whether the third text segment has semantics through the preset analysis model, wherein the third text segment is the text segment in the preset buffer after the second text segment without the preset character has been updated; The determination step and the update step are executed repeatedly until the determination step determines that the second text segment contains the preset character, or the update step determines that the third text segment has semantic meaning; the first text segment is obtained through the preset character or the third text segment with semantic meaning.

3. The streaming audio synthesis method according to claim 2, characterized in that, The first text segment is obtained by means of the preset characters or a third text segment with semantic meaning, including one of the following: Extract a semantically meaningful third text segment from the preset buffer and use the semantically meaningful third text segment as the first text segment; The second text segment containing the preset character is updated to the preset buffer; the first text segment is extracted from the fourth text segment according to the preset character, wherein the fourth text segment is the text segment in the preset buffer after the second text segment containing the preset character has been updated.

4. The streaming audio synthesis method according to claim 1, characterized in that, Sending the first text segment to a speech synthesis model within a first time interval to obtain a target audio package synthesized by the speech synthesis model based on the first text segment includes: During the first time interval, a speech synthesis request is sent to the speech synthesis model to request the speech synthesis model to synthesize the audio of the first text segment, wherein the speech synthesis request includes the first text segment; The target audio package is obtained based on the header information of a preset length output by the speech synthesis model, wherein the header information is used to indicate the length of the audio content of the target audio package, and the target audio package includes: the header information and the audio content, and the audio of the first text segment includes the target audio package.

5. The streaming audio synthesis method according to claim 1, characterized in that, Each segmented audio frame is pushed to the broadcast terminal during the second time interval, including: Each audio frame obtained from the segmentation is encoded, and the segmentation order of each audio frame is determined; If the cutting order is less than or equal to the order threshold, each encoded audio frame is pushed to the broadcast terminal within the second time interval. If the order of being cut is greater than the order threshold, each encoded audio frame is pushed to the broadcast terminal according to the push delay of each encoded audio frame.

6. The streaming audio synthesis method according to claim 5, characterized in that, Each encoded audio frame is pushed to the broadcast terminal according to the push delay of each encoded audio frame, including: Determine the push time of the first audio frame obtained by segmenting the target audio packet; The push delay of each encoded audio frame is determined by the push time, the segmentation order, and the current time. In the case of delaying the push delay based on the current time, each encoded audio frame is pushed to the broadcast terminal within the second time interval.

7. The streaming audio synthesis method according to claim 5, characterized in that, Before pushing each encoded audio frame to the broadcast terminal according to the push delay of each encoded audio frame, the method further includes: Obtain the broadcast speed fed back by the broadcast terminal; If the broadcast speed exceeds the speed threshold, the push delay is reduced by a first preset ratio; If the broadcast speed is lower than the speed threshold, the push delay is increased according to a second preset ratio.

8. A streaming audio synthesis device, characterized in that, include: The first acquisition module is used to acquire a first text fragment that meets a first set condition from the cumulative output of a large language model through preset characters and / or preset analysis models; wherein, the first set condition is that the first text fragment has semantics; The second acquisition module is used to send the first text segment to the speech synthesis model within a first time interval to obtain the target audio package synthesized by the speech synthesis model based on the first text segment. The push module is used to cut the target audio packet according to a second set condition; and push each audio frame obtained by the cutting to the broadcasting terminal within a second time interval, so as to broadcast the target audio packet through the broadcasting terminal; wherein, both the first time interval and the second time interval are less than a time threshold; the second set condition includes: the sampling rate and frame length of the target audio packet.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method of any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method of any one of claims 1 to 7 through the computer program.

Images