Streaming speech generation method

Abstract

The application relates to a streaming voice generation method, prefix unit sequences are constructed based on a voice generation request, the prefix unit sequences are subjected to block processing, and a unique rolling block identifier is generated for each prefix block. Whether a matched block key value exists in a target cache table is searched through the rolling block identifier. If the matched block key value exists, the matched block key value is directly reused, so that repeated calculation is avoided. If the matched block key value does not exist, pre-filling calculation is carried out, and the result is stored in the cache table, so that when other requests encounter the same prefix block, the block key value can be directly obtained from the cache table without the need of re-calculation, first packet delay is significantly reduced, cache occupation is reduced, concurrent reasoning efficiency is improved, and the optimization of system performance is realized.

Description

Technical Field

[0001] This application relates to the field of speech generation technology, and in particular to a streaming speech generation method. Background Technology

[0002] In existing streaming speech generation systems, when faced with scenarios involving concurrent inference with multiple requests, continuous batch processing and incremental decoding are typically employed to improve system throughput. Continuous batch processing integrates multiple requests for processing, while incremental decoding helps generate speech content incrementally, reducing unnecessary redundant computations. However, in practical applications, existing systems exhibit numerous problems when multiple requests share the same text prefix, the same prompt speech prefix, or partially the same historical generated prefix. The system repeatedly performs pre-filling calculations for these identical prefix portions. This process not only increases the latency of the first packet, requiring users to wait longer for the initial speech generation results, but also, due to redundant computation, the cache stores a large number of calculation results corresponding to the same prefix, leading to excessive cache usage and wasting valuable storage resources. Therefore, existing streaming speech generation technologies have significant limitations in handling concurrent inference with multiple requests containing the same prefix and urgently need improvement. Summary of the Invention

[0003] Based on this, the purpose of this application is to provide a streaming speech generation method to solve the problems of increased latency of the first packet and excessive cache usage caused by repeated calculations when multiple requests are concurrently inferred and have the same prefix in existing streaming speech generation systems.

[0004] The streaming speech generation method described in this application includes the following steps:

[0005] Receive speech generation requests and construct a prefix unit sequence; The prefix unit sequence is divided into several prefix blocks; the rolling block identifier corresponding to each prefix block is calculated sequentially according to the order of the prefix blocks. Based on the rolling block identifier of each prefix block, a matching block key value is retrieved from the target cache table; if a matching block key value exists, it is reused as the block key value of the corresponding prefix block; if no matching block key value exists, the corresponding prefix block is pre-filled using a speech generation model to obtain the block key value; the calculated block key value and the corresponding rolling block identifier are stored in the target cache table. Based on the block key values ​​of each of the aforementioned prefix blocks, the speech is decoded and generated using a speech generation model to obtain the speech result.

[0006] In this embodiment, after receiving a speech generation request and constructing a prefix unit sequence, the sequence is divided into prefix blocks and a rolling block identifier is calculated. When a matching block key value is retrieved from the target cache table based on the rolling block identifier, if a match exists, the block key value is directly reused for subsequent decoding and generation, avoiding repeated pre-filling calculations for the same prefix blocks. This mechanism eliminates the need for a complete pre-filling calculation for the same prefix portion of each request when processing multiple requests with the same prefix, significantly reducing computational load and thus significantly lowering the first packet latency. This allows users to receive initial feedback on speech generation faster, improving the real-time performance and smoothness of the interaction. Simultaneously, by reducing redundant calculations, the calculation results of a large number of identical prefixes no longer need to be stored in the cache, effectively reducing cache usage and freeing up valuable storage resources. This allows the cache to more efficiently serve the storage and retrieval of other data, improving overall resource utilization. Furthermore, avoiding redundant calculations means that device resources can be allocated and utilized more rationally, reducing device idleness and resource waste caused by redundant calculations, improving device operating efficiency, and enabling the device to process more requests per unit time, further enhancing the system's throughput and processing capacity. In summary, the embodiments of this application have achieved synergistic optimization in terms of reducing first packet latency, reducing cache usage, and improving device utilization by optimizing prefix processing and cache reuse mechanisms, thereby comprehensively improving the performance and efficiency of the streaming speech generation system in multi-request concurrent inference scenarios.

[0007] To better understand and implement this application, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description

[0008] Figure 1 This is a flowchart illustrating the streaming speech generation method according to an embodiment of this application. Detailed Implementation

[0009] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings. Wherein, when the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements.

[0010] It should be understood that the embodiments described below do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application.

[0011] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application are also intended to include the plural forms unless the context clearly indicates otherwise. Furthermore, in the description of this application, unless otherwise stated, “a plurality” means two or more. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items, for example, A and / or B, which can represent: A alone, A and B together, and B alone; the character “ / ” generally indicates that the preceding and following objects are in an “or” relationship.

[0012] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, this information should not be limited to these terms, and these terms are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances. Depending on the context, the word "if" as used in this application can be interpreted as "when," "when," or "in response to determination."

[0013] Please refer to Figure 1 The streaming speech generation method described in this application includes the following steps: S101: Receive speech generation request and construct prefix unit sequence; S102: Divide the prefix unit sequence into several prefix blocks; calculate the rolling block identifier corresponding to each prefix block in sequence according to the order of the several prefix blocks; S103: Based on the rolling block identifier of each prefix block, retrieve the matching block key value from the target cache table; if a matching block key value exists, reuse the block key value as the block key value of the corresponding prefix block; if no matching block key value exists, pre-fill the corresponding prefix block using the speech generation model to obtain the block key value; store the calculated block key value and the corresponding rolling block identifier into the target cache table; S104: Based on the block key values ​​of each of the aforementioned prefix blocks, the speech is decoded and generated using a speech generation model to obtain the speech result.

[0014] In this embodiment, after receiving a speech generation request and constructing a prefix unit sequence, the sequence is divided into prefix blocks and a rolling block identifier is calculated. When a matching block key value is retrieved from the target cache table based on the rolling block identifier, if a match exists, the block key value is directly reused for subsequent decoding and generation, avoiding repeated pre-filling calculations for the same prefix blocks. This mechanism eliminates the need for a complete pre-filling calculation for the same prefix portion of each request when processing multiple requests with the same prefix, significantly reducing computational load and thus significantly lowering the first packet latency. This allows users to receive initial feedback on speech generation faster, improving the real-time performance and smoothness of the interaction. Simultaneously, by reducing redundant calculations, the calculation results of a large number of identical prefixes no longer need to be stored in the cache, effectively reducing cache usage and freeing up valuable storage resources. This allows the cache to more efficiently serve the storage and retrieval of other data, improving overall resource utilization. Furthermore, avoiding redundant calculations means that device resources can be allocated and utilized more rationally, reducing device idleness and resource waste caused by redundant calculations, improving device operating efficiency, and enabling the device to process more requests per unit time, further enhancing the system's throughput and processing capacity. In summary, the embodiments of this application have achieved synergistic optimization in terms of reducing first packet latency, reducing cache usage, and improving device utilization by optimizing prefix processing and cache reuse mechanisms, thereby comprehensively improving the performance and efficiency of the streaming speech generation system in multi-request concurrent inference scenarios.

[0015] The streaming speech generation method described in this application uses a computer device as the execution entity. The following provides a detailed description of each step.

[0016] For step S101, a speech generation request is received, and a prefix unit sequence is constructed.

[0017] A speech generation request is the instruction carrier that initiates a streaming speech synthesis task, used to transmit speech generation-related configuration and content information to the system. This request must contain at least the text to be synthesized and speech synthesis parameters, which may include timbre, speech rate, emotion, etc., based on which the input sequence can be constructed and subsequent cache retrieval, model inference, and speech output processes can be executed.

[0018] Prefix unit sequence refers to the ordered set of feature units formed by standardized encoding of input content during the streaming speech generation stage, used for model inference. It is generated based on the text to be synthesized and speech synthesis parameters, and consists of several semantically related basic units arranged in text order. It serves as a unified input carrier for block segmentation, cache retrieval, and model pre-filling, providing basic data for subsequent rolling block identifier recursion.

[0019] In this step, after receiving the voice generation request initiated by the user, the relevant content in the request is analyzed and processed. For example, if the request contains text information, the starting part of the text is extracted; if the request contains prompt speech, the leading features of the prompt speech are extracted. Then a prefix unit sequence is constructed based on the extracted information, and this sequence provides basic data for subsequent processing.

[0020] In one embodiment, the voice generation request includes a text to be synthesized and voice synthesis parameters; The step of receiving the voice generation request and constructing the prefix unit sequence in step S101 includes: In step S1011, a voice generation request is received; the text to be synthesized is subjected to serial encoding processing to obtain a plurality of ordered input units.

[0021] Speech synthesis parameters are a set of parameters specified by the user to control the effect of speech generation, including but not limited to speech rate, pitch, emotional style, pause threshold, etc., which are used to custom adjust the prosodic features of speech output.

[0022] Serial encoding processing refers to the process of converting the text to be synthesized into a digital sequence that can be processed by a model. Usually, a text encoder is used to generate ordered input units, and each unit corresponds to a semantic atom of the text (such as a word, a subword, or a phoneme).

[0023] Ordered input units are discrete digital sequences formed after serial encoding, arranged in the original order of the text, retaining structural information such as word order and punctuation, and serving as basic data for constructing the prefix unit sequence.

[0024] After receiving the request containing the text to be synthesized and the speech synthesis parameters in this step, the text to be synthesized is preprocessed (such as removing special symbols and normalizing polyphonic characters) to ensure coding consistency. The cleaned text is converted into an ordered input unit sequence through a pre-trained text encoder. For example, a phoneme encoder is used to map "hello" to the digital sequence corresponding to [n,i,h,a,o], and each unit carries corresponding phoneme features. The speech synthesis parameters are converted into a numerical vector, which together with the ordered input units constitutes the basic data of the model input.

[0025] In step S1012, the plurality of ordered input units are processed based on the speech synthesis parameters to obtain the prefix unit sequence.

[0026] The ordered input units are adapted based on the speech synthesis parameters. For example, the sequence length is adjusted according to the speech rate parameter to generate a prefix unit sequence that meets the target duration; the sequence features are adjusted in combination with the emotional style parameter, and control symbols such as a start symbol and an end symbol are added to the beginning and end of the sequence.

[0027] This embodiment achieves efficient conversion from user input to prefix unit sequences through a collaborative design of serialization coding and parameterization, providing accurate initial data support for streaming speech generation. Specifically, serialization coding ensures the complete digital expression of text semantics, enabling the model to accurately capture the temporal dependencies and semantic details of the input text, avoiding speech distortion caused by coding errors. Customized processing based on speech synthesis parameters ensures that the generated speech strictly conforms to the style requirements specified by the user. For example, adjusting the speech rate parameter achieves natural and fluent acceleration or deceleration effects, and injecting emotion parameters achieves accurate expression of different tones such as pleasantness and seriousness. The construction of prefix unit sequences not only provides a structured input foundation for subsequent block partitioning and cache management but also achieves a direct correlation between the generation process and user needs through parameter fusion.

[0028] In one embodiment, after receiving the speech generation request and constructing the prefix unit sequence in step S101, the method further includes the following step: Step S1013: Create a corresponding task sequence object for the current speech generation request; The task sequence object is an independent management unit created for each speech generation request. It encapsulates the request's context information (such as the text to be synthesized and synthesis parameters), processing status (such as waiting / running), resource requirements (such as cache space and computing units), and associated prefix unit sequence, and is used to track the processing progress and resource consumption of the request throughout the process.

[0029] This step, after receiving a speech generation request and constructing a prefix unit sequence, creates a dedicated task sequence object for that request. This object contains: request metadata: such as request ID, user identifier, text to be synthesized, speech synthesis parameters, and other raw information; processing status tracking: recording the current status of the request, such as waiting queue, running queue, or completed, for easy system monitoring and management. Resource binding information: associating the prefix unit sequence, rolling block identifier generation rules, and subsequent resources such as cache space and model computation units.

[0030] Step S1014: Add the task sequence object to the waiting queue; when the cache resources are ready, move the task sequence object to the running queue to execute the subsequent speech generation steps based on prefix blocks, rolling block identifiers and block key values.

[0031] The waiting queue is a queue that stores a sequence of task objects that have not yet acquired cached resources. It is arranged according to request priority or arrival order to ensure that the system schedules requests in an orderly manner when resources are limited.

[0032] The run queue is a queue that stores task sequence objects that have acquired cached resources and entered the execution phase. The system processes tasks sequentially according to the queue order, and performs subsequent block partitioning, cache retrieval, and decoding generation steps.

[0033] Cache resources are ready when the target cache table contains a block key-value pair that matches the current request prefix unit sequence, or when the cache space has met the new data storage requirements and is ready to execute subsequent steps.

[0034] This step adds the task sequence objects to the waiting queue and executes the following scheduling logic: tasks are queued in the waiting queue according to priority (e.g., higher-priority user requests are prioritized) or arrival order, and the system periodically checks the status of cache resources. When the system's free cache space is sufficient to store the new block key-value pairs required by the current task, the cache resources are considered ready. The task sequence objects with ready resources in the waiting queue are moved to the run queue, and subsequent steps are executed in sequence (e.g., prefix block partitioning, rolling block identifier calculation, cache retrieval, and decoding generation) to ensure that tasks progress quickly when resources are sufficient.

[0035] This embodiment achieves refined management and efficient resource utilization of voice generation requests through the creation of task sequence objects and a queue scheduling mechanism, significantly improving the system's concurrent processing capabilities and response speed. Specifically, the task sequence object tracks the request processing status and resource usage throughout the process, avoiding information chaos and resource conflicts when multiple requests are concurrent. For example, requests from different users can be prioritized through independent objects, ensuring that high-priority tasks get cached resources first. The hierarchical scheduling of the waiting queue and the running queue enables the system to process requests in an orderly manner when cached resources are limited, avoiding task backlog or delays caused by resource contention. The dynamic judgment of cached resource readiness and the task shifting mechanism enable the system to accurately match requests and resources, reducing invalid waiting time.

[0036] In one embodiment, a task sequence object scheduling management step is also included: Step S1015: When cache resources are insufficient, the task sequence objects in the run queue are rolled back to the waiting queue; Insufficient cache resources means that the current cache space is completely occupied, or the remaining space is insufficient to store the prefix block, rolling block identifier, and block key-value data required by the new task, causing the new task to be unable to enter the execution queue.

[0037] Task rollback refers to moving task sequence objects that cannot continue execution due to insufficient cache resources in the run queue back into the waiting queue, waiting for resources to be released before rescheduling.

[0038] This step dynamically monitors the cache resource status during execution. When insufficient cache space is detected, priority is given to executing high-priority tasks or tasks already in the run queue. If a new task cannot enter the run queue due to insufficient resources, no rollback is triggered. If a task in the run queue cannot continue processing due to insufficient resources (e.g., it needs to write to the cache but the space is full), the rollback mechanism is initiated. A task sequence object with lower priority or shorter waiting time is selected from the run queue, its status is marked as "rollback," and it is moved back to the end of the waiting queue. During rollback, key information such as the task's prefix unit sequence and synthesis parameters are preserved to avoid duplicate calculations. After a task rollback, the system releases some of its temporary resources (such as computation units) but retains its associated cache data (such as generated prefix blocks) for rapid recovery during subsequent rescheduling.

[0039] Step S1016: When any speech generation request is completed or canceled, the corresponding task sequence object is removed from the waiting queue and the running queue.

[0040] Task removal refers to the complete removal of the corresponding task sequence object from the waiting queue and running queue when the speech generation request is completed (successfully generating speech) or canceled by the user, thereby releasing the system resources (such as memory and computing units) it occupies.

[0041] This step performs the following cleanup operations upon task completion or cancellation: First, check if the task sequence object exists simultaneously in both the waiting queue and the running queue (e.g., duplicate existence due to rollback). If so, it needs to be processed synchronously. Locate the target task sequence object from both the waiting queue and the running queue, physically delete it from the queue, and update the queue length statistics. Delete all temporary data associated with the task (e.g., intermediate calculation results) and clean up the cache space it occupies (e.g., generated but unused block key-value pairs) to ensure timely reclamation of system resources. Furthermore, the time and reason for task completion or cancellation can be recorded, and notifications (e.g., voice-generated result links or cancellation confirmation messages) can be sent to the user or upstream system.

[0042] This embodiment achieves both efficient utilization of cached resources and system stability through a dynamic task rollback and cleanup mechanism, significantly improving the adaptability of the streaming speech generation system under resource fluctuation scenarios. Specifically, the task rollback mechanism avoids task timeout failures due to long waiting times by adding tasks blocked in the running queue due to insufficient resources back to the waiting queue, while freeing up execution space for high-priority tasks or new tasks. The task cleanup mechanism prevents invalid tasks from occupying queue space and system resources by promptly removing completed or canceled tasks, avoiding performance degradation caused by queue expansion.

[0043] For step S102, the prefix unit sequence is divided into several prefix blocks; according to the order of the several prefix blocks, the rolling block identifier corresponding to each prefix block is calculated in sequence.

[0044] A prefix block is a subsequence obtained by dividing the prefix unit sequence into subsequences of fixed length, which serves as the basic processing unit for cache retrieval and computation reuse.

[0045] The rolling block identifier is a representation identifier calculated for each prefix block, used to quickly match block-based key values ​​in the target cache table.

[0046] After obtaining the prefix unit sequence, this step divides it into multiple prefix blocks of a fixed length. These divided prefix blocks are easier for the system to manage and process. Next, following the order of the prefix blocks, a rolling block identifier is calculated for each prefix block. The rolling block identifier can be calculated using methods such as hash algorithms. By calculating the feature information of the prefix block, a unique identifier is obtained. This identifier accurately reflects the content and characteristics of the prefix block, providing a basis for subsequent retrieval in the cache table.

[0047] In one embodiment, the step of sequentially calculating the scroll block identifier corresponding to each of the plurality of prefix blocks according to their order includes: The corresponding scroll block identifier is calculated based on the content of the first prefix block; starting from the second prefix block, the scroll block identifier corresponding to the current prefix block is recursively calculated based on the content of the current prefix block and the scroll block identifier of the previous prefix block.

[0048] Recursive computation is a method of incremental computation based on the calculated results. It uses the calculation results of the previous step (such as the rolling block identifier of the previous prefix block) and the input of the current step (the content of the current prefix block) to perform combined calculations to quickly obtain the result of the current step (the rolling block identifier of the current prefix block).

[0049] In this embodiment, a hash operation is directly performed on the content of the first prefix block to generate its rolling block identifier. This identifier fully reflects the characteristic information of the first prefix block and serves as the basis for subsequent recursive calculations. Starting from the second prefix block, an incremental calculation strategy is adopted, using the content of the current prefix block and the rolling block identifier of the previous prefix block as input to recursively calculate the rolling block identifier of the current block. This method fully utilizes the calculation result of the previous block, avoids the overhead of rescanning the entire prefix sequence, and ensures the distinguishability and continuity of the identifiers—the identifiers of adjacent blocks are related to reflect the continuity of the sequence, yet they differ due to content differences, ensuring the accuracy of block matching.

[0050] For step S103, based on the rolling block identifier of each prefix block, a matching block key value is retrieved from the target cache table; if a matching block key value exists, the block key value is reused as the block key value of the corresponding prefix block; if no matching block key value exists, the corresponding prefix block is pre-filled using a speech generation model to obtain the block key value; the calculated block key value and the corresponding rolling block identifier are stored in the target cache table.

[0051] The target cache table is a structured cache carrier used to store historical calculation results and support fast retrieval, providing data support for multi-request reuse. In this embodiment, the target cache table uses a scroll block identifier as an index and a block key value as the storage content to avoid redundant calculations and improve the efficiency of streaming speech generation.

[0052] Block-based key values ​​are segmented intermediate feature data output by the speech generation model during the pre-filling stage, providing a basis for reuse in the subsequent decoding stage. They correspond one-to-one with prefix blocks, are calculated by the model from the prefix blocks, and can be uniquely matched using rolling block identifiers. They can be shared and reused by different speech generation requests, reducing the overhead of repetitive inference.

[0053] A speech generation model is a deep learning network model used to perform acoustic feature inference and speech waveform generation. The model consists of two stages: pre-filling computation and decoding generation. The former is used to compute block key-value caches, while the latter performs incremental inference based on cached features to output speech latent variables. Finally, after decoding, a playable continuous streaming speech is obtained.

[0054] Pre-padding computation is a model computation process that performs offline inference on the input segment features before formal speech decoding. By performing context computation on the prefix block, block-based key values ​​are generated, providing initial cached state and context information for decoding.

[0055] This step uses the calculated rolling block identifiers of each prefix block to search the target cache table. The target cache table stores the rolling block identifiers and blockization keys corresponding to previously calculated prefix blocks. If a blockization key matching the current prefix block's rolling block identifier is found in the cache table, it means the prefix block has already been calculated, and this blockization key is directly reused as the current prefix block's blockization key, avoiding duplicate calculations. If no matching blockization key is found in the cache table, it means the prefix block is being processed for the first time, and the speech generation model is invoked to perform pre-filling calculations on the prefix block. Pre-filling calculations perform a series of operations based on the prefix block's content and the speech generation model's requirements to obtain the corresponding blockization key. Then, the newly calculated blockization key and the corresponding rolling block identifier are stored in the target cache table so that they can be directly reused when the same prefix block is encountered subsequently.

[0056] In one embodiment, step S103, which involves retrieving a matching block key value from the target cache table based on the rolling block identifier of each of the prefix blocks, includes: Step S1031: Perform the search sequentially according to the order of the prefix blocks; when a prefix block with no matching block key value is found, stop the search for subsequent prefix blocks; determine the prefix block and all subsequent prefix blocks as prefix blocks with no matching block key value.

[0057] In this step, cache queries are performed strictly according to the order of prefix block division (from the first prefix block to the last) to ensure that the retrieval process conforms to the natural temporal characteristics of speech generation. When the first prefix block without a matching block key value is retrieved, subsequent searches are immediately stopped, and it is directly determined that neither this prefix block nor all its subsequent prefix blocks have a valid cache. This is because there are computational dependencies between prefix blocks; if the previous block misses the cache, the subsequent blocks will inevitably also miss the cache.

[0058] This embodiment achieves significant optimization of computational efficiency and resource utilization in streaming speech generation scenarios through sequential retrieval and early termination mechanisms. On the one hand, it avoids redundant cache lookup operations on subsequent prefix blocks that are bound to mismatch, reducing the number of retrievals from O(n) (in the worst case, all blocks need to be retrieved) to O(k) (where k is the position of the first unmatched block), significantly shortening the latency of the pre-filling stage. On the other hand, in the case of multiple concurrent requests, quickly identifying unmatched blocks can trigger model calculations as early as possible, reducing the waiting time caused by cache misses, thereby reducing the latency of the first packet and improving the real-time performance perceived by the user.

[0059] In one embodiment, if step S103 does not have a matching block key value, the step of pre-filling the corresponding prefix block using a speech generation model to obtain the block key value includes: Step S1032: Based on the preset context structure, organize the block key values ​​of the reused prefix blocks and the contents of the prefix blocks that do not have matching block key values ​​to obtain the second context structure data.

[0060] In speech generation tasks, contextual structure refers to the specific structure used to organize input data (such as text, audio features, etc.). It typically includes historical information, current input, and possible future information to help the model understand the complete semantics and temporal relationships of the sequence. For example, in an autoregressive model, the contextual structure might include the previously generated preceding text and the text segment currently being processed.

[0061] In this step, key-value data corresponding to reused prefix blocks (i.e., blocks with matching block values) are extracted from the target cache table. These key-value pairs have been pre-filled and contain the hidden states and contextual information required by the model to process preceding content. The content of prefix blocks without matching block values ​​(such as text fragments or audio features) is concatenated to the reused key-value data in its original order to form a complete sequence input. According to preset context structure rules, the concatenated data is organized—filling in fields such as position information, cache slot mapping, cache block table, and cumulative query / key length—to form context data for pre-filling calculation, i.e., the second context structure data. Based on this data, the model only performs pre-filling on unmatched parts, without recalculating reused blocks, thus reducing computational load.

[0062] Step S1033: Input the second context structure data into the speech generation model for pre-filling calculation to obtain the block key value corresponding to each prefix block that does not have a matching block key value.

[0063] Pre-filling computation is the process by which a speech generation model initializes the prefix block of the input before formally decoding and generating speech, generating block-based key values ​​containing key information such as hidden states and attention weights, which are then used for fast inference in the subsequent decoding stage.

[0064] Block-based key-value pairs are intermediate results generated after pre-filling computation. They are usually stored in the form of key-value pairs, where the "key" is used for attention queries during model decoding, and the "value" contains the feature representation of the prefix block, which is the basis for decoding and generating coherent speech.

[0065] In this step, if the unmatched block is at the beginning of the sequence, the model needs to start processing from the initial state; if it is in the middle, the hidden state in the reused key is used to initialize the model, avoiding repeated pre-padding calculations of the preceding prefix block. The model processes the second context structure data layer by layer, generating block-based key-value pairs corresponding to each unmatched prefix block. For example, in the Transformer model, the hidden state at each position is calculated using a self-attention mechanism and stored as key-value pairs for the decoder. The generated block-based key-value pairs are separated according to the unmatched prefix block division, ensuring that each block corresponds to independent key-value data, facilitating subsequent caching and decoding.

[0066] This embodiment significantly improves the performance and resource utilization of streaming speech generation by intelligently organizing contextual data and efficiently pre-filling computation. By reusing cached block-based key values, the model only needs to compute the unmatched parts, avoiding full pre-filling from the beginning of the sequence to the unmatched block, which is suitable for long sequences or frequently updated prefix scenarios. The second contextual structure data integrates the reused key values ​​and the content of the unmatched block, ensuring that the model can still access complete historical information when processing unmatched blocks, thereby generating semantically coherent block-based key values ​​and avoiding speech quality degradation (such as repetition, omission, or logical errors) caused by context breaks.

[0067] In one embodiment, step S103, which stores the calculated block key value and the corresponding scroll block identifier into the target cache table, further includes: Step S1034: Initialize the reference count of the block key value stored in the target cache table to a preset value.

[0068] Reference counting is a counting mechanism used in this embodiment to track the number of times a resource (here, a block key value in the cache table) is used. The increase or decrease of the value reflects the reference status of the resource. When the count reaches zero, it indicates that the resource is no longer being used by any requests and can be safely released.

[0069] When a newly generated block key is first stored in the target cache table, its reference count is initialized to a preset value (e.g., 1). This preset value indicates that the current block key is referenced by at least one speech generation request (i.e., the current request) and needs to be retained in the cache for later use. In one embodiment, the preset value is 1.

[0070] Step S1035: When the block key value in the target cache table is reused by other speech generation requests, the reference count of the corresponding block key value is incremented by a preset value.

[0071] If other speech generation requests retrieve and reuse the block key value through the rolling block identifier, the reference count of the corresponding block key value will be increased by a preset value (e.g., incremented by 1). This operation indicates that the scope of reference for the block key value has been expanded, and its lifespan in the cache needs to be extended to avoid premature deletion.

[0072] Step S1036: When the current speech generation request or the other speech generation request is completed, the reference count of the corresponding block key value is reduced by a preset value.

[0073] When the current speech generation request or other request reusing the block key value is completed (such as releasing resources after generating complete speech), the reference count of the corresponding block key value is reduced by a preset value (e.g., by 1). This operation reflects the release of the request's reference to the block key value, and the count needs to be updated to accurately track the remaining reference count.

[0074] Step S1037: When the reference count of the block key value is 0, delete the block key value and its corresponding scroll block identifier from the target cache table.

[0075] Continuously monitor the reference count of each block key value in the target cache table. When the reference count of a block key value drops to 0, it indicates that it is no longer referenced by any active request. At this time, the block key value and its corresponding rolling block identifier are automatically deleted from the cache table to free up storage space for new data.

[0076] This embodiment introduces a reference counting mechanism to dynamically manage the target cache table, achieving efficient utilization and real-time updates of cache resources, and significantly improving the overall performance of the streaming speech generation system. Specifically, the initialization and increment / decrement operations of the reference count accurately track the referenced status of block key-value pairs in multi-request scenarios, avoiding resource waste caused by expired or duplicated cached data. For example, when multiple requests reuse the same block key-value pair, the reference count increases synchronously, ensuring that the data is not mistakenly deleted before all requests are completed; while when requests finish one by one, the count decrement mechanism can release useless data in a timely manner, preventing cache bloat. Secondly, the automatic deletion strategy when the reference count reaches zero optimizes the storage structure of the cache table, ensuring that it always retains frequently used block key-value pairs while eliminating infrequent or invalid data. This dynamic cleanup mechanism reduces interference during cache retrieval, improves the matching efficiency of subsequent requests, and is particularly suitable for the limited cache space constraints in high-concurrency scenarios.

[0077] For step S104, based on the block key values ​​of each of the prefix blocks, the speech is decoded and generated using a speech generation model to obtain the speech result.

[0078] Decoding generation is a process of gradually generating streaming speech content based on pre-filled block key values ​​through a speech generation model.

[0079] After obtaining the block key values ​​of each prefix block, this step uses these block key values ​​as input to call the speech generation model for decoding and generation. The speech generation model will gradually generate the corresponding speech content based on the information contained in the block key values. This process is similar to piecing together a complete picture from jigsaw puzzle pieces, ultimately obtaining a complete speech result, which is then output to the user.

[0080] In one embodiment, step S104, which involves decoding and generating speech results based on the block key values ​​of each prefix block using a speech generation model, includes: Step S1041: Organize the block key values ​​of each prefix block based on a preset context structure to obtain the first context structure data.

[0081] The block key values ​​corresponding to all prefix blocks required for this generation (including block key values ​​reused from the cache and newly calculated block key values ​​pre-filled) are organized according to a preset context structure—filling in fields such as location metadata, cache slot mapping, block table, and total context length—to form context data for decoding inference, i.e., the first context structure data. The model performs step-by-step decoding based on this context, without needing to rebuild the context, ensuring consistency between the pre-filling and decoding execution paths, and improving the stability of streaming generation.

[0082] Step S1042: Input the first context structure data into the speech generation model to obtain the speech result.

[0083] The organized initial contextual structure data is input into the speech generation model, and the decoding process is executed. The model calculates the association weights between the current decoding position and each block key value in the contextual structure data through a self-attention mechanism (such as multi-head attention in Transformer), focusing on the historical information most relevant to the current generation. Based on the attention weights, the model fuses the block key values ​​in the contextual structure data and predicts the speech features at the current time step through a feedforward neural network. For example, the hidden state of the historical block is combined with the text features of the current input to generate corresponding acoustic features. The model generates speech features sequentially time steps until the output terminator is reached or the preset sequence length is reached. Finally, the generated speech feature sequence is converted into a playable speech result (such as waveform synthesis through a vocoder).

[0084] This embodiment significantly improves the coherence, accuracy, and real-time performance of streaming speech generation by refining the organization of contextual structure data and efficiently decoding and generating it. Specifically, the pre-defined contextual structure rules ensure the orderly organization of block-based key values, enabling the model to accurately capture the temporal dependencies and semantic relationships between prefix blocks during decoding. Secondly, the block-based key values, as intermediate results of pre-filled computation, directly contain the feature representations of prefix blocks and attention query information, reducing the burden of repetitive computation during model decoding. Compared with directly inputting raw text or audio features, using block-based key values ​​can significantly shorten decoding latency, making it particularly suitable for real-time voice interaction scenarios with low latency requirements. Finally, the speech generation model performs autoregressive decoding based on the well-organized contextual structure data, dynamically focusing on key historical information through an attention mechanism to generate semantically natural and prosodic speech results.

[0085] In one embodiment, step S1042, which involves inputting the first contextual structure data into the speech generation model to obtain the speech result, includes: Step S10421: Input the first context structure data into the speech generation model, and perform segment-by-segment decoding inference based on the first context structure data through the speech generation model to generate latent variable units for the current segment of speech one by one.

[0086] Segment-by-segment decoding inference is a phased processing method for speech generation models. It breaks down long input sequences into multiple short segments (such as fixed-length audio frame groups) and generates intermediate representations (speech latent variable units) for each segment in turn, avoiding memory overflow or computational delay caused by processing long sequences all at once.

[0087] Speech latent variable units are intermediate feature representations generated by the model during the decoding process. They are usually low-dimensional vectors or hidden states that encode the semantics, prosody, and other information of the current speech segment. They need to be further processed before they can be converted into a playable speech signal.

[0088] In this step, the speech generation model takes the first contextual structure data as input and splits it into multiple subsequences according to a preset segment length (e.g., 16 frames per segment). Decoding and inference are then performed on each segment sequentially. The model uses a self-attention mechanism to focus on contextual information relevant to the current segment (e.g., block-based key values ​​from previous segments) and generates the attention weight distribution for the current segment. Based on these attention weights, the model fuses contextual features and predicts the speech latent variable unit (e.g., hidden state vector) for the current segment. This unit encodes the semantic and acoustic features of the current segment. Each latent variable unit is output immediately after generation, without waiting for all segments to be decoded, supporting real-time speech generation.

[0089] Step S10422: The padding frame of the previous segment of latent variable unit is concatenated with the latent variable unit of the current segment of speech, and the corresponding speech result block is obtained through speech decoding processing.

[0090] Frame completion is used to ensure speech continuity by retaining and reusing the tail frame of the previous latent variable unit of speech, which is then used to splice with the current latent variable unit of speech, eliminating boundary discontinuities, dropouts, or noise caused by block generation.

[0091] Speech decoding is the process of converting latent variable units (including padding frames) of speech into actual speech signals. It is usually achieved through a vocoder or inverse Mel transform, and outputs speech result blocks in waveform or spectrum form.

[0092] A speech result block is a local speech segment generated by a single decoding inference. All result blocks need to be concatenated to form a complete speech output.

[0093] To ensure the continuity of speech between segments, this step extracts several frames (e.g., 2 frames) from the end of the latent variable units of the previous segment as padding frames. These padding frames can contain silence or repeated end features, used to fill in gaps in the transition between segments. The padding frames are then concatenated with the latent variable units of the current segment in chronological order to form a continuous intermediate feature sequence. For example, if the previous segment has 2 padding frames and the current segment has 16 frames, the concatenated sequence will have 18 frames. The concatenated feature sequence is then input into a vocoder, where an inverse transform (e.g., from Mel spectrum to waveform) is used to generate the corresponding speech result block (e.g., 0.5 seconds of audio), ensuring a natural and uninterrupted transition between segments.

[0094] Step S10423: Obtain the final speech result based on all speech result blocks.

[0095] Following the temporal order of the original input sequence, all speech result blocks are sequentially concatenated into a complete speech stream. For example, if N result blocks are generated, each with a duration of T, the final speech duration is N×T, and the semantics and prosody are consistent with the input text.

[0096] This embodiment significantly improves the real-time performance and stability of streaming speech generation while maintaining speech quality through segmented decoding and dynamic concatenation techniques. Specifically, segment-by-segment decoding inference breaks long sequences into shorter segments, reducing the amount of data input to the model per iteration and avoiding computational interruptions due to insufficient memory. Simultaneously, the intra-segment self-attention mechanism still captures local temporal dependencies, ensuring the semantic coherence of each speech segment. Secondly, the introduction of padding frames solves the problem of discontinuity during inter-segment transitions. By adding extra frames to the end of the previous segment, the model can smoothly transition to the current segment during concatenation, avoiding speech jumps or noise caused by fixed segment lengths or decoding delays. Finally, the block-based execution and real-time concatenation of speech decoding processing enable the system to output speech result blocks while decoding, without waiting for all generation to complete, significantly reducing end-to-end latency.

[0097] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of this application, and this application also intends to include these modifications and variations.

Claims

1. A streaming speech generation method, characterized in that, Includes the following steps: Receive speech generation requests and construct a prefix unit sequence; The prefix unit sequence is divided into several prefix blocks; the rolling block identifier corresponding to each prefix block is calculated sequentially according to the order of the prefix blocks. Based on the rolling block identifier of each prefix block, a matching block key value is retrieved from the target cache table; if a matching block key value exists, it is reused as the block key value of the corresponding prefix block; if no matching block key value exists, the corresponding prefix block is pre-filled using a speech generation model to obtain the block key value; the calculated block key value and the corresponding rolling block identifier are stored in the target cache table. Based on the block key values ​​of each of the aforementioned prefix blocks, the speech is decoded and generated using a speech generation model to obtain the speech result.

2. The streaming speech generation method according to claim 1, characterized in that, The step of calculating the rolling block identifier corresponding to each of the prefix blocks in sequence, according to the order of the prefix blocks, includes: The corresponding scroll block identifier is calculated based on the content of the first prefix block; starting from the second prefix block, the scroll block identifier corresponding to the current prefix block is recursively calculated based on the content of the current prefix block and the scroll block identifier of the previous prefix block.

3. The streaming speech generation method according to claim 1 or 2, characterized in that: The step of retrieving a matching block-based key value from the target cache table based on the rolling block identifier of each of the aforementioned prefix blocks includes: The search is performed sequentially according to the order of the prefix blocks; when a prefix block with no matching block key value is found, the search for subsequent prefix blocks is stopped; the prefix block and all subsequent prefix blocks are determined as prefix blocks with no matching block key value.

4. The streaming speech generation method according to claim 1, characterized in that, The step of storing the calculated block key value and the corresponding rolling block identifier into the target cache table further includes: Initialize the reference count of the block key value stored in the target cache table to a preset value; When a block key value in the target cache table is reused by other speech generation requests, the reference count of the corresponding block key value is incremented by a preset value. When the current speech generation request or the other speech generation request is completed, the reference count of the corresponding block key value is decremented by a preset value; When the reference count of a block key is 0, the block key and its corresponding scroll block identifier are deleted from the target cache table.

5. The streaming speech generation method according to claim 1, characterized in that, The steps for decoding and generating speech results based on the block key values ​​of each prefix block using a speech generation model include: Based on a preset context structure, the block key values ​​of each prefix block are organized to obtain the first context structure data; The first context structure data is input into the speech generation model to obtain the speech result.

6. The streaming speech generation method according to claim 5, characterized in that, The steps of inputting the first contextual structure data into the speech generation model to obtain the speech result include: The first context structure data is input into the speech generation model, and the speech generation model performs segment-by-segment decoding inference based on the first context structure data to generate latent variable units for the current segment of speech one by one. The padding frame of the previous latent variable unit is concatenated with the latent variable unit of the current latent variable unit, and the corresponding speech result block is obtained through speech decoding. Based on all the speech result blocks, the final speech result is obtained.

7. The streaming speech generation method according to claim 1, 5, or 6, characterized in that, If no matching block key exists, the block key is obtained by pre-filling the corresponding prefix block using the speech generation model. This process includes: Based on a preset context structure, the block key values ​​of reused prefix blocks and the contents of prefix blocks without matching block key values ​​are organized to obtain the second context structure data. The second context structure data is input into the speech generation model for pre-filling calculation to obtain the block key value corresponding to each prefix block that does not have a matching block key value.

8. The streaming speech generation method according to claim 1, characterized in that, The speech generation request includes the text to be synthesized and speech synthesis parameters; The steps for receiving a speech generation request and constructing a prefix unit sequence include: Receive a speech generation request; perform serialization encoding on the text to be synthesized to obtain several ordered input units; The ordered input units are processed based on the speech synthesis parameters to obtain the prefix unit sequence.

9. The streaming speech generation method according to claim 1, characterized in that, After receiving the speech generation request and constructing the prefix unit sequence, the following steps are also included: Create a corresponding task sequence object for the current speech generation request; The task sequence object is added to the waiting queue; when the cached resources are ready, the task sequence object is moved to the running queue to execute the subsequent speech generation steps based on the prefix block, rolling block identifier and block key value.

10. The streaming speech generation method according to claim 9, characterized in that, It also includes task sequence object scheduling and management steps: When cache resources are insufficient, the task sequence objects in the run queue are rolled back to the waiting queue; When any speech generation request is completed or canceled, the corresponding task sequence object is removed from the waiting queue and the running queue.

Images