A generation-play time interval regulation method and device for real-time interactive audio-video generation

Abstract

The application discloses a generation-play time interval regulation method and device for real-time interactive audio and video generation, and relates to the technical field of artificial intelligence and real-time interactive generation type audio and video stream. The generation-play time interval regulation method and device for real-time interactive audio and video generation comprises the following steps: step 1, generating a lead time interval double attribute modeling and utilization; step 2, generating a rhythm valve closed loop regulation; step 3, voice completion degree triggering content switching; step 4, prompting a supply queue and priority insertion improvement channel scheduling; step 5, zero pre-buffering instant playing execution; and step 6, real-time decoding base stable support. The scheme can stabilize the generation lead time interval in a preset target interval, absorb generation side rate fluctuation, guarantee playing continuity, control interactive response time delay in a sub-second level range, and balance playing fluency and interactive response speed.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence and real-time interactive generative audio and video streaming technology, and in particular to a generation-playback time interval control method and apparatus for real-time interactive audio and video generation. Background Technology

[0002] In recent years, audio and video applications that are "generated in real time by generative models and can be viewed and manipulated by users in real time" have emerged. Typical scenarios include real-time interactive digital humans and AI anchors, live e-commerce, AI emotional companionship and character dialogue, and interactive "playable generation". These scenarios generally require three conditions to be met simultaneously: (i) low interaction latency (the system must respond "immediately" after the user speaks / operates); (ii) real-time interruption / rewriting (barge-in, changing topics, changing camera angles); and (iii) long-term uninterrupted playback (continuous playback for tens of minutes to several hours with stable image quality). The traditional paradigm of "offline rendering before playback" cannot meet these three conditions simultaneously.

[0003] Quantifiable human factors and experience thresholds (used as background constraints): The response interval in natural human conversation generally peaks at around 200 milliseconds and is highly consistent across languages; voice interaction latency is generally almost imperceptible within 300 milliseconds, starts to feel slow around 500 milliseconds, and becomes noticeably "machine-like" after 1 second; the "stop voice output" time when interrupted should ideally be within 150 milliseconds; highly interactive visuals (such as cloud gaming) typically require end-to-end latency of 40–50 milliseconds. On the other hand, streaming media research shows that "stuttering / buffering" has a precipitous impact on the experience—a single stutter can cause about 40% of viewers to drop out, and the longer the startup latency, the higher the abandonment rate. Stuttering is the most weighted negative factor in Quality of Experience (QoE). Therefore, real-time interactive generation must simultaneously consider "low interaction latency" and "zero-stutter continuous playback," which are inherently conflicting. Summary of the Invention

[0004] The purpose of this invention is to at least solve one of the technical problems existing in the prior art, and to provide a generation-playback time interval control method and device for real-time interactive audio and video generation, which can solve the problem of the inherent conflict between "low interactive latency" and "zero stuttering continuous playback".

[0005] To achieve the above objectives, the present invention provides the following technical solution: a generation-playback time interval control method for real-time interactive audio and video generation, comprising the following steps: Step 1: generating advanced time interval dual attribute modeling and utilization; Step 2: generating rhythm valve closed-loop control; Step 3: triggering content switching based on voice completion; Step 4: prompting supply queue and priority insertion channel scheduling; Step 5: zero pre-buffering instant playback execution; and Step 6: real-time decoding base steady-state support. Step 1: generating advanced time interval dual attribute modeling and utilization includes Step 101 generating advanced time interval quantitative definition; Step 102 time interval asset attribute implementation application; Step 103 time interval liability attribute constraint setting; and Step 104 calculating load bearing within the time interval. Step 2: Generating rhythm valve closed-loop control includes step 201 real-time measurement and deviation judgment of time interval, step 202 generating throttling when time interval exceeds target window, step 203 generating rate sprint when time interval is too thin, and step 204 maintaining time interval in playback pause state. Step 3: Voice completion triggers content switching, which includes step 301 tail silence detection and judgment, step 302 script transcription coverage judgment, and step 303 content prompt adaptive switching. Step 4: Prompt supply queue and priority insertion channel scheduling includes step 401 asynchronous management of supply queue, step 402 priority insertion access of user instructions, and step 403 targeted replacement for no generated cycle time. Step 5: Zero-pre-buffered instant playback execution includes step 501 client pre-buffered gate removal and step 502 unit rate uniform playback execution. Step 6: Real-time decoding base steady-state support includes step 601 sliding window constant cost decoding, step 602 converter block-by-block compilation optimization, step 603 multi-process time-domain overlapping scheduling, and step 604 audio window consistency reconstruction.

[0006] Preferably, step 101 generates a leading time interval quantization definition; In systems where the generation frame rate is higher than the playback frame rate, the generation lead time is defined as the time difference between the generation head and the playback head, corresponding to the duration of audio and video content that has been generated but has not yet been viewed by the audience, using time length as the unit of measurement. Step 102: Time-based application of asset attributes; The time interval is used as a buffer asset for scheduling: on the one hand, the time interval is used to absorb the jitter generated by the generation side - the single-step time generated by autoregression will fluctuate with the content complexity, sampling steps and cache status. This fluctuation can be buffered by the time interval and will not be transmitted as playback stuttering; on the other hand, relying on the front window provided by the time interval, the status inspection and anomaly repair operations are completed before the content reaches the audience. Step 103: Setting time-distance liability attribute constraints; Time intervals are managed as latency liabilities: the content of generated and queued frames cannot be modified, and the time interval for each unit of time is added equally to the visible response time of user interaction; based on the human factor threshold that voice interaction is almost imperceptible within 300 milliseconds and begins to show lag after 500 milliseconds, the upper limit of time intervals is constrained to ensure that the overall interaction latency is in the sub-second range. Step 104: Calculate load capacity within the time interval; Execute specific scheduling operations: All intelligent orchestration, online inspection and forward repair calculation tasks of the system are deployed on the generator side and executed within the time range covered by the time interval; the audience will only arrive at the corresponding content after a time interval, so the above-mentioned calculation overhead and generation rate fluctuations are absorbed by the time interval and will not delay the playback progress. The playback end always maintains a unit rate and seamless output.

[0007] Preferably, step 201 involves real-time distance measurement and deviation determination; The specific measurement process is as follows: Based on the real-time timestamp of the viewer's playback head, the latest completion timestamp of the generator head is periodically collected to calculate the real-time generation lead time distance value; the measured value is compared with the preset time distance target window to determine whether the time distance is excessive, insufficient or normal, and corresponding adjustment instructions are generated. Step 202: Throttling is generated when the distance exceeds the target window; When the time interval exceeds the upper limit of the target window, the generation rhythm valve sends a throttling instruction to the generator to gradually reduce the generation frame rate, so that the generation rate is only slightly higher than the playback rate, and the time interval gradually falls back to the target window range, avoiding the continuous expansion of the time interval from increasing the interaction latency. Step 203: When the time interval is relatively thin, the generation rate sprint is performed. When the time interval is close to the lower limit of the target window and there is a risk of underload, the generation rhythm valve issues a sprint command to increase the generation computing power allocation and increase the generation frame rate to more than 30 frames per second. This replenishes the existing content at a rate higher than that of real-time playback, quickly raising the time interval back to the target range and avoiding playback stuttering caused by time interval exhaustion. Step 204: Maintain the paused playback time. When a viewer triggers a playback pause operation, the generation rhythm valve locks the current generation progress, stops the generation and output of new frames, and keeps the duration at the value before the pause, thus preventing the generation end from continuously outputting during the playback pause and causing the duration to grow indefinitely.

[0008] Preferably, step 301 involves tail noise detection and determination; The specific detection process is as follows: real-time amplitude monitoring of the decoded audio stream is performed, and changes in the audio signal are continuously captured after the expected duration of a single speech sentence ends; when a silent segment of more than 150 milliseconds is detected, the current speech sentence is determined to be completed. Step 302: Script transcription coverage determination; The specific matching process is as follows: the real-time generated speech is transcribed synchronously, and the transcribed text is matched with the corresponding prompt script dialogue content by calculating the character matching degree; when the proportion of the transcribed content covering the script dialogue reaches a preset threshold, it is determined that the speech content corresponding to the current prompt has been generated. Step 303 prompts for adaptive switching; When either the tail silence detection or the transcription coverage determination meets the completion condition, the scheduling system triggers a prompt sequence switch, connecting the next prompt to the generation link; replacing the traditional fixed-duration switching method, avoiding the interruption of voice or the occurrence of loop playback.

[0009] Preferably, step 401 prompts asynchronous management of the supply queue; Execute the specific queue management process: establish an asynchronous prompt supply queue, and queue the prompt content to be executed in the order of generation; calculate the total generation time of all prompts in the queue in real time, and control the total time within the current real time interval to ensure that all content in the queue is in the "unplayed" pre-release interval; Step 402: User commands are prioritized for insertion and modification access; A priority insertion channel, independent of the regular queue, is set up. User-initiated control commands enter the scheduling layer directly through this channel, without participating in the queuing and waiting of the regular queue, and obtain the highest scheduling priority. Step 403 did not generate a beat-oriented replacement; The specific replacement process is as follows: After receiving the priority instruction, the scheduling system locates all beat nodes in the prompt queue that have not yet entered the generation process, and only replaces the generated content corresponding to these nodes, while keeping the generated but unplayed content unchanged; the effective duration of the instruction at the generation head is the sum of the arrangement delay and the remaining duration of the current speech, and the content update effect corresponding to the instruction will reach the audience after one time interval, without waiting for the entire queue to be emptied.

[0010] Preferably, in step 501, the client pre-buffer gate is removed; The playback client eliminates the traditional pre-buffering threshold setting for streaming media, does not set the initial buffering duration or buffer level, and does not back up and wait for arriving media segments. Once the segment data arrives, it can enter the rendering process. Step 502: Play at a constant unit rate. The playback end renders and plays at a constant speed with a fixed unit rate. The continuity of playback is fully guaranteed by the time interval buffer on the generation side, without the need for client-side buffering to absorb fluctuations, thus avoiding additional end-to-end interaction latency caused by client-side buffering.

[0011] Preferably, step 601 involves sliding window constant-cost decoding; A sliding window decoder is used to perform block-by-block decoding. A fixed-length attention sliding window is set, and each decoding process only processes the context data within the window. This ensures that the memory usage and computing power consumption of the decoding process do not increase linearly with the total generation time, maintaining a constant cost for long-term operation and supporting continuous generation from tens of minutes to several hours. Step 602: Transformer block-by-block compilation and optimization; Each generation unit of the transformer model is individually compiled and optimized to reduce the noise reduction processing latency of a single block of content, improve the generation efficiency of a single block, and ensure that the average generation frame rate is consistently higher than the playback frame rate. Step 603: Multi-process time-domain overlapping scheduling; The three types of tasks—decoding computation, video encoding, and audio reconstruction—are deployed in separate dedicated processes; this allows the post-processing task of the current data block to overlap with the generation and decoding task of the next data block in time, thus compressing the end-to-end processing time of a single data block. Step 604: Audio window consistency reconstruction; The audio stream is reconstructed according to the rule that "the audio result of each output window is consistent with the complete offline decoding result". The audio data at the junction of the sliding windows is smoothed and calibrated to eliminate the audio discontinuity problem caused by the naive sliding window decoding, so that the final output byte stream is consistent with the output quality of the offline rendering of the same content.

[0012] Preferably, a generation-playback time interval control device for real-time interactive audio and video generation includes a time interval estimation and modeling module, a generation rhythm valve, a speech completion detection and switching module, a prompt supply queue and priority insertion channel module, a zero pre-buffered playback client, and a real-time decoding base.

[0013] Preferably, the real-time decoding base includes a sliding window decoder, block-by-block compilation, a dedicated decoding process, and an audio window.

[0014] Preferably, the real-time decoding base serves as the underlying computing power unit, outputting the encoded audio and video frame data. One transmission is sent to the time interval estimation and modeling module for progress statistics, and the other transmission is sent to the zero-pre-buffered playback client for playback. The time interval estimation and modeling module simultaneously collects real-time playback progress data from the zero-pre-buffered playback client, calculates the real-time generated advance time interval, and then feeds the time interval data back to the generation rhythm valve. The generation rhythm valve generates a rate control command based on the time interval deviation and sends it to the real-time decoding base to adjust the generation speed. The voice completion detection and switching module collects the audio stream output by the real-time decoding base. After completing the voice completion judgment, it sends the switching command to the prompt supply queue and the priority insertion channel module. The prompt supply queue and priority insertion channel module simultaneously receive external business prompts and user interaction instructions. After completing the content scheduling according to priority, a prompt input real-time decoding base will be generated to control the direction of the generated content.

[0015] Compared with the prior art, the beneficial effects of the present invention are: 1. The generation-playback time interval control method and device for real-time interactive audio and video generation can stabilize the generation lead time interval within a preset target range. While absorbing the generation rate fluctuation and ensuring continuous playback, it controls the interactive response latency in the sub-second range, taking into account both playback smoothness and interactive response speed.

[0016] 2. The generation-playback time interval control method and device for real-time interactive audio and video generation can absorb the scheduling, status inspection, anomaly repair calculations, and natural fluctuations in the generation rate during system operation by the generation advance time interval buffer, without adding additional visible latency on the playback side, and the playback process remains uniform and seamless.

[0017] 3. This generation-playback time interval control method and device for real-time interactive audio and video generation allows user interaction commands to replace ungenerated content nodes through a priority channel without waiting for the generated queue to be emptied. The effective delay of the command at the generation end is determined only by the arrangement delay and the remaining duration of the current utterance, resulting in higher response efficiency.

[0018] 4. The generation-playback time interval control method and device for real-time interactive audio and video generation adopts a content switching mechanism driven by voice completion, replacing the traditional fixed clock switching method. This can avoid the problem of voice being truncated or looping playback, and the content connection is more in line with the logic of natural expression.

[0019] 5. The generation-playback time interval control method and device for real-time interactive audio and video generation, based on a constant cost decoding architecture with a sliding window, can ensure the stability of computing power and video memory costs during long-term operation, support continuous generation and playback from tens of minutes to several hours, and the final output audio and video quality is consistent with the offline rendering result of the same content. Attached Figure Description

[0020] The present invention will be further described below with reference to the accompanying drawings and embodiments: Figure 1 This is a schematic diagram illustrating the generation of the lead time interval in this invention; Figure 2 This is a schematic diagram of the throttling / sprint closed-loop control of the rhythm valve generated in this invention; Figure 3 This is a schematic diagram illustrating the speech completion trigger switching and priority insertion channel replacement only for ungenerated beats in this invention; Figure 4This is a schematic diagram of the real-time decoding base of the present invention. Detailed Implementation

[0021] This section will describe in detail specific embodiments of the present invention. Preferred embodiments of the present invention are shown in the accompanying drawings. The purpose of the drawings is to supplement the textual description with graphics, so that people can intuitively and vividly understand each technical feature and overall technical solution of the present invention, but they should not be construed as limiting the scope of protection of the present invention.

[0022] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0023] In the description of this invention, terms such as greater than, less than, and exceeding are understood to exclude the stated number, while terms such as above, below, and within are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0024] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0025] Please see Figure 1-4 This invention provides a technical solution: a generation-playback time interval control method for real-time interactive audio and video generation, comprising the following steps: Step 1: Generating and utilizing the dual attributes of advance time interval. This step completes the quantitative definition of time interval, attribute decomposition, and value realization, providing a core logical benchmark for the entire regulatory mechanism; Step 101 generates the lead time quantization definition; In systems where the generation frame rate is higher than the playback frame rate (more than 30 frames per second on the generation side and 25 frames per second on the playback side), the generation lead time is defined as the time difference between the generation head (the timestamp of the latest frame that has been generated and can enter the playback queue) and the playback head (the timestamp of the frame that the viewer is currently watching), which corresponds to the duration of audio and video content that has been generated but has not yet been watched by the viewer. Step 102: Time-based application of asset attributes; The time interval is used as a buffer asset for scheduling: on the one hand, the time interval is used to absorb the jitter generated by the generation side - the single-step time generated by autoregression will fluctuate with the content complexity, sampling steps and cache status. This fluctuation can be buffered by the time interval and will not be transmitted as playback stuttering; on the other hand, relying on the front window provided by the time interval, the status inspection and anomaly repair operations are completed before the content reaches the audience. Step 103: Setting time-distance liability attribute constraints; Time intervals are managed as latency liabilities: the content of generated and queued frames cannot be modified, and the time interval for each unit of time is added equally to the visible response time of user interaction; based on the human factor threshold that voice interaction is almost imperceptible within 300 milliseconds and begins to show lag after 500 milliseconds, the upper limit of time intervals is constrained to ensure that the overall interaction latency is in the sub-second range. Step 104: Calculate load capacity within the time interval; Specific scheduling operations are performed: all intelligent orchestration, online inspection and forward repair calculation tasks of the system are deployed on the generator side and executed within the time range covered by the time interval; viewers need to wait for a time interval before they will reach the corresponding content, so the above-mentioned calculation overhead and generation rate fluctuations are absorbed by the time interval and will not delay the playback progress. The playback end always maintains a unit rate and seamless output. Step 2: Generate closed-loop control of the rhythm valve; This step uses the viewer's playback head as a reference and dynamically adjusts the generation rate to achieve closed-loop stable control of the time interval. Step 201: Real-time time distance measurement and deviation determination; The specific measurement process is as follows: Based on the real-time timestamp of the viewer's playback head, the latest completion timestamp of the generator head is periodically collected to calculate the real-time generation lead time distance value; the measured value is compared with the preset time distance target window to determine whether the time distance is excessive, insufficient or normal, and corresponding adjustment instructions are generated. Step 202: Throttling is generated when the distance exceeds the target window; When the time interval exceeds the upper limit of the target window, the generation rhythm valve sends a throttling instruction to the generator to gradually reduce the generation frame rate, so that the generation rate is only slightly higher than the playback rate (e.g., from 35 frames per second to 27 frames per second), so that the time interval gradually falls back to the target window range, avoiding the continuous expansion of the time interval leading to an increase in interaction latency. Step 203: When the time interval is relatively thin, the generation rate sprint is performed. When the time interval is close to the lower limit of the target window and there is a risk of underload, the generation rhythm valve issues a sprint command to increase the generation computing power allocation and increase the generation frame rate to more than 30 frames per second. This replenishes the existing content at a rate higher than that of real-time playback, quickly raising the time interval back to the target range and avoiding playback stuttering caused by time interval exhaustion. Step 204: Maintain the paused playback time. When a viewer triggers a playback pause operation, the generation rhythm valve locks the current generation progress, stops the generation and output of new frames, and keeps the duration at the value before the pause, thus preventing the generation end from continuously outputting during the playback pause and causing the duration to grow indefinitely. Step 3: Content switching is triggered based on voice completion. This step uses voice-driven prompt switching based on the actual completion status, replacing the fixed clock switching mechanism and ensuring natural transitions in voice content. Step 301: Rear end noise detection and determination; The specific detection process is as follows: real-time amplitude monitoring of the decoded audio stream is performed, and changes in the audio signal are continuously captured after the expected duration of a single speech sentence ends; when a silent segment of more than 150 milliseconds is detected, the current speech sentence is determined to be completed. Step 302: Script transcription coverage determination; The specific matching process is as follows: the real-time generated speech is transcribed synchronously, and the transcribed text is matched with the corresponding prompt script dialogue content by calculating the character matching degree; when the proportion of the transcribed content covering the script dialogue reaches a preset threshold, it is determined that the speech content corresponding to the current prompt has been generated. Step 303 prompts for adaptive switching; When either the tail silence detection or the transcription coverage determination meets the completion condition, the scheduling system triggers a prompt sequence switch, connecting the next prompt to the generation link; replacing the traditional fixed-duration switching method, avoiding the voice being cut off midway or looping; Step 4: Prompt the supply queue and priority insertion channel scheduling; This step enables the orderly scheduling of routine prompts and the rapid activation of user interaction commands, balancing content continuity and interaction response speed. Step 401 prompts for asynchronous management of the supply queue; Execute the specific queue management process: establish an asynchronous prompt supply queue, and queue the prompt content to be executed in the order of generation; calculate the total generation time of all prompts in the queue in real time, and control the total time within the current real time interval to ensure that all content in the queue is in the "unplayed" pre-release interval; Step 402: User commands are prioritized for insertion and modification access; A priority insertion channel, independent of the regular queue, is set up. User-initiated control commands enter the scheduling layer directly through this channel, without participating in the queuing and waiting of the regular queue, and obtain the highest scheduling priority. Step 403 did not generate a beat-oriented replacement; The specific replacement process is as follows: After receiving the priority instruction, the scheduling system locates all beat nodes in the prompt queue that have not yet entered the generation process, and only replaces the generated content corresponding to these nodes, leaving the generated but unplayed content unchanged; the effective duration of the instruction at the generation head is the sum of the arrangement delay and the remaining duration of the current speech, and the content update effect corresponding to the instruction will reach the audience after one time interval, without waiting for the entire queue to be emptied; Step 5: Zero-pre-buffering instant playback execution; This step achieves low-latency continuous playback through the bufferless design on the playback end, combined with time interval adjustment on the generation side; Step 501: Remove the client-side pre-buffered gate; The playback client eliminates the traditional pre-buffering threshold setting for streaming media, does not set the initial buffering duration or buffer level, and does not back up and wait for arriving media segments. Once the segment data arrives, it can enter the rendering process. Step 502: Play at a constant unit rate. The playback end renders and plays at a constant speed at a fixed unit rate (25 frames per second). The playback continuity is completely guaranteed by the time interval buffer on the generation side, without the need for the client buffer to absorb fluctuations, thus avoiding the client buffer adding extra end-to-end interaction latency. Step 6: Real-time decoding base steady-state support; This step optimizes the underlying decoding architecture to ensure the stability of the generated computing power and output quality over a long period, providing a continuous computing power foundation for time interval adjustment; Step 601: Sliding window constant cost decoding; A sliding window decoder is used to perform block-by-block decoding. A fixed-length attention sliding window is set, and each decoding process only processes the context data within the window. This ensures that the memory usage and computing power consumption of the decoding process do not increase linearly with the total generation time, maintaining a constant cost for long-term operation and supporting continuous generation from tens of minutes to several hours. Step 602: Transformer block-by-block compilation and optimization; Each generation unit of the transformer model is individually compiled and optimized to reduce the noise reduction processing latency of a single block of content, improve the generation efficiency of a single block, and ensure that the average generation frame rate is consistently higher than the playback frame rate. Step 603: Multi-process time-domain overlapping scheduling; The three types of tasks—decoding computation, video encoding, and audio reconstruction—are deployed in separate dedicated processes; this allows the post-processing task of the current data block to overlap with the generation and decoding task of the next data block in time, thus compressing the end-to-end processing time of a single data block. Step 604: Audio window consistency reconstruction; The audio stream is reconstructed according to the rule that "the audio result of each output window is consistent with the complete offline decoding result". The audio data at the junction of the sliding windows is smoothed and calibrated to eliminate the audio discontinuity problem caused by the naive sliding window decoding, so that the final output byte stream is consistent with the output quality of the offline rendering of the same content. Furthermore, a generation-playback time interval control device for real-time interactive audio and video generation includes a time interval estimation and modeling module, a generation rhythm valve, a speech completion detection and switching module, a prompt supply queue and priority insertion channel module, a zero pre-buffered playback client, and a real-time decoding base. The real-time decoding platform includes a sliding window decoder, block-by-block compilation, a dedicated decoding process, and an audio window; The real-time decoding base serves as the underlying computing unit, outputting encoded audio and video frame data. One transmission is sent to the time-distance estimation and modeling module for progress statistics, and the other is sent to the zero-pre-buffered playback client for playback. The time interval estimation and modeling module simultaneously collects real-time playback progress data from the zero-pre-buffered playback client, calculates the real-time generated advance time interval, and then feeds the time interval data back to the generation rhythm valve. The generation rhythm valve generates a rate control command based on the time interval deviation and sends it to the real-time decoding base to adjust the generation speed. The voice completion detection and switching module collects the audio stream output by the real-time decoding base. After completing the voice completion judgment, it sends the switching command to the prompt supply queue and the priority insertion channel module. The prompt supply queue and priority insertion channel module simultaneously receive external business prompts and user interaction instructions. After completing the content scheduling according to priority, a prompt input real-time decoding base is generated to control the direction of the generated content. Time-distance estimation and modeling module: It is responsible for generating real-time calculations and attribute modeling of advance time intervals, synchronously collecting progress data from the generation and playback ends, and outputting real-time time interval values ​​at fixed intervals; at the same time, it maintains the target window parameters and warning thresholds of the time intervals, providing benchmark data and judgment basis for closed-loop control. Generate rhythm valve: As the core execution unit of closed-loop control, it receives real-time time distance data output by the time distance estimation and modeling module, compares it with the target window to determine the deviation status, and generates three types of control instructions: throttling, sprinting, and holding. These instructions are then sent to the generation end to dynamically adjust the generation rate and stabilize the time distance within the target range. Voice completion detection and switching module: It is responsible for determining the completion status of the generated speech, identifying the end of the sentence through two methods: end silence detection and transcription coverage verification, outputting a content switching signal, driving the prompt queue update, replacing the fixed duration switching mechanism, and ensuring natural speech content connection. Prompt for supply queue and priority insertion channel module: On the one hand, it undertakes the queuing and scheduling function of regular prompt content, and manages the total queue duration to match the current time interval; on the other hand, it provides an independent high-priority interaction channel to process real-time user commands, and replace content nodes that have not been generated in the queue in a targeted manner, so as to realize the rapid effect of interaction commands. Zero-prebuffering playback client: As a user-facing playback terminal, it does not have a pre-buffering mechanism. It starts rendering immediately after receiving audio and video segments and outputs them at a constant playback rate. The playback continuity relies entirely on the time interval buffering guarantee on the generation side to avoid additional interaction latency caused by client-side buffering. Real-time decoding dock: As the underlying generation unit of the device, it consists of four parts: a sliding window decoder, a block-by-block compilation unit, a dedicated decoding process, and an audio window reconstruction unit. The sliding window decoder achieves constant-cost autoregressive generation, ensuring stable computing power during long-term operation. The block-by-block compilation unit optimizes the denoising latency of single-block data, improving generation efficiency. The dedicated decoding process compresses the total processing time through multi-process temporal overlap. The audio window reconstruction unit ensures that the audio quality of streaming output is consistent with offline rendering. Overall, it provides stable and efficient real-time generation capabilities for the entire control device. Furthermore, this solution can stabilize the generation lead time within a preset target range, absorbing generation rate fluctuations and ensuring playback continuity while controlling the interaction response latency to the sub-second range, thus balancing playback smoothness and interaction response speed. Furthermore, the scheduling, status inspection, anomaly repair calculations, and natural fluctuations in the generation rate during system operation can all be absorbed by the generation advance time buffer, without adding any visible latency to the playback side, and the playback process remains uniform and seamless. Furthermore, user interaction commands can replace ungenerated content nodes through priority channels without waiting for the generated queue to be emptied. The effective latency of the command at the generation end is determined only by the orchestration latency and the remaining duration of the current utterance, resulting in higher response efficiency. Furthermore, by adopting a content switching mechanism driven by voice completion, instead of the traditional fixed clock switching method, the problem of voice being truncated or looping can be avoided, and the content connection is more in line with the logic of natural expression. Furthermore, the constant-cost decoding architecture based on sliding windows can ensure stable computing power and memory costs during long-term operation, support continuous generation and playback from tens of minutes to several hours, and ensure that the final output audio and video quality is consistent with the offline rendering results of the same content.

[0026] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A generation-playback time interval control method for real-time interactive audio and video generation, comprising the following steps: Step 1: Generate advanced time-distance dual attribute modeling and utilization; Step 2: Generate rhythm valve closed-loop control; Step 3: Voice completion trigger content switching; Step 4: Prompt supply queue and priority insertion channel scheduling; Step 5: Zero pre-buffering instant playback execution; and Step 6: Real-time decoding base steady-state support. The characteristic feature is that Step 1: Generate advanced time-distance dual attribute modeling and utilization includes Step 101: Generate advanced time-distance quantitative definition; Step 102: Time-distance asset attribute implementation application; Step 103: Time-distance liability attribute constraint setting; and Step 104: Calculate load bearing within the time distance. Step 2: Generating rhythm valve closed-loop control includes step 201 real-time measurement and deviation judgment of time interval, step 202 generating throttling when time interval exceeds target window, step 203 generating rate sprint when time interval is too thin, and step 204 maintaining time interval in playback pause state. Step 3: Voice completion triggers content switching, which includes step 301 tail silence detection and judgment, step 302 script transcription coverage judgment, and step 303 content prompt adaptive switching. Step 4: Prompt supply queue and priority insertion channel scheduling includes step 401 asynchronous management of supply queue, step 402 priority insertion access of user instructions, and step 403 targeted replacement for no generated cycle time. Step 5: Zero-pre-buffered instant playback execution includes step 501 client pre-buffered gate removal and step 502 unit rate uniform playback execution. Step 6: Real-time decoding base steady-state support includes step 601 sliding window constant cost decoding, step 602 converter block-by-block compilation optimization, step 603 multi-process time-domain overlapping scheduling, and step 604 audio window consistency reconstruction.

2. The generation-playback time interval control method for real-time interactive audio and video generation according to claim 1, characterized in that: Step 101 generates a leading time interval quantization definition; In systems where the generation frame rate is higher than the playback frame rate, the generation lead time is defined as the time difference between the generation head and the playback head, corresponding to the duration of audio and video content that has been generated but has not yet been viewed by the audience, using time length as the unit of measurement. Step 102: Time-based application of asset attributes; The time interval is used as a buffer asset for scheduling: on the one hand, the time interval is used to absorb the jitter generated by the generation side - the single-step time generated by autoregression will fluctuate with the content complexity, sampling steps and cache status. This fluctuation can be buffered by the time interval and will not be transmitted as playback stuttering; on the other hand, relying on the front window provided by the time interval, the status inspection and anomaly repair operations are completed before the content reaches the audience. Step 103: Setting time-distance liability attribute constraints; Time intervals are managed as latency liabilities: the content of generated and queued frames cannot be modified, and the time interval for each unit of time is added equally to the visible response time of user interaction; based on the human factor threshold that voice interaction is almost imperceptible within 300 milliseconds and begins to show lag after 500 milliseconds, the upper limit of time intervals is constrained to ensure that the overall interaction latency is in the sub-second range. Step 104: Calculate load capacity within the time interval; Execute specific scheduling operations: All intelligent orchestration, online inspection and forward repair calculation tasks of the system are deployed on the generator side and executed within the time range covered by the time interval; the audience will only arrive at the corresponding content after a time interval, so the above-mentioned calculation overhead and generation rate fluctuations are absorbed by the time interval and will not delay the playback progress. The playback end always maintains a unit rate and seamless output.

3. The generation-playback time interval control method for real-time interactive audio and video generation according to claim 1, characterized in that: Step 201 involves real-time time distance measurement and deviation determination. The specific measurement process is as follows: Based on the real-time timestamp of the viewer's playback head, the latest completion timestamp of the generator head is periodically collected to calculate the real-time generation lead time distance value; the measured value is compared with the preset time distance target window to determine whether the time distance is excessive, insufficient or normal, and corresponding adjustment instructions are generated. Step 202: Throttling is generated when the distance exceeds the target window; When the time interval exceeds the upper limit of the target window, the generation rhythm valve sends a throttling instruction to the generator to gradually reduce the generation frame rate, so that the generation rate is only slightly higher than the playback rate, and the time interval gradually falls back to the target window range, avoiding the continuous expansion of the time interval from increasing the interaction latency. Step 203: When the time interval is relatively thin, the generation rate sprint is performed. When the time interval is close to the lower limit of the target window and there is a risk of underload, the generation rhythm valve issues a sprint command to increase the generation computing power allocation and increase the generation frame rate to more than 30 frames per second. This replenishes the existing content at a rate higher than that of real-time playback, quickly raising the time interval back to the target range and avoiding playback stuttering caused by time interval exhaustion. Step 204: Maintain the paused playback time. When a viewer triggers a playback pause operation, the generation rhythm valve locks the current generation progress, stops the generation and output of new frames, and keeps the duration at the value before the pause, thus preventing the generation end from continuously outputting during the playback pause and causing the duration to grow indefinitely.

4. The generation-playback time interval control method for real-time interactive audio and video generation according to claim 1, characterized in that: Step 301 involves tail noise detection and determination. The specific detection process is as follows: real-time amplitude monitoring of the decoded audio stream is performed, and changes in the audio signal are continuously captured after the expected duration of a single speech sentence ends; when a silent segment of more than 150 milliseconds is detected, the current speech sentence is determined to be completed. Step 302: Script transcription coverage determination; The specific matching process is as follows: the real-time generated speech is transcribed synchronously, and the transcribed text is matched with the corresponding prompt script dialogue content by calculating the character matching degree; when the proportion of the transcribed content covering the script dialogue reaches a preset threshold, it is determined that the speech content corresponding to the current prompt has been generated. Step 303 prompts for adaptive switching; When either the tail silence detection or the transcription coverage determination meets the completion condition, the scheduling system triggers a prompt sequence switch, connecting the next prompt to the generation link; replacing the traditional fixed-duration switching method, avoiding the interruption of voice or the occurrence of loop playback.

5. The generation-playback time interval control method for real-time interactive audio and video generation according to claim 1, characterized in that: Step 401 prompts for asynchronous management of the supply queue; Execute the specific queue management process: establish an asynchronous prompt supply queue, and queue the prompt content to be executed in the order of generation; calculate the total generation time of all prompts in the queue in real time, and control the total time within the current real time interval to ensure that all content in the queue is in the "unplayed" pre-release interval; Step 402: User commands are prioritized for insertion and modification access; A priority insertion channel, independent of the regular queue, is set up. User-initiated control commands enter the scheduling layer directly through this channel, without participating in the queuing and waiting of the regular queue, and obtain the highest scheduling priority. Step 403 did not generate a beat-oriented replacement; Execution of specific replacement process: After receiving the priority instruction, the scheduling system locates all beat nodes in the prompt queue that have not yet entered the generation process, and only replaces the generated content corresponding to these nodes, leaving the generated but unplayed content unchanged. The effective duration of the instruction in the generation header is the sum of the orchestration delay and the remaining duration of the current utterance. The content update effect corresponding to the instruction will reach the audience after one time interval, without waiting for the entire queue to be emptied.

6. The generation-playback time interval control method for real-time interactive audio and video generation according to claim 1, characterized in that: Step 501: Remove the client-side pre-buffered gate; The playback client eliminates the traditional pre-buffering threshold setting for streaming media, does not set the initial buffering duration or buffer level, and does not back up and wait for arriving media segments. Once the segment data arrives, it can enter the rendering process. Step 502: Play at a constant unit rate. The playback end renders and plays at a constant speed with a fixed unit rate. The continuity of playback is fully guaranteed by the time interval buffer on the generation side, without the need for client-side buffering to absorb fluctuations, thus avoiding additional end-to-end interaction latency caused by client-side buffering.

7. The generation-playback time interval control method for real-time interactive audio and video generation according to claim 1, characterized in that: Step 601 is constant-cost decoding using a sliding window; A sliding window decoder is used to perform block-by-block decoding. A fixed-length attention sliding window is set, and each decoding operation only processes the context data within the window. This ensures that the memory usage and computing power consumption during the decoding process do not increase linearly with the total generation time, maintaining a constant cost for long-term operation and supporting continuous generation from tens of minutes to several hours. Step 602: Transformer block-by-block compilation and optimization; Each generation unit of the transformer model is individually compiled and optimized to reduce the noise reduction processing latency of a single block of content, improve the generation efficiency of a single block, and ensure that the average generation frame rate is consistently higher than the playback frame rate. Step 603: Multi-process time-domain overlapping scheduling; The three types of tasks—decoding computation, video encoding, and audio reconstruction—are deployed in separate dedicated processes; this allows the post-processing task of the current data block to overlap with the generation and decoding task of the next data block in time, thus compressing the end-to-end processing time of a single data block. Step 604: Audio window consistency reconstruction; The audio stream is reconstructed according to the rule that "the audio result of each output window is consistent with the complete offline decoding result". The audio data at the junction of the sliding windows is smoothed and calibrated to eliminate the audio discontinuity problem caused by the naive sliding window decoding, so that the final output byte stream is consistent with the output quality of the offline rendering of the same content.

8. A generation-playback time interval control device for real-time interactive audio and video generation, characterized in that: It includes a time interval estimation and modeling module, a rhythm generation valve, a voice completion detection and switching module, a prompt supply queue and priority channel insertion module, a zero-pre-buffered playback client, and a real-time decoding base.

9. The generation-playback time interval control device for real-time interactive audio and video generation according to claim 8, characterized in that: The real-time decoding base includes a sliding window decoder, block-by-block compilation, a dedicated decoding process, and an audio window.

10. The generation-playback time interval control device for real-time interactive audio and video generation according to claim 8, characterized in that: The real-time decoding base serves as the underlying computing power unit, outputting the encoded audio and video frame data. One transmission is sent to the time distance estimation and modeling module for progress statistics, and the other transmission is sent to the zero-pre-buffered playback client for playback. The time interval estimation and modeling module simultaneously collects real-time playback progress data from the zero-pre-buffered playback client, calculates the real-time generated advance time interval, and then feeds the time interval data back to the generation rhythm valve. The generation rhythm valve generates a rate control command based on the time interval deviation and sends it to the real-time decoding base to adjust the generation speed. The voice completion detection and switching module collects the audio stream output by the real-time decoding base. After completing the voice completion judgment, it sends the switching command to the prompt supply queue and the priority insertion channel module. The prompt supply queue and priority insertion channel module simultaneously receive external business prompts and user interaction instructions. After completing the content scheduling according to priority, a prompt input real-time decoding base will be generated to control the direction of the generated content.

Images