Streaming video super-resolution method and system

Abstract

The application discloses a kind of streaming video super-resolution method and system, they are corresponding scheme, in scheme: each low-resolution input frame is only through once forward calculation output high-resolution frame, does not depend on future frame or whole video offline processing, can realize single-step streaming video super-resolution;According to the demand of different layers to time context, learn layer-by-layer cache reading strategy, avoid all layers save and read complete history key-value cache, can reduce history cache overhead;Recycle the super-resolution latent variable of last time step as the current frame condition, form lightweight causal propagation path, reduce local texture flicker, can improve time consistency;Use the last super-resolution latent variable generated by itself as condition during training, make training condition distribution consistent with streaming inference condition distribution, can alleviate training inference inconsistency;By multi-range query discriminator, the consistency of overall realism, local spatial texture and time tubular region is simultaneously constrained, and thus the generation quality can be improved.

Description

Technical Field

[0001] This invention relates to the field of video super-resolution technology, and in particular to a streaming video super-resolution method, system, device and storage medium. Background Technology

[0002] Video super-resolution (VSR) aims to recover high-resolution video from low-resolution video, and has important applications in online video transmission, video conferencing, cloud gaming, mobile terminal shooting, old video restoration, and video enhancement. Compared with single-frame super-resolution, video super-resolution not only needs to improve the spatial sharpness of a single frame, but also needs to make full use of the temporal correlation between consecutive frames to avoid problems such as flickering, texture drift, and cross-frame structural inconsistencies in the output video.

[0003] Existing video super-resolution methods can be broadly categorized into traditional restoration methods and generative methods. Traditional deep learning-based video super-resolution methods typically utilize neighboring frame information through alignment, propagation, and aggregation mechanisms, such as EDVR, BasicVSR, BasicVSR++, RealBasicVSR, and video restoration models (VRT) based on Transformer or spatiotemporal attention. These methods can achieve good results on objective fidelity metrics, but in scenarios with complex realism degradation, strong compression, low bitrates, and severe texture loss, the restoration results are often too smooth and struggle to generate natural details and textures.

[0004] In recent years, diffusion models and generative priors have been introduced into video super-resolution tasks. Methods such as Upscale-A-Video, DiffVSR, and VideoGigaGAN utilize the generative capabilities of pre-trained image or video diffusion models to achieve more natural subjective visual effects in regions such as faces, text, logos, textures, and edges. However, conventional diffusion models typically rely on multi-step iterative sampling, resulting in slow inference speeds and high memory consumption. Some methods also require offline processing of the entire video or access to future frames, making it difficult to meet the low-latency streaming requirements of applications such as real-time video enhancement, online streaming media, and edge deployment.

[0005] To improve inference efficiency, existing research has further explored single-step generative video super-resolution and autoregressive video generation mechanisms. For example, methods like DOVE and SeedVR2 focus on single-step diffusion-based video restoration, while FlashVSR and Stream-DiffVSR further target low-latency streaming video super-resolution. One type of method compresses multi-step diffusion models into single-step generative models through distillation or adversarial post-training; another type reuses historical frame key-value caches in the video Transformer, allowing the current frame to access past information without recalculating historical frames. These methods provide a foundation for real-time generative video super-resolution, but directly applying the complete historical cache to all Transformer layers incurs high GPU memory and attention computation costs.

[0006] Furthermore, in video super-resolution tasks, the current low-resolution frame itself provides explicit observation constraints, and the high-resolution reconstruction result of the previous frame also contains reusable local structure and texture states. Therefore, not all network layers need to access the same range of historical key-value caches. Shallow, medium, and deep layers may have different needs for historical information; some layers only need the current frame and short-term context, some layers need recent windows, and some layers need sparse keyframes as long-term references. Existing fixed-window or full-cache strategies are difficult to reflect these layer-by-layer differences.

[0007] Furthermore, streaming video super-resolution suffers from inconsistencies between training and inference conditions. If the training phase uses latent variables or real historical features from the previous high-resolution frame as conditions, while the inference phase can only use the results from the previous frame generated by the model itself, the model is prone to error accumulation, detail jitter, and decreased temporal consistency. Existing single global discriminators also struggle to simultaneously constrain overall realism, local texture quality, and cross-frame stability.

[0008] In summary, the existing technology has the following main shortcomings: 1) Insufficient streaming efficiency: Multi-step diffusion sampling or offline video modeling results in high latency, making it difficult to meet the requirements of real-time video super-resolution.

[0009] 2) Redundancy of historical cache: Existing autoregressive or Transformer streaming methods often use the same historical cache range in all layers, resulting in large overhead for memory and attention computation.

[0010] 3) Insufficient layer-by-layer time modeling capability: Different network layers have different needs for short-term, long-term or no historical context, and the existing fixed caching strategy lacks the ability to adaptively select layers.

[0011] 4) Inconsistency between training and inference: Relying on the real conditions of the previous frame during the training phase can cause teacher-forced bias, and error accumulation is likely to occur when the model generates its own results during the inference phase.

[0012] 5) It is difficult to balance generation quality and temporal stability: a single loss or a single discrimination range cannot simultaneously optimize the overall visual realism, local texture and continuous frame stability.

[0013] Therefore, there is an urgent need for a video super-resolution system that can be adapted for streaming scenarios, while maintaining the quality of generative recovery, reducing historical caching overhead, and improving temporal consistency.

[0014] In view of this, the present invention is hereby proposed. Summary of the Invention

[0015] The purpose of this invention is to provide a streaming video super-resolution method and system that can achieve high-quality, low-latency, and low-buffering-overhead streaming video super-resolution processing within a single-step diffusion generation framework.

[0016] The objective of this invention is achieved through the following technical solution: A streaming video super-resolution system includes: a low-resolution causal condition construction module, a super-resolution latent variable retrieval module, a static cache inference module, a single-step diffusion generation module, and a decoder; The low-resolution causal condition construction module is used to extract the causal hidden conditions of the current frame and historical frames from the input raw resolution video stream; where the current frame corresponds to the current time step. The super-resolved latent variable recovery module is used to save the super-resolved latent variables generated by the single-step diffusion generation module in the previous time step, and input the super-resolved latent variables of the previous time step as recovery conditions into the single-step diffusion generation module in the current time step. The static cache inference module is used to determine the actions of each layer of the single-step diffusion generation module according to the static cache scheduling table, read the recent frame cache and / or key frame anchor point cache according to the actions, or not read the historical cache, and input them into the single-step diffusion generation module. The single-step diffusion generation module is used to generate the super-resolved latent variables of the current time step by combining the current frame, the causal hiding conditions, the super-resolved latent variables of the previous time step, and the output of the static cache inference module. The decoder is used to transform the super-resolution latent variables of the current time step to the image space to obtain the super-resolution video frame of the current time step. The static cache scheduling table is generated during the training phase. It stores the actions of each layer of the single-step diffusion generation module. During the training phase, the single-step diffusion generation module also uses the super-resolved latent variables of the previous time step as the retrieval condition. The single-step diffusion generation module is trained by performing multi-dimensional discrimination on the super-resolved latent variables generated by the single-step diffusion generation module and constructing adversarial training loss.

[0017] A streaming video super-resolution method, based on the aforementioned system implementation, includes: Extract the causal latent conditions of the current frame and historical frames from the input raw resolution video stream; where the current frame corresponds to the current time step; Save the super-resolved latent variables generated by the single-step diffusion generation module in the previous time step, and input the super-resolved latent variables from the previous time step into the single-step diffusion generation module at the current time step. The actions of each layer of the single-step diffusion generation module are determined according to the static cache scheduling table, and the recent frame cache and / or key frame anchor cache are read according to the actions, or the historical cache is not read and input into the single-step diffusion generation module. The super-resolved latent variables for the current time step are generated by the single-step diffusion generation module, which combines the current frame, causal hidden conditions, super-resolved latent variables from the previous time step, and the output of the static cache inference module. Transform the super-resolution latent variables of the current time step into the image space to obtain the super-resolution video frame of the current time step; The static cache scheduling table is generated during the training phase. It stores the actions of each layer of the single-step diffusion generation module. During the training process, the single-step diffusion generation module is trained by performing multi-dimensional discrimination on the super-resolution latent variables generated by the single-step diffusion generation module and constructing adversarial training loss.

[0018] As can be seen from the technical solutions provided by the present invention above: (1) Each low-resolution input frame outputs a high-resolution frame through the generator forward calculation only once, without relying on future frames or offline processing of the entire video, thereby achieving single-step streaming video super-resolution; (2) According to the needs of different network layers for temporal context, a layer-by-layer cache reading strategy is learned to avoid all layers saving and reading the complete historical Key-Value cache, thereby reducing historical cache overhead; (3) The super-resolution latent variables generated by the model in the previous frame are recycled as conditions for the current frame, forming a lightweight causal propagation path, reducing local texture flicker, thereby improving temporal consistency; (4) During the training phase, the online self-generated super-resolution latent variables of the previous frame are used as conditions to make the training condition distribution consistent with the streaming inference condition distribution, thereby alleviating the inconsistency between training and inference; (5) By using a multi-range query discriminator, the consistency of overall realism, local spatial texture and temporal tubular region is simultaneously constrained, thereby improving the generation quality. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of a streaming video super-resolution system provided in an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram of the training and inference stages of a streaming video super-resolution system provided in an embodiment of the present invention.

[0022] Figure 3 This is a schematic diagram of a layer-by-layer caching route and static export provided in an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0024] First, the following explanations are provided for the terms that may be used in this article: The term "and / or" means that either or both can be achieved simultaneously. For example, X and / or Y means that it includes both "X" or "Y" as well as the three cases of "X and Y".

[0025] The terms “comprising,” “including,” “containing,” “having,” or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, “comprising a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.)” should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0026] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0027] The following is a detailed description of a streaming video super-resolution method and system provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they are performed according to conventional conditions in the art or conditions recommended by the manufacturer. Reagents or instruments used in the embodiments of this invention, unless otherwise specified, are all commercially available conventional products.

[0028] Example 1 like Figure 1 As shown, an embodiment of the present invention provides a streaming video super-resolution system, which mainly includes: a low-resolution causal condition construction module, a super-resolution latent variable recovery module, a static cache inference module, a single-step diffusion generation module, and a decoder; The low-resolution causal condition construction module is used to extract the causal hidden conditions of the current frame and historical frames from the input raw resolution video stream; where the current frame corresponds to the current time step. The super-resolved latent variable recovery module is used to save the super-resolved latent variables generated by the single-step diffusion generation module in the previous time step, and input the super-resolved latent variables of the previous time step as recovery conditions into the single-step diffusion generation module in the current time step. The static cache inference module is used to determine the actions of each layer of the single-step diffusion generation module according to the static cache scheduling table, read the recent frame cache and / or key frame anchor point cache according to the actions, or not read the historical cache, and input them into the single-step diffusion generation module. The single-step diffusion generation module is used to generate the super-resolved latent variables of the current time step by combining the current frame, the causal hiding conditions, the super-resolved latent variables of the previous time step, and the output of the static cache inference module. The decoder is used to transform the super-resolution latent variables of the current time step to the image space to obtain the super-resolution video frame of the current time step. The static cache scheduling table is generated during the training phase. It stores the actions of each layer of the single-step diffusion generation module. During the training phase, the single-step diffusion generation module also uses the super-resolved latent variables of the previous time step as the retrieval condition. The single-step diffusion generation module is trained by performing multi-dimensional discrimination on the super-resolved latent variables generated by the single-step diffusion generation module and constructing adversarial training loss.

[0029] Preferably, the step of extracting the causal hidden conditions of the current frame and historical frames from the input original resolution video stream includes: designating the current frame as the k-th frame, extracting the causal hidden conditions from the 1st frame to the k-th frame from the original resolution video stream, and inputting them as observation evidence of the original resolution into the single-step diffusion generation module.

[0030] Preferably, when the current time step is the first time step, the super-resolved latent variables of the previous time step use the zero vector.

[0031] Preferably, during the training process, the single-step diffusion generation module also uses the super-resolved latent variables from the previous time step as the retrieval condition, including: during the training phase, adopting online self-generation conditions, that is, forming a super-resolved latent variable sequence from the super-resolved latent variables generated by the single-step diffusion generation module, then shifting the super-resolved latent variable sequence one frame and stopping gradient propagation, as the retrieval condition for the next frame of training.

[0032] Preferably, the step of determining the actions of each layer of the single-step diffusion generation module according to the static cache scheduling table, and reading the recent frame cache and / or keyframe anchor cache according to the actions, or not reading the historical cache, includes: for layers whose actions are not to read the historical cache, the corresponding layer's historical key-value information is not materialized; for layers whose actions are to read the most recent r-frame window, the r-frame window is read from the recent frame cache library; for layers whose actions are to read the keyframe anchor cache, the keyframe anchor cache is read from the keyframe anchor cache library; for layers that simultaneously read the most recent r-frame window and the keyframe anchor cache, the r-frame window is read from the recent frame cache library, and the keyframe anchor cache is read from the keyframe anchor cache library.

[0033] Preferably, the system further includes: a layer-by-layer cache routing module, used to generate a static cache scheduling table during the training phase; dividing the historical key-value cache into a recent frame cache library and a key frame anchor cache library; the recent frame cache library stores key-value tensors of the most recent frames, and the key frame anchor cache stores key-value tensors of key frames; for each layer in the single-step diffusion generation module, a set of candidate cache actions is defined, including: not reading the historical cache, reading the most recent r-frame window, reading the key frame anchor cache, and reading both the most recent r-frame window and the key frame anchor cache simultaneously; where r takes one or more preset window lengths, and different layers learn to select different actions; during the training phase, the layer-by-layer cache routing module outputs the selection probability of each candidate action according to the layer identifier or layer embedding; cache budget constraints, route sharpening constraints, and temperature annealing are introduced; cache budget constraints are used to control the average number of cache frames read or cache cost of each layer; route sharpening constraints and temperature annealing are used to encourage each layer to eventually form a clear dominant cache action; after training, the action with the highest probability in each layer is selected and exported as a static cache scheduling table.

[0034] Preferably, during the training phase, the layer-by-layer caching routing module incorporates a route-aware attention bias on top of joint causal attention: based on whether a candidate action allows a historical key-value term to be accessed by the current query, the soft visibility probability of the corresponding historical key-value term is calculated, and the logarithm of the soft visibility probability is added to the attention score; thus, layer-by-layer soft route learning is completed in a single attention calculation. The core function of this process is to transform the originally discrete and non-differentiable caching action selection into a differentiable soft route learning process. During training, each layer does not need to perform multiple attention calculations for different candidate actions. Instead, through the route-aware attention bias, the influence of different historical key-value terms on the current query is continuously adjusted in a single joint causal attention calculation, allowing each layer to gradually learn its truly required temporal context. Subsequently, combined with cache budget constraints, route sharpening, and temperature annealing, the soft route can be converged into explicit layer-by-layer caching actions and exported as a static cache scheduling table for the inference phase. This reduces the storage, retrieval, and attention calculation overhead of historical key-value caches while maintaining temporal modeling capabilities and super-resolution quality as much as possible.

[0035] Preferably, the system further includes: a multi-range query discrimination training module, used to perform multi-dimensional discrimination on the super-resolution latent variables generated by the single-step diffusion generation module during the training phase, and to construct adversarial training loss; the multi-range query discrimination training module is configured with a discrimination output head, and during the training phase, for a given real high-resolution latent variable sequence or a super-resolution latent variable sequence composed of super-resolution latent variables generated by the single-step diffusion generation module, video lexical units with time frame index and spatial location index are extracted; and multi-dimensional discrimination results are output through the discrimination output head.

[0036] Preferably, the multi-range query discrimination training module includes three types of learnable queries: global query, spatial window query, and temporal tube query. The global query aggregates the entire video segment to constrain the overall realism. The spatial window query focuses on a local window in a single frame to constrain local textures, text, edges, and target details. The temporal tube query focuses on the lexical units within the same spatial window over a continuous causal time range to constrain the cross-frame stability of local textures. The three types of queries output global discrimination scores, spatial local discrimination scores, and temporal tube discrimination scores, respectively. The three scores are averaged or weighted and fused into the final discrimination result, and an adversarial training loss is constructed.

[0037] The system provided in this embodiment of the invention can achieve high-quality, low-latency, and low-buffering-overhead video super-resolution processing within a single-step diffusion generation framework. Its main advantages are as follows: (1) Achieving single-step streaming video super-resolution: Each low-resolution input frame is processed by the generator only once to output a high-resolution frame, without relying on future frames or offline processing of the entire video. Of course, if higher image quality is required, single-step generation can be extended to multi-step (i.e., using a multi-step diffusion generation module, combining input information to generate the super-resolution latent variable of the current time step through multiple steps). The specific number of steps can be set according to the actual situation, and this invention does not impose any restrictions.

[0038] (2) Reduce historical cache overhead: Based on the time context requirements of different network layers, learn the layer-by-layer cache reading strategy to avoid all layers saving and reading the complete historical Key-Value cache.

[0039] (3) Improve temporal consistency: The super-resolution latent variables generated by the model in the previous frame are recycled as conditions in the current frame, forming a lightweight causal propagation path and reducing local texture flicker.

[0040] (4) Alleviating the inconsistency between training and inference: During the training phase, the super-resolution latent variables of the previous frame generated online are used as conditions to make the distribution of training conditions consistent with the distribution of streaming inference conditions.

[0041] (5) Improve generation quality: Constrain the overall realism, local spatial texture and temporal tubular region consistency simultaneously through a multi-range query discriminator.

[0042] To more clearly demonstrate the technical solution and its effects provided by the present invention, the system provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0043] 1. Introduction to the overall system architecture.

[0044] The system takes the original resolution (low resolution) video stream as input and processes it frame by frame in chronological order. At each time step, it combines the current and historical low resolution conditions, the self-generated super-resolution latent variables of the previous frame, and the historical cache selected by layer to output the high resolution result of the current frame. The overall scheme includes: a low-resolution causal condition construction module, a super-resolution latent variable retrieval module, a single-step diffusion generation module, a layer-by-layer cache routing module, a static cache inference module, and a multi-range query discrimination training module.

[0045] like Figure 2 As shown, this illustrates the training and inference phases of the system.

[0046] Inference phase: The system receives a low-resolution video stream. First, the low-resolution causal condition construction module extracts the low-resolution hidden conditions (causal hidden conditions) of the current frame and historical frames. At the same time, the super-resolution latent variable recovery module reads the super-resolution latent variables generated in the previous time step and projects them into the hidden space of the single-step diffusion generation module.

[0047] The static cache inference module determines the actions of each layer of the single-step diffusion generation module based on the static cache scheduling table, to read the recent frame cache and / or key frame anchor cache, or not to read the historical cache.

[0048] The single-step diffusion generation module can employ video diffusion Transformer, video DiT, or other video generation networks with causal attention capabilities, such as a video generation backbone based on the large-scale video generation model Wan. Under the common constraints of the current frame's latent variables, causal hidden conditions, the super-resolution latent variables from the previous time step, and the historical key-value cache, this single-step diffusion generation module updates the retrieval latent variables and historical key-value cache required for the next time step by making a forward prediction of the current time step's super-resolution latent variables.

[0049] The decoder transforms the super-resolution latent variables at the current time step to obtain the super-resolution video frame at the current time step.

[0050] During the training phase, the layer-by-layer caching routing module learns the historical cache range that each generation layer should read. During the inference phase, the system uses the static cache scheduling table exported after training and no longer executes the router, thereby reducing additional computation. The multi-range query discrimination training module performs discrimination supervision on the generated latent variable fragments and the real latent variable fragments.

[0051] II. Detailed introduction of the system.

[0052] 1. Low-resolution causal condition construction module.

[0053] For the k-th video frame, causal conditions are constructed using only the low-resolution input from frame 1 to frame k, without accessing future information after frame k. These low-resolution causal conditions can be obtained through a lightweight causal projector, causal encoder, or equivalent network, and are used to provide the generator with low-resolution observational evidence from the current and historical frames.

[0054] This low-resolution causal condition building module is more suitable for streaming inference than performing full video encoding on the entire video, and can reduce the computational load and memory usage of the condition building process.

[0055] 2. Over-division latent variable recovery module.

[0056] The system saves the super-resolution latent variables output by the single-step diffusion generation module at the previous time step and inputs these super-resolution latent variables into the single-step diffusion generation module at the current time step. In specific implementation, the super-resolution latent variables of the previous time step can be mapped to the hidden dimensions of the single-step diffusion generation module through a projection layer, and then fused with the current frame latent variable state through addition, splicing, or conditional modulation.

[0057] The first time step can use a zero vector, a learnable initial vector, or a vector generated from the low-resolution conditions of the first frame as the initial latent variables for recycling. Starting from the second time step, the system uses the super-resolution latent variables generated in the previous time step as the conditions for the current frame, enabling the current frame to reuse the most recent high-resolution structure and texture states.

[0058] To avoid conditional distribution bias caused by using high-resolution latent variables from the previous frame during training, this invention employs online self-generated conditions during training. This approach shares a similar motivation with the training ideas used in autoregressive video diffusion to mitigate training-test bias. Specifically, the current single-step diffusion generation module first generates a super-resolution latent variable sequence based on the low-resolution segment. This sequence is then shifted one frame forward and gradient propagation is stopped, serving as the retrieval condition for the next frame's training. This method allows both the training and inference phases to utilize the model-generated conditions from the previous frame, helping to reduce the accumulation of streaming inference errors.

[0059] 3. Layer-by-layer caching routing module.

[0060] like Figure 3 As shown, this invention divides the historical Key-Value cache into a recent frame cache library and a key frame anchor point cache library. The recent frame cache library stores Key-Value tensors of the most recent few frames to provide short-term motion continuity; the key frame anchor point cache library stores the first frame, periodic frames, scene key frames, or other sparse anchor point frames to provide reference information for long-range background, identity, and recurring targets.

[0061] For each layer in the generator, the system defines a set of candidate cache actions, including: not reading the history cache, reading the most recent r-frame window, reading the keyframe anchor cache, and reading both the most recent r-frame window and the keyframe anchor cache. Here, r can take one or more preset window lengths. Different layers can learn to select different actions.

[0062] During the training phase, the caching router outputs the selection probability of each candidate action based on the layer identifier or layer embedding. To ensure route differentiability and avoid repeatedly performing attention calculations for each action, the system adds a route-aware attention bias to the original joint causal attention: based on whether a candidate action allows a certain historical key-value token to be accessed by the current query, the soft visibility probability of that token is calculated, and the logarithm of the soft visibility probability is added to the attention score. Thus, the system can complete layer-by-layer soft route learning in a single attention calculation.

[0063] The routing training introduces cache budget constraints and route sharpening constraints. Cache budget constraints control the average number of cached frames read or cache cost at each layer; route sharpening constraints and temperature annealing encourage each layer to eventually form a clear dominant cache action. After training, each layer selects the cache action with the highest probability, which is exported as a static cache scheduling table.

[0064] 4. Static cache inference module.

[0065] During the inference phase, the layer-by-layer caching routing module is no longer run, nor is the route-aware attention bias calculated. Instead, the historical key-value cache is selectively saved and read for different layers according to the static cache scheduling table.

[0066] For layers assigned not to read the history cache, the system does not materialize the layer's historical key-value pairs; for layers assigned to read the recent window, the system maintains a fixed-length circular cache; for layers assigned to read keyframe anchors, the system maintains a sparse anchor cache; for combined actions, both the recent window cache and the anchor cache are maintained. In this way, the layer-by-layer temporal range differences learned during training can be translated into actual savings in GPU memory, bandwidth, and computation.

[0067] 5. Multi-range query discrimination training module.

[0068] To improve the recovery quality of the single-step generator, this invention introduces a multi-range query discriminant training module (which can be used as a discriminator). The discriminator can use the same or similar video Transformer backbone as the generator, and is configured with independent parameters and a discriminant output header. For real high-resolution latent variable segments or generated latent variable segments, the discriminator extracts video tokens with temporal frame indices and spatial location indices.

[0069] The discriminator includes three types of learnable queries: global queries, spatial window queries, and temporal tubular queries. Global queries aggregate the entire video segment to constrain overall realism; spatial window queries focus on local windows within a single frame to constrain local textures, text, edges, and object details; and temporal tubular queries focus on tokens within the same spatial window over consecutive causal time ranges to constrain the cross-frame stability of local textures.

[0070] The three types of queries output global discrimination scores, spatial local discrimination scores, and temporal tubular discrimination scores, respectively. These three scores can be averaged or weighted and fused into a final discrimination result. An adversarial training loss is then used to optimize the single-step diffusion generation module. This adversarial training can combine the loss construction methods used in diffusion adversarial post-training or autoregressive adversarial post-training; specific details can refer to conventional techniques, which will not be elaborated upon in this invention. The training objectives of the single-step diffusion generation module may include reconstruction loss, adversarial loss, cache budget loss, and route sharpening loss.

[0071] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0072] Example 2 This invention also provides a streaming video super-resolution method, which is mainly used for system implementation based on the foregoing embodiments. The method mainly includes: Extract the causal latent conditions of the current frame and historical frames from the input raw resolution video stream; where the current frame corresponds to the current time step; Save the super-resolved latent variables generated by the single-step diffusion generation module in the previous time step, and input the super-resolved latent variables from the previous time step into the single-step diffusion generation module at the current time step. The actions of each layer of the single-step diffusion generation module are determined according to the static cache scheduling table, and the recent frame cache and / or key frame anchor cache are read according to the actions, or the historical cache is not read and input into the single-step diffusion generation module. The super-resolved latent variables for the current time step are generated by the single-step diffusion generation module, which combines the current frame, causal hidden conditions, super-resolved latent variables from the previous time step, and the output of the static cache inference module. Transform the super-resolution latent variables of the current time step into the image space to obtain the super-resolution video frame of the current time step; The static cache scheduling table is generated during the training phase. It stores the actions of each layer of the single-step diffusion generation module. During the training process, the single-step diffusion generation module is trained by performing multi-dimensional discrimination on the super-resolution latent variables generated by the single-step diffusion generation module and constructing adversarial training loss.

[0073] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0074] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A streaming video super-resolution system, characterized in that, include: The module includes a low-resolution causal condition construction module, a super-resolution latent variable retrieval module, a static cache inference module, a single-step diffusion generation module, and a decoder. The low-resolution causal condition construction module is used to extract the causal hidden conditions of the current frame and historical frames from the input raw resolution video stream; where the current frame corresponds to the current time step. The super-resolved latent variable recovery module is used to save the super-resolved latent variables generated by the single-step diffusion generation module in the previous time step, and input the super-resolved latent variables of the previous time step as recovery conditions into the single-step diffusion generation module in the current time step. The static cache inference module is used to determine the actions of each layer of the single-step diffusion generation module according to the static cache scheduling table, read the recent frame cache and / or key frame anchor point cache according to the actions, or not read the historical cache, and input them into the single-step diffusion generation module. The single-step diffusion generation module is used to generate the super-resolved latent variables of the current time step by combining the current frame, the causal hiding conditions, the super-resolved latent variables of the previous time step, and the output of the static cache inference module. The decoder is used to transform the super-resolution latent variables of the current time step to the image space to obtain the super-resolution video frame of the current time step. The static cache scheduling table is generated during the training phase. It stores the actions of each layer of the single-step diffusion generation module. During the training phase, the single-step diffusion generation module also uses the super-resolved latent variables of the previous time step as the retrieval condition. The single-step diffusion generation module is trained by performing multi-dimensional discrimination on the super-resolved latent variables generated by the single-step diffusion generation module and constructing adversarial training loss.

2. The streaming video super-resolution system according to claim 1, characterized in that, The step of extracting the causal hiding conditions for the current frame and historical frames from the input raw resolution video stream includes: The current frame is designated as frame k. Causal hidden conditions from frame 1 to frame k are extracted from the original resolution video stream and used as observational evidence at the original resolution as input to the single-step diffusion generation module.

3. The streaming video super-resolution system according to claim 1, characterized in that, When the current time step is the first time step, the super-resolution latent variables of the previous time step use a zero vector, a learnable initial vector, or a vector generated by the low-resolution conditions of the first frame.

4. The streaming video super-resolution system according to claim 1, characterized in that, During the training phase, the single-step diffusion generation module also uses the super-latent variable from the previous time step as a retrieval condition, including: During the training phase, an online self-generated condition is adopted, which is to form a super-resolved latent variable sequence by generating super-resolved latent variables from the single-step diffusion generation module, and then shift the super-resolved latent variable sequence one frame and stop gradient propagation, which serves as the retrieval condition for the next frame of training.

5. A streaming video super-resolution system according to claim 1, characterized in that, The step of determining the actions of each layer of the single-step diffusion generation module based on the static cache scheduling table, and reading the recent frame cache and / or keyframe anchor point cache according to the actions, or not reading the historical cache, includes: For layers whose action is not to read the history cache, the corresponding layer's historical key-value information is not materialized; for layers whose action is to read the most recent r-frame window, the r-frame window is read from the recent frame cache library; for layers whose action is to read the keyframe anchor point cache, the keyframe anchor point cache is read from the keyframe anchor point cache library; for layers that simultaneously read the most recent r-frame window and the keyframe anchor point cache, the r-frame window is read from the recent frame cache library, and the keyframe anchor point cache is read from the keyframe anchor point cache library; where r takes one or more preset window lengths.

6. A streaming video super-resolution system according to claim 1, characterized in that, The system also includes: a layer-by-layer cache routing module, used to generate a static cache scheduling table during the training phase; The historical key-value cache is divided into a recent frame cache library and a key frame anchor cache library; the recent frame cache library stores the key-value tensors of the most recent several frames, and the key frame anchor cache stores the key-value tensors of key frames. For each layer in the single-step diffusion generation module, a set of candidate cache actions is defined, including: not reading the historical cache, reading the most recent r-frame window, reading the keyframe anchor cache, and reading both the most recent r-frame window and the keyframe anchor cache; where r takes one or more preset window lengths, and different layers learn to select different actions; During the training phase, the layer-by-layer caching routing module outputs the selection probability of each candidate action based on the layer identifier or layer embedding; it introduces cache budget constraints, route sharpening constraints, and temperature annealing; the cache budget constraints are used to control the average number of cached frames read or the cache cost of each layer; the route sharpening constraints and temperature annealing are used to encourage each layer to eventually form a clear dominant cache action; after training is completed, the action with the highest probability in each layer is selected and exported as a static cache scheduling table.

7. A streaming video super-resolution system according to claim 6, characterized in that, Also includes: During the training phase, the layer-by-layer caching routing module adds a route-aware attention bias on top of the joint causal attention: based on whether a candidate action allows a certain historical key-value term to be accessed by the current query, the soft visibility probability of the corresponding historical key-value term is calculated, and the logarithm of the soft visibility probability is added to the attention score; thus, layer-by-layer soft route learning is completed in a single attention calculation.

8. A streaming video super-resolution system according to claim 1, characterized in that, The system also includes: a multi-range query discrimination training module, used to perform multi-dimensional discrimination on the super-resolution latent variables generated by the single-step diffusion generation module during the training phase, and to construct adversarial training loss; The multi-range query discrimination training module is configured with a discrimination output header. During the training phase... For a given sequence of real high-resolution latent variables or a sequence of super-resolution latent variables generated by a single-step diffusion generation module, extract video terms with temporal frame index and spatial location index; output multi-dimensional discrimination results through the discrimination output head.

9. A streaming video super-resolution system according to claim 8, characterized in that, The multi-range query discrimination training module includes three types of learnable queries: global query, spatial window query, and temporal tubular query. Global queries aggregate the entire video clip to constrain overall realism; spatial window queries focus on local windows within a single frame to constrain local textures, text, edges, and object details; temporal tubular queries focus on terms within the same spatial window over a continuous causal temporal range to constrain cross-frame stability of local textures. The three types of queries output global discrimination scores, spatial local discrimination scores, and temporal tubular discrimination scores, respectively. The three scores are averaged or weighted and fused into the final discrimination result, and an adversarial training loss is constructed.

10. A streaming video super-resolution method, characterized in that, The system implementation based on any one of claims 1 to 9 includes: Extract the causal latent conditions of the current frame and historical frames from the input raw resolution video stream; where the current frame corresponds to the current time step; Save the super-resolved latent variables generated by the single-step diffusion generation module in the previous time step, and input the super-resolved latent variables from the previous time step into the single-step diffusion generation module at the current time step. The actions of each layer of the single-step diffusion generation module are determined according to the static cache scheduling table, and the recent frame cache and / or key frame anchor cache are read according to the actions, or the historical cache is not read and input into the single-step diffusion generation module. The super-resolved latent variables for the current time step are generated by the single-step diffusion generation module, which combines the current frame, causal hidden conditions, super-resolved latent variables from the previous time step, and the output of the static cache inference module. Transform the super-resolution latent variables of the current time step into the image space to obtain the super-resolution video frame of the current time step; The static cache scheduling table is generated during the training phase. It stores the actions of each layer of the single-step diffusion generation module. During the training process, the single-step diffusion generation module is trained by performing multi-dimensional discrimination on the super-resolution latent variables generated by the single-step diffusion generation module and constructing adversarial training loss.

Images