Video super-resolution model training method and system based on double-layer time sequence constraint

Abstract

The invention provides a video super-resolution model training method and system based on double-layer time sequence constraint. The method comprises the following steps: extracting a low-definition video clip and a high-definition truth value clip from a training data set according to a time sequence; inputting the low-definition video clip into the generative network, and outputting a video restoration clip; performing inter-frame differential calibration on the high-Muslim true-value fragment and the video restoration fragment to obtain a calibration signal; performing cross-segment statistical alignment on the high-Muslim true-value segment and the video restoration segment to obtain an alignment signal; performing weighted summation on the basic loss, the calibration signal and the alignment signal of the generative network according to the dynamically adjusted weight to obtain a loss feedback signal; and inputting the loss feedback signal into a back propagation link of the generative network, driving parameters of the generative network to perform back update, and enabling the generative network to converge to a stable solution so as to complete training of the generative network. According to the method, the stability of the generative video super-resolution model in a long time span can be improved on the premise of not changing an inference period structure and a computational graph.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and multimedia processing, in particular, to a video super-resolution model training method and system based on double-layer time sequence constraints. BACKGROUND

[0002] In the task of video restoration facing real scenes, the model should not only improve the spatial quality of a single frame, but also maintain stable output in the time dimension. Existing technologies usually revolve around two types of ideas: one type is the local smoothing constraint taking adjacent frames as the object, which suppresses flicker and random jitter through the difference, optical flow or neighborhood alignment of adjacent frames; the other type is focused on the maintenance of a stable state for a longer time span, such as introducing external memory, identity maintenance modules or cross-segment feature alignment, etc. The former can effectively improve the transition between adjacent frames, but it is difficult to fundamentally avoid semantic drift and style cumulative offset on long videos; the latter often relies on additional inference overhead, complex access structure or needs to process the whole sequence globally, which has high engineering cost and deployment difficulty. With the widespread use of streaming segment inference, long sequences are divided into several segments to be generated in sequence. How to balance the local stability within the segment and the global consistency across segments without increasing the inference period burden has become a key problem restricting the quality and feasibility of generative video restoration.

[0003] Through literature retrieval of existing technologies, it is found that the Chinese patent with publication number CN120343361A proposes a video generation system based on a video large model, which effectively eliminates the inter-frame jump and logical contradiction in video generation through the optical flow constraint and motion interpolation technology of the time sequence consistency unit. This combination of technologies makes the generated video have considerable coherence with real video, meeting the high-standard demand of film and television production. However, the above-mentioned video generation system needs to dynamically calculate the inter-frame difference and adjust the generation result during inference, increasing the computational complexity.

[0004] Therefore, there is an urgent need for a network model that can effectively improve the transition between adjacent frames while avoiding semantic drift and style cumulative offset on long videos. SUMMARY

[0005] In view of the defects in the prior art, the purpose of the present application is to provide a video super-resolution model training method and system based on double-layer time sequence constraints.

[0006] According to a first aspect of the present application, a video super-resolution model training method based on double-layer time sequence constraints is provided, comprising: extracting low-definition video segments and their corresponding high-definition ground truth segments from the training data set in time sequence and segment by segment; The low-resolution video clip is input into the generator network to be trained, and the generator network is used to restore the low-resolution video clip to output the corresponding restored video clip. Inter-frame differential calibration is performed on high-definition true value segments and video restoration segments to obtain calibration signals at the local level; Cross-segment statistical alignment is performed on high-definition ground truth segments and video restoration segments to obtain alignment signals at the global level; The loss feedback signal is obtained by weighting and summing the base loss, calibration signal, and alignment signal of the generator network according to the dynamically adjusted weights. The loss feedback signal is input to the backpropagation link of the generator network to drive the parameters of the generator network to be updated in reverse, so that the generator network converges to a stable solution, thereby completing the training of the generator network.

[0007] Optionally, the step of performing inter-frame difference calibration on the high-definition ground truth segment and the video restoration segment to obtain a local-level calibration signal includes: Two adjacent frames are randomly selected from the low-definition video clip as low-definition frame pairs, two frames from the high-definition ground truth clip corresponding to the low-definition frame pairs are taken as ground truth frame pairs, and two frames from the video restoration clip corresponding to the low-definition frame pairs are taken as generated frame pairs. Alignment compensation is applied to both the true frame pair and the generated frame pair using an alignment compensation method. In each iteration of the training of the generative network, the inter-frame change of the ground truth frame pair after alignment compensation is obtained as the ground truth inter-frame difference, and the inter-frame change of the generated frame pair after alignment compensation is obtained as the generated inter-frame difference. A robust metric is used to evaluate the consistency between the ground truth inter-frame difference and the generated inter-frame difference, thereby quantifying the inter-frame difference between the ground truth inter-frame difference and the generated inter-frame difference. The inter-frame differences are weighted using a confidence-based mask, so that the weight of the inter-frame differences corresponding to unreliable regions in low-definition video clips is lower than that corresponding to reliable regions. The weighted inter-frame differences and their corresponding weights are then used as calibration signals at the local level.

[0008] Optionally, the alignment compensation method includes at least one of the following: a block matching method and a displacement estimation method.

[0009] Optionally, the measurement method with anomaly resistance includes at least one of the following: robust statistics and outlier adaptive weighting.

[0010] Optionally, the step of performing cross-segment statistical alignment on the high-definition ground truth segment and the video restoration segment to obtain a global-level alignment signal includes: By sequentially splicing several high-definition ground truth segments in chronological order, a ground truth sequence is obtained; by sequentially splicing several video restoration segments in chronological order, a generated sequence is obtained. In each iteration of the training of the generative network, the high-level representation information of the truth sequence is obtained as the truth representation, and the high-level representation information of the generated sequence is obtained as the generated representation. The high-level representation information is a semantic-level representation of the sequence. A distribution proximity metric is used to obtain the distribution proximity metric result between the true value representation and the generated representation, which serves as an alignment signal at the global level. The distribution proximity metric result is used to quantify the degree of alignment between the generated sequence and the true value sequence in the same time dimension.

[0011] Optionally, the distribution proximity metric includes at least one of the following: a divergence metric and a distribution distance metric, wherein the divergence metric measures the difference between the true representation and the generated representation through divergence, and the distribution distance metric measures the difference between the true representation and the generated representation through distribution distance.

[0012] Optionally, the step of weighting and summing the base loss, calibration signal, and alignment signal of the generator network according to dynamically adjusted weights to obtain the loss feedback signal includes: In the early stages of training the generative network, the weights of the calibration signal are set to the maximum value to suppress flicker and artifacts between adjacent frames. During the middle and later stages of training the generative network, the weight of the alignment signal is gradually increased and the weight of the calibration signal is decreased, so that the generative network can form a stable and consistent expression over a long period of time. Throughout the entire training process of the generative network, the weights of the basic loss remain unchanged.

[0013] According to a second aspect of this application, a video super-resolution model training system based on two-layer temporal constraints is provided, comprising: The data extraction module is used to extract low-resolution video segments and their corresponding high-resolution ground truth segments from the training dataset in chronological order. A sub-network module is provided, which provides a generation network that performs restoration processing on the low-definition video clip and outputs the corresponding restored video clip. The inter-frame difference calibration module is used to perform inter-frame difference calibration on high-definition true value segments and video restoration segments to obtain calibration signals at the local level. The characterization and statistical alignment module is used to perform cross-segment statistical alignment on high-definition ground truth segments and video restoration segments to obtain alignment signals at the global level. The training scheduling module is used to perform a weighted summation of the basic loss, calibration signal, and alignment signal of the generator network according to dynamically adjusted weights to obtain the loss feedback signal; The optimization module is used to input the loss feedback signal into the backpropagation link of the generator network, drive the parameters of the generator network to be updated in reverse, and make the generator network converge to a stable solution to complete the training of the generator network.

[0014] According to a third aspect of this application, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described video super-resolution model training method based on two-layer temporal constraints.

[0015] According to a fourth aspect of this application, an electronic device is provided, comprising: At least one memory for storing program instructions; At least one processor is configured to call program instructions stored in the memory and execute the steps of the video super-resolution model training method based on two-layer temporal constraints according to the obtained program instructions.

[0016] This application provides a video super-resolution model training method based on two-layer temporal constraints. By simultaneously applying inter-frame difference calibrator constraints and cross-segment statistical alignment constraints during the training phase, the inter-frame difference calibrator constraints reduce inter-frame flicker and fine-grained noise at the local level, while the cross-segment statistical alignment constraints suppress cross-segment semantic shift and identity style drift at the global level. This method maintains natural compatibility with online inference paradigms such as segmented bootstrapping and single-step generation. This application presents a two-layer temporal constraint method that simultaneously considers local temporal stability and long-range semantic steady-state, improving the stability of generative video super-resolution models over long time spans without altering the inference-era structure and computation graph.

[0017] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0018] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart of a video super-resolution model training method in one embodiment of this application; Figure 2 This is a schematic diagram of a dual-temporal constraint method combining frame-level differential and cross-segment statistical consistency in one embodiment of this application; Figure 3 This is a schematic diagram of a video super-resolution model training system in one embodiment of this application. Detailed Implementation

[0019] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0020] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations.

[0021] In video restoration tasks targeting real-world scenarios, models should not only improve the spatial quality of individual frames but also maintain stable output over time. Existing technologies cannot effectively improve the transition between neighboring frames while avoiding semantic drift and style accumulation shifts in long videos. To address these issues, this application provides a video super-resolution model training method based on two-layer temporal constraints to resolve these problems.

[0022] Reference Figure 1 As shown in the figure, this application provides a video super-resolution model training method based on two-layer temporal constraints, the method including: S1. Extract low-resolution video segments and their corresponding high-resolution ground truth segments from the training dataset in chronological order. S2. Input the low-definition video clips into the generator network to be trained, use the generator network to restore the low-definition video clips, and output the corresponding restored video clips. S3. Perform inter-frame difference calibration on the high-definition true value segment and the video restoration segment to obtain the calibration signal at the local level; S4. Perform cross-segment statistical alignment on the high-definition true value segment and the video restoration segment to obtain the alignment signal at the global level; S5. The basic loss, calibration signal, and alignment signal of the generator network are weighted and summed according to the dynamically adjusted weights to obtain the loss feedback signal. S6. Input the loss feedback signal into the backpropagation link of the generator network to drive the parameters of the generator network to be updated in reverse, so that the generator network converges to a stable solution, thus completing the training of the generator network.

[0023] Of course, the above embodiment is one implementation of this application. Steps S3-S4 are not required to be executed in strict order. In other embodiments, steps S3-S4 can be performed in any order or in parallel.

[0024] For example, the video super-resolution model training method trains the generator network based on a video super-resolution model training system. The training system obtains low-resolution video clips and their corresponding high-resolution ground truth clips from the existing training dataset in chronological order. The low-resolution video clips and high-resolution ground truth clips are pre-associated paired data in the training dataset, and their time spans and frame rates are consistent. Low-resolution video clips can be selected from video clips with a resolution of 480p or lower, while high-resolution ground truth clips can be selected from video clips with a resolution of 720p or higher. The generator network (i.e., the video super-resolution model) receives the low-resolution video clips and outputs video reconstruction clips. For the same low-resolution video clip, its corresponding video reconstruction clip and high-resolution ground truth clip have a one-to-one temporal correspondence. The generator network converges to a stable solution simultaneously in both the short-time inter-frame dimension and the long-time sequence dimension, indicating that the generator network training is complete. The short-time inter-frame dimension refers to the temporal correlation and feature consistency dimension between adjacent frames within a short time span, while the long-time sequence dimension refers to the global feature distribution and overall temporal logic dimension of the complete sequence over a long time span.

[0025] The embodiments described above in this application simultaneously apply inter-frame difference calibrator constraints and cross-segment statistical alignment constraints during the training phase. By applying inter-frame difference calibrator constraints, adjacent frame flicker and fine-grained noise are reduced at the local level. By applying cross-segment statistical alignment constraints, cross-segment semantic shift and identity style drift are suppressed at the global level, while maintaining natural compatibility with online inference paradigms such as segmented bootstrapping and single-step generation. This application is a two-layer temporal constraint method that simultaneously considers local temporal stability and long-range semantic steady-state, which can improve the stability of generative video super-resolution models over long time spans without changing the inference-era structure and computation graph.

[0026] The technical solution proposed in this application runs during the training phase, but can be "inherited at zero cost" during the inference phase, meaning that no additional computation or structure is added during the inference phase. (See reference...) Figure 2 As shown, the overall concept is to introduce two complementary types of temporal consistency constraints while keeping the original generative network unchanged: the first is an inter-frame difference calibration constraint oriented towards adjacent frames, used to suppress high-frequency flicker, minor artifacts, and random jitter in the short temporal domain; the second is a cross-segment statistical alignment constraint oriented towards long time spans, used to limit semantic drift and style fragmentation caused by segmented inference. These two types of constraints are optimized together with the base loss in a weighted manner during training. After training, no additional modules need to be retained during the inference period, enabling stable visuals and consistent identity / style expression in complex, long-duration real-world videos with scene transitions.

[0027] In some specific embodiments of this application, in order to suppress errors such as flickering and random jitter in video clips, inter-frame difference calibration is performed on the high-definition true value clips and video restoration clips to obtain calibration signals at the local level, which may further include: S31. Randomly select two adjacent frames from the low-definition video clip as low-definition frame pairs, take the two frames in the high-definition true value clip that correspond to the low-definition frame pairs as true value frame pairs, and take the two frames in the video restoration clip that correspond to the low-definition frame pairs as generated frame pairs. S32. Alignment compensation method is used to perform alignment compensation on true frame pairs and generated frame pairs respectively; S33. In each iteration of the training of the generative network, the inter-frame change of the ground truth frame pair after alignment compensation is obtained as the ground truth frame difference, and the inter-frame change of the generated frame pair after alignment compensation is obtained as the generated frame difference. S34. Using a measurement method with anomaly resistance, the consistency between the true inter-frame difference and the generated inter-frame difference is evaluated, thereby quantifying the inter-frame difference between the true inter-frame difference and the generated inter-frame difference. S35. Use a confidence-based mask to weight the inter-frame differences so that the weight of the inter-frame differences corresponding to unreliable regions in the low-definition video clip is lower than the weight of the inter-frame differences corresponding to reliable regions, and use the weighted inter-frame differences and their corresponding weights as calibration signals at the local level.

[0028] For example, for video reconstruction clips and high-definition ground truth clips within the same time period, the training system first evaluates the consistency of inter-frame changes within a short time range to form a local calibration signal. The short time range can be set to 10-20 frames of the current low-definition video clip. Each frame image in the high-definition ground truth clip is a high-definition ground truth image, and the ground truth frame pair is composed of two adjacent high-definition ground truth images. Each frame image in the video reconstruction clip is a video reconstruction image, and the generated frame pair is composed of two adjacent video reconstruction images.

[0029] In the embodiments described above, inter-frame difference calibration refers to applying an inter-frame difference calibration sub-constraint, which focuses on "how changes should occur between adjacent frames". In each iteration, the system selects adjacent frame pairs from the current training segment, calculates the changes in the video restoration image frame pairs and the high-definition ground truth image frame pairs respectively, and compares whether the two changes are consistent. The changes in the frame pairs can be the differences between adjacent frames at the pixel level, such as the difference in corresponding pixel values ​​between adjacent frames. For occluded regions (i.e., unreliable regions), the system introduces a confidence-based mask to weaken the impact of unreliable regions on the overall loss. Specifically, for occluded regions in the video segment, a confidence-based mask is introduced, assigning lower weights to the inter-frame differences corresponding to unreliable regions to weaken their impact on the overall training loss, and assigning higher weights to the inter-frame differences corresponding to reliable regions (e.g., a weight of 1 for reliable regions and a weight close to 0 for unreliable regions), thus achieving differentiated processing for different regions. This inter-frame difference calibration sub-constraint only relies on short-term neighborhood information, does not require access to complete long sequences, has stable computational overhead, and is easy to implement in parallel.

[0030] In some specific embodiments of this application, in order to eliminate slight displacement between adjacent frames, alignment compensation can be used to eliminate the above-mentioned displacement interference. The alignment compensation method includes at least one of the following: block matching method and displacement estimation method.

[0031] The embodiments described above allow for lightweight alignment compensation within a small neighborhood. Alignment compensation can employ pixel-level block matching or a very low-cost displacement estimation method, thereby making the calibration results closer to actual physical changes.

[0032] In some specific embodiments of this application, in order to quantify the difference in the amount of change between true frame pairs and generated frame pairs, the measurement method with anomaly resistance includes at least one of the following: robust statistical method and outlier adaptive weighting method.

[0033] In the embodiments described above, to avoid training dominated by occasional errors caused by jitter, occlusion, or sudden changes in brightness, the system adopts a measurement method with anomaly resistance to evaluate the consistency of inter-frame changes, so as to avoid interference from outliers.

[0034] In some specific embodiments of this application, in order to achieve a stable and consistent representation of the model over a long period of time, cross-segment statistical alignment is performed on the high-definition ground truth segments and video restoration segments to obtain a global-level alignment signal, which may further include: S41. Sequentially splice several high-definition true value segments in chronological order to obtain a true value sequence; sequentially splice several video restoration segments in chronological order to obtain a generated sequence; S42. In each iteration of the training of the generative network, the high-level representation information of the truth sequence is obtained as the truth representation, and the high-level representation information of the generated sequence is obtained as the generation representation. The high-level representation information is the semantic-level representation of the sequence. S43. Using a distribution proximity metric, obtain the distribution proximity metric result between the true representation and the generated representation as a global alignment signal. The distribution proximity metric result is used to quantify the degree of alignment between the generated sequence and the true sequence in the same time dimension.

[0035] For example, the training system constructs a sequence composed of multiple segments spliced ​​together in chronological order over a longer time span, extracts high-level representations from the generated sequence spliced ​​from video reconstruction segments and the ground truth sequence spliced ​​from high-definition ground truth segments respectively, and compares the statistical similarity between the two to obtain the alignment signal at the global level.

[0036] In the embodiments described above, cross-segment statistical alignment refers to applying cross-segment statistical alignment sub-constraints. Cross-segment statistical alignment focuses on whether "the semantics and style remain in a correct stable state over a longer time span." The system sequentially splices multiple adjacent segments on the timeline to form a sequence window covering a longer time span. Subsequently, the system extracts high-level representations from the generated sequence and the ground truth sequence, respectively. Low-level representations are closer to the details of the original data, such as low-level visual features like pixel values, edges, and textures in video frames. These low-level representations usually contain a lot of redundant information and have weak semantic correlation. High-level representations are features obtained by abstracting and aggregating low-level representations layer by layer, focusing more on the semantic meaning, global patterns, or abstract attributes of the data. The high-level representation network can be a frozen visual encoder or a lightweight, adjustable module can be added to the end of the visual encoder to adapt to differences in the data domain; the above network is only called during the training phase and does not participate in the computation during the inference phase. After obtaining the representations of the two sequences in the high-level space (i.e., high-level representations), the system compares their closeness in a statistical sense (i.e., the distribution closeness measurement result). To improve adaptability to cross-fragment boundaries, sequence windows are constructed in a sliding manner, allowing overlap between adjacent windows. This ensures that the same boundary frame is repeatedly constrained in different contexts, thus more comprehensively eliminating boundary effects. If a significant scene change is detected, the system truncates or resets the window to avoid forcibly aligning unrelated scenes.

[0037] In some specific embodiments of this application, the distribution proximity measurement method includes at least one of the following: divergence measurement and distribution distance measurement. The divergence measurement measures the difference between the true representation and the generated representation by divergence, and the distribution distance measurement measures the difference between the true representation and the generated representation by distribution distance.

[0038] In the embodiments described above, the distribution proximity metric results include at least one of the following: divergence and distribution distance. Divergence and distribution distance can quantify the degree of statistical alignment between the generated sequence and the true sequence across segments within the sequence window.

[0039] In some specific embodiments of this application, in order to achieve a time-series consistency mechanism that inherits at zero cost during the inference phase, and to enable the model to simultaneously receive two complementary signals during the learning process, the loss feedback signal is obtained by weighted summation of the basic loss, calibration signal, and alignment signal of the generator network according to dynamically adjusted weights. This can further include: S51. In the early stages of training the generative network, the weights of the calibration signal are set to the maximum value to suppress flickering and artifacts between adjacent frames. S52. In the middle and later stages of training the generative network, gradually increase the weight of the alignment signal and decrease the weight of the calibration signal so that the generative network can form a stable and consistent expression over a long period of time. Specifically, the weights of the base loss remain unchanged throughout the entire training process of the generative network.

[0040] In the embodiments described above, the basic loss, calibration signal, and alignment signal are jointly optimized and their weights are scheduled. The joint optimization phase integrates these three types of signals: the basic reconstruction and perceptual loss (i.e., the basic loss), the local calibration signal from inter-frame differential calibration, and the global alignment signal from cross-segment statistical alignment. In the early stages of training, the system increases the weight of the local calibration signal to quickly suppress flickering and minor artifacts between neighboring frames. As training progresses, the system gradually increases the weight of the global alignment signal, enabling the model to form a stable and consistent representation over a long period. Based on the preset total number of iterations (Epochs), the network training process can be divided into early training, mid-training, and late training phases. The early training phase completes 20%-40% of the total number of iterations (Epochs), the mid-training phase completes 40%-70%, and the late training phase completes 70%-100%. The weight of the basic loss remains unchanged. In the early training phase, the weight of the local calibration signal is increased to its maximum value. As training progresses, in the mid- and late-training phases of the generative network, the weight of the alignment signal is gradually increased while the weight of the calibration signal is decreased, until the weight of the alignment signal is greater than the weight of the calibration signal. To avoid excessive constraints on local textures caused by global alignment, the system introduces temperature-based optimization or subspace filtering in the feature space. This allows alignment to focus more on long-term stable elements such as identity, style, and scene structure, while minimizing negative impacts on detailed textures. Weight scheduling can be automatically adjusted based on the training phase, convergence speed, validation set performance, or stability metrics.

[0041] This application provides a video super-resolution model training method based on two-layer temporal constraints, which can be applied to video restoration tasks, such as generative networks. A video restoration method includes: Step 1: Obtain the video to be restored and generate the network; Step 2: Train the generator network using the video super-resolution model training method of any of the above embodiments to determine the trained generator network; For details, please refer to steps S1 to S6 above, which will not be repeated here.

[0042] Step 3: Input the video to be restored into the trained generative network to determine the restored video.

[0043] Specifically, "zero-cost inheritance" can be achieved in the video restoration process (i.e., the inference stage) in step three, meaning that no additional computation or structure is added during the inference stage. After the generator network is trained, there is no need to retain any additional modules in the inference stage, enabling it to obtain a stable visual experience and consistent identity / style expression in complex, long-duration real videos with scene transitions.

[0044] This application provides a video super-resolution model training method based on two-layer temporal constraints, which can be applied to video super-resolution tasks. A video super-resolution processing method includes: Step 1: Obtain the video to be super-resolution processed and the generation network; Step 2: Train the generator network using the video super-resolution model training method of any of the above embodiments to determine the trained generator network; For details, please refer to steps S1 to S6 above, which will not be repeated here.

[0045] Step 3: Input the video to be super-resolution processed into the trained generator network to determine the video that has undergone super-resolution processing.

[0046] The following examples and comparative examples will be used to further illustrate this application in order to better understand the above-mentioned technical solutions. It should be understood that the following are only some examples and are not intended to limit this application.

[0047] To verify the performance of the proposed method in various video super-resolution tasks, a unified testing process was constructed, and evaluation experiments were conducted on publicly available benchmark datasets such as UDM10, SPMCS, MVSR4x, and VideoLQ. Comparison methods included traditional multi-frame enhancement strategies (such as RealBasicVSR and RealViFormer) and representative generative model-based methods (such as Upscale-A-Video, MGLD-VSR, STAR, SeedVR, and SeedVR2), all of which are currently mainstream technologies. Evaluation metrics covered the following three categories: (1) PSNR and SSIM, used to measure reconstruction accuracy; (2) Perceptual quality metrics: LPIPS, DISTS, MUSIQ, CLIP-IQA, and DOVER, used to measure perceptual quality, especially suitable for realistic degraded input scenarios; (3) Temporal consistency related metrics: Indicators reflecting temporal consistency are used to evaluate the model's ability to model cross-frame consistency. The direction of each indicator is uniformly represented by an arrow, where ↑ indicates higher is better, and ↓ indicates lower is better.

[0048]

[0049] As shown in Table 1, the proposed method demonstrates excellent overall performance across all metrics. It significantly outperforms most comparative methods in perceptual quality metrics, producing results that more closely resemble subjective perception. It also exhibits a significant advantage in temporal consistency metrics, indicating strong cross-frame continuous modeling capabilities. Furthermore, it maintains a good level of performance in traditional reconstruction accuracy metrics, achieving a balance between visual quality, coherence, and accuracy. Experimental results demonstrate that the proposed method is robust across diverse video types and evaluation dimensions, possessing good generalization ability and practical application value.

[0050] Based on the same inventive concept, another embodiment of this application provides a video super-resolution model training system based on two-layer temporal constraints, referring to... Figure 3 As shown, the video super-resolution model training system 100 includes: Data extraction module 110 is used to extract low-definition video segments and their corresponding high-definition ground truth segments from the training dataset in chronological order. The generation sub-network module 120 provides a generation network that performs restoration processing on low-definition video clips and outputs corresponding restored video clips. The inter-frame difference calibration module 130 is used to perform inter-frame difference calibration on high-definition true value segments and video restoration segments to obtain calibration signals at the local level. The characterization and statistical alignment module 140 is used to perform cross-segment statistical alignment on high-definition ground truth segments and video restoration segments to obtain alignment signals at the global level. The training scheduling module 150 is used to perform a weighted summation of the basic loss, calibration signal, and alignment signal of the generator network according to dynamically adjusted weights to obtain the loss feedback signal. The optimization module 160 is used to input the loss feedback signal into the backpropagation link of the generator network, drive the parameters of the generator network to be updated in reverse, and make the generator network converge to a stable solution to complete the training of the generator network.

[0051] For example, the training system also includes a data access and caching module, which is responsible for organizing continuous frames and long program column windows; an inter-frame difference calibration module completes the evaluation of the consistency of changes between adjacent frames within a segment; a representation and statistical alignment module performs high-level representation extraction and distribution proximity calculation over a long period of time; and a training scheduling and optimization module is responsible for weight management, learning rate planning, and overall reverse update.

[0052] The embodiments described above introduce complementary two-layer temporal constraints during the training phase, which significantly improves the temporal stability and identity / style consistency of long videos without changing the complexity of the inference phase. Inter-frame differential calibration effectively suppresses random flicker and minor artifacts in the short-term neighborhood, making edge, texture, and brightness transitions more natural; cross-segment statistical alignment limits the drift trend of semantic expression over a longer time range, reducing boundary breaks and style drift caused by segmented inference. The two work together to ensure smooth continuity within segments and maintain global steady state across segments. Since the representation and statistical alignment modules are only used during training, they do not need to be loaded during the inference deployment phase. The inference phase maintains the original structure and latency, making it easy to deploy directly in streaming systems or single-step generation paradigms. The system's runtime latency and memory usage remain consistent with the original network. Furthermore, the implementation of this application relies on a modular interface, facilitating integration with different generation networks and quality evaluation systems, demonstrating good engineering feasibility and scalability. This application provides a temporal consistency mechanism that is introduced only during the training phase but can be "inherited at zero cost" during the inference phase, enabling the model to receive two complementary signals simultaneously during the learning process, thereby improving the quality of the video without changing the inference complexity.

[0053] It should be noted that the modules in the video super-resolution model training system based on two-layer temporal constraints provided in the above embodiments of this application correspond to the steps of the video super-resolution model training method based on two-layer temporal constraints in any of the above embodiments. Those skilled in the art can refer to the step features of the video super-resolution model training method based on two-layer temporal constraints to implement the corresponding modules in the video super-resolution model training system based on two-layer temporal constraints, which will not be elaborated here.

[0054] In another embodiment of this application, a non-transitory computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described video super-resolution model training method based on two-layer temporal constraints.

[0055] In another embodiment of this application, an electronic device is also provided, including a memory and a processor; the memory is used to store program instructions; the processor is used to call the program instructions stored in the memory and execute the steps of the above-described video super-resolution model training method based on two-layer temporal constraints according to the obtained program instructions.

[0056] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0057] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0058] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0059] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0060] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0061] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0062] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0063] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0064] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0065] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A video super-resolution model training method based on two-layer temporal constraints, characterized in that, include: Extract low-resolution video segments and their corresponding high-resolution ground truth segments from the training dataset in chronological order. The low-resolution video clip is input into the generator network to be trained, and the generator network is used to restore the low-resolution video clip to output the corresponding restored video clip. Inter-frame differential calibration is performed on high-definition true value segments and video restoration segments to obtain calibration signals at the local level; Cross-segment statistical alignment is performed on high-definition ground truth segments and video restoration segments to obtain alignment signals at the global level; The loss feedback signal is obtained by weighting and summing the base loss, calibration signal, and alignment signal of the generator network according to the dynamically adjusted weights. The loss feedback signal is input to the backpropagation link of the generator network to drive the parameters of the generator network to be updated in reverse, so that the generator network converges to a stable solution, thereby completing the training of the generator network.

2. The video super-resolution model training method based on two-layer temporal constraints according to claim 1, characterized in that, The step of performing inter-frame difference calibration on the high-definition true value segment and the video restoration segment to obtain the calibration signal at the local level includes: Two adjacent frames are randomly selected from the low-definition video clip as low-definition frame pairs, two frames from the high-definition ground truth clip corresponding to the low-definition frame pairs are taken as ground truth frame pairs, and two frames from the video restoration clip corresponding to the low-definition frame pairs are taken as generated frame pairs. Alignment compensation is applied to both the true frame pair and the generated frame pair using an alignment compensation method. In each iteration of the training of the generative network, the inter-frame change of the ground truth frame pair after alignment compensation is obtained as the ground truth inter-frame difference, and the inter-frame change of the generated frame pair after alignment compensation is obtained as the generated inter-frame difference. A robust metric is used to evaluate the consistency between the ground truth inter-frame difference and the generated inter-frame difference, thereby quantifying the inter-frame difference between the ground truth inter-frame difference and the generated inter-frame difference. The inter-frame differences are weighted using a confidence-based mask, so that the weight of the inter-frame differences corresponding to unreliable regions in low-definition video clips is lower than that corresponding to reliable regions. The weighted inter-frame differences and their corresponding weights are then used as calibration signals at the local level.

3. The video super-resolution model training method based on two-layer temporal constraints according to claim 2, characterized in that, The alignment compensation method includes at least one of the following: block matching method and displacement estimation method.

4. The video super-resolution model training method based on two-layer temporal constraints according to claim 2, characterized in that, The measurement method with anomaly resistance includes at least one of the following: robust statistics and outlier adaptive weighting.

5. The video super-resolution model training method based on two-layer temporal constraints according to claim 1, characterized in that, The step of performing cross-segment statistical alignment on high-definition ground truth segments and video restoration segments to obtain global-level alignment signals includes: By sequentially splicing several high-definition ground truth segments in chronological order, a ground truth sequence is obtained; by sequentially splicing several video restoration segments in chronological order, a generated sequence is obtained. In each iteration of the training of the generative network, the high-level representation information of the truth sequence is obtained as the truth representation, and the high-level representation information of the generated sequence is obtained as the generated representation. The high-level representation information is a semantic-level representation of the sequence. A distribution proximity metric is used to obtain the distribution proximity metric result between the true value representation and the generated representation, which serves as an alignment signal at the global level. The distribution proximity metric result is used to quantify the degree of alignment between the generated sequence and the true value sequence in the same time dimension.

6. The video super-resolution model training method based on two-layer temporal constraints according to claim 5, characterized in that, The distribution proximity metric includes at least one of the following: divergence metric and distribution distance metric, wherein the divergence metric measures the difference between the true representation and the generated representation through divergence, and the distribution distance metric measures the difference between the true representation and the generated representation through distribution distance.

7. The video super-resolution model training method based on two-layer temporal constraints according to claim 1, characterized in that, The loss feedback signal is obtained by weighted summation of the basic loss, calibration signal, and alignment signal of the generator network according to dynamically adjusted weights, including: In the early stages of training the generative network, the weights of the calibration signal are set to the maximum value to suppress flicker and artifacts between adjacent frames. During the middle and later stages of training the generative network, the weight of the alignment signal is gradually increased and the weight of the calibration signal is decreased, so that the generative network can form a stable and consistent expression over a long period of time. Throughout the entire training process of the generative network, the weights of the basic loss remain unchanged.

8. A video super-resolution model training system based on two-layer temporal constraints, characterized in that, include: The data extraction module is used to extract low-resolution video segments and their corresponding high-resolution ground truth segments from the training dataset in chronological order. A sub-network module is provided, which provides a generation network that performs restoration processing on the low-definition video clip and outputs the corresponding restored video clip. The inter-frame difference calibration module is used to perform inter-frame difference calibration on high-definition true value segments and video restoration segments to obtain calibration signals at the local level. The characterization and statistical alignment module is used to perform cross-segment statistical alignment on high-definition ground truth segments and video restoration segments to obtain alignment signals at the global level. The training scheduling module is used to perform a weighted summation of the basic loss, calibration signal, and alignment signal of the generator network according to dynamically adjusted weights to obtain the loss feedback signal; The optimization module is used to input the loss feedback signal into the backpropagation link of the generator network, drive the parameters of the generator network to be updated in reverse, and make the generator network converge to a stable solution to complete the training of the generator network.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-7.

10. An electronic device, characterized in that, include: At least one memory for storing program instructions; At least one processor is configured to invoke program instructions stored in the memory and execute the steps of the method described in any one of claims 1-7 according to the obtained program instructions.

Images