A video super-resolution method and system based on a diffusion model

Abstract

The application provides a video super-resolution method and system based on a diffusion model, which comprises the following steps: inputting a video calibration dataset into a pre-trained full-precision diffusion model, performing an inference process, and collecting model running data as calibration data by using a callback function at runtime; determining the contribution degree or sensitivity index of each network layer in the full-precision diffusion model based on the calibration data; performing layer-by-layer quantization resource allocation and federal optimization on the full-precision diffusion model according to the calibration data and the contribution degree or sensitivity index, generating and saving quantization parameters; loading the quantization parameters, performing light-weight inference processing on a low-quality video, reconstructing video details, and obtaining a super-resolution recovery result. The application can still successfully recover a high-resolution video from a low-resolution video in a low-bit quantization scene, and has significant advantages in spatial quality, structural consistency, detail texture recovery and the like compared with other low-bit quantization methods.

Description

Technical Field

[0001] This application relates to the fields of computer vision and video processing technology, and more specifically, to a video super-resolution method and system based on a diffusion model. Background Technology

[0002] With the rapid development of video shooting technology and streaming media-related services, online scenarios are placing higher demands on the restoration and super-resolution (VSR) of low-definition videos. Diffusion models, with their excellent generative capabilities and training stability, have been widely used in various visual generation tasks. In the field of video super-resolution, diffusion models can leverage the natural priors and additional module designs in pre-trained models to achieve better restoration results. Meanwhile, lightweight model quantization techniques significantly reduce the computational and storage overhead during deployment by compressing weights and activation information. However, although low-bit quantization can effectively reduce the model's inference burden, existing techniques still have significant shortcomings when applied to video super-resolution tasks using diffusion models.

[0003] First, diffusion models typically rely on multi-step iterative inference to generate high-quality results. In high-resolution restoration scenarios, the computational cost and memory usage of multi-step inference increase rapidly. The backbone network of diffusion models is large, with considerable computational demands and high inference costs, limiting practical applications. Second, deploying diffusion models on resource-constrained edge devices using lightweight methods places more stringent demands on hardware resources than traditional multi-step models. Existing mass-production methods are mostly geared towards general scenarios, often resulting in severe performance degradation and loss of generated quality in video super-resolution tasks, making them impractical for deployment. Therefore, a method is needed that can fully utilize the model's generative priors to process low-resolution videos, reduce inference and memory overhead, and maintain spatial detail and temporal consistency, thus overcoming the shortcomings of existing technologies in terms of low efficiency and compromised generated quality.

[0004] Chinese patent application CN202410620027.4 discloses a diffusion model quantization method, which allocates different quantization bit widths to each layer to achieve mixed-precision quantization by evaluating the sensitivity of each network layer to different quantization bit widths. However, this method mainly allocates bit widths based on the independence of a single layer, failing to fully consider the coupling propagation effect of quantization errors between layers and lacking a global joint optimization mechanism. Furthermore, this method does not introduce additional compensation branches to address the problem of lost key information and is also unable to effectively solve the quantization distortion problem caused by the unbalanced distribution of activation value channels in diffusion model inference. When applied to video super-resolution scenarios, this method neglects the accumulation of video temporal errors, the fragility of spatiotemporal condition information, and the extreme distribution of video activation values, leading to fatal defects in the generated video such as inter-frame flicker, motion ghosting, and local artifacts. Summary of the Invention

[0005] In view of the deficiencies in the prior art, the purpose of this application is to provide a video super-resolution method and system based on a diffusion model.

[0006] A first aspect of this application provides a video super-resolution method based on a diffusion model, comprising: The video calibration dataset is input into a pre-trained full-precision diffusion model to perform the inference process, and the model running data is collected as calibration data at runtime using a callback function. Based on the calibration data, determine the contribution or sensitivity index of each network layer in the full-precision diffusion model; Based on the calibration data and the contribution or sensitivity index, the full-precision diffusion model is subjected to layer-by-layer quantization resource allocation and federated optimization to generate and save quantization parameters; By loading the quantization parameters, lightweight inference processing is performed on the low-quality video to reconstruct video details and obtain super-resolution restoration results.

[0007] Optionally, the step of inputting the video calibration dataset into a pre-trained full-precision diffusion model, performing the inference process, and collecting model running data as calibration data using a callback function during runtime includes: Load the pre-trained full-precision diffusion model weights and the video calibration dataset, and perform inference on the full-precision diffusion model using the video calibration dataset; During the inference process, a callback function is used to record the statistical information of the activation value distribution of each network layer in the full-precision diffusion model. After inference, the captured activation value distribution statistics are saved as calibration data; wherein, the activation value distribution statistics include the numerical distribution, dynamic range and histogram statistics of the input or output feature maps of each network layer.

[0008] Optionally, determining the contribution or sensitivity index of each network layer in the full-precision diffusion model based on the calibration data includes: Based on the calibration data, each network layer of the full-precision diffusion model is simulated and quantized, and the local quantization error of each network layer is calculated. By utilizing the interlayer error propagation factor, the local quantization error is mapped to the degree of influence on the video super-resolution restoration result; The degree of influence is normalized to obtain the contribution or sensitivity index of each network layer.

[0009] Optionally, the step of performing layer-by-layer quantization resource allocation and federated optimization on the full-precision diffusion model based on the calibration data and the evaluated contribution or sensitivity index, and saving the quantization parameters, includes: Load the pre-trained full-precision diffusion model weights and determine the available quantization space based on the specified quantization configuration; For each network layer, based on the determined contribution or sensitivity index, an initial quantization method and preset hyperparameters are matched for each layer from the quantization candidate space; The federated optimization algorithm is executed, taking into account the coupling effect of inter-layer quantization errors, and the initial quantization method and preset hyperparameters are adjusted in a coordinated manner to find the optimal hyperparameter configuration. After multiple rounds of collaborative optimization, the final quantization parameters of each network layer are obtained and saved.

[0010] Optionally, the available quantization space includes: Global default quantization parameters, including default weight quantization bit width, activation value quantization bit width, quantization granularity, and calibration method; A list of optional strategies, which includes a variety of quantization schemes with different quantization precision, quantization granularity or structural configuration; Layer-by-layer coverage rules set mandatory quantization strategies for specific structural layers in the model; wherein, the specific structural layers include the input layer, the output layer, or the attention mechanism layer, and the mandatory quantization strategies include maintaining full precision or adopting a high-bit quantization scheme.

[0011] Optionally, for each network layer, based on the determined contribution or sensitivity index, matching an initial quantization method and preset hyperparameters from the quantization candidate space for each layer includes: The value range of the contribution or sensitivity index is divided into several preset sensitivity intervals; Establish a mapping relationship between the preset sensitivity range and the quantization schemes in the list of optional strategies; Based on the preset sensitivity range into which the contribution or sensitivity index of each network layer falls, a corresponding initial quantization method and preset hyperparameters are assigned to each layer; wherein, the mapping relationship is configured such that the higher the sensitivity index, the higher the assigned quantization accuracy.

[0012] Optionally, the federated optimization algorithm, considering the coupling effect of inter-layer quantization errors, coordinates the initial quantization method and preset hyperparameters to find the optimal hyperparameter configuration, including: Based on the calibration data, the error propagation process is simulated and the error propagation factor of each network layer is calculated. Based on the error propagation factor and the contribution or sensitivity index, determine the key locations where additional computational branches need to be introduced; Configure additional computation branches at the critical locations and initialize the structural parameters of the computation branches; A joint optimization objective function, which includes quantization error loss and inference efficiency constraints, is constructed to perform collaborative iterative optimization on the quantization parameters and preset hyperparameters of the initial quantization method and the structural parameters of the computation branch. Through multiple rounds of collaborative search, the globally optimal hyperparameter configuration is determined. The hyperparameter configuration includes the quantization parameters of each network layer, preset hyperparameters, and the structural configuration parameters of the computational branches.

[0013] A second aspect of this application provides a video super-resolution system based on a diffusion model, comprising: The calibration module inputs the video calibration dataset into the pre-trained full-precision diffusion model, performs the inference process, and uses a callback function to collect model running data as calibration data during runtime. The evaluation module, based on the calibration data, determines the contribution or sensitivity index of each network layer in the full-precision diffusion model; The quantization module performs quantization resource allocation and federated optimization layer by layer on the full-precision diffusion model based on the calibration data and the contribution or sensitivity index, and generates and saves quantization parameters. The inference module loads the quantization parameters, performs lightweight inference processing on the low-quality video, reconstructs video details, and obtains super-resolution restoration results.

[0014] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements a video super-resolution method based on a diffusion model as described in any one of the claims.

[0015] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements a video super-resolution method based on a diffusion model as described in any one of the claims.

[0016] The video super-resolution method based on the diffusion model provided in this application effectively reduces quantization distortion by collecting calibration data through callback functions and combining quantization with inference; it avoids quality loss caused by uniform quantization by determining the sensitivity index of each network layer and allocating quantization resources layer by layer; and it reduces computation and storage overhead by loading quantization parameters to perform lightweight inference, thereby achieving efficient video super-resolution deployment in resource-constrained scenarios and overcoming the problems of performance degradation and generation quality loss in existing technologies.

[0017] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0018] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating a video super-resolution method based on a diffusion model according to an exemplary embodiment; Figure 2 This is a flowchart illustrating the calibration inference process according to an exemplary embodiment; Figure 3 This is a flowchart illustrating the post-training quantization process according to an exemplary embodiment; Figure 4 This is a structural diagram of a video super-resolution system based on a diffusion model, according to an exemplary embodiment. Detailed Implementation

[0019] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0020] Applying diffusion models to video super-resolution tasks makes it difficult to maintain or even approach the video restoration quality (spatial detail, temporal consistency) of a full-precision model while significantly reducing model inference overhead (computational cost, memory usage). To address these issues, this application provides a video super-resolution method based on a diffusion model.

[0021] Reference Figure 1 As shown, in one embodiment of this application, a video super-resolution method based on a diffusion model is provided, including steps S100-S400: S100 inputs the video calibration dataset into the pre-trained full-precision diffusion model, performs the inference process, and uses a callback function to collect model running data as calibration data during runtime; S200, based on calibration data, determines the contribution or sensitivity index of each network layer in the full-precision diffusion model; S300 performs layer-by-layer quantization resource allocation and federated optimization on the full-precision diffusion model based on calibration data and contribution or sensitivity indicators, generating and saving quantization parameters; The S400, with its quantization parameters loaded, performs lightweight inference processing on low-quality videos to reconstruct video details and obtain super-resolution restoration results.

[0022] Specifically, low-quality video refers to a damaged video sequence used as input for model inference. Due to equipment limitations, transmission compression, or environmental interference, it suffers from one or more visual degradations such as low spatial resolution, blurring, noise, or compression artifacts. It is necessary to reconstruct high-frequency details and improve the overall image quality through a super-resolution model.

[0023] The above embodiments use a callback function in S100 to collect calibration data, accurately capture the dynamic range of activation values, and reduce quantization distortion; determine sensitivity indicators in S200, and combine layer-by-layer quantization resource allocation and federated optimization in S300 to achieve differentiated quantization configuration and global parameter coordination, avoiding the loss of key information; and load quantization parameters in S400 for lightweight inference, reducing computation and memory usage while restoring high-frequency details and temporal consistency in the video, achieving efficient video super-resolution deployment in resource-constrained scenarios.

[0024] To ensure the spatial sharpness and temporal coherence of the final super-resolution restored video, calibration data needs to be obtained to accurately calculate quantization parameters, avoiding quantization errors that could damage high-frequency detail reconstruction. In some specific embodiments of this application, when performing step S100, the following steps S101-S103 can be used to obtain the required calibration data, such as... Figure 2 As shown: S101, load the pre-trained full-precision diffusion model weights and video calibration dataset, and perform inference on the full-precision diffusion model using the video calibration dataset; Specifically, the video calibration dataset is used to provide the model with a representative distribution of input features during the post-training quantization (PTQ) phase to guide subsequent quantization parameter calculations. Unlike the model training phase, the video calibration dataset does not need to contain the full training set size, nor does it need to be equipped with corresponding high-resolution ground truth labels.

[0025] It can usually be obtained through one of the following methods: Subset sampling: A small subset of data (typically 1% to 5% of the total training set) is randomly or hierarchically sampled from the large-scale video dataset used for the original training of the full-precision diffusion model.

[0026] Publicly available standard datasets: directly use publicly available validation or test sets in the video super-resolution field (such as REDS, Vimeo-90K, Vid4, SPMCS, etc.).

[0027] Real-world data collection for specific scenarios: Real low-quality video clips collected on-site for specific business scenarios where the model will be deployed (such as security monitoring, old film restoration, and medical endoscopy videos).

[0028] S102, During the inference process, a callback function is used to record the statistical information of the activation value distribution of each network layer in the full-precision diffusion model; S103, After inference is completed, save the statistical information of the captured activation value distribution as calibration data.

[0029] Specifically, the calibration data includes various indicators such as the numerical distribution, dynamic range (including minimum and maximum values), and histogram statistics of the input or output feature maps of each module (e.g., convolutional layer, self-attention layer) that needs to be evaluated in the model.

[0030] The acquisition of this data provides a foundation for subsequent steps: on the one hand, it is used to calculate the sensitivity index of each layer to identify the network layers that have a critical impact on the super-resolution quality of the video; on the other hand, it serves as a quantization calibration statistic to provide a dynamic range reference for the activation values ​​of the quantization processing unit, thereby reducing quantization errors.

[0031] Through steps S101 to S103 of the above embodiment, this embodiment can obtain the activation value distribution characteristics of the model under real video data at low cost without retraining or backpropagation. Using callback functions to collect data non-intrusively avoids the additional overhead of modifying the model structure; this provides a reliable data foundation for accurately calculating the sensitivity of each layer and rationally allocating the quantization budget for the federated optimization algorithm in subsequent steps, thereby ensuring quantization efficiency while preserving the detailed information required for the video super-resolution task to the greatest extent.

[0032] Because the network layers in the diffusion model exhibit significant heterogeneity in their responses to input latent variables, adopting a uniform quantization strategy can easily lead to the loss of key information and a decline in output quality. Therefore, in some specific embodiments of this application, step S200 is performed to identify the differences in tolerance to quantization errors among different layers by evaluating the quantization contribution or sensitivity index of each network layer, thereby guiding differentiated quantization resource allocation. Specifically, based on calibration data, determining the contribution or sensitivity index of each network layer in the full-precision diffusion model can be achieved using the following steps S201-S203: S201, based on calibration data, simulates and quantizes each network layer of the full-precision diffusion model, and calculates the local quantization error of each network layer; S202 utilizes the inter-layer error propagation factor to map local quantization errors to the degree of impact on the video super-resolution restoration result; S203 normalizes the degree of influence to obtain the contribution or sensitivity index of each network layer.

[0033] For example, let's take a model containing layers A, B, and C as an example: First, 4-bit quantization is simulated to calculate the local quantization error (layer A: 0.1, layer B: 0.5, layer C: 0.2). Second, error propagation factors (layer A: 5.0, layer B: 0.1, layer C: 1.0) are used to map the local error to the degree of influence on the super-resolution result (layer A: 0.5, layer B: 0.05, layer C: 0.2). Finally, normalization is performed to obtain the sensitivity index (layer A: 1.0, layer B: 0.1, layer C: 0.4). This example shows that layer B, although having the largest local error, has the lowest sensitivity and can be assigned low-bit quantization; layer A, although having the smallest local error, has the highest sensitivity and needs to retain high precision, thus achieving a refined quantization configuration based on actual impact.

[0034] The above embodiments accurately identify key network layers through simulated quantization and error propagation analysis, avoid ineffective compression of sensitive layers, and achieve efficient and lightweight model while ensuring the quality of video super-resolution.

[0035] Of course, the above example mainly illustrates the implementation by calculating quantization sensitivity. In other embodiments, it can also be obtained by calculating gradient contribution, feature importance, etc., and is not limited to calculating quantization sensitivity.

[0036] Based on the sensitivity of each network layer obtained in the above embodiments, step S300 is implemented to perform quantized resource allocation optimization. In some specific embodiments of this application, such as… Figure 2 and Figure 3 As shown in S300, based on calibration data and contribution or sensitivity indicators, the full-precision diffusion model is quantized and optimized layer by layer to generate and save quantization parameters. This can be achieved through the following steps S301-S304: S301, Load the pre-trained full-precision diffusion model weights and determine the available quantization space based on the specified quantization configuration.

[0037] Specifically, the available quantization space includes: Global default quantization parameters, including default weight quantization bit width, activation value quantization bit width, quantization granularity, and calibration method; The list of optional strategies includes a variety of quantization schemes with different quantization precision, quantization granularity, or structural configurations; Layer-by-layer coverage rules set mandatory quantization strategies for specific structural layers in the model; wherein, the specific structural layers include the input layer, the output layer, or the attention mechanism layer, and the mandatory quantization strategies include maintaining full precision or adopting a high-bit quantization scheme.

[0038] For example, the global default quantization parameter is configured as follows: The weighted quantization bit width is set to 8 bits, symmetric quantization is used, and the granularity is per-channel. The activation value quantization bit width is set to 8 bits, asymmetric quantization is used, the granularity is per-tensor, and the calibration method is KL divergence.

[0039] The list of optional strategies includes various schemes with different quantization precisions or structure configurations, such as: Strategy A allows weight bit width to be selected from 4 bits or 6 bits, activation bit width to be selected from 4 bits, 6 bits or 8 bits, and quantization granularity to be per-tensor or per-channel. Strategy B allows the weight bit width to be selected from 4 bits, 6 bits, or 8 bits, the activation bit width to be selected from 4 bits or 6 bits, and enables residual branching; Strategy C skips weight quantization, and the activation value uses 16-bit or 32-bit high precision.

[0040] Layer-by-layer coverage rules set mandatory policies for specific structural layers, for example: The first layer, i.e. the input layer, forces the use of strategy C; The intermediate attention layer allows for trying either strategy A or strategy B; The final layer, the output layer, is forced to maintain full precision.

[0041] S302, for each network layer, based on the determined contribution or sensitivity index, matches an initial quantization method and preset hyperparameters for each layer from the quantization candidate space.

[0042] For example, several sensitivity ranges are preset, and each range corresponds to a quantization scheme. For instance: The sensitivity range is 0.0~0.2. The weight bit width is 4, the activation bit width is 6, the granularity is per-tensor, and the error propagation method is MinMax calibration. The sensitivity range of 0.2 to 0.5 uses the weight bit width of 6 and the activation bit width of 4 in Strategy B, with error propagation method: KL divergence calibration; For a sensitivity range of 0.5 to 0.8, strategy C is used, with the error propagation method being: minimization of MSE. The sensitivity range of 0.8 to 1.0 is forcibly preserved with full precision and no error propagation is performed.

[0043] Of course, the above is just one example of this application. The specific sensitivity range and the corresponding quantification scheme can be determined or selected according to actual needs, and are not limited to the above example.

[0044] S303 executes the federated optimization algorithm, considers the coupling effect of inter-layer quantization errors, and coordinates the initial quantization method and preset hyperparameters to find the optimal hyperparameter configuration; S304, after multiple rounds of collaborative optimization from S302 to S303, obtains the final quantization parameters of each network layer and saves them.

[0045] The above embodiments, using S301-304, flexibly configure the quantization intensity and find the optimal parameters in the post-training quantization stage, solving the problem of key information loss caused by the unified quantization strategy in the single-step diffusion model, and improving the video reconstruction quality while ensuring the compression rate; the federated optimization algorithm is adopted, which uses different quantization algorithms in a coordinated manner and achieves reasonable configuration, effectively reducing the quantization error of weights and activation values, and maintaining high output video quality even in lightweight scenarios.

[0046] Continue to refer to Figure 3 In some preferred embodiments of this application, the federated optimization algorithm is executed in step S303 to find the optimal hyperparameter configuration, which can be achieved by the following steps S3031-S3035: S3031, based on calibration data, simulates and quantizes the error propagation process to calculate the error propagation factor of each network layer; S3032, based on the error propagation factor and contribution or sensitivity index, determine the key locations where additional computational branches need to be introduced; S3033, configures additional computation branches at critical locations and initializes the structural parameters of the computation branches; S3034, Construct a joint optimization objective function that includes quantization error loss and inference efficiency constraints, and based on the joint optimization objective function, perform collaborative iterative optimization on the quantization parameters and preset hyperparameters of the initial quantization method and the structural parameters of the computational branches; S3035, through multiple rounds of collaborative iterative optimization of step S3034, determines the globally optimal hyperparameter configuration. The hyperparameter configuration includes the quantization parameters of each network layer, preset hyperparameters, and structural configuration parameters of computation branches.

[0047] The embodiments described above address the coupling effect between multi-layer quantization schemes through a federated optimization algorithm. When different layers employ different quantization methods, inter-layer errors can propagate and affect each other. This embodiment jointly considers the quantization schemes and additional branch configurations of all layers, rather than optimizing independently, and finds the globally optimal hyperparameter combination through multiple rounds of collaborative search. Under given computational resource constraints, the model achieves an optimal balance between video super-resolution quality and inference efficiency, ensuring that the model maintains high output video quality even after low-bit quantization.

[0048] Based on the post-training quantization of the diffusion model completed in steps S100 to S300, step S400 performs lightweight inference by loading quantization parameters. In some specific embodiments of this application, step S400 may further include the following steps S401-S403: S401, the globally optimal hyperparameter configuration and quantization configuration file (as obtained in step S3035 of the preferred embodiment above) are loaded, specifically the quantization parameters, preset hyperparameters and structural configuration parameters of each network layer and computational branch. S402: Before inference begins, the globally optimal hyperparameter configuration and quantization configuration file are loaded into the corresponding network module that needs to be quantized, and the quantization configuration of weights and activation values ​​is completed to obtain the quantized model module. S403 performs forward inference on low-quality video input based on the configured model, reconstructs video details, and outputs super-resolution restoration results.

[0049] This embodiment, through the above S401-S403, can effectively suppress quantization noise caused by the imbalance of activation value channels during the inference stage, ensuring that high-frequency details of the video can still be restored under low bit quantization conditions, and significantly improving the visual reconstruction quality of the lightweight model in actual deployment.

[0050] Based on the same technical concept, other embodiments of this application provide a video super-resolution system 100 based on a diffusion model, such as... Figure 4 As shown, it includes: The calibration module 110 inputs the video calibration dataset into the pre-trained full-precision diffusion model, performs the inference process, and uses a callback function to collect model running data as calibration data during runtime. Evaluation module 120, based on calibration data, determines the contribution or sensitivity index of each network layer in the full-precision diffusion model; The quantization module 130 performs quantization resource allocation and federated optimization layer by layer on the full-precision diffusion model based on calibration data and contribution or sensitivity indicators, and generates and saves quantization parameters. Inference module 140 loads quantization parameters, performs lightweight inference processing on low-quality videos, reconstructs video details, and obtains super-resolution restoration results.

[0051] The specific implementation techniques of each module / unit in the above examples of this application can be referred to the steps of the methods in the above embodiments, and will not be repeated here.

[0052] The above embodiments, by employing a quantized model for inference, can effectively determine the optimal quantization strategy based on video input features during the inference process. Combined with hardware design, this effectively reduces quantization distortion and significantly improves the output quality and inference speed of the quantized model. The use of a quantization module allows for flexible configuration of quantization strength and search for optimal parameters during the post-training quantization stage, solving the problem of key information loss caused by a uniform quantization strategy in single-step diffusion models, thus improving video reconstruction quality while maintaining compression ratio. Furthermore, the use of a federated optimization algorithm during quantization, which collaboratively utilizes different quantization algorithms and achieves reasonable configuration, effectively reduces quantization errors in weights and activation values, maintaining high output video quality even in lightweight scenarios.

[0053] By organically combining the above-mentioned technical means, efficient and lightweight deployment of diffusion models for video super-resolution has been achieved, effectively overcoming the problems of significant performance degradation and severe loss of generation quality in existing technologies under resource-constrained scenarios.

[0054] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0055] The following examples and comparative examples will be used to further illustrate this application in order to better understand the above-mentioned technical solutions. It should be understood that the following are only some examples and are not intended to limit this application.

[0056] To verify the effectiveness and feasibility of the technical solution of this application, quantitative and qualitative evaluations were conducted on multiple public datasets such as UDM10, SPMCS, and YouHQ40. As shown in Table 1, Method 1 represents the full-precision model, and Method 2 represents the method of this application embodiment. In terms of spatial quality (PSNR, SSIM) and perceptual consistency (LPIPS, MUSIQ, etc.) metrics, the performance of this method is close to that of the full-precision model, and it shows comparable or even better competitiveness than the full-precision scenario in some perceptual metrics.

[0057] In terms of visual reconstruction, the super-resolution video output by this method can effectively restore the true structural information and significantly suppress the distortion noise caused by low-bit quantization. Compared with traditional low-bit quantization methods, the video generated by this method has more realistic textures, clearer details, and higher temporal consistency and better dynamic continuity.

[0058] The experimental results above demonstrate that this method can successfully recover high-resolution video from low-resolution video in low-bit quantization scenarios. While significantly reducing computational load and video memory usage, it achieves significant advantages over other low-bit quantization methods in terms of spatial quality, structural consistency, and detail texture recovery, thus achieving the best balance between efficiency and quality.

[0059] Table 1

[0060] The terms "comprising" and "having," and any variations thereof, in the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or devices.

[0061] Based on the same technical concept, in other embodiments of this application, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute the above-described lightweight diffusion model method suitable for video super-resolution.

[0062] Based on the same technical concept, in other embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, can be used to perform the above-described lightweight diffusion model method for video super-resolution.

[0063] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0064] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0065] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0066] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0067] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0068] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0069] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0070] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0071] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A video super-resolution method based on a diffusion model, characterized in that, include: The video calibration dataset is input into a pre-trained full-precision diffusion model to perform the inference process, and the model running data is collected as calibration data at runtime using a callback function. Based on the calibration data, determine the contribution or sensitivity index of each network layer in the full-precision diffusion model; Based on the calibration data and the contribution or sensitivity index, the full-precision diffusion model is subjected to layer-by-layer quantization resource allocation and federated optimization to generate and save quantization parameters; By loading the quantization parameters, lightweight inference processing is performed on the low-quality video to reconstruct video details and obtain super-resolution restoration results.

2. The video super-resolution method based on a diffusion model according to claim 1, characterized in that, The process of inputting the video calibration dataset into a pre-trained full-precision diffusion model, executing the inference process, and collecting model running data as calibration data using a callback function during runtime includes: Load the pre-trained full-precision diffusion model weights and the video calibration dataset, and perform inference on the full-precision diffusion model using the video calibration dataset; During the inference process, a callback function is used to record the statistical information of the activation value distribution of each network layer in the full-precision diffusion model. After inference, the captured activation value distribution statistics are saved as calibration data; wherein, the activation value distribution statistics include the numerical distribution, dynamic range and histogram statistics of the input or output feature maps of each network layer.

3. The video super-resolution method based on a diffusion model according to claim 1, characterized in that, The step of determining the contribution or sensitivity index of each network layer in the full-precision diffusion model based on the calibration data includes: Based on the calibration data, each network layer of the full-precision diffusion model is simulated and quantized, and the local quantization error of each network layer is calculated. By utilizing the interlayer error propagation factor, the local quantization error is mapped to the degree of influence on the video super-resolution restoration result; The degree of influence is normalized to obtain the contribution or sensitivity index of each network layer.

4. The video super-resolution method based on a diffusion model according to claim 1, characterized in that, The process involves performing layer-by-layer quantization resource allocation and federated optimization on the full-precision diffusion model based on the calibration data and the evaluated contribution or sensitivity index, and saving the quantization parameters, including: Load the pre-trained full-precision diffusion model weights and determine the available quantization space based on the specified quantization configuration; For each network layer, based on the determined contribution or sensitivity index, an initial quantization method and preset hyperparameters are matched for each layer from the quantization space; The federated optimization algorithm is executed, taking into account the coupling effect of inter-layer quantization errors, and the initial quantization method and preset hyperparameters are adjusted in a coordinated manner to find the optimal hyperparameter configuration. After multiple rounds of collaborative optimization, the final quantization parameters of each network layer are obtained and saved.

5. The video super-resolution method based on a diffusion model according to claim 4, characterized in that, The available quantization space includes: Global default quantization parameters, including default weight quantization bit width, activation value quantization bit width, quantization granularity, and calibration method; A list of optional strategies, which includes a variety of quantization schemes with different quantization precision, quantization granularity or structural configuration; Layer-by-layer coverage rules set mandatory quantization strategies for specific structural layers in the model; wherein, the specific structural layers include the input layer, the output layer, or the attention mechanism layer, and the mandatory quantization strategies include maintaining full precision or adopting a high-bit quantization scheme.

6. The video super-resolution method based on a diffusion model according to claim 5, characterized in that, For each network layer, based on the determined contribution or sensitivity index, an initial quantization method and preset hyperparameters are matched for each layer from the quantization candidate space, including: The value range of the contribution or sensitivity index is divided into several preset sensitivity intervals; Establish a mapping relationship between the preset sensitivity range and the quantization schemes in the list of optional strategies; Based on the preset sensitivity range into which the contribution or sensitivity index of each network layer falls, a corresponding initial quantization method and preset hyperparameters are assigned to each layer; wherein, the mapping relationship is configured such that the higher the sensitivity index, the higher the assigned quantization accuracy.

7. The video super-resolution method based on a diffusion model according to claim 6, characterized in that, The federated optimization algorithm, considering the coupling effect of inter-layer quantization errors, coordinates the initial quantization method and preset hyperparameters to find the optimal hyperparameter configuration, including: Based on the calibration data, the error propagation process is simulated and the error propagation factor of each network layer is calculated. Based on the error propagation factor and the contribution or sensitivity index, determine the key locations where additional computational branches need to be introduced; Configure additional computation branches at the critical locations and initialize the structural parameters of the computation branches; A joint optimization objective function, including quantization error loss and inference efficiency constraints, is constructed to perform collaborative iterative optimization on the quantization parameters and preset hyperparameters of the initial quantization method, as well as the structural parameters of the computational branch. Through multiple rounds of collaborative search, the globally optimal hyperparameter configuration is determined. The hyperparameter configuration includes the quantization parameters of each network layer, preset hyperparameters, and the structural configuration parameters of the computational branches.

8. A video super-resolution system based on a diffusion model, characterized in that, include: The calibration module inputs the video calibration dataset into the pre-trained full-precision diffusion model, performs the inference process, and uses a callback function to collect model running data as calibration data during runtime. The evaluation module, based on the calibration data, determines the contribution or sensitivity index of each network layer in the full-precision diffusion model; The quantization module performs quantization resource allocation and federated optimization layer by layer on the full-precision diffusion model based on the calibration data and the contribution or sensitivity index, and generates and saves quantization parameters. The inference module loads the quantization parameters, performs lightweight inference processing on the low-quality video, reconstructs video details, and obtains super-resolution restoration results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images