Quantization method and system of video super-division model, medium and equipment

Abstract

The invention provides a quantification method and system of a video super-division model, a medium and equipment, and the method comprises the steps: determining a pre-trained real-world video super-division model; determining an integer version quantization layer corresponding to a to-be-quantized layer according to the to-be-quantized layer of the pre-trained real world video super-division model; replacing a to-be-quantized layer of the pre-trained real-world video super-division model with an integer version quantization layer, and determining a quantized real-world video super-division model; and carrying out calibration processing and training processing on the quantization parameters of the quantized real-world video super-division model by adopting a preset calibration data set, and determining the quantized real-world video super-division model. According to the method and the device, the performance of the original video super-division model is reserved, the model parameters are converted into the low-bit integer from the high-bit floating point number, storage occupation is reduced, the reasoning speed is increased, and the model reasoning efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and more specifically, to a quantization method, system, medium, and device for a video super-resolution model. Background Technology

[0002] Super-resolution (SR) is an important topic in the field of computer vision. With the development of deep learning, especially the emergence of diffusion models, super-resolution technology has made breakthrough progress, and in recent years, research has expanded from image super-resolution to video super-resolution.

[0003] Some video super-resolution models, such as BasicVSR and MGLD-VSR, have achieved superior results in video super-resolution tasks. However, these models typically have high computational complexity and resource consumption, especially when processing high-definition video, where their inference process places enormous demands on storage and computing power. This leads to hardware resource limitations in practical applications, making deployment on real-time video processing and edge devices particularly difficult.

[0004] Model quantization, a commonly used model compression method, has achieved significant results in fields such as large language models, image super-resolution, and video generation. However, in the field of video super-resolution, research and application of quantization techniques remain somewhat lacking, and the exploration and implementation of related methods are relatively limited. Some existing quantization methods, such as MaxMin, Vidit-q, and SVDQuant, exhibit significant accuracy loss when transferred to video super-resolution models. Furthermore, activation outliers also contribute to reduced quantization accuracy.

[0005] In the patent "CN117274049A; A video super-resolution method based on post-training quantization", an FP32 model is trained using a dataset, and the model is deployed using TensorRT. Finally, the model is quantized and calibrated using the dataset, and post-training static quantization is performed based on FRVSR. Group quantization is performed for layers with large errors. However, in the quantization process, the impact of activation outlier anomalies on quantization accuracy is not considered. Summary of the Invention

[0006] In view of the deficiencies in the prior art, the purpose of this application is to provide a quantization method, system, medium and device for video super-resolution models.

[0007] A first aspect of this application provides a quantization method for a video super-resolution model, comprising: Determine the pre-trained real-world video super-resolution model; Based on the layer to be quantized in the pre-trained real-world video super-resolution model, determine the integer version quantization layer corresponding to the layer to be quantized; The integer version of the quantization layer is used to replace the unquantized layer of the pre-trained real-world video super-resolution model to determine the quantized real-world video super-resolution model; The quantization parameters of the quantized real-world video super-resolution model are calibrated and trained using a preset calibration dataset to determine the quantized real-world video super-resolution model.

[0008] Optionally, determining the integer version of the quantization layer corresponding to the layer to be quantized based on the layer to be quantized in the pre-trained real-world video super-resolution model includes: A first equivalent transformation is performed on the activation values ​​and weights of the unquantized layer of the pre-trained real-world video super-resolution model using a preset smoothing factor to determine the activation values ​​after the first smoothing process and the weights after the second smoothing process. A second equivalent transformation is performed on the activation value after the first smoothing process and the weight after the first smoothing process using a preset rotation matrix to determine the activation value after the second smoothing process and the weight after the second smoothing process. The activation value and the weight after the second smoothing process are quantized into integers to determine a new quantization layer; Each new quantization layer is summed with a preset learnable bias correction amount to determine the integer version quantization layer corresponding to each layer to be quantized.

[0009] Optionally, the method further includes: The pre-defined video super-resolution dataset is sampled to determine the sample data; The sample data is subjected to downsampling, noise addition, and blurring operations in sequence to determine the preset calibration dataset.

[0010] Optionally, the step of calibrating and training the quantization parameters of the quantized real-world video super-resolution model using a preset calibration dataset to determine the completed quantization of the real-world video super-resolution model includes: The preset calibration dataset is input into the quantized real-world video super-resolution model for forward propagation, and the exponential moving average method is used to calibrate the smoothing factor. The parameters of the non-preset learnable bias correction of the quantized real-world video super-resolution model are frozen. Based on the preset loss function, the preset learnable bias correction of each integer version quantization layer is trained using the preset calibration dataset until the parameters converge, thus determining the quantized real-world video super-resolution model.

[0011] A second aspect of this application provides a quantization system for a video super-resolution model, comprising: The module for determining the model to be processed is used to determine the pre-trained real-world video super-resolution model; The integer version quantization layer determination module is used to determine the integer version quantization layer corresponding to the layer to be quantized based on the layer to be quantized in the pre-trained real-world video super-resolution model. The quantization layer replacement module is used to replace the unquantized layer of the pre-trained real-world video super-resolution model with the integer version of the quantization layer, thereby determining the quantized real-world video super-resolution model. The calibration module is used to calibrate and train the quantization parameters of the quantized real-world video super-resolution model using a preset calibration dataset, thereby determining the quantized real-world video super-resolution model.

[0012] A third aspect of this application discloses a video super-resolution processing method, comprising: Determine the pre-trained MGLD-VSR video super-resolution model; The pre-trained MGLD-VSR video super-resolution model is quantized using the quantization method of the video super-resolution model described in any one of the first aspects above, to determine the quantized MGLD-VSR video super-resolution model; The video to be processed is input into the quantized MGLD-VSR video super-resolution model to determine the video after super-resolution processing.

[0013] A fourth aspect of this application discloses a video super-resolution processing method, comprising: Determine the pre-trained floating-point super-resolution model; The pre-trained floating-point super-resolution model is quantized using the quantization method of the video super-resolution model described in any one of the first aspects above, to determine the quantized floating-point super-resolution model.

[0014] The video to be processed is input into the quantized floating-point super-resolution model to determine the video that has undergone super-resolution processing.

[0015] A fifth aspect of this application provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of any of the methods provided in the first aspect of this application.

[0016] A sixth aspect of this application provides an electronic device comprising: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of any of the methods provided in the first aspect of this application.

[0017] The quantization method of the video super-resolution model in this application determines the corresponding integer version quantization layer based on the layer to be quantized in the pre-trained real-world video super-resolution model, and replaces the original layer to be quantized with the integer version quantization layer to achieve quantization of the video super-resolution model. This quantizes the model parameters from high-bit floating-point to low-bit integer, uniformly distributes weights and activations, reduces the impact of activation outliers on quantization, and uses a pre-set calibration dataset to correct and train the quantization parameters, reducing the accuracy loss caused by low-bit quantization. While preserving the performance of the original video super-resolution model, this method reduces storage usage, speeds up inference, and improves model inference efficiency.

[0018] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0019] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating a quantization method for a video super-resolution model according to an exemplary embodiment.

[0020] Figure 2 This is a schematic diagram illustrating a quantization process for a layer to be quantized according to an exemplary embodiment.

[0021] Figure 3 This is a schematic diagram illustrating a comparison of quantization effects according to an exemplary embodiment.

[0022] Figure 4 This is a schematic diagram illustrating the structure of a quantization system for a video super-resolution model according to an exemplary embodiment. Detailed Implementation

[0023] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0024] The terms "comprising" and "having," and any variations thereof, in the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or devices.

[0025] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature.

[0026] Existing model quantization methods, such as MaxMin, Vidit-q, and SVDQuant, suffer significant accuracy loss when transferred to video super-resolution models. Activation outlier anomalies also lead to reduced quantization accuracy. To address these issues, this application provides a quantization method for video super-resolution models.

[0027] Figure 1 This is a flowchart illustrating a quantization method for a video super-resolution model according to an exemplary embodiment.

[0028] Reference Figure 1 As shown in one embodiment of this application, a quantization method for a video super-resolution model includes: S11 to S14.

[0029] S11, determine the pre-trained real-world video super-resolution model.

[0030] S12, based on the layer to be quantized in the pre-trained real-world video super-resolution model, determine the integer version of the quantization layer corresponding to the layer to be quantized.

[0031] S13, replace the unquantized layer of the pre-trained real-world video super-resolution model with an integer version of the quantization layer to determine the quantized real-world video super-resolution model.

[0032] S14. The quantization parameters of the quantized real-world video super-resolution model are calibrated and trained using a pre-set calibration dataset to determine the quantized real-world video super-resolution model.

[0033] In the embodiments described above, the corresponding integer version of the quantization layer is determined based on the quantization layer of the pre-trained real-world video super-resolution model, and the original quantization layer is replaced with the integer version of the quantization layer to achieve quantization of the video super-resolution model. This quantizes the model parameters from high-bit floating-point to low-bit integer, uniformly distributes weights and activations, reduces the impact of activation outliers on quantization, and uses a preset calibration dataset to correct and train the quantization parameters, reducing the accuracy loss caused by low-bit quantization. While retaining the performance of the original video super-resolution model, this reduces storage usage and speeds up inference, thereby improving model inference efficiency.

[0034] To determine the quantization object, in some specific embodiments of this application, the pre-trained real-world video super-resolution model may include, but is not limited to: RealBasicVSR, Upscale-A-Video, MGLD-VSR, STAR, SeedVR, etc.

[0035] To achieve quantization of real-world video super-resolution models, in some specific embodiments of this application, for S12, the integer version quantization layer corresponding to the layer to be quantized is determined based on the layer to be quantized in the pre-trained real-world video super-resolution model, which can be implemented as S121 to S124.

[0036] S121, a first equivalent transformation is performed on the activation values ​​and weights of the unquantized layer of the pre-trained real-world video super-resolution model using a preset smoothing factor, and the activation values ​​and weights after the first smoothing process are determined.

[0037] Specifically, the preset smoothing factor s is: ; in, X represents the hyperparameter, W represents the activation value, and W represents the weight.

[0038] The activation values ​​and weights of the unquantized layers in the pre-trained real-world video super-resolution model are subjected to a first equivalent transformation using a preset smoothing factor s: ; Where Y represents the output, X represents the activation value, W represents the weight, and s represents the smoothing factor.

[0039] The activation value after the first equivalent transformation and the first smoothing process is: The weights after the first smoothing process are .

[0040] S122, a preset rotation matrix is ​​used to perform a second equivalent transformation on the activation value and weight after the first smoothing process, and the activation value and weight after the second smoothing process are determined.

[0041] Specifically, a second equivalent transformation is performed on the activation values ​​and weights after the first smoothing process using a preset rotation matrix: ; Where Y represents the output, X represents the activation value, W represents the weight, and XH represents the activation value after the second smoothing process. The weights after the second smoothing process are represented by , and Q represents the preset rotation matrix.

[0042] Preset rotation matrix The Hadmar matrix can be used.

[0043] In the steps S121 to S122 of this application, the activation values ​​and weights of the quantized layer of the pre-trained real-world video super-resolution model are smoothed using a preset smoothing factor and a preset rotation matrix. The order is not important. The preset smoothing factor can be used first for the first smoothing process, and then the preset rotation matrix can be used for the second smoothing process. Alternatively, the preset rotation matrix can be used first for the second smoothing process, and then the preset smoothing factor can be used for the first smoothing process.

[0044] In this embodiment, a first smoothing process is performed using a preset smoothing factor, and a second smoothing process is performed using a preset rotation matrix.

[0045] S123, quantize the activation values ​​and weights after the second smoothing process into integers to determine the new quantization layer.

[0046] Specifically, the objects quantized into integers in this step S123 are the activation values ​​and weights that have undergone the first and second smoothing processes.

[0047] S124, sum each new quantization layer with a preset learnable bias correction amount to determine the integer version quantization layer corresponding to each layer to be quantized.

[0048] Specifically, the integer version of the quantization layer is: ; Where y represents the output of the integer version of the quantization layer. This indicates that the activation value is being quantized. This indicates that the weights are quantized, and bc represents the preset learnable bias correction amount.

[0049] bc is the same as the bias parameter in the original layer to be quantized, and is used to learn the mean loss of the quantization process: ; Where E represents the mean, X represents the full-precision activation value, and W represents the full-precision weight. Indicates the quantized activation value. Indicates quantization weights, This represents the difference between the full-precision tensor and the quantized tensor.

[0050] In the embodiments described above, a preset smoothing factor is used to perform a first smoothing process on the activation values ​​and weights, and a preset rotation matrix is ​​used to perform a second smoothing process on the activation values ​​and weights. This effectively removes outliers from the activation values, resulting in a more uniform distribution of activation values ​​and weights, and reducing the impact of abnormal activation values ​​on the quantization process. Low-bit integer quantization achieves low computational cost, reduces storage usage, and speeds up inference, thereby improving the model's inference efficiency. A preset learnable bias correction amount is set in each quantization layer to align the quantization layer and the full-precision layer, further reducing the accuracy loss caused by low-bit quantization.

[0051] To quantize the real-world video super-resolution model, in some specific embodiments of this application, for S13, an integer version of the quantization layer is used to replace the layer to be quantized in the pre-trained real-world video super-resolution model to determine the real-world video super-resolution model to be quantized. This can be achieved by: The integer version of the quantization layer is replaced with the corresponding layer to be quantized to achieve the quantization of the pre-trained real-world video super-resolution model and determine the quantized real-world video super-resolution model.

[0052] In the embodiments described above, an integer version of the quantization layer is used to replace the corresponding layer to be quantized, converting the real-world video super-resolution model from a high-bit floating-point number to a low-bit integer. This enables the quantization of the pre-trained real-world video super-resolution model, reduces the computational complexity and resource consumption of the pre-trained model, improves the inference efficiency of the model, and overcomes the accuracy loss caused by low-bit quantization.

[0053] To achieve calibration and training of quantization parameters, in some specific embodiments of this application, a quantization method for a video super-resolution model may further include S15 to S16.

[0054] S15: Sample the preset video super-resolution dataset to determine the sample data.

[0055] Specifically, the preset video super-resolution dataset includes, but is not limited to, the clear video dataset REDS30.

[0056] S16, perform downsampling, noise addition, and blurring operations on the sample data in sequence to determine the preset calibration dataset.

[0057] Specifically, real-world degradation can be simulated through manual downsampling, noise addition, and blurring operations, with the preset calibration dataset being a video dataset simulating real-world degradation.

[0058] To achieve the calibration and training of quantization parameters, in some specific embodiments of this application, for S14, a preset calibration dataset is used to calibrate and train the quantization parameters of the quantized real-world video super-resolution model to determine the quantized real-world video super-resolution model, which can be implemented using S141 to S142.

[0059] S141, input the preset calibration dataset into the quantized real-world video super-resolution model for forward propagation, and use the exponential moving average method to calibrate the smoothing factor.

[0060] Specifically: ; in, Indicates the updated smoothing factor. This represents the smoothing factor corresponding to the activation of the i-th layer. Indicates the weighting factor. This represents the smoothing factor of the previous iteration step.

[0061] The weighting factor is used to control the degree of weighting between the old and new data.

[0062] S142, freeze the parameters of the non-preset learnable bias correction of the quantized real-world video super-resolution model, and train the preset learnable bias correction of each integer version quantization layer using a preset calibration dataset according to the preset loss function until the parameters converge, thus determining the quantized real-world video super-resolution model.

[0063] Specifically, the preset calibration dataset is input into the quantized real-world video super-resolution model, and the predicted super-resolution video is output. Based on the super-resolution video output by the full-precision model and the super-resolution video output by the quantized real-world video super-resolution model, as well as the preset loss function, the loss is determined, and the parameters of the preset learnable bias correction are optimized through backpropagation until convergence, thus determining the quantized real-world video super-resolution model.

[0064] The preset loss function is: ; in, This represents the super-resolution video output by the full-precision model. This represents the super-resolution video output by the quantized real-world video super-resolution model, where MSE represents the mean squared error.

[0065] The embodiments described above in this application improve the accuracy of the quantized real-world video super-resolution model by calibrating the model using a preset calibration dataset and training the parameters of the preset learnable bias correction amount. This aligns the results of the quantization layer with the full-precision layer, thereby reducing the accuracy loss caused by low-bit quantization and improving the accuracy of the super-resolution video.

[0066] To obtain low-bit super-resolution video, in some specific embodiments of this application, a quantization method for a video super-resolution model may further include S16.

[0067] S16, input the video to be processed into the real-world video super-resolution model that has been quantized, and determine the super-resolution video.

[0068] The embodiments described above employ a quantization method for a video super-resolution model provided in this application to perform low-bit quantization of a real-world video super-resolution model, and then use the quantized real-world video super-resolution model to perform super-resolution processing on the video to be processed. This effectively reduces the computational complexity and resource consumption of the model, improves the inference efficiency of the model, and reduces accuracy loss.

[0069] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0070] The following examples and comparative examples will be used to further illustrate this application in order to better understand the above-mentioned technical solutions. It should be understood that the following are only some examples and are not intended to limit this application.

[0071] The video super-resolution model MGLD-VSR is quantized to 4 bits using the quantization method of the video super-resolution model provided in this application.

[0072] Step 1: Determine the pre-trained real-world video super-resolution model as MGLD-VSR.

[0073] Step 2: Decompose the original MGLD-VSR model into Unet+VAE parts. The VAE part has a small time consumption in the multi-step diffusion model and can be ignored. Therefore, only the structure of the Unet part is quantized.

[0074] The layers to be quantized in the Unet part of the structure are two-dimensional convolutional layers, three-dimensional convolutional layers, and linear layers. Based on the above layers to be quantized, the integer version quantization layer corresponding to the layers to be quantized is determined.

[0075] Figure 2 This is a schematic diagram illustrating a quantization process for a layer to be quantized according to an exemplary embodiment.

[0076] Reference Figure 2 As shown, specifically, for quantized convolutional layers / quantized linear layers, taking a two-dimensional convolutional layer as an example, a smoothing factor is used to smooth the layer to be quantized, and a rotation matrix is ​​used to rotate the layer to be quantized. The convolutional / linear layer is quantized into integers, and a learnable bias correction amount is added to obtain an integer version of the quantized layer. The specific steps can be referred to steps S121 to S124 above, and will not be repeated here.

[0077] Specifically, the quantization settings are as follows: 4 bits; asymmetric quantization method; per-channel fine-grained quantization weights; and per-tensor activation.

[0078] Step 3: Replace the layer to be quantized with an integer version of the quantization layer to determine the MGLD-VSR model to be quantized.

[0079] Step 4: Select 30 videos from the commonly used video super-resolution dataset REDS as sample data, and perform degradation operations such as 4x downsampling, blurring, compression, and noise addition on the ground truth videos of the sample data to simulate the degradation of real-world videos and obtain the calibration dataset.

[0080] The calibration dataset is input into the quantized real-world video super-resolution model for forward propagation, and the smoothing factor is calibrated using the exponential moving average method. Activation of calibration data in each calibration dataset And the weight W, and its corresponding smoothing factor for: ; in, Let W represent the activation of the i-th layer, and W represent the weights. This represents the smoothing factor corresponding to the activation of the i-th layer. This represents hyperparameters.

[0081] In this embodiment, The value is 0.5.

[0082] The smoothing factor was calibrated using the exponential moving average method. ; in, Indicates the updated smoothing factor. This represents the smoothing factor corresponding to the activation of the i-th layer. Indicates the weighting factor. This represents the smoothing factor of the previous iteration step.

[0083] In this embodiment, The value is 0.1.

[0084] The parameters of the non-preset learnable bias correction of the quantized MGLD-VSR model are frozen, and the parameters of the learnable bias correction are trained to obtain the quantized MGLD-VSR model.

[0085] Specifically, the batch size is set to 4, and the training rounds are two.

[0086] Step 5: Perform inference verification using the fully quantized MGLD-VSR model and the full-precision model to obtain the super-resolution video output by the fully quantized MGLD-VSR model and the super-resolution video output by the full-precision model.

[0087] Figure 3 This is a schematic diagram illustrating a comparison of quantization effects according to an exemplary embodiment.

[0088] Reference Figure 3 As shown, from top to bottom, the images are: LR low-resolution video image, FP full-precision model output super-resolution video image, and W4A4 quantized MGLD-VSR model output super-resolution video image.

[0089] Depend on Figure 3 It can be seen that the video super-resolution model quantized using the quantization method of the video super-resolution model provided in this application can achieve the purpose of video super-resolution, and the difference from the full-precision model is small.

[0090] The following comparison of the results of the method provided in this application with those of the comparative methods MaxMin, Q-Diffusion, QuaRot, and ViDit-Q on the quantitative indicators PSNR, SSIM, LPIPS, DISTS, MUSIQ, NIQE, and MANIQA. Here, Ours indicates the method provided in this application.

[0091]

[0092] Referring to Table 1, the method provided in this application achieves excellent results on the above quantitative indicators. The method provided in this application can reduce the accuracy loss caused by low-bit quantization, reduce storage occupation and speed up inference while retaining the performance of the original video super-resolution model, and improve the inference efficiency of the model.

[0093] In some specific embodiments of this application, a video super-resolution processing method includes: Step 1: Determine the pre-trained MGLD-VSR video super-resolution model.

[0094] Step 2: The pre-trained MGLD-VSR video super-resolution model is quantized using a quantization method for a video super-resolution model provided in this application to determine the quantized MGLD-VSR video super-resolution model.

[0095] Step 3: Input the video to be processed into the quantized MGLD-VSR video super-resolution model to determine the video that has undergone super-resolution processing.

[0096] In the above embodiments of this application, an integer version of the quantization layer is used to replace the layer to be quantized, and the quantization parameters are calibrated and trained using a calibration dataset. This achieves the goal of quantizing the model parameters from high-bit floating-point to low-bit integer while retaining the performance of the original video super-resolution model, thereby reducing storage usage, accelerating inference speed, and improving model inference efficiency.

[0097] In some specific embodiments of this application, a video super-resolution processing method includes: Step 1: Determine the pre-trained floating-point super-resolution model.

[0098] Specifically, the pre-trained floating-point super-resolution model represents the floating-point super-resolution model that needs to be accelerated.

[0099] For example, the floating-point super-resolution models that need to be accelerated may include, but are not limited to, BasicVSR, MGLD-VSR, STAR, DOVE, and SeedVR.

[0100] Step 2: The pre-trained floating-point super-resolution model is quantized using a quantization method for a video super-resolution model provided in this application to determine the quantized floating-point super-resolution model.

[0101] Specifically, the quantized floating-point super-resolution model represents the quantization to a low-bit integer model, that is, it can be quantized to a 4-bit integer model.

[0102] Step 3: Input the video to be processed into the quantized floating-point super-resolution model to determine the video after super-resolution processing.

[0103] Specifically, under the same video input conditions, the 4-bit quantization model can ideally achieve a speedup of 8 times compared to the original model.

[0104] The embodiments described above employ a quantization method for a video super-resolution model provided in this application. The integer version of the quantization layer replaces the layer to be quantized, and the quantization parameters are calibrated and trained using a calibration dataset. This achieves the conversion of model parameters from high-bit floating-point to low-bit integer while preserving the performance of the original video super-resolution model, thereby reducing storage usage, accelerating inference speed, and improving model inference efficiency.

[0105] Figure 4This is a schematic diagram illustrating the structure of a quantization system for a video super-resolution model according to an exemplary embodiment.

[0106] Reference Figure 4 As shown in one embodiment of this application, a quantization system for a video super-resolution model is provided, including: a model determination module 110, an integer version quantization layer determination module 120, a quantization layer replacement module 130, and a calibration module 140.

[0107] The model determination module 110 is used to determine the pre-trained real-world video super-resolution model; The integer version quantization layer determination module 120 is used to determine the integer version quantization layer corresponding to the layer to be quantized based on the layer to be quantized in the pre-trained real-world video super-resolution model. The quantization layer replacement module 130 is used to replace the unquantized layer of the pre-trained real-world video super-resolution model with an integer version of the quantization layer, thereby determining the quantized real-world video super-resolution model. The calibration module 140 is used to calibrate and train the quantization parameters of the quantized real-world video super-resolution model using a preset calibration dataset, and to determine the quantized real-world video super-resolution model.

[0108] In the embodiments described above, the corresponding integer version of the quantization layer is determined based on the quantization layer of the pre-trained real-world video super-resolution model, and the original quantization layer is replaced with the integer version of the quantization layer to achieve quantization of the video super-resolution model. This quantizes the model parameters from high-bit floating-point to low-bit integer, uniformly distributes weights and activations, reduces the impact of activation outliers on quantization, and uses a preset calibration dataset to correct and train the quantization parameters, reducing the accuracy loss caused by low-bit quantization. While retaining the performance of the original video super-resolution model, this reduces storage usage and speeds up inference, thereby improving model inference efficiency.

[0109] Regarding the embodiments of the above system, the specific ways in which each module performs operations have been described in detail in the embodiments of the method, and will not be elaborated here.

[0110] Based on the same technical concept, in some specific embodiments of this application, a terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and a method that the processor can use to execute when executing the program.

[0111] Based on the same technical concept, in some specific embodiments of this application, a computer-readable storage medium is provided on which a computer program is stored, which can be used to execute a method when the program is executed by a processor.

[0112] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0113] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0114] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0115] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0116] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0117] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0118] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0119] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0120] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A method for quantizing a video super-resolution model, characterized in that, The method comprises the steps of: determining a pre-trained real-world video super-resolution model; determining an integer version quantization layer corresponding to a to-be-quantized layer of the pre-trained real-world video super-resolution model according to the to-be-quantized layer; replacing the to-be-quantized layer of the pre-trained real-world video super-resolution model with the integer version quantization layer to determine a quantized real-world video super-resolution model; performing calibration processing and training processing on a quantization parameter of the quantized real-world video super-resolution model by using a preset calibration data set to determine a completed quantized real-world video super-resolution model.

2. The method of quantizing a video super-resolution model according to claim 1, wherein, The step of determining an integer version quantization layer corresponding to a to-be-quantized layer of the pre-trained real-world video super-resolution model according to the to-be-quantized layer comprises the steps of: performing first equivalent conversion processing on activation values and weights of the to-be-quantized layer of the pre-trained real-world video super-resolution model by using a preset smoothing factor to determine activation values after first smoothing processing and weights after first smoothing processing; performing second equivalent conversion processing on the activation values after first smoothing processing and the weights after first smoothing processing by using a preset rotation matrix to determine activation values after second smoothing processing and weights after second smoothing processing; quantizing the activation values after second smoothing processing and the weights after second smoothing processing into integers to determine new quantization layers; performing sum operation on each of the new quantization layers and a preset learnable bias correction quantity to determine an integer version quantization layer corresponding to each of the to-be-quantized layers.

3. The method of claim 1, wherein, The method further comprises the steps of: performing sampling processing on a preset video super-resolution data set to determine sample data; performing, in sequence, downsampling operation, noise adding operation and blurring operation on the sample data to determine the preset calibration data set.

4. The method of claim 1, wherein, The step of performing calibration processing and training processing on a quantization parameter of the quantized real-world video super-resolution model by using a preset calibration data set to determine a completed quantized real-world video super-resolution model comprises the steps of: inputting the preset calibration data set into the quantized real-world video super-resolution model for forward propagation, and performing calibration on a smoothing factor by using an exponential moving average method; freezing parameters of non-pre-set learnable bias correction quantities of the quantized real-world video super-resolution model, and performing training processing on a preset learnable bias correction quantity of each of the integer version quantization layers by using the preset calibration data set according to a preset loss function until the parameters converge, so as to determine the completed quantized real-world video super-resolution model.

5. The method of claim 1, wherein, The step of inputting the preset calibration data set into the to-be-quantized real-world video super-resolution model for forward propagation and performing calibration on a smoothing factor by using an exponential moving average method comprises the steps of: ； wherein, denotes an updated smoothing factor, denotes the i-th layer activation corresponding smoothing factor, denotes a weighting factor, denotes the smoothing factor of the previous iteration step.

6. A system for quantizing a video super-resolution model, the system comprising: The device comprises: a to-be-processed model determination module configured to determine a pre-trained real-world video super-resolution model; an integer version quantization layer determination module configured to determine an integer version quantization layer corresponding to a to-be-quantized layer of the pre-trained real-world video super-resolution model according to the to-be-quantized layer; a quantization layer replacement module configured to replace the to-be-quantized layer of the pre-trained real-world video super-resolution model with the integer version quantization layer to determine a quantized real-world video super-resolution model; and a calibration data set input module configured to perform calibration processing and training processing on a quantization parameter of the quantized real-world video super-resolution model by using a preset calibration data set to determine a completed quantized real-world video super-resolution model. A calibration module is configured to calibrate and train the quantized parameters of the quantized real-world video super-resolution model using a preset calibration data set, and determine the completed quantized real-world video super-resolution model.

7. A method of video super-resolution processing, characterized by, The method comprises the steps of: determining a pre-trained MGLD-VSR video super-resolution model; quantizing the pre-trained MGLD-VSR video super-resolution model using the quantization method of the video super-resolution model according to any one of claims 1 to 5, and determining a quantized MGLD-VSR video super-resolution model; inputting a video to be processed into the quantized MGLD-VSR video super-resolution model, and determining a video processed by super-resolution.

8. A method of video super-resolution processing, characterized by, The method comprises the steps of: determining a pre-trained floating-point super-resolution model; quantizing the pre-trained floating-point super-resolution model using the quantization method of the video super-resolution model according to any one of claims 1 to 5, and determining a quantized floating-point super-resolution model; inputting a video to be processed into the quantized floating-point super-resolution model, and determining a video processed by super-resolution.

9. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by a processor to implement the steps of the method according to any one of claims 1 to 5.

10. An electronic device, comprising: The method comprises the steps of: a memory having a computer program stored thereon; a processor configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 5.

Images