Video variable bit rate coding method and device based on implicit neural representation model

Abstract

The application provides a video variable code rate coding method and device based on an implicit neural representation model. The method comprises the following steps: step 1, constructing and training an implicit neural representation model; step 2, calculating possible quantization parameter combinations by using the trained implicit neural representation model according to a specified target code rate; step 3, calculating quantization sensitivity for all quantization parameter combinations and selecting a combination with minimum quantization sensitivity for calibration; and step 4, encoding the calibrated model parameters by using an entropy encoder. The application realizes variable code rate coding by adjusting quantization parameters instead of retraining the model, greatly reduces the training cost and improves the coding efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video coding, in particular to a video variable bit rate coding method and device based on a neural network implicit representation model. BACKGROUND

[0002] With the rapid development of digital media, video coding technology has become an important part of Internet data transmission and storage. Traditional video coding techniques are mostly based on motion compensation prediction, transform coding, quantization and entropy coding, etc. in order to achieve efficient compression of data under the premise of meeting certain visual quality. However, with the development of deep learning, especially the introduction of implicit neural representation (INR) model, a new video coding paradigm has gradually emerged. The implicit neural representation model provides a new method to replace the traditional coding framework by learning the mapping from coordinates to pixel values, which shows great potential in the field of video coding.

[0003] Implicit neural representation (INR) is an emerging technology that differs from the variational autoencoder (VAE) based framework, which relies on a general model trained on large datasets to create compact representations of various input signals. In contrast, INR video coding (INR-VC) treats each video as a unique neural representation that maps directly from frame indices to pixel values (such as color). Such a mapping allows the spatio-temporal structure of video content to be more closely associated with model parameters, enabling more refined video compression and reconstruction.

[0004] In existing video coding techniques, in order to achieve variable bit rate coding, it is often necessary to retrain the model weights for different target bit rates. However, this method not only consumes time and computational resources, but also requires a large number of iterative training each time the bit rate is adjusted. In addition, traditional quantization methods assume that network layers are independent of each other, but in non-generalized INR-VC models, there is significant dependence between layers, which invalidates this assumption. This dependence not only leads to the accumulation of quantization errors, affecting coding efficiency, but also can negatively impact the final reconstruction quality of the video. Therefore, a quantization strategy specifically designed for variable bit rate INR-VC is needed to address the current bottlenecks in coding technology.

[0005] In this context, developing new coding frameworks to support flexible bit rate management, especially in the area of post-training quantization (PTQ), has become a focus of current research. By combining theoretical analysis and experimental verification, exploring how to adjust the quantization parameters without retraining the model to achieve efficient and accurate variable bit rate coding will lay the foundation for the future development of implicit neural representation video coding technology. SUMMARY

[0006] In view of the above deficiencies of the prior art, the present application provides a video variable bit rate coding method and device based on an implicit neural representation model.

[0007] The technical scheme of the method of the present application is as follows:

[0008] The video variable bit rate coding method based on an implicit neural representation model comprises the following steps:

[0009] Step 1, training phase: first, the learning ability of a neural network is utilized to map input video data into implicit features of model parameters; then, the video is coded at a high bit rate, and an implicit neural representation model is constructed and trained to learn the spatial and temporal characteristics of video frames;

[0010] Step 2, testing phase: according to a specified target bit rate, the implicit neural representation model trained in step 1 is utilized to calculate possible combinations of quantization parameters;

[0011] Step 3, quantization sensitivity analysis: for all combinations of quantization parameters generated in step 2, the calculation and analysis of quantization sensitivity are performed one by one, and finally the combination with the minimum quantization sensitivity is selected as the quantized model parameters; then, the quantized model parameters are calibrated;

[0012] Step 4, entropy coding: the calibrated model parameters in step 3 are further compressed by an entropy encoder.

[0013] The present application also provides a video variable bit rate coding device based on an implicit neural representation model, which comprises:

[0014] A training module is configured to map input video data into implicit features of model parameters, and to code the video at a high bit rate and train an implicit neural representation model;

[0015] A quantization parameter calculation module is configured to calculate possible combinations of quantization parameters by using the implicit neural representation model according to a specified target bit rate;

[0016] A quantization sensitivity analysis module is configured to calculate and analyze the quantization sensitivity one by one for all combinations of quantization parameters generated by the quantization parameter calculation module, and to select the combination with the minimum quantization sensitivity for calibration;

[0017] An entropy coding module is configured to further compress the model parameters calibrated by the quantization sensitivity analysis module.

[0018] The present application also provides a computer program product comprising computer readable instructions, which, when executed on an electronic device, cause the electronic device to implement the above-mentioned coding method.

[0019] Considering the implicit neural network video compression task, the present application can perform hybrid quantization on the trained model parameters, and by using the calibration method and quantization sensitivity criterion proposed in the present application, the optimal quantization combination can be determined by using the quantization sensitivity under different quantization combinations, and the model parameters after quantization can be optimized by the calibration method, and the model has the code rate adjustment capability within a certain code rate range, compared with retraining the code rate point model, the method can save about 5-8 times of retraining time. Therefore, the present application not only solves the shortcomings of the prior art, but also lays a theoretical foundation for future research and promotes the development of video coding technology. BRIEF DESCRIPTION OF DRAWINGS

[0020] The above and / or other aspects of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0021] Figure 1 is a commonly used INR-VC video compression process flowchart.

[0022] Figure 2 is the overall method flowchart of the present application.

[0023] Figure 3 is the specific flowchart of the method of the present application.

[0024] Figure 4 is the quantitative test result of the method of the present application on the UVG dataset.

[0025] Figure 5 is the visual effect of the method of the present application on the UVG dataset, (a), (b), (c) are the effects under three different scenes. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical scheme and advantages of the present application more clear, the implementation method of the present application will be described in further detail below in conjunction with the drawings.

[0027] The implicit neural representation (INR for short) used in video coding is fundamentally different from traditional video coding methods. Traditional video coding methods usually process video data based on a series of steps such as motion compensation prediction, transform coding, quantization and entropy coding, and finally generate a bitstream that can be decoded by a standard decoder. INR video coding, on the other hand, uses neural networks to directly learn the intrinsic representation of video content, i.e., stores information in network parameters, thereby bypassing many complex steps in the traditional coding process. Although this method simplifies the coding process, it often has a large overhead in coding time due to the need to train complex neural network models.

[0028] Currently, there are many studies on implicit neural networks for video coding, but few consider how to reduce the coding time of INR-VC, especially in different network scenarios and bandwidth-limited situations. The existing INR-VC video compression process as shown in Figure 1 , its research mainly focuses on how to improve the quality of video reconstruction, and less involves how to optimize the coding time, especially the solution when facing network bandwidth limitation.

[0029] To solve this problem, the present application proposes a video variable bit rate coding method and device based on INR. This method realizes variable bit rate coding by adjusting the quantization parameter instead of retraining the model, greatly reducing the training cost and improving the coding efficiency. Specifically, as shown in Figure 2 , first, a high bit rate network is trained using video images; in the bit rate control stage, the quantization combination is calculated according to the specified bit rate and the bit rate error operating range, all quantization combinations are traversed, the quantization sensitivity is calculated, and the quantization combination with the minimum quantization sensitivity is selected for calibration. The calibrated model is compressed and saved through entropy coding. The present application proposes a mixed precision quantization strategy and introduces quantization sensitivity analysis and network calibration technology to further optimize the quantization process and ensure high-quality reconstruction of videos at different bit rates.

[0030] As shown in Figure 3 , the video variable bit rate coding method based on INR of the present embodiment has the following steps:

[0031] Step 1, training stage: using the representation ability of the learnable neural network model, first use the learning ability of the neural network to map the input video data to the implicit features of the model parameters, which can better reflect the spatio-temporal structure of the video content. By encoding the video at a high bit rate, an implicit neural representation model is constructed and trained to learn the accurate spatio-temporal characteristics of video frames. This model learns the spatial and temporal characteristics of video frames based on training data and optimizes network weights through backpropagation, so that the model can accurately represent video content and provide support for subsequent variable bit rate coding.

[0032] Step 2, test stage: according to the target bit rate specified by the user or system, calculate the possible quantization parameter combination using the implicit neural representation model trained in step 1 to ensure that the model can adapt to different bit rate coding requirements.

[0033] Step 3, Quantization Sensitivity Analysis: For all the combinations of quantization parameters generated in Step 2, the calculation and analysis of quantization sensitivity are performed one by one. The calculation of quantization sensitivity is based on the sensitivity of model parameters in the video encoding process to the change of visual quality, to identify the specific impact of different quantization schemes on visual quality and compression efficiency. Finally, the combination with the smallest quantization sensitivity is selected as the quantized model parameters, to ensure the optimal visual quality of the encoding result under a certain code rate, while ensuring the compression efficiency. Then, considering the dependence between layers and blocks in the implicit neural network in the video compression task, unlike the traditional layer-by-layer calibration, the global calibration method is adopted to better capture the cross-layer dependence. Specifically, for the case where the weights of different layers or even different channels in the same layer differ greatly, the same quantization bit width is applied between channels in the same layer, and all layer parameters are updated in one optimization process.

[0034] Step 4, Entropy Coding: The calibrated model parameters in Step 3 are input into the entropy encoder. Through entropy coding techniques, these quantized parameters are further compressed to reduce the storage and transmission of redundant information. The entropy encoder analyzes the probability distribution of the parameters and selects the appropriate encoding method to optimize the overall coding efficiency, reducing the code rate while minimizing information loss.

[0035] In Step 1, the implicit neural network model can be specifically divided into two categories: one is (encoding layer + embedding vector + decoding layer), and the other is (embedding vector + decoding layer). Among them, the function of the encoding layer is to convert the input data into a higher-level representation by extracting features to capture the structure and relationship of the data; the embedding vector represents the mapping of discrete input to continuous vector space, so that similar inputs are closer in this space for subsequent processing and learning; the function of the decoding layer is to convert the high-level representation generated by the encoding layer back to the original data format or target format. In the video compression task of implicit neural networks, code rate calculation usually only considers the network parameter quantity of the decoding layer and the code stream size of the embedding vector, and the encoding layer is not considered. The subsequent quantization refers to the quantization of the decoding layer and the embedding vector.

[0036] This embodiment takes the classic implicit neural network model HNeRV as an example, which contains five convolutional layers in the decoding layer. However, it is emphasized that the method of the present application is not limited to a specific type of implicit neural network model, but is applicable to all INR-VC methods. That is, no matter which specific INR architecture is used, as long as it involves implicit representation and encoding of video, the technology of the present application can play a role. In step 1, the frame number and resolution of the encoded video and other information are collected to ensure that the model structure matches the characteristics of the input data. According to this information, the network parameters between the layers of the model are adjusted, such as setting appropriate input and output dimensions. In addition, for different feature extraction needs, the size and stride of the convolution kernel are reasonably selected to ensure that the network can fully capture the spatial and temporal information of the video. During the training process, the learning of the model can be regarded as the encoding process of the video. By encoding the video at a high code rate, the representation ability of the neural network is utilized to map the input video data to the implicit features of the corresponding model parameters. These features not only accurately reflect the spatio-temporal structure of the video content, but also, in the process of backpropagation optimization, by continuously adjusting the network weights, the representation ability of the model for the spatio-temporal characteristics of the video frames is improved, thereby realizing high-quality video reconstruction.

[0037] In step 2, according to the target code rate specified by the user or the system, the possible quantization parameter combinations are calculated to ensure that the model can adapt to the encoding needs of different code rates. To ensure the quality of video encoding, considering the error caused by low bit quantization, the lowest quantization bit is set to 3. In specific implementation, taking the classic full 8-bit quantization as the upper limit and the full 3-bit quantization as the lower limit, the code rate combinations are calculated within this range. For example, assuming that the size of HNERV at full 8-bit quantization in step 1 is 2M, then the size at 3-bit quantization is about For the convenience of subsequent description of the quantization scheme, taking a 5-layer HNERV as an example, (b1, b2, b3, b4, b5) represents the quantization combination of the 1st to 5th convolutional layers, where b1 to b5 represent the quantization bit width of the 1st to 5th layers, respectively, then represents the code rate under this quantization combination. Then the size of the video can be defined as

[0038]

[0039] where Param(·) represents the number of parameters, w l represents the weight of the lth layer, represents the quantization bit width of the weight of the lth layer, represents the quantization bit width of the embedding vector of the tth frame, e t represents the embedding vector of the tth frame, T represents the total number of frames, and L represents the number of layers.

[0040] The calculation process can be defined as

[0041] R = S ± ε

[0042] where R represents the specified code rate, and ε represents the allowed code rate error.

[0043] By traversing the possible combinations of (b1, b2, b3, b4, b5), the quantization combination within the allowed code rate error range is obtained.

[0044] In step 3, according to the calculated quantization combination, the performance of each combination is evaluated one by one. Specifically, the system applies each quantization combination to the model parameters and performs a quantization sensitivity analysis to determine the impact of each combination on the visual quality in the video encoding process.

[0045] The model loss under each quantization combination can be represented as,

[0046]

[0047] w represents the model weight, and Δw represents the impact on the weight due to quantization, Δw T represents the transpose of Δw, represents the model loss, respectively represent the first-order gradient and the second-order Hessian matrix. In the task of implicit neural network video compression, it is generally believed that the first-order gradient tends to 0, so this method evaluates the quantization sensitivity through the Hessian matrix. However, since the direct calculation of the Hessian matrix has high cost in terms of calculation and storage, this method provides a near-sighted method to estimate the Hessian matrix, avoiding explicit calculation. Specifically:

[0048] Let Ω = Δw T H w Δw, and construct where g is the loss of w. Therefore The first-order gradient of w can be represented as:

[0049]

[0050] The final quantization sensitivity can be modeled as

[0051] where

[0052] By comparing the quantization sensitivity of the quantization results under different combinations, the optimal quantization scheme is identified, ensuring that the encoding result can achieve the best balance between visual effect and compression efficiency under a specific code rate. This analysis process not only improves the stability of the encoding quality, but also provides a reliable quantization basis for the subsequent encoding process.

[0053] In step 3, after determining the optimal quantization combination by the above method, the quantized model parameters are calibrated to reduce the distortion introduced by quantization, thereby improving the visual quality of the encoded video, and this process is defined as a continuous optimization framework.

[0054] The dequantization process is defined as

[0055]

[0056] Where b represents the quantization bit width, s represents the quantization bit width, and clip(a, b, c) represents limiting the value a to [b, c], returning a value not less than b and not greater than c.

[0057] The optimization objective is defined as

[0058]

[0059] Where v i represents the continuous variable to be optimized, s represents the proportional relationship, x represents the model input, represents the output of the model under the input x and the weight w, h(v i ) is an arbitrary differentiable function constrained in [0, 1], argmin is the minimization process, and λ represents the regularization weight, generally 0.1-0.01. By introducing the annealing β, h(·) is guided to converge to 0 or 1, thereby accelerating the convergence of the calibration process and improving the compression performance and visual quality of the network.

[0060] In step 4, after selecting and calibrating the quantized model parameters, the final entropy encoding is performed. First, the processed model parameters in step 3 are input into the entropy encoder. The entropy encoder analyzes the probability distribution of these parameters and selects the optimal encoding method to compress the redundant information to a minimum. This process not only effectively reduces the number of bits required for transmission or storage, but also maximizes the visual quality of the video while reducing the code rate. The encoder models the statistical characteristics of the model parameters to achieve efficient compression under limited bit number, while ensuring that the decoded video data can be restored to close to the original quality, ensuring that high-quality decoding results can be maintained at low code rates.

[0061] As can be seen from Table 1, the application of the method of the present application to different INR-VC methods can improve the encoding speed.

[0062] Table 1 Encoding time of different INR-VC methods

[0063]

[0064] Figure 4The present application HNeRV-VR shows the decrease of the required code rate compared with the traditional HNeRV at the same PSNR value. By introducing the variable code rate control technology of the present application, HNeRV-VR significantly reduces the code rate while ensuring video quality, with an average of 25.5% reduction in bit rate requirement. This means that at the same PSNR (image quality), HNeRV-VR can achieve similar video reconstruction effect at a lower code rate, improving the coding efficiency. Figure 5 (a) of the present application shows the performance improvement effect brought by the calibration method of the present application under the same quantization configuration (full int4); (b) and (c) further show the effect optimization achieved by applying hybrid precision quantization combined with quantization calibration technology.

[0065] The embodiment also provides a video variable code rate coding device based on an implicit neural representation model, comprising: a training module configured to map input video data to implicit features of model parameters, and encode the video at a high code rate to train the implicit neural representation model; a quantization parameter calculation module configured to calculate possible quantization parameter combinations by using the implicit neural representation model according to a specified target code rate; a quantization sensitivity analysis module configured to calculate and analyze the quantization sensitivity of all quantization parameter combinations generated by the quantization parameter calculation module one by one, and select the combination with the minimum quantization sensitivity for calibration; and an entropy coding module configured to further compress the model parameters calibrated by the quantization sensitivity analysis module.

[0066] In the above embodiment, the method can be implemented wholly or partially by software, hardware, firmware or any combination thereof. When implemented by software, the method can be implemented wholly or partially in the form of a computer program product. The computer program product comprises computer readable instructions, which, when executed on an electronic device, cause the electronic device to implement the above-mentioned coding method of the present application.

[0067] There are many methods and ways to implement the technical solutions of the present application, and the above description is only the preferred embodiment of the present application. It should be noted that for ordinary skilled persons in the art, without departing from the principles of the present application, several improvements and refinements can be made, which should also be considered as the protection scope of the present application. The components not explicitly described in the embodiment can be implemented by existing technology.

Claims

1. A method for video variable bit rate coding based on an implicit neural representation model, characterized in that, The method comprises the following steps: Step 1, training phase: first, the learning ability of the neural network is used to map the input video data to the implicit features of the model parameters; Then encode the video at a high code rate, build and train the implicit neural representation model to learn the spatial and temporal characteristics of the video frames; Step 2, test phase: according to the specified target code rate, use the implicit neural representation model trained in step 1 to calculate the possible quantization parameter combination; Specifically, the size of the video is defined as where Param(·) denotes the number of parameters, w l denotes the weight of the l-th layer, denotes the quantization bit width of the weight of the l-th layer, denotes the quantization bit width of the embedding vector of the t-th frame, e t denotes the embedding vector of the t-th frame, T denotes the total number of frames of the video, and L denotes the number of layers of the model. Then the process of calculating the quantization parameter combination is defined as R=S±ε Where R represents the specified code rate, and ε represents the allowed code rate error; Step 3, quantization sensitivity analysis: for all the quantization parameter combinations generated in step 2, the quantization sensitivity is calculated and analyzed one by one, and the combination with the smallest quantization sensitivity is finally selected as the quantized model parameter; Then the calibrated quantized model parameter; Specifically: apply each quantization parameter combination to the implicit neural representation model and analyze its quantization sensitivity, and the model loss under each quantization parameter combination is represented as: where w represents the model weights, Aw represents the impact on the weights due to quantization, Aw T represents the transpose of Aw, represents the model loss, respectively represent the first order gradient and the second order Hessian matrix; Therefore, the quantization sensitivity modeling is: wherein By comparing the quantization sensitivity of the quantization results under different combinations, the combination with the smallest quantization sensitivity is identified; The process of calibrating the quantized model parameter is defined as a continuous optimization framework, and the optimization target is: where v i represents an optimized continuous variable, s represents a scaling relationship, x represents a model input, represents an output of the model under input x and weight w, h(v i ) is an arbitrary differentiable function constrained in [0, 1], arg min represents a minimization process, and λ represents a regularization weight, b represents a quantization bit width, s represents a quantization bit width, and clip(a, b, c) represents limiting the value a to be between [b, c], and returning a value not less than b and not greater than c. Step 4, entropy encoding: further compress the calibrated model parameters in step 3 using an entropy encoder.

2. The method of claim 1, wherein the implicit neural representation model is trained using a video dataset. Step 1 further comprises: first, pre-processing the input video data to extract the spatial and temporal features of the video frames; Then map the pre-processed video data to implicit features, which is specifically to encode the video data at a high code rate to generate an implicit neural representation model at a high code rate point; and optimize the neural network weights through the back propagation algorithm, so that the model can accurately represent the video content at the high code rate point.

3. The method of claim 1, wherein the implicit neural representation model is trained using a video dataset. In step 1, the implicit neural representation model can be divided into two categories: one is: encoding layer + embedding vector + decoding layer, and the other is: embedding vector + decoding layer; Wherein, the encoding layer is used to convert the input data into a higher level representation, and capture the structure and relationship of the data by extracting features; The embedding vector is used to map discrete input to continuous vector space, so that similar input is closer in this space, to facilitate subsequent processing and learning; The decoding layer is used to convert the high-level representation generated by the encoding layer back to the original data format or the target format.

4. The method of claim 1, wherein the implicit neural representation model is trained using a video dataset. Step 3 includes: for the case that the weights between different layers of the implicit neural representation model or even between different channels of the same layer are significantly different, the same quantization bit width is applied between the channels of the same layer, and all layer parameters are updated in one optimization process.

5. The method of claim 1, wherein the implicit neural representation model is trained using a video dataset. Step 4 includes: inputting the calibrated model parameters in step 3 into the entropy encoder, and the entropy encoder analyzes the probability distribution of these parameters to select the optimal encoding method to minimize the redundant information.

6. Apparatus for video variable bit rate coding based on an implicit neural representation model, for implementing the coding method according to claim 1, characterized in that, The device comprises: A training module is configured to map input video data to implicit features of model parameters, and encode the video at a high code rate to train an implicit neural representation model; A quantization parameter calculation module is configured to calculate possible quantization parameter combinations by using an implicit neural representation model according to a specified target code rate; A quantization sensitivity analysis module is configured to calculate and analyze the quantization sensitivity of all quantization parameter combinations generated by the quantization parameter calculation module one by one, and select a combination with the minimum quantization sensitivity for calibration; An entropy encoding module is configured to further compress the model parameters calibrated by the quantization sensitivity analysis module.

7. A computer program product, characterised in that, Computer readable instructions are included, and when the computer readable instructions run on an electronic device, the electronic device implements the encoding method in any one of claims 1 to 5.

Images