Video variable code rate coding method and device based on implicit neural representation model

Abstract

The invention provides a video variable code rate coding method and device based on an implicit neural representation model. The method comprises the following steps: step 1, constructing and training an implicit neural representation model; step 2, according to a specified target code rate, calculating a possible quantization parameter combination by using the trained implicit neural representation model; step 3, aiming at all quantization parameter combinations, quantization sensitivity is calculated, and a combination with the minimum quantization sensitivity is selected for calibration; and step 4, encoding the calibrated model parameters by using an entropy encoder. According to the invention, the variable code rate coding is realized by adjusting the quantization parameter instead of retraining the model, so that the training cost is greatly reduced and the coding efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of video coding, and in particular to a method and device for video variable bit rate coding based on a neural network implicit representation model. Background Art

[0002] With the rapid development of digital media, video coding technology has become an important part of Internet data transmission and storage. Traditional video coding technologies are mostly based on steps such as motion compensation prediction, transform coding, quantization, and entropy coding, in order to achieve efficient data compression while meeting certain visual quality requirements. However, with the development of deep learning, especially the introduction of the implicit neural representation (INR) model, a new video coding paradigm has gradually emerged. The implicit neural representation model provides a new method to replace the traditional coding framework by learning the mapping from coordinates to pixel values. This method shows great potential in the field of video coding.

[0003] Implicit neural representation (INR) is an emerging technique that differs from the variational autoencoder (VAE)-based framework, which relies on a general model trained on a large-scale dataset to create a compact representation of a variety of input signals. In contrast, INR-VC treats each video as a unique neural representation that maps directly from frame index to pixel value (such as color). This mapping allows the spatiotemporal structure of the video content to be more closely associated with the model parameters, thereby achieving more refined video compression and reconstruction.

[0004] In existing video coding technologies, in order to achieve variable bitrate coding, it is often necessary to retrain the model weights for different target bitrates. However, this method is not only time-consuming and computationally expensive, but also requires a large amount of iterative training every time the bitrate is adjusted. In addition, traditional quantization methods assume that network layers are independent of each other, but in the non-generalized INR-VC model, there is a significant dependency between the layers, which makes this assumption invalid. This dependency not only leads to the accumulation of quantization errors and affects coding efficiency, but may also have a negative impact on the final reconstruction quality of the video. Therefore, a quantization strategy designed specifically for variable bitrate INR-VC is needed to address the bottleneck in current coding technology.

[0005] In this context, developing new coding frameworks to support flexible bitrate management, especially in post-training quantization (PTQ), has become the focus of current research. By combining theoretical analysis with experimental verification, exploring how to adjust quantization parameters without retraining the model to achieve efficient and accurate variable bitrate coding will lay the foundation for the future development of implicit neural representation video coding technology. Summary of the invention

[0006] In view of the above deficiencies in the prior art, the present invention proposes a video variable bit rate encoding method and device based on an implicit neural representation model.

[0007] The technical solution adopted by the method of the present invention is as follows:

[0008] The video variable bit rate encoding method based on the implicit neural representation model includes the following steps:

[0009] Step 1, training phase: First, the learning ability of the neural network is used to map the input video data into implicit features of the model parameters; then the video is encoded at a high bit rate, and an implicit neural representation model is constructed and trained to learn the spatial and temporal characteristics of the video frames;

[0010] Step 2, testing phase: Based on the specified target bit rate, the implicit neural representation model trained in step 1 is used to calculate possible quantization parameter combinations;

[0011] Step 3, quantization sensitivity analysis: for all the quantization parameter combinations generated in step 2, the quantization sensitivity is calculated and analyzed one by one, and finally the combination with the smallest quantization sensitivity is selected as the quantized model parameter; then the quantized model parameter is calibrated;

[0012] Step 4, entropy coding: The model parameters calibrated in step 3 are further compressed using an entropy encoder.

[0013] The present invention also provides a video variable bit rate encoding device based on an implicit neural representation model, the device comprising:

[0014] A training module is used to map the input video data into implicit features of the model parameters and encode the video at a high bit rate to train the implicit neural representation model;

[0015] A quantization parameter calculation module is used to calculate possible quantization parameter combinations using an implicit neural representation model according to a specified target bit rate;

[0016] A quantization sensitivity analysis module, used to calculate and analyze the quantization sensitivity of all quantization parameter combinations generated by the quantization parameter calculation module, and select the combination with the smallest quantization sensitivity for calibration;

[0017] The entropy coding module is used to further compress the model parameters calibrated by the quantization sensitivity analysis module.

[0018] The present invention also provides a computer program product, comprising computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the above encoding method.

[0019] Considering the implicit neural network video compression task, the present invention can perform hybrid quantization on the trained model parameters. By using the calibration method and quantization sensitivity criteria proposed by the present invention, the optimal quantization combination can be determined by using the quantization sensitivity under different quantization combinations, and the quantized model parameters can be optimized by the calibration method. The model has the ability to adjust the bit rate within a certain bit rate range. Compared with retraining the bit rate point model, this method can save about 5 to 8 times the retraining time. Therefore, the present invention not only solves the shortcomings of the prior art, but also lays a theoretical foundation for future research and promotes the development of video coding technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0021] Figure 1 This is a flowchart of the commonly used INR-VC video compression process.

[0022] Figure 2 It is a flow chart of the overall method of the present invention.

[0023] Figure 3 It is a specific flow chart of the method of the present invention.

[0024] Figure 4 It is the quantitative test result of the method of the present invention on the UVG data set.

[0025] Figure 5 This is the visualization effect of the method of the present invention on the UVG dataset. (a), (b), and (c) are the effects in three different scenarios. DETAILED DESCRIPTION

[0026] In order to make the objectives, technical solutions and advantages of the present invention more clear, the implementation method of the present invention will be further described in detail below with reference to the accompanying drawings.

[0027] The technology of implicit neural representation (INR) for video coding is fundamentally different from traditional video coding methods. Traditional video coding methods usually process video data based on a series of steps such as motion compensation prediction, transform coding, quantization, and entropy coding, and finally generate a bit stream that can be decoded by a standard decoder. INR video coding uses neural networks to directly learn the intrinsic representation of video content, that is, to store information in network parameters, thereby bypassing many complex steps in the traditional coding process. Although this method simplifies the coding process, it often has a large overhead in encoding time because it needs to train complex neural network models.

[0028] At present, there are many studies on implicit neural networks for video coding, but few consider how to reduce the encoding time of INR-VC. In particular, for different network scenarios, the time efficiency of video encoding is particularly important under bandwidth constraints. The existing INR-VC video compression process is as follows: Figure 1 As shown, their research mainly focuses on how to improve the quality of video reconstruction, and less on how to optimize encoding time, especially solutions when facing network bandwidth limitations.

[0029] In order to solve this problem, the present invention proposes a video variable bit rate coding method and device based on INR. The method realizes variable bit rate coding by adjusting quantization parameters instead of retraining the model, which greatly reduces the training cost and improves the coding efficiency. Specifically, Figure 2 As shown, firstly, a high bit rate network is trained using video images; in the bit rate control stage, the quantization combination is calculated according to the specified bit rate and bit rate error operating range, all quantization combinations are traversed, the quantization sensitivity is calculated, and the quantization combination with the minimum quantization sensitivity is selected for calibration. The calibrated model is compressed and saved through entropy coding. The present invention proposes a mixed precision quantization strategy and introduces quantization sensitivity analysis and network calibration technology to further optimize the quantization process and ensure high-quality reconstruction of the video at different bit rates.

[0030] like Figure 3 As shown, a video variable bit rate encoding method based on INR in this embodiment comprises the following steps:

[0031] Step 1, training phase: Using the representation capabilities of the learnable neural network model, first use the learning ability of the neural network to map the input video data into implicit features of the model parameters, which can better reflect the spatiotemporal structure of the video content. By encoding the video at a high bit rate, an implicit neural representation model is constructed and trained to learn the precise spatiotemporal characteristics of the video frames. The model learns the spatial and temporal characteristics of the video frames based on the training data, and optimizes the network weights through back propagation, so that the model can accurately represent the video content and provide support for subsequent variable bit rate encoding.

[0032] Step 2, testing phase: Based on the target bit rate specified by the user or system, the implicit neural representation model trained in step 1 is used to calculate possible quantization parameter combinations to ensure that the model can adapt to encoding requirements of different bit rates.

[0033] Step 3, quantization sensitivity analysis: For all quantization parameter combinations generated in step 2, the quantization sensitivity is calculated and analyzed one by one. The calculation of quantization sensitivity is based on the sensitivity of model parameters to changes in visual quality during video encoding to identify the specific impact of different quantization schemes on visual quality and compression efficiency. Finally, the combination with the smallest quantization sensitivity is selected as the quantized model parameter to ensure the optimal visual quality of the encoding result at a specific bit rate while ensuring compression efficiency. Then, considering the dependency between layers and blocks of implicit neural networks in video compression tasks, a global calibration method is adopted, which is different from the traditional layer-by-layer calibration, to better capture the dependencies across layers. Specifically, for the case where the weights between different layers of the implicit neural network, or even between different channels in the same layer, are large, the same quantization bit width is applied between channels in the same layer, and all layer parameters are updated in one optimization process.

[0034] Step 4, entropy coding: Input the model parameters calibrated in step 3 into the entropy encoder. These quantized parameters are further compressed through entropy coding technology to reduce the storage and transmission of redundant information. The entropy encoder analyzes the probability distribution of the parameters and selects the appropriate encoding method to optimize the overall coding efficiency, while reducing the bit rate and minimizing information loss.

[0035] In step 1, the implicit neural network model can be specifically divided into two categories: one is (encoding layer + embedding vector + decoding layer), and the other is (embedding vector + decoding layer). Among them, the function of the encoding layer is to convert the input data into a higher-level representation, capturing the structure and relationship of the data by extracting features; the embedding vector represents the mapping of discrete inputs to a continuous vector space, so that similar inputs are closer in this space, which is convenient for subsequent processing and learning; the function of the decoding layer is to convert the high-level representation generated by the encoding layer back to the original data format or target format. In the video compression task of the implicit neural network, the bit rate calculation usually only considers the network parameter quantity of the decoding layer and the bit stream size of the embedding vector, while the encoding layer is not taken into consideration. The subsequent quantization refers to the quantization of the decoding layer and the embedding vector.

[0036] This embodiment takes the classic implicit neural network model HNeRV as an example, and the decoding layer of the model includes five convolutional layers. However, it should be emphasized that the method of the present invention is not limited to a specific type of implicit neural network model, but is applicable to all INR-VC methods. In other words, no matter which specific INR architecture is adopted, as long as it involves implicit representation and encoding of video, the technology of the present invention can play a role. In step 1, information such as the number of frames and resolution of the encoded video is collected to ensure that the model structure matches the characteristics of the input data. According to this information, the network parameters between the layers of the model are adjusted, such as setting appropriate input and output dimensions. In addition, for different feature extraction requirements, the convolution kernel size and stride are reasonably selected to ensure that the network can fully capture the spatial and temporal information of the video. During the training process, the learning of the model can be regarded as the encoding process of the video. The video is encoded at a high bit rate to utilize the representation ability of the neural network to map the input video data to the implicit features of the corresponding model parameters. These features can not only accurately reflect the spatiotemporal structure of the video content, but also can improve the model's representation ability of the spatiotemporal characteristics of the video frame by continuously adjusting the network weights during the back propagation optimization process, thereby achieving high-quality video reconstruction.

[0037] In step 2, according to the target bit rate specified by the user or system, possible quantization parameter combinations are calculated to ensure that the model can adapt to the encoding requirements of different bit rates. To ensure the quality of video encoding, the error caused by low-bit quantization is considered and the lowest quantization bit is set to 3. In the specific implementation, the classic full 8-bit quantization is used as the upper limit and the full 3-bit quantization is used as the lower limit. The bit rate combination is calculated within this range. For example, assuming that the size of HNERV in step 1 under full 8-bit quantization is 2M, the size under 3-bit quantization is approximately To facilitate the explanation of the subsequent quantization scheme, take the 5-layer HNERV as an example, (b1, b2, b3, b4, b5) represents the quantization combination of the 1st to 5th convolutional layers, where b1 to b5 represent the quantization bit widths of the 1st to 5th layers respectively. represents the bit rate under this quantization combination. Then the size of the video can be defined as

[0038]

[0039] Where Param(·) represents the number of parameters, w l represents the weight of the lth layer, represents the quantization bit width of the weight of layer l, represents the quantization bit width of the embedding vector of the tth frame, e t Represents the embedding vector of the tth frame, T represents the total number of video frames, and L represents the number of model layers.

[0040] The calculation process can be defined as

[0041] R=S±ε

[0042] Where R represents the specified bit rate and ε represents the allowed bit rate error.

[0043] By traversing the possible combinations of (b1, b2, b3, b4, b5), a quantization combination within the allowable bit rate error range is obtained.

[0044] In step 3, the performance of each quantization combination is evaluated one by one based on the calculated quantization combinations. Specifically, the system applies each quantization combination to the model parameters and performs quantization sensitivity analysis on them to determine the impact of each combination on visual quality during video encoding.

[0045] The model loss under each quantization combination can be expressed as,

[0046]

[0047] w represents the model weight, and Δw represents the impact of quantization on the weight. T represents the transpose of Δw, represents the model loss, They represent the first-order gradient and the second-order Hessian Matrix, respectively. In implicit neural network video compression tasks, it is usually assumed that the first-order gradient tends to 0, so this method evaluates the quantization sensitivity through the Hessian matrix. However, since the direct calculation of the Hessian matrix is ​​very expensive in terms of computation and storage, this method provides a short-sighted method to estimate the Hessian matrix and avoid explicit calculation. Specifically:

[0048] Let Ω=Δw T H w Δw, and construct Where g is the loss The first-order gradient of w. Therefore The first-order gradient of w can be expressed as:

[0049]

[0050] The final quantitative sensitivity can be modeled as

[0051] in

[0052] By comparing the quantization sensitivity of the quantization results under different combinations, the optimal quantization scheme is identified to ensure that the encoding result can achieve the best balance between visual effect and compression efficiency at a specific bit rate. This analysis process not only improves the stability of encoding quality, but also provides a reliable quantization basis for subsequent encoding processes.

[0053] In step 3, after the optimal quantization combination is determined by the above method, the quantized model parameters are calibrated to reduce the distortion introduced by quantization, thereby improving the visual quality after encoding, and this process is defined as a continuous optimization framework.

[0054] The dequantization process is defined as

[0055]

[0056] Where b represents the quantization bit width, s represents the quantization bit width, and clip(a,b,c) means limiting the value a to between [b,c] and returning a value that is not less than b and not greater than c.

[0057] The optimization objective is defined as

[0058]

[0059] where v i represents the optimized continuous variable, s represents the proportional relationship, and x represents the model input. represents the output of the model when the input is x and the weight is w, h(v i ) is an arbitrary differentiable function constrained in [0,1], argmin is the minimization process, λ represents the regularization weight, which is generally 0.1-0.01. By introducing the regularization term annealing β, h(·) is guided to converge to 0 or 1, thereby accelerating the convergence of the calibration process and improving the compression performance and visual quality of the network.

[0060] In step 4, after selecting and calibrating the quantized model parameters, the final entropy coding is performed. First, the model parameters processed in step 3 are input into the entropy encoder. The entropy encoder selects the optimal coding method by analyzing the probability distribution of these parameters to compress the redundant information to a minimum. This process can not only effectively reduce the number of bits required for transmission or storage, but also maximize the visual quality of the video while reducing the bit rate. The encoder models the statistical characteristics of the model parameters to achieve efficient compression with a limited number of bits, while ensuring that the video data can be restored to a quality close to the original during decoding, ensuring that high-quality decoding results can be maintained at low bit rates.

[0061] It can be seen from Table 1 that the coding speed can be improved by applying the method of the present invention to different INR-VC methods.

[0062] Table 1 Encoding time of different INR-VC methods

[0063]

[0064] Figure 4The figure shows the decrease in the bit rate required for the HNeRV-VR of the present invention compared to the traditional HNeRV at the same PSNR value. By introducing the variable bit rate control technology of the present invention, HNeRV-VR significantly reduces the bit rate while ensuring the video quality, reducing the bit rate requirement by 25.5% on average. This means that at the same PSNR (image quality), HNeRV-VR can achieve similar video reconstruction effects at a lower bit rate, improving coding efficiency. Figure 5 (a) shows the performance improvement effect brought by the calibration method of the present invention under the same quantization configuration (all int4); (b) and (c) further show the effect optimization achieved by applying mixed precision quantization combined with quantization calibration technology.

[0065] The present embodiment also provides a video variable bit rate encoding device based on an implicit neural representation model, including: a training module, used to map input video data into implicit features of model parameters, and encode the video at a high bit rate to train the implicit neural representation model; a quantization parameter calculation module, used to calculate possible quantization parameter combinations based on a specified target bit rate using the implicit neural representation model; a quantization sensitivity analysis module, used to calculate and analyze the quantization sensitivity of all quantization parameter combinations generated by the quantization parameter calculation module, and select the combination with the smallest quantization sensitivity for calibration; an entropy coding module, used to further compress the model parameters calibrated by the quantization sensitivity analysis module.

[0066] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes computer readable instructions, and when the computer readable instructions are executed on an electronic device, the electronic device implements the above encoding method of the present invention.

[0067] There are many methods and approaches to implement the technical solution of the present invention. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be considered as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A video variable bit rate coding method based on an implicit neural representation model, characterized in that: The following steps are involved: Step 1, training phase: First, the learning ability of the neural network is used to map the input video data into implicit features of the model parameters; The video is then encoded at a high bitrate, and an implicit neural representation model is built and trained to learn the spatial and temporal characteristics of the video frames; Step 2, testing phase: according to the specified target bit rate, the implicit neural representation model trained in step 1 is used to calculate possible quantization parameter combinations; Step 3, quantization sensitivity analysis: for all the quantization parameter combinations generated in step 2, the quantization sensitivity is calculated and analyzed one by one, and finally the combination with the smallest quantization sensitivity is selected as the quantized model parameter; then the quantized model parameter is calibrated; Step 4, entropy coding: The model parameters calibrated in step 3 are further compressed using an entropy encoder.

2. The video variable bit rate coding method based on implicit neural representation model according to claim 1 is characterized in that: Step 1 also includes: first preprocessing the input video data to extract the spatial features and temporal features of the video frames; then mapping the preprocessed video data into implicit features, specifically encoding the video data at a high bit rate point to generate an implicit neural representation model at a high bit rate point; and optimizing the neural network weights through a back-propagation algorithm so that the model can accurately represent the video content at a high bit rate point.

3. The video variable bit rate coding method based on implicit neural representation model according to claim 1 is characterized in that: In step 1, the implicit neural representation model can be specifically divided into two categories: one is: encoding layer + embedding vector + decoding layer, and the other is: embedding vector + decoding layer; wherein, the encoding layer is used to convert the input data into a higher-level representation, capturing the structure and relationship of the data by extracting features; the embedding vector is used to map discrete inputs to a continuous vector space, so that similar inputs are closer in the space, for subsequent processing and learning; the decoding layer is used to convert the high-level representation generated by the encoding layer back to the original data format or the target format.

4. The video variable bit rate coding method based on implicit neural representation model according to claim 1, characterized in that: Step 2 includes: defining the size of the video as Where Param(·) represents the number of parameters, w l represents the weight of the lth layer, represents the quantization bit width of the weight of the lth layer, represents the quantization bit width of the embedding vector of the tth frame, e t represents the embedding vector of the tth frame, T represents the total number of video frames, and L represents the number of model layers; The process of calculating the quantization parameter combination is defined as R = S ± ε Where R represents the specified bit rate and ε represents the allowed bit rate error.

5. The video variable bit rate coding method based on implicit neural representation model according to claim 1, characterized in that: Step 3 includes: applying each quantization parameter combination to the implicit neural representation model and performing quantization sensitivity analysis on it. The model loss under each quantization parameter combination is expressed as: Where w represents the model weight, Δw represents the impact of quantization on the weight, and Δw T represents the transpose of Δw, represents the model loss, denote the first-order gradient and second-order Hessian matrix respectively; Therefore, the quantitative sensitivity is modeled as: in By comparing the quantitative sensitivity of the quantitative results under different combinations, the combination with the smallest quantitative sensitivity is identified.

6. The video variable bit rate coding method based on implicit neural representation model according to claim 1, characterized in that: Step 3 includes: in case of large differences in weights between different layers of the implicit neural representation model or even between different channels in the same layer, the same quantization bit width is applied between channels in the same layer, and all layer parameters are updated in one optimization process.

7. The video variable bit rate coding method based on implicit neural representation model according to claim 1, characterized in that: Step 3 includes: the process of calibrating the quantized model parameters is defined as a continuous optimization framework, and the optimization goal is: where v i represents the optimized continuous variable, s represents the proportional relationship, and x represents the model input. represents the output of the model when the input is x and the weight is w, h(v i ) is any differentiable function constrained in [0,1], argmin is the minimization process, λ represents the regularization weight, b represents the quantization bit width, s represents the quantization bit width, and clip(a,b,c) means limiting the value a to [b,c] and returning a value that is not less than b and not greater than c.

8. The video variable bit rate coding method based on implicit neural representation model according to claim 1, characterized in that: Step 4 includes: inputting the model parameters calibrated in step 3 into an entropy encoder, and the entropy encoder selects the optimal encoding method by analyzing the probability distribution of these parameters to compress the redundant information to a minimum.

9. A video variable bit rate encoding device based on an implicit neural representation model, characterized in that: The device includes: A training module is used to map the input video data into implicit features of the model parameters and encode the video at a high bit rate to train the implicit neural representation model; A quantization parameter calculation module is used to calculate possible quantization parameter combinations using an implicit neural representation model according to a specified target bit rate; A quantization sensitivity analysis module, used to calculate and analyze the quantization sensitivity of all quantization parameter combinations generated by the quantization parameter calculation module, and select the combination with the smallest quantization sensitivity for calibration; The entropy coding module is used to further compress the model parameters calibrated by the quantization sensitivity analysis module.

10. A computer program product, characterized in that It comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the encoding method as claimed in any one of claims 1 to 8.

Images