Video quantization coding method and device, equipment, storage medium and program product
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- MIGU COMIC CO LTD
- Filing Date
- 2024-10-31
- Publication Date
- 2026-05-01
AI Technical Summary
In existing end-to-end video coding models, the standard rounding quantization method results in insufficient adaptability of latent variable distribution, affecting the rate-distortion performance of video compression.
The input information is encoded and transformed by the first encoding neural network, and adaptive non-uniform quantization is performed by the encoding and decoding neural network of the super-prior link. The non-uniform quantization coefficients are obtained by entropy encoding and decoding, and finally adaptive non-uniform quantization of the latent variables is achieved.
It improves encoding quality under linear complexity, enhances rate-distortion performance of video compression, and significantly improves video compression results.
Smart Images
Figure CN121967692A_ABST
Abstract
Description
Video quantization encoding methods, devices, equipment, storage media, and software products Technical Field
[0001] This invention relates to the field of video processing technology, and in particular to a video quantization encoding method, apparatus, device, storage medium, and program product. Background Technology
[0002] During video storage and transmission, compression encoding is typically required to reduce storage capacity and transmission bandwidth. Currently, end-to-end video coding models are a common encoding and decoding scheme. However, existing end-to-end video coding models usually employ standard rounding quantization to discretize elements in the latent space. While this discretization facilitates subsequent entropy coding, standard rounding quantization, being a uniform quantization method, while simple and efficient, suffers from insufficient adaptability to real-world latent variable distributions, impacting the rate-distortion performance of video compression. Summary of the Invention
[0003] To address the problems existing in the prior art, embodiments of the present invention provide a video quantization encoding method, apparatus, device, storage medium, and program product. By adaptive non-uniform quantization of latent variables, it can effectively address the problem of insufficient adaptability of latent variable distribution, improve encoding quality under linear complexity, and enhance the rate-distortion performance of video compression.
[0004] In a first aspect, embodiments of the present invention provide a video quantization encoding method, comprising:
[0005] The input information is encoded and transformed using a first encoding neural network to obtain latent variables;
[0006] Based on the latent variables, a priori encoding and decoding transformation is performed through a priori link encoding and decoding neural network to adaptively non-uniformly quantize the latent variables.
[0007] Entropy encoding and decoding are performed on the latent variables after non-uniform quantization to obtain the latent variables after entropy decoding.
[0008] The hidden variables of entropy decoding are decoded and transformed by the first decoding neural network to obtain the decoded input information.
[0009] As an improvement to the above scheme, the encoding and decoding neural network includes: a second encoding neural network for the super-prior link and a second decoding neural network for the super-prior link;
[0010] The step of performing a priori encoding / decoding transformation on the latent variables using a priori link encoding / decoding neural network to adaptively non-uniformly quantize the latent variables includes:
[0011] The latent variables are transformed by a second coding neural network of the super-prior link to obtain super-prior information.
[0012] Entropy encoding and decoding are performed on the aforementioned prior information;
[0013] The non-uniform quantization coefficients are obtained by performing a priori decoding transformation on the entropy-decoded priori information through the second decoding neural network of the priori link.
[0014] Based on the non-uniform quantization coefficient, the latent variable is subjected to adaptive non-uniform quantization.
[0015] As an improvement to the above scheme, the second decoding neural network includes: multiple first activation functions, multiple downsampled convolutional layers, and multiple non-downsampled convolutional layers; wherein the downsampled convolutional layers and the non-downsampled convolutional layers are arranged alternately to form a convolutional layer network; the activation functions are arranged alternately with each convolutional layer in the convolutional layer network.
[0016] The step of performing a priori decoding transformation on the entropy-decoded priori information through the second decoding neural network of the priori link to obtain non-uniform quantization coefficients includes:
[0017] The entropy-decoded prior information is input into the second decoding neural network for prior decoding transformation to obtain non-uniform coefficients based on context information.
[0018] In the second decoding neural network, the entropy-decoded prior information is processed by the non-downsampling convolutional layer, the spatial size of the tensor is changed by the downsampling convolutional layer, and the first activation function is used alternately so that the output of the second decoding neural network is aligned with the size of the latent variable, and the non-uniform coefficients most suitable for the pixel are learned from the surrounding pixel context of a pixel.
[0019] The non-uniform coefficients are quantized to obtain non-uniform quantization coefficients.
[0020] As an improvement to the above scheme, the step of adaptively non-uniformly quantizing the latent variable based on the non-uniform quantization coefficient includes:
[0021] The first quantization interval and the first quantization center are calculated based on the non-uniform quantization coefficients and the preset tensor multiplier coefficients.
[0022] Based on the first quantization interval and the first quantization center, the latent variable is subjected to non-uniform quantization.
[0023] As an improvement to the above scheme, the tensor multiplier coefficients are set to 1.
[0024] As an improvement to the above scheme, the quantization of the non-uniform coefficient to obtain the non-uniform quantization coefficient includes:
[0025] The non-uniform coefficients are activated by a second activation function to obtain non-uniform activation coefficients;
[0026] The non-uniform activation coefficients are quantized into a preset set of values to obtain non-uniform quantization coefficients.
[0027] Secondly, embodiments of the present invention provide a video quantization encoding apparatus, comprising:
[0028] The encoding transformation module is used to encode and transform the input information through the first encoding neural network to obtain latent variables;
[0029] The non-uniform quantization module is used to perform a priori encoding and decoding transformation on the latent variables through a priori link encoding and decoding neural network to perform adaptive non-uniform quantization on the latent variables.
[0030] The entropy encoding / decoding module is used to entropy encode / decode the latent variables after non-uniform quantization to obtain the latent variables after entropy decoding.
[0031] The decoding transformation module is used to decode and transform the latent variables of entropy decoding through the first decoding neural network to obtain the decoded input information.
[0032] Thirdly, embodiments of the present invention provide a video quantization encoding device, comprising: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the video quantization encoding method as described in any one of the first aspects.
[0033] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform a video quantization encoding method as described in any one of the first aspects.
[0034] Fifthly, embodiments of the present invention provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the video quantization encoding method as described in any one of the first aspects.
[0035] Compared to existing technologies, the present invention provides a video quantization encoding method, apparatus, device, storage medium, and program product. It uses a first encoding neural network to encode and transform input information to obtain latent variables. Then, based on the latent variables, it performs a priori encoding and decoding transformation using a super-prior link encoding and decoding neural network to adaptively non-uniformly quantize the latent variables. The non-uniformly quantized latent variables are then subjected to entropy encoding and decoding to obtain entropy-decoded latent variables. Finally, the entropy-decoded latent variables are decoded using a first decoding neural network to obtain the decoded input information. This invention, through adaptive non-uniform quantization of latent variables, effectively addresses the problem of insufficient adaptability in latent variable distribution, improves encoding quality under linear complexity, and enhances the rate-distortion performance of video compression. Attached Figure Description
[0036] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0037] Figure 1 is a flowchart of a video quantization encoding method provided in an embodiment of the present invention;
[0038] Figure 2 is a schematic diagram of an end-to-end coding framework with non-uniform quantization provided in an embodiment of the present invention;
[0039] Figure 3 is a schematic diagram of the network structure of the second decoding neural network of the super-prior link provided in an embodiment of the present invention;
[0040] Figure 4 is a schematic diagram of the non-uniform quantization process provided in an embodiment of the present invention;
[0041] Figure 5 is a schematic diagram of an example of the video quantization encoding process provided in an embodiment of the present invention;
[0042] Figure 6 is a structural block diagram of a video quantization encoding device provided in an embodiment of the present invention;
[0043] Figure 7 is a structural block diagram of a video quantization encoding device provided in an embodiment of the present invention. Detailed Implementation
[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0045] It is understood that the various numerical designations used in the embodiments of this invention are merely for descriptive convenience and are not intended to limit the scope of this application. The order of the process numbers does not imply the order of execution; the execution order of each process should be determined by its function and internal logic.
[0046] In embodiments of the invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The term "a plurality or several" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.
[0047] Please refer to Figure 1, which is a flowchart of a video quantization encoding method provided by an embodiment of the present invention. The video quantization encoding method specifically includes:
[0048] S11: The input information is encoded and transformed using the first encoding neural network to obtain latent variables;
[0049] S12: Based on the latent variables, perform a priori encoding and decoding transformation through a priori link encoding and decoding neural network to adaptively non-uniformly quantize the latent variables;
[0050] S13: Perform entropy encoding and decoding on the latent variables after non-uniform quantization to obtain the latent variables after entropy decoding;
[0051] S14: The hidden variables of entropy decoding are decoded and transformed by the first decoding neural network to obtain the decoded input information.
[0052] It should be noted that the embodiments of the present invention are applied to end-to-end video compression scenarios and can be executed by terminal devices. The terminal devices can be mobile phones, tablets, computers, servers, etc., and are not specifically limited in the embodiments of the present invention.
[0053] The input information may be images, video sequences, or other similar information, and is not specifically limited in this embodiment of the invention.
[0054] This invention, through a priori encoding / decoding transformation of latent variables in input information (such as video sequences), introduces context-based adaptive non-uniform quantization. This achieves the use of contextual information from video textures to adaptively select the non-uniform quantization codebook for the current element using encoded information as context, without compromising the linear complexity of scalar quantization. This enables vector quantization of latent variables in end-to-end video coding with low-complexity adaptive codebooks, effectively addressing the problem of insufficient latent variable distribution adaptability, improving coding quality under linear complexity, and thus significantly enhancing the rate-distortion performance of video compression.
[0055] Specifically, S13: Perform entropy encoding and decoding on the latent variables after non-uniform quantization to obtain the latent variables after entropy decoding, including:
[0056] Entropy encoding is performed on the latent variables after non-uniform quantization to obtain the encoded video stream;
[0057] The video stream is entropy decoded to obtain the latent variables of the entropy decoding.
[0058] Specifically, the encoding / decoding neural network includes: a second encoding neural network for the super-prior link and a second decoding neural network for the super-prior link;
[0059] Step S12: Based on the latent variables, perform a priori encoding / decoding transformation using a priori link encoding / decoding neural network to adaptively non-uniformly quantize the latent variables, including:
[0060] The latent variables are transformed by a second coding neural network of the super-prior link to obtain super-prior information.
[0061] Entropy encoding and decoding are performed on the aforementioned prior information;
[0062] The non-uniform quantization coefficients are obtained by performing a priori decoding transformation on the entropy-decoded priori information through the second decoding neural network of the priori link.
[0063] Based on the non-uniform quantization coefficient, the latent variable is subjected to adaptive non-uniform quantization.
[0064] As shown in Figure 2, the end-to-end coding framework incorporates non-uniform quantization. The first coding neural network is network g in Figure 2. a The first decoding neural network is network g in Figure 2. s The second encoding neural network of the hyperprior link is network h in Figure 2. aThe second decoding neural network of the hyperprior link is the network h in Figure 2 that obtains the non-uniform quantization coefficient β. s First, the input information is fed into the first encoding neural network g. a The encoding transformation is performed to obtain the latent variables of the input information. These latent variables are output to a non-uniform quantization layer, awaiting non-uniform quantization; and simultaneously fed into the second encoding neural network h of the prior knowledge link. a A super-prior coding transformation is performed to obtain super-prior information, which is then fed into an entropy encoding / decoding layer for entropy encoding and decoding. The entropy-decoded super-prior information is then fed into a second decoding neural network of the super-prior link for super-prior decoding transformation to obtain non-uniform quantization coefficients β based on context information. These non-uniform quantization coefficients β are then fed into a non-uniform quantization layer, and adaptive non-uniform quantization is performed on the latent variables based on these coefficients, achieving low-complexity adaptive codebook vector quantization of the latent variables in end-to-end video encoding. The non-uniformly quantized latent variables are then fed into another entropy encoding / decoding layer. In this layer, entropy encoding is first performed on the non-uniformly quantized latent variables to obtain the encoded video stream. During decoding, entropy decoding is performed on the video stream to obtain the entropy-decoded latent variables. Finally, the entropy-decoded latent variables are fed into a first decoding neural network for decoding transformation to obtain the decoded input information, completing the end-to-end video encoding.
[0065] In this embodiment of the invention, after entropy encoding and decoding the obtained prior information, the second decoding neural network h of the prior link is used. s The prior information obtained from entropy decoding is subjected to a priori decoding transformation to obtain non-uniform quantization coefficients β based on context information. Then, the latent variables are adaptively non-uniformly quantized using the non-uniform quantization coefficients β. After that, the non-uniformly quantized latent variables are subjected to entropy encoding and decoding to obtain the entropy-decoded latent variables. The entropy-decoded latent variables are then input into the decoding neural network g. s Obtain the input information for decoding. By applying the super-prior decoding network to the generation of non-uniform quantization coefficients, the remaining encoder-decoder neural networks can all adopt existing self-encoders, with the first encoding neural network g... a First decoding neural network g s The second encoding neural network h of the hyperprior link a The second decoding neural network h of the hyperprior link s In this end-to-end coding framework, all networks are optimized synchronously during training, eliminating the need for staged pre-training of specific parts of the network. This improves performance of newly added modules (including the second coding neural network h of the super-prior link). a The second decoding neural network h of the hyperprior link s ) and the original module (including the first encoding neural network g) aFirst decoding neural network g s High degree of harmony.
[0066] The second decoding neural network includes: multiple first activation functions, multiple downsampled convolutional layers, and multiple non-downsampled convolutional layers; wherein the downsampled convolutional layers and the non-downsampled convolutional layers are arranged alternately to form a convolutional layer network; the activation functions are arranged alternately with each convolutional layer in the convolutional layer network.
[0067] The step of performing a priori decoding transformation on the entropy-decoded priori information through the second decoding neural network of the priori link to obtain non-uniform quantization coefficients includes:
[0068] The entropy-decoded prior information is input into the second decoding neural network for prior decoding transformation to obtain non-uniform coefficients based on context information.
[0069] In the second decoding neural network, the entropy-decoded prior information is processed by the non-downsampling convolutional layer, the spatial size of the tensor is changed by the downsampling convolutional layer, and the first activation function is used alternately so that the output of the second decoding neural network is aligned with the size of the latent variable, and the non-uniform coefficients most suitable for the pixel are learned from the surrounding pixel context of a pixel.
[0070] The non-uniform coefficients are quantized to obtain non-uniform quantization coefficients.
[0071] It should be noted that the embodiments of the present invention do not specifically limit the number of the first activation function, the downsampled convolutional layer, and the non-downsampled convolutional layer, nor do the embodiments of the present invention specifically limit the function used for the first activation function.
[0072] In this embodiment of the invention, the number of downsampled convolutional layers is less than the number of non-downsampled convolutional layers, and the number of non-downsampled convolutional layers is less than the number of first activation functions. In the second decoding neural network of the hyper-prior link, multiple first activation functions, multiple downsampled convolutional layers, and multiple non-downsampled convolutional layers can be arranged alternately in the manner of one non-downsampled convolutional layer and one downsampled convolutional layer, with the first activation functions interspersed between the non-downsampled convolutional layers and the downsampled convolutional layers. For example, the second decoding neural network of the hyper-prior link can be arranged sequentially in the order of "one non-downsampled convolutional layer + one first activation function + one downsampled convolutional layer + one first activation function..." to form a network structure in which convolutional layers and activation functions are arranged alternately.
[0073] In some embodiments, the second decoding neural network of the hyper-prior link may include four first activation functions, two downsampled convolutional layers, and three non-downsampled convolutional layers. The first activation function may be the Leaky ReLU activation function. The network structure of the second decoding neural network of the hyper-prior link is shown in Figure 3. The two downsampled convolutional layers have the same convolutional kernel of 128×128×2, while the three non-downsampled convolutional layers (also called ordinary convolutional layers) have different convolutional kernels of 64×128, 128×64, and 128×128, respectively. Therefore, the network structure of the second decoding neural network of the hyper-prior link is: a 64×128 non-downsampled convolutional layer + a Leaky ReLU activation function + a 128×128×2 downsampled convolutional layer + a Leaky ReLU activation function + a 128×64 non-downsampled convolutional layer + a Leaky ReLU activation function. ReLU activation function + 128×128×2 downsampled convolutional layer + LeakyReLU activation function + 128×128 non-downsampled convolutional layer.
[0074] It should be understood that in training large-scale natural video data, convolutional layers can better capture the spatial neighborhood information of pixels, thereby learning the most suitable non-uniform coefficients for a pixel from the context of its surrounding pixels. In this embodiment of the invention, in the second decoding neural network of the super-prior link, a non-downsampled convolutional layer (i.e., a regular convolutional layer) is first applied to process the super-prior signal input to the second decoding neural network of the super-prior link. The Leaky ReLU activation function is used alternately to increase the non-linearity of the network, and three downsampled convolutional layers are added in the middle to change the spatial size of the tensor, so that the final output of the second decoding neural network of the super-prior link is aligned with the size of the latent variable, and the most suitable non-uniform coefficients for a pixel are learned from the context of its surrounding pixels. Then, the non-uniform coefficients output by the second decoding neural network of the super-prior link are quantized to obtain the final non-uniform quantization coefficients. This embodiment of the invention can adaptively select the non-uniform quantization codebook of the current element using the encoded information as context without destroying the linear complexity of scalar quantization, thereby achieving context-based adaptive non-uniform quantization and improving the coding quality under linear complexity.
[0075] Specifically, quantizing the non-uniform coefficients to obtain non-uniform quantization coefficients includes:
[0076] The non-uniform coefficients are activated by a second activation function to obtain non-uniform activation coefficients;
[0077] The non-uniform activation coefficients are quantized into a preset set of values to obtain non-uniform quantization coefficients.
[0078] It should be noted that the embodiments of the present invention do not specifically limit the function used for the second activation function. The second activation function is used to map the non-uniform coefficients to around 1. For example, the second activation function is g(x) = exp(0.01x).
[0079] For example, for the non-uniform coefficient x output by the second decoding neural network of the super-prior link, the second activation function g(x) = exp(0.01x) is used for activation. The second activation function maps the non-uniform parameter x to near 1, preventing negative impacts such as training instability due to the excessively large exponent of the power function. Then, the non-uniform activation coefficient g(x) is quantized to a preset set of values, such as {g(-2), g(-1), g(0), g(1), g(2)}, to obtain the final non-uniform quantization coefficient β. If the output non-uniform coefficient is 1.7, β = g(1.7), β = g(1.7) is quantized to the five closest values g(-2), g(-1), g(0), g(1), g(2), and the CDF table is preprocessed for each value of the final quantized non-uniform quantization coefficient β.
[0080] It should be noted that the embodiments of the present invention do not specifically limit the quantization method of non-uniform coefficients. For example, the ste_round method can be used to quantize non-uniform coefficients. The ste_round method refers to using ordinary rounding quantization operation for quantization in both training and testing. However, in training, the gradient of the rounding quantization operation is set to 1 so that the gradient will not vanish during the backpropagation of the neural network.
[0081] It should be understood that end-to-end video compression differs from image compression. In end-to-end video compression scenarios, due to the massive amount of video data, existing end-to-end video compression schemes typically require preprocessing of the CDF table needed for entropy coding. It should be noted that CDF table preprocessing can also be called CDF table pre-computation, and the two terms can be used interchangeably.
[0082] CDF table preprocessing refers to pre-calculating the cumulative probability density function (CDD) of all possible Gaussian distribution standard deviations σ and possible integer characters u under a Gaussian distribution, thus avoiding probability calculations for each latent variable element encoded. In scenarios with large amounts of video data, CDF table preprocessing effectively reduces computational complexity. Since the probability of a character in non-uniform quantization is no longer the integral of a Gaussian distribution over an integer neighborhood, the direct introduction of non-uniform quantization would disrupt this conventional CDF preprocessing practice. Therefore, to avoid disrupting existing CDF table preprocessing due to the direct introduction of non-uniform quantization, this embodiment of the invention quantizes the non-uniform coefficients to take values from a finite number of predefined discrete values, and then performs CDF table preprocessing on each value of the resulting non-uniform quantization coefficient β, achieving compatibility with CDF table preprocessing in end-to-end video coding.
[0083] Specifically, the step of performing adaptive non-uniform quantization on the latent variable based on the non-uniform quantization coefficient includes:
[0084] The first quantization interval and the first quantization center are calculated based on the non-uniform quantization coefficients and the preset tensor multiplier coefficients.
[0085] Based on the first quantization interval and the first quantization center, the latent variable is subjected to non-uniform quantization.
[0086] Further, the step of calculating the first quantization interval and the first quantization center based on the non-uniform quantization coefficient includes:
[0087] Based on the non-uniform quantization coefficients and the preset tensor multiplier coefficients, calculate the second quantization interval of each element in the latent variable and the second quantization center of the corresponding second quantization interval;
[0088] The first quantization interval is determined based on the second quantization interval of all elements in the latent variable;
[0089] The first quantization center is determined based on the second quantization center of all elements in the latent variable.
[0090] In some embodiments, the tensor multiplier coefficients may be set to 1 to achieve compatibility with existing end-to-end video coding.
[0091] For example, suppose the i-th element of the latent variable y is y i For element y i Performing non-uniform quantization involves decisions about the quantization interval and the quantization center.
[0092] Define the i-th element as y i The first quantization interval is B i，± Then we have:
[0093] B i，± ={b i，j± |j∈Z} (1);
[0094] Among them, b i，j± Represents the i-th element y i The left and right boundaries of the j-th second quantization interval; Z represents the number of elements in the latent variable y;
[0095] Define the i-th element as y i The first quantization center is C i Then we have:
[0096] C i ={c i，j |j∈Z} (2);
[0097] Among them, c i，j It represents the second quantization center of the second quantization interval of the i-th element and the j-th element.
[0098] This invention achieves nonlinearity based on a power function prior, where the i-th element y... i The left and right boundaries b of the j-th second quantization interval i，j± The second quantization center c of the j-th second quantization interval of the i-th element i，j Specifically, it can be modeled as the following function:
[0099]
[0100]
[0101] Where sgn() represents the sign function, β i Represents the i-th element y i Non-uniform quantization coefficient, β i >0, β i These are the parameters learned and quantized through the second decoding neural network of the aforementioned hyperprior link; α i Represents the i-th element y i Tensor-type multiplier coefficients.
[0102] In current end-to-end video coding, a tensor multiplier is typically learned using prior information and applied to latent variables. This is because it inherently possesses coefficients α. i Therefore, in this embodiment of the invention, the tensor multiplier coefficient α is used to achieve the desired effect. i Set to 1 to achieve the effect of learning a tensor multiplier using advanced prior information in existing end-to-end video coding and applying it to latent variables.
[0103] To ensure the sharing of non-uniform quantization parameters at the encoding and decoding ends, this embodiment of the invention uses quantized prior information. To generate the non-uniform quantization parameter β, i.e. Where F() represents the second decoding neural network h of the aforementioned prior link. s The function expression.
[0104] The above non-uniform quantization operation performs non-uniform quantization on each element of the latent variable, which has linear complexity. At the same time, since it is based on the context-based calculation of the non-uniform quantization paradigm, it has strong adaptability to the source distribution and can effectively address the problem of insufficient adaptability of latent variable distribution.
[0105] Non-uniform quantization can be introduced into motion vector compression and residual compression in end-to-end video coding. Compared with existing technologies that only apply non-uniform quantization to the compression of single-frame images, it can realize vector quantization of latent variables in end-to-end video coding with low complexity adaptive codebook, which significantly improves the rate-distortion performance of video compression.
[0106] For example, as illustrated in Figure 4, the non-uniform quantization process maps the latent space (i.e., latent variables, shown on the horizontal axis in Figure 4) to another space (shown on the vertical axis in Figure 4) using a non-linear characteristic function f(y). After rounding and quantizing in the latter space, the inverse function of the non-linear characteristic function f(y) is used to map it back to the latent space. Y takes the values of elements of the latent variables, such as y0 and y1. After the non-uniform quantization, the elements in the latent interval (y0, y1) are quantized to y′0.
[0107] In this embodiment of the invention, a parameterized power function is used as the nonlinear characteristic function f(y), then:
[0108]
[0109] Where, β i The i-th element y of the latent variable is obtained by learning and quantizing the second decoding neural network of the aforementioned prior link. i Non-uniform quantization coefficient; α i =1.
[0110] Taking a video sequence as the input information as an example, and combining the end-to-end coding framework of non-uniform quantization shown in Figure 2, the quantization coding process of video is explained in detail, as shown in Figure 5. The specific process includes:
[0111] Step a: Use the first encoding neural network g a The input video sequence is encoded and transformed to obtain latent variables.
[0112] Step b: Use the second encoding neural network h of the super-prior link a Perform a priori encoding transformation on the latent variables obtained in step a to obtain prior information.
[0113] Step c: After entropy encoding and decoding of the prior information obtained in step b, use the second decoding neural network h of the prior link. s A hyperprior decoding transformation is performed on the hyperprior information obtained from entropy decoding to obtain non-uniform quantization coefficients β based on context information. It should be noted that the second decoding neural network h in this hyperprior link... s This is the network for obtaining non-uniform quantization coefficients, as shown in Figure 3.
[0114] Step d: Use the non-uniform quantization coefficient β to perform adaptive non-uniform quantization on the latent variables; that is, based on the non-uniform quantization coefficient β, use the modeling method shown in the above formula (1-4) to calculate the quantization interval (including the first quantization interval and the second quantization interval) and quantization center (including the first quantization center and the second quantization center) of each element of the latent variables.
[0115] Step e: Entropy encoding is performed on the latent variables after non-uniform quantization to obtain the encoded video stream. During the decoding process, entropy decoding is performed on the video stream to obtain the decoded latent variables.
[0116] Step f: Input the latent variables of entropy decoding into the first decoding neural network g. s , thus obtaining the decoded video sequence.
[0117] Compared with the prior art, the beneficial effects of the embodiments of the present invention are as follows:
[0118] By performing a priori coding transformation on the latent variables of input information (such as images and video sequences), priori information is obtained. Then, entropy coding and decoding are performed on the priori information. The second decoding neural network of the priori link performs a priori decoding transformation on the entropy-decoded priori information to obtain non-uniform quantization coefficients based on context information. Subsequently, the non-uniform quantization coefficients are used to perform adaptive non-uniform quantization on the latent variables. This achieves low-complexity adaptive codebook vector quantization of the latent variables of end-to-end video coding by utilizing the context information of video texture, improving coding quality under linear complexity, and thus significantly improving the rate-distortion performance of video compression.
[0119] In the process of performing a priori decoding transformation on the priori information of entropy decoding through the second decoding neural network of the priori link, the second decoding neural network of the priori link includes multiple activation functions, downsampling convolutional layers and non-downsampling convolutional layers, which increases the nonlinearity of the network, aligns the final output with the size of the latent variable, and learns the most suitable non-uniform coefficient of the pixel from the surrounding pixel context.
[0120] The output of the second neural network of the super-prior link is activated using the activation function g(x) = exp(0.01x) and quantized to multiple selected values. The values of each quantized non-uniform quantization coefficient are preprocessed using a CDF table, which achieves compatibility with CDF table pre-computation in end-to-end video coding.
[0121] In latent variable quantization modeling, the tensor multiplier coefficient is fixed at 1, realizing the effect of learning a tensor multiplier using advanced prior information in existing end-to-end video coding and applying it to latent variables.
[0122] Furthermore, compared to conventional scalar quantization, which is unable to adapt to varying source distributions due to its fixed quantization paradigm and has exponential complexity and memory consumption, the embodiments of this invention adopt an element-wise quantization method for latent variables, which has linear complexity. At the same time, because it is based on context-based computation of a non-uniform quantization paradigm, it has strong adaptability to source distributions and can effectively address the problem of insufficient adaptability of latent variable distributions.
[0123] Referring to Figure 6, which is a structural block diagram of a video quantization encoding device provided in an embodiment of the present invention, the video quantization encoding device includes:
[0124] The encoding transformation module 11 is used to encode and transform the input information through the first encoding neural network to obtain latent variables;
[0125] The non-uniform quantization module 12 is used to perform a priori encoding and decoding transformation on the latent variables through a priori link encoding and decoding neural network to perform adaptive non-uniform quantization on the latent variables.
[0126] Entropy encoding / decoding module 13 is used to entropy encode / decode the latent variables after non-uniform quantization to obtain the latent variables after entropy decoding.
[0127] The decoding transformation module 14 is used to perform decoding transformation on the latent variables of entropy decoding through the first decoding neural network to obtain the decoded input information.
[0128] In an optional embodiment, the entropy encoding / decoding module 13 includes:
[0129] Entropy coding unit is used to entropy code the latent variables after non-uniform quantization to obtain the encoded video stream;
[0130] The entropy decoding unit is used to perform entropy decoding on the video stream to obtain the latent variables of the entropy decoding.
[0131] In one optional embodiment, the encoding / decoding neural network includes: a second encoding neural network for the super-prior link and a second decoding neural network for the super-prior link;
[0132] The non-uniform quantization module 12 includes:
[0133] The advanced prior coding transformation unit is used to perform advanced prior coding transformation on the latent variables through the second coding neural network of the advanced prior link to obtain advanced prior information;
[0134] An entropy encoding / decoding unit is used to entropy encode / decode the aforementioned prior information;
[0135] The super-prior decoding transformation unit is used to perform super-prior decoding transformation on the entropy-decoded super-prior information through the second decoding neural network of the super-prior link to obtain non-uniform quantization coefficients.
[0136] The non-uniform quantization operation unit is used to perform adaptive non-uniform quantization on the latent variable according to the non-uniform quantization coefficient.
[0137] In one optional embodiment, the second decoding neural network includes: a plurality of first activation functions, a plurality of downsampled convolutional layers, and a plurality of non-downsampled convolutional layers; wherein the downsampled convolutional layers and the non-downsampled convolutional layers are arranged alternately to form a convolutional layer network; the activation functions are arranged alternately with each convolutional layer in the convolutional layer network;
[0138] The advanced priori decoding transform unit includes:
[0139] The super-prior decoding transformation subunit is used to input the entropy-decoded super-prior information into the second decoding neural network to perform super-prior decoding transformation and obtain non-uniform coefficients based on context information.
[0140] In the second decoding neural network, the entropy-decoded prior information is processed by the non-downsampling convolutional layer, the spatial size of the tensor is changed by the downsampling convolutional layer, and the first activation function is used alternately so that the output of the second decoding neural network is aligned with the size of the latent variable, and the non-uniform coefficients most suitable for the pixel are learned from the surrounding pixel context of a pixel.
[0141] The first coefficient quantization subunit is used to quantize the non-uniform coefficients to obtain non-uniform quantized coefficients.
[0142] In one optional embodiment, the non-uniform quantization operation unit includes:
[0143] The quantization interval center calculation subunit is used to calculate the first quantization interval and the first quantization center based on the non-uniform quantization coefficients and the preset tensor multiplier coefficients.
[0144] The latent variable non-uniform quantization subunit is used to perform non-uniform quantization operation on the latent variable based on the first quantization interval and the first quantization center.
[0145] In one optional embodiment, the quantization interval center calculation subunit includes:
[0146] The quantization calculation subunit is used to calculate the second quantization interval of each element in the latent variable and the second quantization center of the corresponding second quantization interval based on the non-uniform quantization coefficient and the preset tensor multiplier coefficient.
[0147] The first quantization interval determination sub-unit is used to determine the first quantization interval based on the second quantization interval of all elements in the latent variable;
[0148] The first quantization center determination sub-unit is used to determine the first quantization center based on the second quantization centers of all elements in the latent variable.
[0149] In an optional embodiment, the tensor multiplier coefficients are set to 1.
[0150] In one optional embodiment, the hidden variable non-uniform quantization subunit includes:
[0151] The coefficient activation subunit is used to activate the non-uniform coefficients through a second activation function to obtain non-uniform activation coefficients.
[0152] The second coefficient quantization subunit is used to quantize the non-uniform activation coefficients into a preset value set to obtain non-uniform quantization coefficients.
[0153] It should be noted that the working process of each module in the video quantization encoding device described in the embodiments of the present invention can refer to the working process of the video quantization encoding method described in the above embodiments, and the technical effect achieved is the same as that of the video quantization encoding method described in the above embodiments, so it will not be repeated here.
[0154] Referring to Figure 7, which is a structural block diagram of a video quantization encoding device provided in an embodiment of the present invention, the video quantization encoding device includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, it implements the steps in the above-described embodiments of the video quantization encoding methods, such as step S11: encoding and transforming the input information through a first encoding neural network to obtain latent variables; step S12: performing a priori encoding and decoding transformation on the latent variables through a priori link encoding and decoding neural network to adaptively non-uniformly quantize the latent variables; step S13: performing entropy encoding and decoding on the non-uniformly quantized latent variables to obtain entropy-decoded latent variables; and step S14: performing decoding transformation on the entropy-decoded latent variables through a first decoding neural network to obtain decoded input information.
[0155] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, the instruction segments describing the execution process of the computer program in the video quantization encoding device.
[0156] The video quantization encoding device may include, but is not limited to, processor 21 and memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of a video quantization encoding device and does not constitute a limitation on the video quantization encoding device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the video quantization encoding device may also include input / output devices, network access devices, buses, etc.
[0157] The processor 21 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or any conventional processor. The processor 21 is the control center of the video quantization encoding device, connecting various parts of the entire video quantization encoding device through various interfaces and lines.
[0158] The memory 22 can be used to store the computer program and / or modules. The processor 21 implements various functions of the video quantization encoding device by running or executing the computer program and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.
[0159] Wherein, if the module / unit integrated into the video quantization encoding device is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor 21, it can implement the steps of the above-described method embodiments. Wherein, the computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.
[0160] It should be noted that the working process of each module in the video quantization encoding device described in the embodiments of the present invention can refer to the working process of the video quantization encoding method described in the above embodiments, and the technical effect achieved is the same as that of the video quantization encoding method described in the above embodiments, which will not be repeated here.
[0161] The present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the steps in the above-described embodiments of the video quantization encoding methods.
[0162] Another computer program product provided by this invention includes a computer program / instruction, which, when executed by a processor, implements the steps in the above-described embodiments of the video quantization encoding methods.
[0163] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.
[0164] The above description is a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, many improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.
Claims
1. A video quantization encoding method, characterized in that, include: The input information is encoded and transformed using a first encoding neural network to obtain latent variables; Based on the latent variables, a priori encoding and decoding transformation is performed through a priori link encoding and decoding neural network to adaptively non-uniformly quantize the latent variables; entropy encoding and decoding are performed on the non-uniformly quantized latent variables to obtain entropy-decoded latent variables; and the entropy-decoded latent variables are decoded through a first decoding neural network to obtain the decoded input information.
2. The video quantization encoding method as described in claim 1, characterized in that, The encoding / decoding neural network includes: a second encoding neural network for a priori links and a second decoding neural network for a priori links; the step of performing a priori encoding / decoding transformation on the latent variables through the encoding / decoding neural network for a priori links to perform adaptive non-uniform quantization on the latent variables includes: performing a priori encoding transformation on the latent variables through the second encoding neural network for a priori links to obtain priori information; performing entropy encoding / decoding on the priori information; performing a priori decoding transformation on the entropy-decoded priori information through the second decoding neural network for a priori links to obtain non-uniform quantization coefficients; and performing adaptive non-uniform quantization on the latent variables based on the non-uniform quantization coefficients.
3. The video quantization encoding method as described in claim 2, characterized in that, The second decoding neural network includes: multiple first activation functions, multiple downsampled convolutional layers, and multiple non-downsampled convolutional layers; wherein the downsampled convolutional layers and the non-downsampled convolutional layers are alternately arranged to form a convolutional layer network; the activation functions are alternately arranged with each convolutional layer in the convolutional layer network; the step of performing a priori decoding transformation on the entropy-decoded prior information through the second decoding neural network via the prior link to obtain non-uniform quantization coefficients includes: inputting the entropy-decoded prior information into the second decoding neural network for a priori decoding transformation to obtain non-uniform coefficients based on context information; wherein, in the second decoding neural network, the entropy-decoded prior information is processed by the non-downsampled convolutional layers, the spatial size of the tensor is changed by the downsampled convolutional layers, and the first activation functions are used alternately to align the output of the second decoding neural network with the size of the latent variable, and the non-uniform coefficients most suitable for the pixel are learned from the surrounding pixel context of a pixel; the non-uniform coefficients are quantized to obtain non-uniform quantization coefficients.
4. The video quantization encoding method as described in claim 2, characterized in that, The step of performing adaptive non-uniform quantization on the latent variable based on the non-uniform quantization coefficient includes: calculating a first quantization interval and a first quantization center based on the non-uniform quantization coefficient and preset tensor multiplier coefficients; and performing non-uniform quantization operation on the latent variable based on the first quantization interval and the first quantization center.
5. The video quantization encoding method as described in claim 4, characterized in that, The tensor multiplier coefficients are set to 1.
6. The video quantization encoding method as described in claim 3, characterized in that, The step of quantizing the non-uniform coefficients to obtain non-uniform quantization coefficients includes: activating the non-uniform coefficients through a second activation function to obtain non-uniform activation coefficients; and quantizing the non-uniform activation coefficients into a preset set of values to obtain non-uniform quantization coefficients.
7. A video quantization encoding device, characterized in that, include: The encoding transformation module is used to encode and transform the input information through the first encoding neural network to obtain latent variables; The non-uniform quantization module is used to perform a priori encoding and decoding transformation on the latent variables through a priori link encoding and decoding neural network to adaptively non-uniformly quantize the latent variables; the entropy encoding and decoding module is used to perform entropy encoding and decoding on the non-uniformly quantized latent variables to obtain the entropy-decoded latent variables. The decoding transformation module is used to decode and transform the latent variables of entropy decoding through the first decoding neural network to obtain the decoded input information.
8. A video quantization encoding device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the video quantization encoding method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the video quantization encoding method as described in any one of claims 1 to 6.
10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the video quantization encoding method according to any one of claims 1 to 6.