Decoding method, device and equipment

Abstract

The invention provides a decoding method, device and equipment. The method comprises the following steps: decoding a target feature corresponding to a current image block from a code stream corresponding to the current image block; determining a first input feature of the target decoding network based on the target feature; obtaining a target eigenvalue quantization bit width and a target eigenvalue quantization super parameter of the target decoding network, and converting the first input feature into a second input feature based on the target eigenvalue quantization bit width and the target eigenvalue quantization super parameter; processing the second input feature based on the fixed-point weight of the target decoding network to obtain an output feature of the target decoding network; wherein the fixed-point weight is determined based on the target weight quantization bit width and the target weight quantization super parameter; and determining a reconstructed image block corresponding to the current image block based on the output characteristics of the target decoding network. Through the technical scheme of the invention, the decoding calculation amount can be reduced on the premise of ensuring the decoding quality.

Description

Technical Field

[0001] The present application relates to the field of coding and decoding technology, and in particular to a decoding method, device and equipment thereof. Background Art

[0002] In order to save space, video images are transmitted after being encoded. Complete video encoding can include prediction, transformation, quantization, entropy coding, filtering and other processes. For the prediction process, the prediction process can include intra-frame prediction and inter-frame prediction. Inter-frame prediction refers to the use of the correlation in the video time domain to use the pixels of the adjacent encoded images to predict the current pixel, so as to effectively remove the redundancy in the video time domain. Intra-frame prediction refers to the use of the correlation in the video space domain to use the pixels of the encoded blocks of the current frame image to predict the current pixel, so as to remove the redundancy in the video space domain.

[0003] With the rapid development of deep learning technology, deep learning technology has achieved success in many high-level computer vision problems, such as image classification and target detection. Deep learning technology has also gradually begun to be applied in the field of encoding and decoding, that is, neural networks can be used to encode and decode images. Although the encoding and decoding methods based on neural networks have shown great performance potential, they still have problems such as poor decoding performance and high complexity. Summary of the invention

[0004] In view of this, the present application provides a decoding method, apparatus and device thereof to improve decoding performance and reduce decoding complexity.

[0005] The present application provides a decoding method, the method comprising:

[0006] Decoding a target feature corresponding to the current image block from a bitstream corresponding to the current image block;

[0007] Determine a first input feature of a target decoding network based on the target feature;

[0008] Obtain a target eigenvalue quantization bit width and a target eigenvalue quantization hyperparameter of a target decoding network, and convert the first input feature into a second input feature based on the target eigenvalue quantization bit width and the target eigenvalue quantization hyperparameter;

[0009] Processing the second input feature based on the fixed-point weight of the target decoding network to obtain the output feature of the target decoding network; wherein the fixed-point weight is determined based on the target weight quantization bit width and the target weight quantization hyperparameter;

[0010] A reconstructed image block corresponding to the current image block is determined based on the output features of the target decoding network.

[0011] The present application provides a decoding device, the device comprising:

[0012] A decoding module, used for decoding a target feature corresponding to a current image block from a code stream corresponding to the current image block;

[0013] A determination module, configured to determine a first input feature of a target decoding network based on the target feature;

[0014] A processing module, used to obtain a target feature value quantization bit width and a target feature value quantization hyperparameter of a target decoding network, convert the first input feature into a second input feature based on the target feature value quantization bit width and the target feature value quantization hyperparameter; process the second input feature based on a fixed-point weight of the target decoding network to obtain an output feature of the target decoding network; wherein the fixed-point weight is determined based on a target weight quantization bit width and a target weight quantization hyperparameter;

[0015] The determination module is used to determine the reconstructed image block corresponding to the current image block based on the output characteristics of the target decoding network.

[0016] The present application provides a decoding end device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

[0017] The processor is used to execute machine executable instructions to implement the above decoding method.

[0018] The present application provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the above-mentioned decoding method is implemented.

[0019] It can be seen from the above technical scheme that in the embodiment of the present application, the target feature corresponding to the current image block is decoded from the code stream corresponding to the current image block, the first input feature of the target decoding network is determined based on the target feature, the first input feature is converted into a second input feature based on the target feature value quantization bit width and the target feature value quantization hyperparameter, the second input feature is processed based on the fixed-point weight of the target decoding network to obtain the output feature of the target decoding network, and the reconstructed image block corresponding to the current image block is determined based on the output feature, thereby proposing an end-to-end video image compression method, which can realize the decoding of video images based on the decoding network, and achieve the purpose of improving the encoding efficiency and decoding efficiency. The decoding network with fixed-point weights is constructed by using the target weight quantization bit width and the target weight quantization hyperparameter, and the target feature value quantization bit width and the target feature value quantization hyperparameter are used to generate fixed-point input features, so as to realize adaptive decoding acceleration and reduce the decoding calculation amount while ensuring the decoding quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a schematic diagram of a three-dimensional feature matrix in one embodiment of the present application;

[0021] Figure 2 is a flowchart of a decoding method in one embodiment of the present application;

[0022] Figure 3 is a schematic diagram of a processing process of an encoding end in one implementation manner of the present application;

[0023] Figure 4 is a schematic diagram of a processing process of a decoding end in one embodiment of the present application;

[0024] Figure 5A and Figure 5B is a schematic diagram of a decoding network in one embodiment of the present application;

[0025] Fig. 6A and Figure 6B is a schematic diagram of a decoding network in one embodiment of the present application;

[0026] Figure 7 It is a hardware structure diagram of a decoding end device in one implementation manner of the present application. DETAILED DESCRIPTION

[0027] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, rather than for limiting the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application and the claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items. It should be understood that, although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information, depending on the context. In addition, the word "if" used can be interpreted as "at ... ", or "when ... ", or "in response to determination".

[0028] In the embodiments of the present application, a decoding method is proposed, which may involve the following technical terms:

[0029] JPEG (Joint Photographic Experts Group): JPEG is a standard for continuous tone static image compression. The file extension can be .jpg or .jpeg. It is a commonly used image file format. JPEG uses a joint coding method of predictive coding (DPCM), discrete cosine transform (DCT) and entropy coding to remove redundant image and color data. It is a lossy compression format that can compress images in a very small storage space, which will cause damage to image data to a certain extent. In particular, when using too high a compression ratio, the quality of the image restored after the final decompression will be reduced. If you pursue high-quality images, JPEG should not use too high a compression ratio.

[0030] JPEG-AI (Joint Photographic Experts Group Artificial Intelligence): The scope of JPEG-AI is to create a learning-based image coding standard that provides a single-stream, compact compressed domain representation that significantly improves compression efficiency over commonly used image coding standards at the same subjective quality, effectively improving performance in image processing and computer vision tasks. JPEG-AI targets a wide range of applications such as cloud storage, vision management, self-driving cars and devices, image acquisition storage and management, real-time management of visual data, and media distribution. The goal of JPEG-AI is to design a codec solution that needs to significantly improve compression efficiency at the same subjective quality and provide efficient compressed domain processing for machine learning-based image processing and computer vision tasks. JPEG-AI requires hardware- and software-friendly encoding and decoding, support for 8-bit and 10-bit depths, and efficient encoding and progressive decoding of images with text and graphics.

[0031] Entropy coding: Entropy coding is a method of coding that does not lose any information in accordance with the entropy principle. Information entropy is the average amount of information in the source (a measure of uncertainty). Entropy coding methods may include, but are not limited to, Shannon coding, Huffman coding, and arithmetic coding.

[0032] Neural Network (NN): Neural network refers to artificial neural network. Neural network is a computing model composed of a large number of nodes (or neurons) connected to each other. In a neural network, neuron processing units can represent different objects, such as features, letters, concepts, or some meaningful abstract patterns. The types of processing units in a neural network can be divided into three categories: input units, output units, and hidden units. Input units receive signals and data from the outside world; output units realize the output of processing results; hidden units are units between input and output units and cannot be observed from outside the system. The connection weights between neurons reflect the connection strength between units, and the representation and processing of information are reflected in the connection relationship of processing units. Neural network is a non-programmed, brain-like information processing method. The essence of neural network is to obtain a parallel distributed information processing function through the transformation and dynamic behavior of neural network, and imitate the information processing function of human brain nervous system to different degrees and levels. In the field of video processing, commonly used neural networks may include but are not limited to: convolutional neural network (CNN), recurrent neural network (RNN), fully connected network, etc.

[0033] Convolutional Neural Network (CNN): Convolutional neural network is a feedforward neural network and one of the most representative network structures in deep learning technology. The artificial neurons of convolutional neural network can respond to surrounding units within a certain coverage range and have excellent performance in large-scale image processing. The basic structure of convolutional neural network can include two layers. One is the feature extraction layer (also called convolution layer). The input of each neuron is connected to the local receptive field of the previous layer and extracts the local features. Once the local feature is extracted, its positional relationship with other features is also determined. The second is the feature mapping layer (also called activation layer). Each computing layer of the neural network consists of multiple feature maps. Each feature map is a plane, and the weights of all neurons on the plane are equal. The feature mapping structure can use Sigmoid function, ReLU function, Leaky-ReLU function, PReLU function, GDN function, etc. as the activation function of the convolutional network. In addition, since neurons on a mapping surface share weights, the number of free parameters of the network is reduced.

[0034] For example, one of the advantages of convolutional neural networks over image processing algorithms is that they avoid the complex pre-processing of images (extracting artificial features, etc.), and can directly input the original image for end-to-end learning. One of the advantages of convolutional neural networks over ordinary neural networks is that ordinary neural networks use a fully connected approach, that is, all neurons from the input layer to the hidden layer are connected, which will result in a huge number of parameters, making network training time-consuming or even difficult to train, while convolutional neural networks avoid this difficulty through local connections, weight sharing and other methods.

[0035] For example, convolutional neural networks can process large images. Convolutional neural networks usually include convolutional layers, pooling layers, fully connected layers, etc. Convolutional neural networks have made great progress in image classification, target detection, semantic segmentation and other fields.

[0036] Deconvolution layer: Deconvolution layer is also called transposed convolution layer. The working process of deconvolution layer and convolution layer is very similar. The main difference is that deconvolution layer will make the output larger than the input through padding (of course, it can also remain the same). If stride is 1, it means that the output size is equal to the input size; if stride is N, it means that the width of the output feature is N times the width of the input feature, and the height of the output feature is N times the height of the input feature.

[0037] Model fixed-point / model quantization: Model fixed-point can also be called model quantization. It is a commonly used neural network model acceleration algorithm. By fixing the model's float type parameters and input and output feature values ​​into fixed-bit values, such as 8-bit and 4-bit, the network model's computational cost, data bandwidth, and storage space can be reduced, allowing the network model to be applied faster and better on hardware with limited memory and computing power. The fixed-point model includes the fixed-point model weights and the fixed-point feature values. The lower the number of bits for the fixed-point model, the more obvious the acceleration on the hardware, and the performance will be reduced accordingly.

[0038] Image coding and decoding: The purpose of image coding and decoding technology is to encode and compress images in order to reduce the transmission and storage costs of image data. At the same time, it can be decoded and restored to image content by the decoding end. The higher the coding compression rate, the lower the transmission and storage consumption, but it also brings great difficulties to image restoration. With the successful performance of artificial intelligence in various fields, artificial intelligence has also been applied to the field of image coding and decoding to obtain lower compression rates and better image restoration effects.

[0039] Feature: The feature involved in this application can be a three-dimensional feature matrix of C*W*H, see Figure 1As shown, it is a schematic diagram of a three-dimensional feature matrix. In the three-dimensional feature matrix, C represents the number of channels, H represents the feature height, and W represents the feature width. The three-dimensional feature matrix can be the input of a neural network or the output of a neural network.

[0040] Rate-Distortion Optimized: There are two major indicators for evaluating coding efficiency: bit rate and PSNR (Peak Signal to Noise Ratio). The smaller the bit stream, the greater the compression rate, and the greater the PSNR, the better the quality of the reconstructed image. When selecting a mode, the discriminant formula is essentially a comprehensive evaluation of the two. For example, the cost corresponding to the mode: J(mode) = D + λ*R, where D represents Distortion, which can usually be measured using the SSE indicator. SSE refers to the mean square sum of the difference between the reconstructed image block and the source image. In order to achieve cost considerations, the SAD indicator can also be used. SAD refers to the sum of the absolute values ​​of the difference between the reconstructed image block and the source image; λ is the Lagrange multiplier, and R is the actual number of bits required for encoding the image block in this mode, including the sum of bits required for encoding mode information, motion information, residuals, etc. When selecting a mode, if the rate-distortion principle is used to make a comparison decision on the encoding mode, the best encoding performance can usually be guaranteed.

[0041] A large number of coding tools have been proposed for each module of the encoding end, and each tool often has multiple modes. For different video sequences, the coding tools that can achieve the best coding performance are often different. Therefore, in the encoding process, RDO (Rate-Distortion Optimize) is usually used to compare the coding performance of different tools or modes to select the best mode. After determining the optimal tool or mode, the decision information of the tool or mode is transmitted by encoding the marker information in the bitstream. Although this method brings higher coding complexity, it can adaptively select the optimal mode combination for different content to obtain the best coding performance. The decoding end can obtain relevant mode information by directly parsing the flag information, and the complexity impact is small.

[0042] The decoding method in the embodiments of the present application is described in detail below in conjunction with several specific embodiments.

[0043] Example 1: A decoding method is proposed in the embodiment of the present application, see Figure 2 As shown, the method may include:

[0044] Step 201: Decode the target feature corresponding to the current image block from the code stream corresponding to the current image block.

[0045] Step 202: Determine a first input feature of a target decoding network based on the target feature.

[0046] Step 203: Obtain a target eigenvalue quantization bit width and a target eigenvalue quantization hyperparameter of a target decoding network, and convert the first input feature into a second input feature based on the target eigenvalue quantization bit width and the target eigenvalue quantization hyperparameter.

[0047] Step 204: Process the second input feature based on the fixed-point weights of the target decoding network to obtain the output feature of the target decoding network; wherein the fixed-point weights are determined based on the target weight quantization bit width and the target weight quantization hyperparameters.

[0048] Step 205: Determine a reconstructed image block corresponding to the current image block based on the output features of the target decoding network.

[0049] Exemplarily, the target decoding network includes at least one target network layer, and the target network layer can be a network layer using fixed-point weights. For each target network layer in the target decoding network: based on the target feature value quantization bit width and the target feature value quantization hyperparameter of the target network layer, the first input feature of the target network layer is converted into a second input feature; based on the fixed-point weight of the target network layer, the second input feature is processed to obtain the output feature of the target network layer; wherein the fixed-point weight of the target network layer is determined based on the target weight quantization bit width of the target network layer and the target weight quantization hyperparameter of the target network layer; wherein the target feature value quantization bit widths of different target network layers can be the same or different, the target feature value quantization hyperparameters of different target network layers can be the same or different, the target weight quantization bit widths of different target network layers can all be fixed quantization bit widths, and the target weight quantization hyperparameters of different target network layers can be the same or different.

[0050] Exemplarily, based on the target feature value quantization bit width and the target feature value quantization hyperparameter of the target network layer, the first input feature of the target network layer is converted into the second input feature, including but not limited to: using the following formula to convert the first input feature of the target network layer into the second input feature: quant = clip (round (data × 2 param ),-2 bw-1 ,2 bw-1 -1); or, quant = clip (round (data ÷ param), -2 bw-1 ,2 bw-1 -1); where quant is used to represent the second input feature, data is used to represent the first input feature, param is used to represent the target feature value quantization hyperparameter, bw is used to represent the target feature value quantization bit width, clip is used to represent the truncation function, and round is used to represent rounding.

[0051] Among them, through the operation of the truncation function clip, if round(data×2 param ) is less than -2 bw-1 , then quant is -2 bw-1 , if round(data×2 param ) is greater than 2 bw-1 -1, then quant is 2 bw-1 -1, if round(data×2 param ) is located at -2 bw-1 with 2 bw-1 -1, then quant is round(data×2 param ).

[0052] Among them, through the operation of the truncation function clip, if round(data÷param) is less than -2 bw-1 , then quant is -2 bw-1 , if round(data÷param) is greater than 2 bw-1 -1, then quant is 2 bw-1 -1, if round(data÷param) is -2 bw-1 with 2 bw-1 -1, then quant is round(data÷param).

[0053] In the subsequent embodiments, the meaning of the clip operation is similar and will not be repeated in the subsequent process.

[0054] Exemplarily, the sample decoding network may include multiple network layers using floating-point weights, and the multiple network layers may include a target network layer. For each target network layer: a weight quantization hyperparameter candidate list corresponding to the target network layer is obtained, and the weight quantization hyperparameter candidate list may include multiple candidate weight quantization hyperparameters; for each candidate weight quantization hyperparameter, a fixed quantization bit width and the candidate weight quantization hyperparameter are used to simulate and quantize the floating-point weights of the target network layer to obtain candidate simulation weights; based on the candidate simulation weights, the floating-point input features of the target network layer are processed to obtain the floating-point output features of the target network layer, and the quantization error corresponding to the candidate weight quantization hyperparameter is determined based on the floating-point output features of the target network layer; based on the quantization error corresponding to each candidate weight quantization hyperparameter, the candidate weight quantization hyperparameter corresponding to the minimum quantization error is determined as the target weight quantization hyperparameter, and the fixed quantization bit width is determined as the target weight quantization bit width; based on the target weight quantization bit width and the target weight quantization hyperparameter, the floating-point weights of the target network layer are fixed-point quantized to obtain the fixed-point weights of the target network layer; and a target decoding network is generated based on the fixed-point weights of each target network layer.

[0055] Exemplarily, obtaining a weight quantization hyperparameter candidate list corresponding to the target network layer may include, but is not limited to: determining a maximum weight value based on all floating-point weights of the target network layer; wherein each floating-point weight includes multiple weight values, and the maximum weight value is the maximum value of all weight values ​​of all floating-point weights; generating an initial weight quantization hyperparameter corresponding to the target network layer based on the maximum weight value; and constructing a weight quantization hyperparameter candidate list based on the initial weight quantization hyperparameter.

[0056] Exemplarily, generating the initial weight quantization hyperparameter corresponding to the target network layer based on the maximum weight value may include but is not limited to: generating the initial weight quantization hyperparameter using the following formula: param = bw-1-ceil(log2(max)); or, param = (max) / 2 bw-1 ; Among them, param is used to represent the initial weight quantization hyperparameter corresponding to the target network layer, bw is used to represent the fixed quantization bit width, max is used to represent the maximum weight value, and ceil is used to represent the rounding up operation.

[0057] Exemplarily, using a fixed quantization bit width and the candidate weight quantization hyperparameter to simulate and quantize the floating-point weight of the target network layer to obtain the candidate simulation weight may include: using the following formula to simulate and quantize the floating-point weight of the target network layer to obtain the candidate simulation weight: quant = 2 -param ×clip(round(data×2 param ),-2 bw-1 ,2 bw -1 -1); or, quant = param × clip (round (data ÷ param), -2 bw-1 ,2 bw-1 -1); where quant is used to represent the candidate simulation weight, param is used to represent the candidate weight quantization hyperparameter, data is used to represent the floating-point weight of the target network layer, bw is used to represent the fixed quantization bit width, clip is used to represent the truncation function, and round is used to represent rounding.

[0058] Exemplarily, determining the quantization error corresponding to the candidate weight quantization hyperparameter based on the floating-point output feature of the target network layer may include but is not limited to: determining the sample output feature based on the floating-point output feature of the target network layer, the sample output feature may be the output feature of the reference network layer in the sample decoding network, and the reference network layer is any network layer located after the target network layer; determining the quantization error corresponding to the candidate weight quantization hyperparameter based on the sample output feature and the reference output feature; wherein, the method for obtaining the reference output feature may include but is not limited to: decoding the sample feature from the sample bitstream, determining the floating-point input feature corresponding to the sample decoding network based on the sample feature, and determining the reference output feature based on the floating-point input feature, the reference output feature may be the output feature of the reference network layer in the sample decoding network.

[0059] Exemplarily, the sample decoding network may include multiple network layers using floating-point weights, the multiple network layers include a target network layer, and for each target network layer: obtaining a candidate list of eigenvalue quantization hyperparameters and a quantization bit width set corresponding to the target network layer, the candidate list of eigenvalue quantization hyperparameters includes multiple candidate eigenvalue quantization hyperparameters, and the quantization bit width set includes at least two quantization bit widths; traversing the current quantization bit width from the quantization bit width set, for each candidate eigenvalue quantization hyperparameter, using the current quantization bit width and the candidate eigenvalue quantization hyperparameter to simulate and quantize the floating-point input features of the target network layer to obtain a candidate Input feature; Process the candidate input feature based on the pseudo weight of the target network layer to obtain the floating-point output feature of the target network layer, and determine the quantization error corresponding to the candidate eigenvalue quantization hyperparameter based on the floating-point output feature of the target network layer; Based on the quantization error corresponding to each candidate eigenvalue quantization hyperparameter, if the minimum quantization error is less than a preset threshold, determine the candidate eigenvalue quantization hyperparameter corresponding to the minimum quantization error as the target eigenvalue quantization hyperparameter, determine the current quantization bit width as the target eigenvalue quantization bit width, and record the target eigenvalue quantization hyperparameter and the target eigenvalue quantization bit width for the target network layer.

[0060] Exemplarily, based on the quantization error corresponding to each candidate eigenvalue quantization hyperparameter, if the minimum quantization error is not less than a preset threshold, determine whether the current quantization bit width is the last quantization bit width in the quantization bit width set; if so, determine the candidate eigenvalue quantization hyperparameter corresponding to the minimum quantization error as the target eigenvalue quantization hyperparameter, determine the current quantization bit width as the target eigenvalue quantization bit width, and record the target eigenvalue quantization hyperparameter and the target eigenvalue quantization bit width for the target network layer; if not, traverse the next quantization bit width of the current quantization bit width as the current quantization bit width, and return to execute the operation of simulating quantizing the floating-point input features of the target network layer using the current quantization bit width and the candidate eigenvalue quantization hyperparameter to obtain the candidate input features.

[0061] Exemplarily, obtaining a candidate list of eigenvalue quantization hyperparameters corresponding to the target network layer includes: determining a maximum eigenvalue based on all floating-point input features of the target network layer; wherein each floating-point input feature includes multiple eigenvalues, and the maximum eigenvalue is the maximum value of all eigenvalues ​​of all floating-point input features; generating an initial eigenvalue quantization hyperparameter corresponding to the target network layer based on the maximum eigenvalue; and constructing a candidate list of eigenvalue quantization hyperparameters based on the initial eigenvalue quantization hyperparameters.

[0062] Exemplarily, generating the initial eigenvalue quantization hyperparameter corresponding to the target network layer based on the maximum eigenvalue may include but is not limited to: generating the initial eigenvalue quantization hyperparameter using the following formula: param = bw-1-ceil(log2(max)); or, param = (max) / 2 bw-1 ; Among them, param represents the initial eigenvalue quantization hyperparameter corresponding to the target network layer, bw represents the current quantization bit width, max represents the maximum eigenvalue, and ceil represents the rounding up operation.

[0063] Exemplarily, the current quantization bit width and the candidate feature value quantization hyperparameter are used to simulate and quantize the floating-point input features of the target network layer to obtain the candidate input features, which may include but is not limited to: the following formula is used to simulate and quantize the floating-point input features of the target network layer to obtain the candidate input features: quant = 2 -param ×clip(round(data×2 param ),-2 bw-1 ,2 bw-1 -1); or, quant = param × clip (round (data ÷ param), -2 bw-1 ,2 bw -1 -1); where quant represents the candidate input feature, param represents the candidate feature value quantization hyperparameter, data represents the floating-point input feature, bw represents the current quantization bit width, clip represents the truncation function, and round represents rounding.

[0064] Exemplarily, before processing the candidate input features based on the pseudo-weights of the target network layer to obtain the floating-point output features of the target network layer, the floating-point weights of the target network layer can be simulated and quantized using the target weight quantization hyperparameters of the target network layer and the target weight quantization bit width of the target network layer to obtain the pseudo-weights of the target network layer.

[0065] Exemplarily, determining the quantization error corresponding to the candidate eigenvalue quantization hyperparameter based on the floating-point output feature of the target network layer may include but is not limited to: determining the sample output feature based on the floating-point output feature of the target network layer, where the sample output feature may be the output feature of a reference network layer in a sample decoding network, and the reference network layer may be any network layer located after the target network layer; determining the quantization error corresponding to the candidate eigenvalue quantization hyperparameter based on the sample output feature and the reference output feature; wherein the method for obtaining the reference output feature may include but is not limited to: decoding the sample feature from the sample bitstream, determining the floating-point input feature corresponding to the sample decoding network based on the sample feature, and determining the reference output feature based on the floating-point input feature, where the reference output feature may be the output feature of the reference network layer in the sample decoding network.

[0066] Exemplarily, the above execution order is only for the convenience of describing the examples given. In practical applications, the execution order between the steps can also be changed, and there is no limitation on this execution order. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification, and the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; multiple steps described in this specification may also be combined into a single step for description in other embodiments.

[0067] It can be seen from the above technical scheme that in the embodiment of the present application, the target feature corresponding to the current image block is decoded from the code stream corresponding to the current image block, the first input feature of the target decoding network is determined based on the target feature, the first input feature is converted into a second input feature based on the target feature value quantization bit width and the target feature value quantization hyperparameter, the second input feature is processed based on the fixed-point weight of the target decoding network to obtain the output feature of the target decoding network, and the reconstructed image block corresponding to the current image block is determined based on the output feature, thereby proposing an end-to-end video image compression method, which can realize the decoding of video images based on the decoding network, and achieve the purpose of improving the encoding efficiency and decoding efficiency. The decoding network with fixed-point weights is constructed by using the target weight quantization bit width and the target weight quantization hyperparameter, and the target feature value quantization bit width and the target feature value quantization hyperparameter are used to generate fixed-point input features, so as to realize adaptive decoding acceleration and reduce the decoding calculation amount while ensuring the decoding quality.

[0068] Example 2: The processing process of the encoding end can be seen in Figure 3 As shown, Figure 3 This is just an example and is not intended to be limiting.

[0069] After obtaining the current image block x (the current image block x may be the original image block x, i.e., the input image block), the encoder may analyze and transform the current image block x through an analysis transformation network (i.e., a neural network) to obtain an image feature y corresponding to the current image block x. Among them, performing feature transformation on the current image block x through the analysis transformation network means: transforming the current image block x to an image feature y in the latent domain, thereby facilitating all subsequent processes to operate on the latent domain.

[0070] The image can be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can also be the current image, that is, the encoding and decoding process for the image block can also be directly used for the image.

[0071] After obtaining the image feature y, the encoding end performs coefficient hyperparameter feature transformation on the image feature y to obtain coefficient hyperparameter feature z. For example, the image feature y can be input into a hyperparameter encoding network (i.e., a neural network), and the hyperparameter encoding network performs coefficient hyperparameter feature transformation on the image feature y to obtain coefficient hyperparameter feature z. The hyperparameter encoding network can be a trained neural network, and there is no restriction on the training process of the hyperparameter encoding network, as long as the coefficient hyperparameter feature transformation can be performed on the image feature y. The image feature y in the latent domain obtains the super prior latent information z after passing through the hyperparameter encoding network.

[0072] After obtaining the coefficient hyperparameter feature z, the encoder can quantize the coefficient hyperparameter feature z to obtain the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, that is, Figure 3 The Q operation in is the quantization process. After obtaining the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, the hyperparameter quantization feature is encoded to obtain Bitstream#1 corresponding to the current image block (i.e., the first bitstream, which belongs to the main bitstream corresponding to the current image block), that is, Figure 3 The AE operation in represents the encoding process, such as the entropy encoding process. Alternatively, after obtaining the coefficient hyperparameter feature z, the encoder can also directly encode the coefficient hyperparameter feature z to obtain the Bitstream#1 corresponding to the current image block without involving the quantization process. Among them, the hyperparameter quantization feature or coefficient hyperparameter feature z carried in Bitstream#1 is mainly used to obtain the mean and probability distribution parameters of the probability distribution model.

[0073] After obtaining the Bitstream#1 corresponding to the current image block, the encoder may send the Bitstream#1 corresponding to the current image block to the decoder. For the processing process of the decoder for the Bitstream#1 corresponding to the current image block, refer to the subsequent embodiments.

[0074] After obtaining the Bitstream#1 corresponding to the current image block, the encoder can also decode Bitstream#1 to obtain the hyperparameter quantization feature, that is, Figure 3 AD in it represents the decoding process. Then, the hyperparameter quantization feature is dequantized to obtain the coefficient hyperparameter feature z_hat. The coefficient hyperparameter feature z_hat can be the same as or different from the coefficient hyperparameter feature z. Figure 3 The IQ operation in is the inverse quantization process. Alternatively, after obtaining the Bitstream#1 corresponding to the current image block, the encoder can also decode the Bitstream#1 to obtain the coefficient hyperparameter feature z_hat without involving the inverse quantization process of the coefficient hyperparameter feature z_hat.

[0075] For the encoding process of Bitstream#1, the encoding method of the fixed probability density model can be adopted, and for the decoding process of Bitstream#1, the decoding method of the fixed probability density model can be adopted. There is no restriction on this encoding and decoding process.

[0076] After obtaining the coefficient hyperparameter feature z_hat, the encoding end can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the image feature y_hat of the previous image block (the determination process of the image feature y_hat is described in the subsequent embodiments) to obtain the prediction value mu (i.e., mean mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the image feature y_hat are input to the mean prediction network, and the mean prediction network determines the prediction value mu based on the coefficient hyperparameter feature z_hat and the image feature y_hat. There is no restriction on this prediction process. Among them, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded image feature y_hat, and the two are jointly input to obtain a more accurate prediction value mu. The prediction value mu is used to obtain a residual by subtracting the original feature, and the residual is added to the decoded feature to obtain the reconstructed y.

[0077] It should be noted that the mean prediction network is an optional neural network, that is, there is no mean prediction network, that is, there is no need to determine the predicted value mu through the mean prediction network. Figure 3 The dashed box in indicates that the mean prediction network is optional.

[0078] After obtaining the image feature y, the encoder can determine the residual feature r based on the image feature y and the prediction value mu, such as taking the difference between the image feature y and the prediction value mu as the residual feature r. Then, the residual feature r is feature processed to obtain the image feature s. There is no restriction on this feature processing process, and it can be any feature processing method. In this case, it is necessary to deploy a mean prediction network, and the mean prediction network provides the prediction value mu. Alternatively, after obtaining the image feature y, the encoder can perform feature processing on the image feature y to obtain the image feature s. There is no restriction on this feature processing process, and it can be any feature processing method. In this case, there is no need to deploy a mean prediction network, and the dotted box indicates that the residual process is an optional process.

[0079] After obtaining the image feature s, the encoder can quantize the image feature s to obtain the image quantization feature corresponding to the image feature s, that is, Figure 3 The Q operation in is the quantization process. After obtaining the image quantization feature corresponding to the image feature s, the encoder can encode the image quantization feature to obtain Bitstream#2 (i.e., the second bitstream) corresponding to the current image block, i.e. Figure 3 The AE operation in represents an encoding process, such as an entropy encoding process. Alternatively, the encoding end may directly encode the image feature s to obtain Bitstream#2 corresponding to the current image block without involving the quantization process of the image feature s.

[0080] After obtaining Bitstream#2 corresponding to the current image block, the encoder may send Bitstream#2 corresponding to the current image block to the decoder. For the processing process of the decoder for Bitstream#2 corresponding to the current image block, refer to the subsequent embodiments.

[0081] After obtaining the Bitstream#2 corresponding to the current image block, the encoder can also decode Bitstream#2 to obtain the image quantization feature, that is, Figure 3 AD in the figure represents the decoding process. Then, the encoder can dequantize the image quantization feature to obtain the image feature s'. The image feature s' can be the same as or different from the image feature s. Figure 3 The IQ operation in is the inverse quantization process. Alternatively, after obtaining the Bitstream#2 corresponding to the current image block, the encoder can also decode the Bitstream#2 to obtain the image feature s' without involving the inverse quantization process of the image quantization feature.

[0082] After obtaining the image feature s', the encoder can perform feature recovery on the image feature s' (i.e., the inverse process of feature processing). There is no restriction on this feature recovery process, and it can be any feature recovery method to obtain the residual feature r_hat. The residual feature r_hat and the residual feature r can be the same or different. After obtaining the residual feature r_hat, the encoder determines the image feature y_hat based on the residual feature r_hat and the prediction value mu. The image feature y_hat and the image feature y can be the same or different. For example, the sum of the residual feature r_hat and the prediction value mu is used as the image feature y_hat. In this case, it is necessary to deploy a mean prediction network, and the mean prediction network provides the prediction value mu. Alternatively, after obtaining the image feature s', the encoder can perform feature recovery on the image feature s' (i.e., the inverse process of feature processing) to obtain the image feature y_hat. The image feature y_hat and the image feature y can be the same or different. In this case, it is not necessary to deploy a mean prediction network, and the dotted box indicates that the residual process is an optional process.

[0083] After obtaining the image feature y_hat, the encoding end can perform a synthetic transformation on the image feature y_hat to obtain a reconstructed image block x_hat corresponding to the current image block x. For example, the image feature y_hat is input into the synthetic transformation network, and the synthetic transformation network performs a synthetic transformation on the image feature y_hat to obtain the reconstructed image block x_hat. At this point, the image reconstruction process is completed.

[0084] In a possible implementation, when the encoding end encodes the image quantization feature or the image feature s to obtain the Bitstream#2 corresponding to the current image block, the encoding end needs to first determine the probability distribution model, and then encode the image quantization feature or the image feature s based on the probability distribution model. In addition, when the encoding end decodes Bitstream#2, it also needs to first determine the probability distribution model, and then decode Bitstream#2 based on the probability distribution model.

[0085] To obtain the probability distribution model, continue to refer to Figure 3 As shown, after obtaining the coefficient hyperparameter feature z_hat, the encoding end can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat is input to the probability hyperparameter decoding network, and the probability hyperparameter decoding network performs a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p. Among them, the probability hyperparameter decoding network can be a trained neural network, and there is no restriction on the training process of this probability hyperparameter decoding network, and it is sufficient to be able to perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat.

[0086] In a possible implementation, the processing process at the encoding end may be performed by a deep learning model or a neural network model, thereby realizing an end-to-end image compression and encoding process, without any limitation on the encoding process.

[0087] Example 3: The processing process of the decoding end can be seen in Figure 4 As shown, Figure 4 This is just an example and is not limiting.

[0088] After obtaining the Bitstream#1 corresponding to the current image block, the decoding end can also decode Bitstream#1 to obtain the hyperparameter quantization feature, that is, Figure 4 AD in it represents the decoding process. Then, the hyperparameter quantization feature is dequantized to obtain the coefficient hyperparameter feature z_hat. The coefficient hyperparameter feature z_hat can be the same as or different from the coefficient hyperparameter feature z. Figure 4 The IQ operation in is the inverse quantization process. Alternatively, after obtaining the Bitstream#1 corresponding to the current image block, the decoder can also decode the Bitstream#1 to obtain the coefficient hyperparameter feature z_hat without involving the inverse quantization process of the coefficient hyperparameter feature z_hat.

[0089] For the decoding process of Bitstream#1, a decoding method of a fixed probability density model may be used, and there is no restriction on this.

[0090] The image can be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can also be the current image, that is, the decoding process for the image block can also be directly used for the image.

[0091] After obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the image feature y_hat of the previous image block (the determination process of the image feature y_hat is described in the subsequent embodiments) to obtain the prediction value mu (i.e., mean mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the image feature y_hat are input to the mean prediction network, and the mean prediction network determines the prediction value mu based on the coefficient hyperparameter feature z_hat and the image feature y_hat. There is no restriction on this prediction process. Among them, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded image feature y_hat, and the two are jointly input to obtain a more accurate prediction value mu.

[0092] It should be noted that the mean prediction network is an optional neural network, that is, there is no mean prediction network, that is, there is no need to determine the predicted value mu through the mean prediction network. Figure 4 The dashed box in indicates that the mean prediction network is optional.

[0093] After obtaining the Bitstream#2 corresponding to the current image block, the decoding end can also decode Bitstream#2 to obtain the image quantization feature, that is, Figure 4 AD in the figure represents the decoding process. Then, the decoding end can dequantize the image quantization feature to obtain the image feature s'. The image feature s' can be the same as or different from the image feature s. Figure 4 The IQ operation in is the inverse quantization process. Alternatively, after obtaining the Bitstream#2 corresponding to the current image block, the decoder can also decode the Bitstream#2 to obtain the image feature s' without involving the inverse quantization process of the image quantization feature.

[0094] After obtaining the image feature s', the decoder can perform feature recovery on the image feature s' (i.e., the inverse process of feature processing) to obtain the residual feature r_hat, which is the same as or different from the residual feature r. After obtaining the residual feature r_hat, the decoder determines the image feature y_hat based on the residual feature r_hat and the prediction value mu, and the image feature y_hat is the same as or different from the image feature y, such as taking the sum of the residual feature r_hat and the prediction value mu as the image feature y_hat. In this case, it is necessary to deploy a mean prediction network, and the prediction value mu is provided by the mean prediction network. Alternatively, after obtaining the image feature s', the decoder can perform feature recovery on the image feature s' to obtain the image feature y_hat, which is the same as or different from the image feature y. In this case, it is not necessary to deploy a mean prediction network, and the dotted box indicates that the residual process is an optional process.

[0095] After obtaining the image feature y_hat, the decoding end can perform a synthetic transformation on the image feature y_hat to obtain a reconstructed image block x_hat corresponding to the current image block x. For example, the image feature y_hat is input into the synthetic transformation network, and the synthetic transformation network performs a synthetic transformation on the image feature y_hat to obtain the reconstructed image block x_hat. At this point, the image reconstruction process is completed.

[0096] In a possible implementation, when decoding Bitstream#2, the decoding end needs to first determine the probability distribution model, and then decode Bitstream#2 based on the probability distribution model. To obtain the probability distribution model, continue to refer to Figure 4As shown, after obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat is input into the probability hyperparameter decoding network, and the probability hyperparameter decoding network performs a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p. Among them, the probability hyperparameter decoding network can be a trained neural network, and there is no restriction on the training process of this probability hyperparameter decoding network. It can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p.

[0097] In a possible implementation, the processing process at the above decoding end can be performed by a deep learning model or a neural network model, thereby realizing an end-to-end image compression and encoding process, without limiting the decoding process.

[0098] Example 4: For Example 2 and Example 3, the encoding end involves an analysis transformation network, a hyperparameter encoding network, a mean prediction network, a probability hyperparameter decoding network, and a synthetic transformation network, and the decoding end involves a mean prediction network, a probability hyperparameter decoding network, and a synthetic transformation network. The above networks can all be neural networks. All neural networks involved in the decoding end can be used as decoding networks, and some neural networks involved in the decoding end can be used as decoding networks, without limitation. For example, all networks in the mean prediction network, the probability hyperparameter decoding network, and the synthetic transformation network can be used as decoding networks, and some networks in the mean prediction network, the probability hyperparameter decoding network, and the synthetic transformation network can be used as decoding networks. Of course, when other neural networks are involved in the decoding end, other neural networks can also be used as decoding networks, without limitation.

[0099] In a possible implementation, all neural networks involved in the decoding end are taken as a decoding network for illustration, see Figure 5A As shown, the decoding network may include a probability super-parameter decoding network and a synthesis transformation network. Figure 5B As shown, the decoding network can include a mean prediction network, a probability hyperparameter decoding network, and a synthetic transformation network. Figure 5A and Figure 5B In the embodiment, the decoding network may also include only the synthesis transformation network but not other networks, and there is no limitation on this.

[0100] In a possible implementation, the encoding end or the decoding end may decode the target feature corresponding to the current image block from the code stream corresponding to the current image block (such as the first code stream and the second code stream), determine the input feature of the decoding network based on the target feature, process the input feature based on the decoding network to obtain the output feature of the decoding network, and after obtaining the output feature, determine the reconstructed image block corresponding to the current image block based on the output feature. This process is described below.

[0101] See also Figure 5A As shown, after obtaining Bitstream#1 (i.e., the first bitstream) corresponding to the current image block, the encoder or decoder can decode Bitstream#1 to obtain the hyperparameter quantization feature (i.e., the target feature of the current image block), dequantize the hyperparameter quantization feature, and obtain the coefficient hyperparameter feature z_hat (i.e., the input feature of the probability hyperparameter decoding network is determined based on the target feature). Alternatively, after obtaining Bitstream#1 corresponding to the current image block, the encoder or decoder can decode Bitstream#1 to obtain the coefficient hyperparameter feature z_hat (i.e., the target feature of the current image block, and the target feature is directly used as the input feature of the probability hyperparameter decoding network), without involving the dequantization process of the coefficient hyperparameter feature z_hat.

[0102] After obtaining the coefficient hyperparameter feature z_hat (i.e., the input feature of the probability hyperparameter decoding network), the coefficient hyperparameter feature z_hat is input to the probability hyperparameter decoding network (i.e., the decoding network), and the probability hyperparameter decoding network performs a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p corresponding to the current image block (i.e., the output feature of the probability hyperparameter decoding network). After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p.

[0103] After obtaining Bitstream#2 (i.e., the second bitstream) corresponding to the current image block, the encoder or decoder may also decode Bitstream#2 to obtain image quantization features (i.e., the target features of the current image block), dequantize the image quantization features to obtain image features s', perform feature recovery on image features s', and obtain image features y_hat (i.e., determining the input features of the synthetic transformation network based on the target features). Alternatively, after obtaining Bitstream#2 corresponding to the current image block, the encoder or decoder may also decode Bitstream#2 to obtain image features s' (i.e., the target features of the current image block), without involving the dequantization process of image quantization features, perform feature recovery on image features s', and obtain image features y_hat (i.e., determining the input features of the synthetic transformation network based on the target features). When decoding Bitstream#2, the encoder or decoder may decode Bitstream#2 based on a probability distribution model, and this process will not be described in detail.

[0104] After obtaining the image feature y_hat, the encoding end or the decoding end inputs the image feature y_hat to the synthesis transformation network (i.e., the decoding network), and the synthesis transformation network performs a synthesis transformation on the image feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x (i.e., the output feature of the synthesis transformation network). In summary, it can be seen that the image feature y_hat can be synthesized based on the synthesis transformation network to obtain the reconstructed image block x_hat corresponding to the current image block x.

[0105] After obtaining the output features of the synthetic transformation network, the output features of the synthetic transformation network can be used as the reconstructed image block x_hat corresponding to the current image block x, that is, the reconstructed image block x_hat corresponding to the current image block x is determined based on the output features.

[0106] See also Figure 5B As shown, after obtaining the Bitstream#1 corresponding to the current image block, the encoding end or the decoding end can decode Bitstream#1 to obtain the hyperparameter quantization feature (i.e., the target feature of the current image block), dequantize the hyperparameter quantization feature, and obtain the coefficient hyperparameter feature z_hat (i.e., the input feature of the probability hyperparameter decoding network is determined based on the target feature). Alternatively, after obtaining the Bitstream#1 corresponding to the current image block, the Bitstream#1 is decoded to obtain the coefficient hyperparameter feature z_hat (i.e., the target feature of the current image block, and the target feature is directly used as the input feature of the probability hyperparameter decoding network).

[0107] After obtaining the coefficient hyperparameter feature z_hat (i.e., the input feature of the probability hyperparameter decoding network), the coefficient hyperparameter feature z_hat is input to the probability hyperparameter decoding network (i.e., the decoding network), and the probability hyperparameter decoding network performs a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p corresponding to the current image block (i.e., the output feature of the probability hyperparameter decoding network). After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p.

[0108] After obtaining the coefficient hyperparameter feature z_hat (i.e., the input feature of the mean prediction network), the coefficient hyperparameter feature z_hat and the image feature y_hat of the previous image block are input into the mean prediction network (i.e., the decoding network), and the mean prediction network determines the prediction value mu (i.e., the output feature of the mean prediction network) based on the coefficient hyperparameter feature z_hat and the image feature y_hat. That is, the mean prediction network performs context-based prediction to obtain the prediction value mu (i.e., mean mu) corresponding to the current image block.

[0109] After obtaining Bitstream#2 corresponding to the current image block, the encoder or decoder may also decode Bitstream#2 to obtain image quantization features (i.e., target features of the current image block), dequantize the image quantization features to obtain image features s', perform feature recovery on the image features s' (i.e., residual recovery, which is the inverse process of residual processing) to obtain residual features r_hat, and determine image features y_hat based on the residual features r_hat and the predicted value mu, such as using the sum of the residual features r_hat and the predicted value mu as the image feature y_hat (i.e., determining the input features of the synthetic transformation network based on the target features).

[0110] Alternatively, after obtaining Bitstream#2 corresponding to the current image block, the encoder or decoder may also decode Bitstream#2 to obtain image feature s' (i.e., target feature of the current image block), perform feature recovery on image feature s' to obtain residual feature r_hat, and determine image feature y_hat based on residual feature r_hat and prediction value mu, such as using the sum of residual feature r_hat and prediction value mu as image feature y_hat (i.e., determining input feature of synthetic transformation network based on the target feature).

[0111] After obtaining the image feature y_hat, the encoding end or the decoding end inputs the image feature y_hat to the synthesis transformation network (i.e., the decoding network), and the synthesis transformation network performs a synthesis transformation on the image feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x (i.e., the output feature of the synthesis transformation network). In summary, it can be seen that the image feature y_hat can be synthesized based on the synthesis transformation network to obtain the reconstructed image block x_hat corresponding to the current image block x.

[0112] After obtaining the output features of the synthetic transformation network, the output features of the synthetic transformation network can be used as the reconstructed image block x_hat corresponding to the current image block x, that is, the reconstructed image block x_hat corresponding to the current image block x is determined based on the output features.

[0113] Embodiment 5: For Embodiment 4, the decoding network can be a floating-point decoding network, that is, the decoding network can use floating-point weights (i.e., floating-point parameters), that is, the weights of the decoding network are all float-type weights, the input eigenvalues ​​of the decoding network are float-type eigenvalues, and the output eigenvalues ​​of the decoding network are float-type eigenvalues. For example, for each network layer of the decoding network, the network layer can use floating-point weights, that is, the weights of the network layer are all float-type weights, the input eigenvalues ​​of the network layer are float-type eigenvalues, and the output eigenvalues ​​of the network layer are float-type eigenvalues.

[0114] However, in order to implement a decoding network with floating-point weights, the decoder needs to store float-type weights, which takes up a lot of storage resources. Moreover, when the decoder uses float-type weights to process float-type feature values, it takes up a lot of computing resources and the processing speed of the decoder is relatively slow.

[0115] In view of the above findings, this embodiment proposes an efficient reasoning method for image encoding and decoding, which can use a fixed-point decoding network to process image features. The decoding network involved in Example 4 is a fixed-point decoding network, that is, the decoding network can use fixed-point weights (i.e., fixed-point parameters), that is, the weights of the decoding network are all fixed-bit weights, the input feature values ​​of the decoding network are fixed-bit feature values, and the output feature values ​​of the decoding network are fixed-bit feature values. For example, for the network layer of the decoding network, the network layer can use fixed-point weights, that is, the weights of the network layer are fixed-bit weights, the input feature values ​​of the network layer are fixed-bit feature values, and the output feature values ​​of the network layer are fixed-bit feature values.

[0116] Obviously, in order to realize the decoding network of fixed-point weights, the decoding end needs to store fixed-bit weights (such as 8 bits, 16 bits, etc.), and when storing fixed-bit weights, only a small amount of storage resources are needed, thereby saving storage resources. Moreover, when the decoding end uses fixed-bit weights to process fixed-bit eigenvalues, only a small amount of computing resources are needed, thereby saving computing resources. Moreover, when the decoding end uses fixed-bit weights to process fixed-bit eigenvalues, the processing speed can be increased, realizing adaptive acceleration of the decoding process.

[0117] This embodiment is a solution that does not require training and can be quickly integrated into any floating-point decoding network to obtain a decoding network with fixed-point weights. The decoding network with fixed-point weights can be directly inferred on the hardware platform. When obtaining the decoding network with fixed-point weights, the sensitivity of each network layer of the decoding network to quantization will be evaluated first to determine a more appropriate quantization bit width for each network layer, thereby ensuring the quantization compression rate of the decoding network while ensuring the quantization performance of the decoding network.

[0118] Exemplarily, when the floating-point parameters of the decoding network are quantized into fixed-point type parameters, the storage consumption of the decoding network can be reduced. When the floating-point eigenvalues ​​of the decoding network are quantized into fixed-point type eigenvalues, the bandwidth during forward calculation can also be reduced. In addition, when the floating-point operations in the decoding network are converted to fixed-point operations, the amount of calculation of the decoding network can be reduced. At the same time, the calculation under the fixed-point type also contributes to the decoding consistency between different devices. However, the quantization of the decoding network itself is achieved by reducing the data type and the accuracy of the calculation, which affects the performance of the decoding network, which is unacceptable in image encoding and decoding. In view of the above findings, considering that for a decoder based on a neural network, when decoding a bit stream, the sensitivity to quantization is different in each network layer of the neural network, that is, some network layers can maintain quantization performance at a lower bit, while some network layers require a higher bit to ensure quantization accuracy. This is related to the parameters and feature distribution learned by the neural network. In this embodiment, in order to achieve the maximum compression of the bit width while ensuring the performance after compression, an efficient compression and inference method is proposed, which can use the target weight quantization bit width and the target weight quantization hyperparameter to construct a fixed-point weight decoding network, and use the target eigenvalue quantization bit width and the target eigenvalue quantization hyperparameter to generate fixed-point input features, thereby achieving adaptive decoding acceleration and reducing the amount of decoding calculation while ensuring the decoding quality.

[0119] Example 6: See Fig. 6A As shown, in Figure 5A Based on the decoding network shown in Figure 1 (such as the probabilistic super-parameter decoding network and the synthetic transformation network), three modules are added: a bit stream extraction module, a quantization analysis module, and a decoding network quantization module; see Figure 6BAs shown, in Figure 5B Based on the decoding network shown (such as mean prediction network, probability hyperparameter decoding network, and synthetic transformation network), three modules are added: code stream extraction module, quantization analysis module, and decoding network quantization module.

[0120] Exemplarily, after the decoding network training is completed (the decoding network is obtained by training with floating-point weights), the floating-point weights of the decoding network are fixed. For the sake of distinction, the decoding network using floating-point weights is called a sample decoding network. The sample decoding network may include multiple network layers using floating-point weights. The multiple network layers may include a target network layer. The target network layer is a network layer that needs to use fixed-point weights (i.e., the floating-point weights of the target network layer need to be converted into fixed-point weights). Assuming that the sample decoding network includes network layer 1 and network layer 2, network layer 1 needs to use fixed-point weights, while network layer 2 does not need to use fixed-point weights, then network layer 1 is the target network layer, and network layer 2 is not the target network layer.

[0121] In a possible implementation, all network layers in the sample decoding network may be used as target network layers, or some network layers in the sample decoding network may be used as target network layers, or all convolutional layers in the sample decoding network may be used as target network layers. Of course, the above are just a few examples and are not limited to this. Any one or more network layers in the sample decoding network may be used as target network layers, taking all network layers as target network layers as an example.

[0122] Exemplarily, in order to convert a sample decoding network into a target decoding network with fixed-point weights, the following steps may be used:

[0123] Step S11: The code stream extraction module obtains a sample code stream and inputs the sample code stream to the quantization analysis module.

[0124] For example, we can collect several frames of images in real scenes (such as the application scenes of image acquisition equipment), such as 100 frames of images, 200 frames of images, etc. Figure 3 The process shown processes each frame of the image to obtain the first bitstream (Bitstream#1) and the second bitstream (Bitstream#2) corresponding to the frame of the image. The first bitstream and the second bitstream can be called sample bitstreams, or the first bitstream can be called sample bitstreams, or the second bitstream can be called sample bitstreams. For the convenience of description, in the subsequent process, the first bitstream and the second bitstream are called sample bitstreams as an example. The bitstream extraction module can obtain the sample bitstreams corresponding to each frame of the image and input these sample bitstreams to the quantization analysis module.

[0125] Step S12: The quantization analysis module receives the sample code stream corresponding to each frame of the image, and stores the sample code stream corresponding to each frame of the image.

[0126] Step S13, the quantization analysis module decodes the sample features from the sample bitstream, determines the floating-point input features corresponding to the sample decoding network based on the sample features, and determines the reference output features corresponding to the sample features based on the floating-point input features, and the reference output features can be the output features of the reference network layer in the sample decoding network. The reference network layer can be any network layer in the sample decoding network, for example, the last network layer in the sample decoding network, or the second to last network layer in the sample decoding network, or the last target network layer in the sample decoding network, or any network layer in the sample decoding network located after the last target network layer, and there is no restriction on the position of the reference network layer.

[0127] For example, for each sample code stream, the sample code stream can be decoded to obtain sample features. The process of obtaining the target features in Example 4 is referred to and will not be described in detail here. After obtaining the sample features, the floating-point input features corresponding to the sample decoding network are determined based on the sample features. The process of determining the input features of the decoding network based on the target features in Example 4 is referred to and will not be described in detail here. After obtaining the floating-point input features corresponding to the sample decoding network, the floating-point input features are input to the sample decoding network, and pass through each network layer of the sample decoding network in turn. After passing through the reference network layer of the sample decoding network, the output features of the reference network layer are used as the reference output features corresponding to the sample features.

[0128] Obviously, for each sample code stream, a reference output feature corresponding to the sample code stream can be obtained. Based on this, the quantitative analysis module can store the average value of the reference output features corresponding to all sample code streams, and can also store the maximum value of the reference output features corresponding to all sample code streams. It can also store the minimum value of the reference output features corresponding to all sample code streams. Taking the average value of the reference output features corresponding to all sample code streams as an example, this average value is stored as the reference output feature.

[0129] Step S14: for each target network layer in the sample decoding network: the quantization analysis module obtains a weight quantization super-parameter candidate list corresponding to the target network layer, and the weight quantization super-parameter candidate list may include multiple candidate weight quantization super-parameters.

[0130] First, the quantitative analysis module determines the maximum weight value based on all floating-point weights of the target network layer; wherein each floating-point weight includes multiple weight values, and the maximum weight value is the maximum value of all weight values ​​of all floating-point weights. For example, the target network layer may include multiple floating-point weights, each of which includes multiple weight values. For each floating-point weight, the maximum value can be selected from all weight values ​​corresponding to the floating-point weight, and then the maximum weight value can be selected from the maximum values ​​corresponding to all floating-point weights. For example, assuming that the target network layer is network layer l, the maximum weight value can be recorded as max l =max(abs(w 0_l ),abs(w 1_l ),…,abs(w n_l )), abs(w 0_l ) represents the maximum value corresponding to the first floating-point weight of network layer l, and so on. l Indicates the maximum weight value among all floating-point weights.

[0131] The target network layer may include multiple channels. If all channels correspond to the same target weight quantization hyperparameter, the maximum weight value is the maximum value of all weight values ​​of the floating-point weights of all channels. If each channel corresponds to a target weight quantization hyperparameter separately, each channel corresponds to a maximum weight value, that is, the maximum weight value is the maximum value of all weight values ​​of the floating-point weights of the channel. For the convenience of description, the following explanation will be given by taking the example that all channels correspond to the same target weight quantization hyperparameter. The implementation method of each channel corresponding to the target weight quantization hyperparameter is similar. The same method can be used to determine the target weight quantization hyperparameter corresponding to each channel separately. The determination method is the same as the determination method of the target weight quantization hyperparameter corresponding to all channels.

[0132] Then, the quantization analysis module generates the initial weight quantization hyperparameter corresponding to the target network layer based on the maximum weight value. For example, the following formula can be used to generate the initial weight quantization hyperparameter corresponding to the target network layer: param = bw-1-ceil(log2(max)); where param is used to represent the initial weight quantization hyperparameter corresponding to the target network layer, bw is used to represent the fixed quantization bit width, max is used to represent the maximum weight value, and ceil is used to represent the rounding up operation.

[0133] For example, the initial weight quantization hyperparameter corresponding to the target network layer is param, param = get_param (bw, max), bw is used to indicate the quantization bit width corresponding to the target network layer, and the quantization bit width corresponding to the target network layer can be a fixed quantization bit width, which can be configured based on experience, such as 4bit, 8bit, 16bit, etc., and the fixed quantization bit width is 8bit as an example. param = get_param (bw, max) indicates the initial weight quantization hyperparameter calculated based on the data range max under the bw quantization bit width. Since the method of calculating the initial weight quantization hyperparameter is different under different quantization algorithms, an example of the initial weight quantization hyperparameter is given here: param = bw-1-ceil (log2 (max)).

[0134] Then, the quantization analysis module constructs a weight quantization hyperparameter candidate list based on the initial weight quantization hyperparameter, and the weight quantization hyperparameter candidate list may include multiple candidate weight quantization hyperparameters, that is, multiple candidate weight quantization hyperparameters corresponding to the target network layer.

[0135] For example, considering that there are usually outliers (abnormal weight values, the numerical range of abnormal weight values ​​is usually very large) in the weight values ​​of the neural network, these outliers are too large and will affect the quantization accuracy. Therefore, a weight quantization hyperparameter candidate list can be constructed based on the initial weight quantization hyperparameter. For example, an example of the weight quantization hyperparameter candidate list can be: param l_list =[param l -2,param l -1,param l ,param l +1,param l +2]; Of course, the above is only an example of the weight quantization hyperparameter candidate list, and there is no restriction on the candidate weight quantization hyperparameters in this weight quantization hyperparameter candidate list. Taking the target network layer as network layer l as an example, param l Indicates the initial weight quantization hyperparameter corresponding to the network layer l, param l -2,param l -1,param l ,param l +1,param l +2 are the five candidate weight quantization hyperparameters corresponding to network layer l.

[0136] Step S15: For each candidate weight quantization hyperparameter, the quantization analysis module uses a fixed quantization bit width and the candidate weight quantization hyperparameter to simulate and quantize the floating-point weight of the target network layer to obtain a candidate simulation weight.

[0137] Exemplarily, the quantization bit width corresponding to the target network layer may be a fixed quantization bit width, and the fixed quantization bit width may be configured based on experience, such as 4 bits, 8 bits, 16 bits, etc., and the fixed quantization bit width of 8 bits is taken as an example for explanation.

[0138] For example, each candidate weight quantization hyperparameter in the weight quantization hyperparameter candidate list is traversed in turn. For the currently traversed candidate weight quantization hyperparameter, the floating-point weight of the target network layer can be simulated and quantized using a fixed quantization bit width and the candidate weight quantization hyperparameter to obtain the candidate simulation weight corresponding to the candidate weight quantization hyperparameter. For example, the following formula can be used to simulate and quantize the floating-point weight of the target network layer to obtain the candidate simulation weight: quant = 2 -param ×clip(round(data×2 param ),-2 bw-1 ,2 bw-1 -1). Among them, quant is used to represent the candidate simulation weight, param is used to represent the candidate weight quantization hyperparameter, data is used to represent the floating-point weight of the target network layer, bw is used to represent the fixed quantization bit width, clip is used to represent the truncation function, and round is used to represent the rounding operation. Since the simulation quantization method is different under different quantization algorithms, the above formula is just an example of simulation quantization.

[0139] To sum up, for each candidate weight quantization hyperparameter, after substituting the candidate weight quantization hyperparameter param into the above formula, the candidate simulation weight quant corresponding to the candidate weight quantization hyperparameter can be obtained.

[0140] Step S16: After obtaining the candidate simulation weights, the quantization analysis module processes the floating-point input features of the target network layer based on the candidate simulation weights to obtain the floating-point output features of the target network layer. The quantization analysis module determines the quantization error corresponding to the candidate weight quantization hyperparameter based on the floating-point output features of the target network layer.

[0141] First, the floating-point input features of the target network layer can be obtained. For example, if the target network layer is the first network layer of the sample decoding network, the input features of the sample decoding network are the floating-point input features of the target network layer. Alternatively, if the target network layer is not the first network layer of the sample decoding network, the output features of the previous network layer of the target network layer are used as the floating-point input features of the target network layer. For the network layer before the target network layer (referred to as network layer A), if network layer A is not a network layer that requires fixed-point weights, network layer A directly processes the input features to obtain floating-point output features, and this floating-point output feature is used as the floating-point input feature of the target network layer. If network layer A is a network layer that needs to use fixed-point weights, after obtaining the target weight quantization hyperparameters and target weight quantization bit width of network layer A, the target weight quantization hyperparameters and the target weight quantization bit width can also be used to simulate and quantize the floating-point weights of network layer A to obtain the pseudo-weights of network layer A. The floating-point input features of network layer A are processed through the pseudo-weights to obtain the floating-point output features of network layer A. The floating-point output features are used as the floating-point input features of the target network layer.

[0142] Then, the floating-point input features of the target network layer are processed based on the candidate simulation weights to obtain the floating-point output features of the target network layer. For example, if the target network layer is a convolutional layer, the floating-point input features of the target network layer are convolved based on the candidate simulation weights to obtain the floating-point output features of the target network layer. If the target network layer is a pooling layer, the floating-point input features of the target network layer are pooled based on the candidate simulation weights to obtain the floating-point output features of the target network layer. Of course, the above is just an example, and there is no limitation on this processing method, which is related to the functions supported by the target network layer.

[0143] Then, the quantization error corresponding to the candidate weight quantization hyperparameter is determined based on the floating-point output feature of the target network layer. For example, the sample output feature is determined based on the floating-point output feature of the target network layer, and the sample output feature can be the output feature of the reference network layer in the sample decoding network, and the reference network layer is any network layer located after the target network layer.

[0144] For example, the floating-point output features of the target network layer are used as the input features of the next network layer (the next network layer still uses floating-point weights), and the next network layer processes the input features to obtain the floating-point output features of the next network layer, and so on, all the way to the reference network layer in the sample decoding network, the reference network layer processes the input features to obtain the output features of the reference network layer, and the output features of the reference network layer are used as the sample output features.

[0145] Then, the quantization error corresponding to the candidate weight quantization hyperparameter is determined based on the sample output features and the reference output features; for example, based on the sample output features and the reference output features, the quantization error can be determined using an error loss function, and the error loss function may include but is not limited to mse, cosine similarity, KL divergence and other loss functions, without limitation.

[0146] Among them, the sample output feature may be the average value of the sample output features corresponding to all sample code streams, and the reference output feature may be the average value of the reference output features corresponding to all sample code streams; or, the sample output feature may be the maximum value of the sample output features corresponding to all sample code streams, and the reference output feature may be the maximum value of the reference output features corresponding to all sample code streams; or, the sample output feature may be the minimum value of the sample output features corresponding to all sample code streams, and the reference output feature may be the minimum value of the reference output features corresponding to all sample code streams, and there is no limitation on this.

[0147] Step S17: Based on the quantization error corresponding to each candidate weight quantization hyperparameter, the quantization analysis module determines the candidate weight quantization hyperparameter corresponding to the minimum quantization error as the target weight quantization hyperparameter corresponding to the target network layer, and determines the fixed quantization bit width as the target weight quantization bit width corresponding to the target network layer. At this point, the quantization analysis module can obtain the target weight quantization hyperparameter corresponding to the target network layer and the target weight quantization bit width corresponding to the target network layer.

[0148] Step S18: The quantization analysis module uses the target weight quantization hyperparameter of the target network layer and the target weight quantization bit width of the target network layer to simulate and quantize the floating-point weights of the target network layer to obtain the pseudo weights of the target network layer.

[0149] Exemplarily, after obtaining the target weight quantization hyperparameter and the target weight quantization bit width, the target weight quantization hyperparameter and the target weight quantization bit width can also be used to simulate and quantize the floating-point weights of the target network layer to obtain the pseudo-weights of the target network layer, that is, the pseudo-weights of the target network layer are used to replace the floating-point weights of the target network layer. For the floating-point input features of the target network layer, the target network layer processes the floating-point input features of the target network layer through the pseudo-weights to obtain the floating-point output features of the target network layer, and the floating-point output features are used as the floating-point input features of the subsequent network layer.

[0150] For example, based on the target weight quantization hyperparameter and the target weight quantization bit width, the following formula can be used to simulate and quantize the floating-point weight to obtain the pseudo weight: quant = 2 -param ×clip(round(data×2 param ),-2 bw-1 ,2bw-1 -1). Among them, quant is used to represent the pseudo weight of the target network layer, param is used to represent the target weight quantization hyperparameter of the target network layer, data is used to represent the floating-point weight of the target network layer, bw is used to represent the target weight quantization bit width of the target network layer, clip is used to represent the truncation function, and round is used to represent the rounding operation. Among them, since the simulation quantization method is different under different quantization algorithms, the above formula only gives an example of simulation quantization.

[0151] Exemplarily, for each target network layer, after obtaining the target weight quantization hyperparameter and the target weight quantization bit width of the target network layer, the target eigenvalue quantization bit width and the target eigenvalue quantization hyperparameter of the target network layer can also be obtained. For example, the target eigenvalue quantization bit width and the target eigenvalue quantization hyperparameter of the target network layer can be obtained by the following steps:

[0152] Step S19, for each target network layer in the sample decoding network: the quantization analysis module obtains the eigenvalue quantization super-parameter candidate list and the quantization bit width set corresponding to the target network layer, the eigenvalue quantization super-parameter candidate list may include multiple candidate eigenvalue quantization super-parameters, and the quantization bit width set may include at least two quantization bit widths.

[0153] First, the quantitative analysis module can determine the maximum eigenvalue based on all floating-point input features of the target network layer; wherein each floating-point input feature includes multiple eigenvalues, and the maximum eigenvalue is the maximum value of all eigenvalues ​​of all floating-point input features. For example, the target network layer may correspond to multiple floating-point input features (such as 100 sample code streams will correspond to 100 floating-point input features), each floating-point input feature includes multiple eigenvalues, and for each floating-point input feature, the maximum value can be selected from all eigenvalues ​​corresponding to the floating-point input feature, and then the maximum eigenvalue can be selected from the maximum values ​​corresponding to all floating-point input features. For example, assuming that the target network layer is network layer l, the maximum eigenvalue can be recorded as max l =max(abs(f 0_l ),abs(f 1_l ),…,abs(f n_l )), abs(f 0_l ) represents the maximum value corresponding to the first floating-point input feature of network layer l, and so on. l Represents the maximum eigenvalue among all floating-point input features.

[0154] Then, the quantization analysis module generates the initial eigenvalue quantization hyperparameter corresponding to the target network layer based on the maximum eigenvalue. For example, the following formula can be used to generate the initial eigenvalue quantization hyperparameter corresponding to the target network layer: param = bw-1-ceil(log2(max)); where param is used to represent the initial eigenvalue quantization hyperparameter corresponding to the target network layer, bw is used to represent the current quantization bit width, max is used to represent the maximum eigenvalue, and ceil is used to represent the rounding up operation.

[0155] For example, the initial eigenvalue quantization hyperparameter corresponding to the target network layer is param, param = get_param (bw, max), bw is used to represent the quantization bit width corresponding to the target network layer, and the quantization bit width corresponding to the target network layer is the current quantization bit width (that is, the quantization bit width currently traversed from the quantization bit width set). param = get_param (bw, max) means the initial eigenvalue quantization hyperparameter calculated based on the data range max under the bw quantization bit width. Since the methods for calculating the initial eigenvalue quantization hyperparameters are different under different quantization algorithms, this is just an example of the initial eigenvalue quantization hyperparameter, and there is no restriction on this.

[0156] Then, the quantization analysis module constructs a candidate list of eigenvalue quantization hyperparameters based on the initial eigenvalue quantization hyperparameters. The candidate list of eigenvalue quantization hyperparameters may include multiple candidate eigenvalue quantization hyperparameters, that is, multiple candidate eigenvalue quantization hyperparameters corresponding to the target network layer. For example, considering that there are usually outliers (abnormal eigenvalues, the numerical range of abnormal eigenvalues ​​is usually very large) in the eigenvalues ​​of neural networks, these outliers are too large and will affect the quantization accuracy. Therefore, a candidate list of eigenvalue quantization hyperparameters can be constructed based on the initial eigenvalue quantization hyperparameters. For example, an example of the candidate list of eigenvalue quantization hyperparameters can be: param l_list =[param l -2,param l -1,param l ,param l +1,param l +2]; Of course, the above is only an example of the candidate list of eigenvalue quantization hyperparameters, and there is no restriction on the candidate eigenvalue quantization hyperparameters in this candidate list of eigenvalue quantization hyperparameters. Taking the target network layer as network layer l as an example, param l It indicates the initial eigenvalue quantization hyperparameter corresponding to network layer l. The eigenvalue quantization hyperparameter candidate list contains 5 candidate eigenvalue quantization hyperparameters corresponding to network layer l.

[0157] The quantization analysis module can also generate a quantization bit width set, which can include at least two quantization bit widths, such as the quantization bit width set can include two quantization bit widths, such as 8bit, 16bit, or 8bit, 32bit, or 16bit, 32bit, or 4bit, 8bit. For another example, the quantization bit width set can include three quantization bit widths, such as 8bit, 16bit, 32bit, or 4bit, 8bit, 16bit, or 8bit, 16bit, 24bit. For another example, the quantization bit width set can include four quantization bit widths. Of course, the above are just a few examples of the quantization bit width set, and there is no limitation on this.

[0158] Step S20: Based on the order of quantization bit width from small to large, the quantization analysis module traverses the current quantization bit width from the quantization bit width set. For each candidate eigenvalue quantization hyperparameter, the quantization analysis module uses the current quantization bit width and the candidate eigenvalue quantization hyperparameter to simulate and quantize the floating-point input feature of the target network layer to obtain the candidate input feature.

[0159] Exemplarily, based on the order of quantization bit widths from small to large, the quantization analysis module first traverses the first quantization bit width as the current quantization bit width. For example, assuming that the quantization bit width set includes two quantization bit widths, such as 8 bits and 16 bits, the quantization analysis module first traverses the quantization bit width of 8 bits as the current quantization bit width, and executes step S20 based on the current quantization bit width.

[0160] For example, each candidate eigenvalue quantization hyperparameter in the candidate list of eigenvalue quantization hyperparameters is traversed in turn. For the currently traversed candidate eigenvalue quantization hyperparameter, the current quantization bit width and the candidate eigenvalue quantization hyperparameter can be used to simulate and quantize the floating-point input features of the target network layer to obtain the candidate input features corresponding to the candidate eigenvalue quantization hyperparameter. For example, the following formula can be used to simulate and quantize the floating-point input features of the target network layer to obtain the candidate input features: quant = 2 -param ×clip(round(data×2 param ),-2 bw-1 ,2 bw-1 -1); where quant is used to represent the candidate input feature, param is used to represent the candidate feature value quantization hyperparameter, data is used to represent the floating-point input feature, bw is used to represent the current quantization bit width, clip is used to represent the truncation function, and round is used to represent the rounding operation. Since the simulation quantization method is different under different quantization algorithms, the above formula only gives an example of simulation quantization.

[0161] To sum up, for each candidate eigenvalue quantization hyperparameter, after substituting the candidate eigenvalue quantization hyperparameter param into the above formula, the candidate input feature quant corresponding to the candidate eigenvalue quantization hyperparameter can be obtained.

[0162] Step S21, after obtaining the candidate input feature, the quantization analysis module processes the candidate input feature of the target network layer based on the pseudo weight of the target network layer to obtain the floating-point output feature of the target network layer. The quantization analysis module determines the quantization error corresponding to the candidate feature value quantization hyperparameter based on the floating-point output feature of the target network layer.

[0163] First, for the floating-point input feature of the target network layer (the content of the floating-point input feature can be found in step S16), the current quantization bit width and the candidate eigenvalue quantization hyperparameter have been used to simulate and quantize the floating-point input feature to obtain the candidate input feature corresponding to the candidate eigenvalue quantization hyperparameter, and this candidate input feature is used as the input feature of the target network layer.

[0164] Then, the candidate input feature is processed based on the pseudo weight of the target network layer to obtain the floating-point output feature of the target network layer. For example, if the target network layer is a convolutional layer, the candidate input feature is convolutionally processed based on the pseudo weight to obtain the floating-point output feature of the target network layer. If the target network layer is a pooling layer, the candidate input feature is pooled based on the pseudo weight to obtain the floating-point output feature of the target network layer. Of course, the above is just an example, and there is no limitation on this processing method, which is related to the functions supported by the target network layer. Among them, after obtaining the target weight quantization hyperparameter and the target weight quantization bit width corresponding to the target network layer, the floating-point weight of the target network layer will be simulated and quantized to obtain the pseudo weight. Therefore, the candidate input feature is processed based on the pseudo weight of the target network layer to obtain the floating-point output feature.

[0165] Then, the quantization error corresponding to the candidate feature value quantization hyperparameter is determined based on the floating-point output feature of the target network layer. For example, the sample output feature is determined based on the floating-point output feature of the target network layer, and the sample output feature can be the output feature of the reference network layer in the sample decoding network, and the reference network layer is any network layer located after the target network layer.

[0166] For example, the floating-point output features of the target network layer are used as the input features of the next network layer (the next network layer still uses floating-point weights), and the next network layer processes the input features to obtain the floating-point output features of the next network layer, and so on, all the way to the reference network layer in the sample decoding network, the reference network layer processes the input features to obtain the output features of the reference network layer, and the output features of the reference network layer are used as the sample output features.

[0167] Then, the quantization error corresponding to the candidate eigenvalue quantization hyperparameter is determined based on the sample output features and the reference output features; for example, based on the sample output features and the reference output features, an error loss function can be used to determine the quantization error, and the error loss function may include but is not limited to mse, cosine similarity, KL divergence and other loss functions, without limitation.

[0168] Among them, the sample output feature may be the average value of the sample output features corresponding to all sample code streams, and the reference output feature may be the average value of the reference output features corresponding to all sample code streams; or, the sample output feature may be the maximum value of the sample output features corresponding to all sample code streams, and the reference output feature may be the maximum value of the reference output features corresponding to all sample code streams; or, the sample output feature may be the minimum value of the sample output features corresponding to all sample code streams, and the reference output feature may be the minimum value of the reference output features corresponding to all sample code streams, and there is no limitation on this.

[0169] Step S22: Based on the quantization error corresponding to each candidate eigenvalue quantization hyperparameter, the quantization analysis module determines whether the minimum quantization error is less than a preset threshold (which can be configured based on experience). If so, execute step S23; if not, execute step S24.

[0170] Step S23, the quantization analysis module determines the candidate eigenvalue quantization hyperparameter corresponding to the minimum quantization error as the target eigenvalue quantization hyperparameter corresponding to the target network layer, and determines the current quantization bit width as the target eigenvalue quantization bit width corresponding to the target network layer, and records the target eigenvalue quantization hyperparameter and target eigenvalue quantization bit width for the target network layer.

[0171] At this point, the quantization analysis module can obtain the target eigenvalue quantization hyperparameters and target eigenvalue quantization bit width corresponding to the target network layer, and record the target eigenvalue quantization hyperparameters and target eigenvalue quantization bit width for the target network layer.

[0172] Step S24: The quantization analysis module determines whether the current quantization bit width is the last quantization bit width in the quantization bit width set.

[0173] If so, the candidate eigenvalue quantization hyperparameter corresponding to the minimum quantization error is determined as the target eigenvalue quantization hyperparameter corresponding to the target network layer, and the current quantization bit width is determined as the target eigenvalue quantization bit width corresponding to the target network layer, and the target eigenvalue quantization hyperparameter and target eigenvalue quantization bit width are recorded for the target network layer. This process can be referred to step S23.

[0174] If not, the next quantization bit width after the current quantization bit width is traversed as the current quantization bit width, and the process returns to step S20. Based on the current quantization bit width, for each candidate eigenvalue quantization hyperparameter, the quantization analysis module uses the current quantization bit width and the candidate eigenvalue quantization hyperparameter to simulate and quantize the floating-point input features of the target network layer to obtain candidate input features.

[0175] For example, the first traversal outputs a quantization bit width of 8 bits as the current quantization bit width. If the minimum quantization error is not less than a preset threshold, the second traversal outputs a quantization bit width of 16 bits as the current quantization bit width, and the above steps are repeated, and so on.

[0176] Exemplarily, after recording the target eigenvalue quantization hyperparameters and target eigenvalue quantization bit width for the target network layer, the quantization analysis module successfully obtains the target weight quantization hyperparameters, target weight quantization bit width, target eigenvalue quantization hyperparameters and target eigenvalue quantization bit width corresponding to the target network layer. At this point, the processing process of the target network layer is completed, and the above steps are repeated for the next target network layer of the target network layer, and so on, until the target weight quantization hyperparameters, target weight quantization bit width, target eigenvalue quantization hyperparameters and target eigenvalue quantization bit width corresponding to each target network layer are obtained.

[0177] Step S25, after obtaining the target weight quantization hyperparameters, target weight quantization bit width, target eigenvalue quantization hyperparameters and target eigenvalue quantization bit width corresponding to each target network layer, the decoding network quantization module generates a target decoding network based on the sample decoding network. For example, for each target network layer in the sample decoding network: the decoding network quantization module performs fixed-point quantization on the floating-point weights of the target network layer based on the target weight quantization bit width and target weight quantization hyperparameters corresponding to the target network layer to obtain the fixed-point weights of the target network layer. After obtaining the fixed-point weights of each target network layer, the decoding network quantization module can generate a target decoding network based on the fixed-point weights of each target network layer.

[0178] For example, for each target network layer, the quantization analysis module can output the target weight quantization bit width and target weight quantization hyperparameters corresponding to the target network layer to the decoding network quantization module, and the decoding network quantization module performs fixed-point quantization on the floating-point weights of the target network layer based on the target weight quantization bit width and target weight quantization hyperparameters corresponding to the target network layer to obtain the fixed-point weights of the target network layer, and there is no restriction on the fixed-point quantization process of the floating-point weights.

[0179] For each target network layer, the quantization analysis module can also output the target eigenvalue quantization hyperparameters and target eigenvalue quantization bit width corresponding to the target network layer to the decoding network quantization module, and the decoding network quantization module records the target eigenvalue quantization hyperparameters and target eigenvalue quantization bit width for the target network layer. For example, the decoding network quantization module can record the target eigenvalue quantization hyperparameters and the target eigenvalue quantization bit width in the configuration information of the target network layer. When the target network layer performs related processing, the target eigenvalue quantization hyperparameters and the target eigenvalue quantization bit width can be read from the configuration information. Of course, the target eigenvalue quantization hyperparameters and the target eigenvalue quantization bit width can also be recorded in other storage areas, as long as the target network layer can obtain the target eigenvalue quantization hyperparameters and the target eigenvalue quantization bit width.

[0180] After obtaining the fixed-point weights of each target network layer, the fixed-point weights can be used to replace the floating-point weights of the target network layer, so that all the replaced target network layers can be formed into a target decoding network. At this point, the target decoding network is successfully obtained, and the target decoding network can be deployed to the encoding end or the decoding end. For example, the target decoding network may include a probabilistic hyperparameter decoding network, a synthetic transformation network, or the target decoding network may include a mean prediction network, a probabilistic hyperparameter decoding network, and a synthetic transformation network. By deploying these target decoding networks on the encoding end, when the encoding end uses Example 2 to perform processing, the processing is performed based on the target decoding network. In addition, by deploying these target decoding networks on the decoding end, when the decoding end uses Example 3 to perform processing, the processing is performed based on the target decoding network.

[0181] Example 7: See Fig. 6A As shown, in Figure 5A Based on the decoding network shown in the figure, three modules are added: code stream extraction module, quantization analysis module, and decoding network quantization module; see Figure 6B As shown, in Figure 5B Three modules are added to the decoding network shown: a code stream extraction module, a quantization analysis module, and a decoding network quantization module. The sample decoding network includes multiple network layers using floating-point weights, and the multiple network layers may include a target network layer. The target network layer is a network layer that needs to use fixed-point weights (that is, the floating-point weights of the target network layer need to be converted into fixed-point weights). For example, all network layers in the sample decoding network can be used as target network layers, some network layers in the sample decoding network can be used as target network layers, and all convolutional layers in the sample decoding network can be used as target network layers.

[0182] Exemplarily, in order to convert a sample decoding network into a target decoding network with fixed-point weights, the following steps may be used:

[0183] Step S31: The code stream extraction module obtains a sample code stream and inputs the sample code stream to the quantization analysis module.

[0184] Step S32: The quantitative analysis module receives the sample code stream corresponding to each frame of the image, and stores the sample code stream corresponding to each frame of the image.

[0185] Step S33, the quantization analysis module decodes the sample features from the sample bitstream, determines the floating-point input features corresponding to the sample decoding network based on the sample features, and determines the reference output features corresponding to the sample features based on the floating-point input features, and the reference output features can be the output features of the reference network layer in the sample decoding network. The reference network layer can be any network layer in the sample decoding network, for example, the last network layer in the sample decoding network, or the second to last network layer in the sample decoding network, or the last target network layer in the sample decoding network, or any network layer in the sample decoding network located after the last target network layer, and there is no restriction on the position of the reference network layer.

[0186] Exemplarily, steps S31 to S33 are similar to steps S11 to S13 and are not repeated here.

[0187] Step S34: for each target network layer in the sample decoding network: the quantization analysis module obtains a weight quantization super-parameter candidate list corresponding to the target network layer, and the weight quantization super-parameter candidate list may include multiple candidate weight quantization super-parameters.

[0188] First, the quantitative analysis module determines the maximum weight value based on all floating-point weights of the target network layer; wherein each floating-point weight may include multiple weight values, and the maximum weight value may be the maximum value of all weight values ​​of all floating-point weights. For example, the target network layer may include multiple floating-point weights, each floating-point weight may include multiple weight values, and for each floating-point weight, the maximum value may be selected from all weight values ​​corresponding to the floating-point weight, and then the maximum weight value may be selected from the maximum values ​​corresponding to all floating-point weights.

[0189] Then, the quantization analysis module generates the initial weight quantization hyperparameter corresponding to the target network layer based on the maximum weight value. For example, the following formula is used to generate the initial weight quantization hyperparameter corresponding to the target network layer: param = (max) / 2 bw -1; Among them, param is used to indicate the initial weight quantization hyperparameter corresponding to the target network layer, bw is used to indicate the fixed quantization bit width, and max is used to indicate the maximum weight value. Among them, the initial weight quantization hyperparameter corresponding to the target network layer is param, param = get_param (bw, max), bw is used to indicate the quantization bit width corresponding to the target network layer, and the quantization bit width corresponding to the target network layer can be a fixed quantization bit width, such as 4bit, 8bit, 16bit, etc. param = get_param (bw, max) means the initial weight quantization hyperparameter calculated according to the data range max under the bw quantization bit width. Since the method of calculating the initial weight quantization hyperparameter is different under different quantization algorithms, here is just an example of the initial weight quantization hyperparameter.

[0190] Then, the quantization analysis module constructs a weight quantization hyperparameter candidate list based on the initial weight quantization hyperparameter. The weight quantization hyperparameter candidate list may include multiple candidate weight quantization hyperparameters, that is, multiple candidate weight quantization hyperparameters corresponding to the target network layer. For example, considering that there are usually outliers (abnormal weight values) in the weight values ​​of the neural network, a weight quantization hyperparameter candidate list can be constructed based on the initial weight quantization hyperparameter. For example, an example of the weight quantization hyperparameter candidate list can be: Of course, the above is only an example of a weight quantization hyperparameter candidate list, and there is no restriction on the candidate weight quantization hyperparameters in this weight quantization hyperparameter candidate list. l Indicates the initial weight quantization hyperparameter corresponding to network layer l. The weight quantization hyperparameter candidate list shows 7 candidate weight quantization hyperparameters corresponding to network layer l.

[0191] Step S35: For each candidate weight quantization hyperparameter, the quantization analysis module uses a fixed quantization bit width and the candidate weight quantization hyperparameter to simulate and quantize the floating-point weight of the target network layer to obtain a candidate simulation weight.

[0192] Exemplarily, the quantization bit width corresponding to the target network layer can be a fixed quantization bit width, and the fixed quantization bit width can be configured based on experience, and there is no restriction on this. Each candidate weight quantization hyperparameter in the weight quantization hyperparameter candidate list can be traversed in turn. For the currently traversed candidate weight quantization hyperparameter, the fixed quantization bit width and the candidate weight quantization hyperparameter can be used to simulate and quantize the floating-point weights of the target network layer to obtain the candidate simulation weights corresponding to the candidate weight quantization hyperparameters. For example, the following formula can be used to simulate and quantize the floating-point weights of the target network layer to obtain the candidate simulation weights: quant = param × clip (round (data ÷ param), -2 bw-1 ,2 bw-1-1). Among them, quant is used to represent the candidate simulation weight, param is used to represent the candidate weight quantization hyperparameter, data is used to represent the floating-point weight of the target network layer, bw is used to represent the fixed quantization bit width, clip is used to represent the truncation function, and round is used to represent the rounding operation. Since the simulation quantization method is different under different quantization algorithms, the above formula is just an example of simulation quantization.

[0193] To sum up, for each candidate weight quantization hyperparameter, after substituting the candidate weight quantization hyperparameter param into the above formula, the candidate simulation weight quant corresponding to the candidate weight quantization hyperparameter can be obtained.

[0194] Step S36: After obtaining the candidate simulation weights, the quantization analysis module processes the floating-point input features of the target network layer based on the candidate simulation weights to obtain the floating-point output features of the target network layer. The quantization analysis module determines the quantization error corresponding to the candidate weight quantization hyperparameter based on the floating-point output features of the target network layer.

[0195] Step S37: Based on the quantization error corresponding to each candidate weight quantization hyperparameter, the quantization analysis module determines the candidate weight quantization hyperparameter corresponding to the minimum quantization error as the target weight quantization hyperparameter corresponding to the target network layer, and determines the fixed quantization bit width as the target weight quantization bit width corresponding to the target network layer. At this point, the quantization analysis module can obtain the target weight quantization hyperparameter corresponding to the target network layer and the target weight quantization bit width corresponding to the target network layer.

[0196] Exemplarily, step S36-step S37 are similar to step S16-step S17, and will not be repeated here.

[0197] Step S38: The quantization analysis module uses the target weight quantization hyperparameter of the target network layer and the target weight quantization bit width of the target network layer to simulate and quantize the floating-point weights of the target network layer to obtain the pseudo weights of the target network layer.

[0198] Exemplarily, after obtaining the target weight quantization hyperparameter and the target weight quantization bit width, the target weight quantization hyperparameter and the target weight quantization bit width can also be used to simulate and quantize the floating-point weights of the target network layer to obtain the pseudo-weights of the target network layer. For the floating-point input features of the target network layer, the target network layer processes the floating-point input features of the target network layer through the pseudo-weights to obtain the floating-point output features of the target network layer, and the floating-point output features are used as the floating-point input features of the subsequent network layer. For example, the following formula can be used to simulate and quantize the floating-point weights to obtain pseudo-weights: quant = param × clip (round (data ÷ param), -2 bw-1,2 bw-1 -1).

[0199] Among them, quant is used to represent the pseudo weight of the target network layer, param is used to represent the target weight quantization hyperparameter of the target network layer, data is used to represent the floating-point weight of the target network layer, bw is used to represent the target weight quantization bit width of the target network layer, clip is used to represent the truncation function, and round is used to represent the rounding operation. Among them, since the simulation quantization method is different under different quantization algorithms, the above formula only gives an example of simulation quantization.

[0200] Exemplarily, for each target network layer, after obtaining the target weight quantization hyperparameter and the target weight quantization bit width of the target network layer, the target eigenvalue quantization bit width and the target eigenvalue quantization hyperparameter of the target network layer can also be obtained. For example, the target eigenvalue quantization bit width and the target eigenvalue quantization hyperparameter of the target network layer can be obtained by the following steps:

[0201] Step S39, for each target network layer in the sample decoding network: the quantization analysis module obtains the eigenvalue quantization super-parameter candidate list and the quantization bit width set corresponding to the target network layer, the eigenvalue quantization super-parameter candidate list may include multiple candidate eigenvalue quantization super-parameters, and the quantization bit width set may include at least two quantization bit widths.

[0202] First, the quantitative analysis module can determine the maximum eigenvalue based on all floating-point input features of the target network layer; wherein each floating-point input feature includes multiple eigenvalues, and the maximum eigenvalue is the maximum value of all eigenvalues ​​of all floating-point input features. For example, the target network layer can correspond to multiple floating-point input features, each of which includes multiple eigenvalues. For each floating-point input feature, the maximum value can be selected from all eigenvalues ​​corresponding to the floating-point input feature, and then the maximum eigenvalue can be selected from the maximum values ​​corresponding to all floating-point input features.

[0203] Then, the quantization analysis module generates the initial eigenvalue quantization hyperparameter corresponding to the target network layer based on the maximum eigenvalue. For example, the following formula is used to generate the initial eigenvalue quantization hyperparameter corresponding to the target network layer: param = (max) / 2 bw -1; Among them, param is used to represent the initial eigenvalue quantization hyperparameter, bw is used to represent the fixed quantization bit width, and max is used to represent the maximum eigenvalue. The initial eigenvalue quantization hyperparameter is param, param=get_param(bw,max), bw is used to represent the quantization bit width corresponding to the target network layer, and the quantization bit width corresponding to the target network layer is the current quantization bit width (that is, the quantization bit width currently traversed from the quantization bit width set). param=get_param(bw,max) represents the initial eigenvalue quantization hyperparameter calculated according to the data range max under the bw quantization bit width. Since the method of calculating the initial eigenvalue quantization hyperparameter is different under different quantization algorithms, here is just an example of the initial eigenvalue quantization hyperparameter, and there is no restriction on this.

[0204] Then, the quantization analysis module can construct a candidate list of eigenvalue quantization hyperparameters based on the initial eigenvalue quantization hyperparameters. The candidate list of eigenvalue quantization hyperparameters can include multiple candidate eigenvalue quantization hyperparameters, that is, multiple candidate eigenvalue quantization hyperparameters corresponding to the target network layer. For example, considering that there are usually outliers in the eigenvalues ​​of the neural network, these outliers are too large and will affect the quantization accuracy. Therefore, the quantization analysis module can construct a candidate list of eigenvalue quantization hyperparameters based on the initial eigenvalue quantization hyperparameters. For example, an example of the candidate list of eigenvalue quantization hyperparameters can be: Of course, the above is only an example of the eigenvalue quantization hyperparameter candidate list, and there is no restriction on this eigenvalue quantization hyperparameter candidate list.

[0205] The quantization analysis module may also generate a quantization bit width set, which may include at least two quantization bit widths.

[0206] Step S40: Based on the order of quantization bit width from small to large, the quantization analysis module traverses the current quantization bit width from the quantization bit width set. For each candidate feature value quantization hyperparameter, the quantization analysis module uses the current quantization bit width and the candidate feature value quantization hyperparameter to simulate and quantize the floating-point input feature of the target network layer to obtain the candidate input feature.

[0207] Exemplarily, based on the order of quantization bit width from small to large, the quantization analysis module can first traverse the first quantization bit width as the current quantization bit width. Each candidate eigenvalue quantization hyperparameter in the eigenvalue quantization hyperparameter candidate list is traversed in turn. For the currently traversed candidate eigenvalue quantization hyperparameter, the current quantization bit width and the candidate eigenvalue quantization hyperparameter can be used to simulate and quantize the floating-point input features of the target network layer to obtain the candidate input features corresponding to the candidate eigenvalue quantization hyperparameter. For example, the following formula can be used to simulate and quantize the floating-point input features of the target network layer to obtain the candidate input features: quant = param × clip (round (data ÷ param), -2 bw-1 ,2 bw-1 -1); where quant is used to represent the candidate input feature, param is used to represent the candidate feature value quantization hyperparameter, data is used to represent the floating-point input feature, bw is used to represent the current quantization bit width, clip is used to represent the truncation function, and round is used to represent the rounding operation. Since the simulation quantization method is different under different quantization algorithms, the above formula only gives an example of simulation quantization.

[0208] To sum up, for each candidate eigenvalue quantization hyperparameter, after substituting the candidate eigenvalue quantization hyperparameter param into the above formula, the candidate input feature quant corresponding to the candidate eigenvalue quantization hyperparameter can be obtained.

[0209] Step S41: After obtaining the candidate input feature, the quantization analysis module processes the candidate input feature of the target network layer based on the pseudo weight of the target network layer to obtain the floating-point output feature of the target network layer. The quantization analysis module determines the quantization error corresponding to the candidate feature value quantization hyperparameter based on the floating-point output feature of the target network layer.

[0210] Step S42: Based on the quantization error corresponding to each candidate eigenvalue quantization hyperparameter, the quantization analysis module determines whether the minimum quantization error is less than a preset threshold (which can be configured based on experience). If so, execute step S43; if not, execute step S44.

[0211] Step S43, the quantization analysis module determines the candidate eigenvalue quantization hyperparameter corresponding to the minimum quantization error as the target eigenvalue quantization hyperparameter corresponding to the target network layer, and determines the current quantization bit width as the target eigenvalue quantization bit width corresponding to the target network layer, and records the target eigenvalue quantization hyperparameter and target eigenvalue quantization bit width for the target network layer.

[0212] Step S44: The quantization analysis module determines whether the current quantization bit width is the last quantization bit width in the quantization bit width set.

[0213] If so, the candidate eigenvalue quantization hyperparameter corresponding to the minimum quantization error is determined as the target eigenvalue quantization hyperparameter corresponding to the target network layer, and the current quantization bit width is determined as the target eigenvalue quantization bit width corresponding to the target network layer, and the target eigenvalue quantization hyperparameter and target eigenvalue quantization bit width are recorded for the target network layer. This process can be seen in step S43.

[0214] If not, the next quantization bit width after the current quantization bit width is traversed as the current quantization bit width, and the process returns to step S40. Based on the current quantization bit width, for each candidate eigenvalue quantization hyperparameter, the quantization analysis module uses the current quantization bit width and the candidate eigenvalue quantization hyperparameter to simulate and quantize the floating-point input features of the target network layer to obtain candidate input features.

[0215] Step S45, after obtaining the target weight quantization hyperparameters, target weight quantization bit width, target eigenvalue quantization hyperparameters and target eigenvalue quantization bit width corresponding to each target network layer, the decoding network quantization module generates a target decoding network based on the sample decoding network. For example, for each target network layer in the sample decoding network: the decoding network quantization module performs fixed-point quantization on the floating-point weights of the target network layer based on the target weight quantization bit width and target weight quantization hyperparameters corresponding to the target network layer to obtain the fixed-point weights of the target network layer. After obtaining the fixed-point weights of each target network layer, the decoding network quantization module can generate a target decoding network based on the fixed-point weights of each target network layer.

[0216] Exemplarily, steps S41 to S45 are similar to steps S21 to S25 and are not repeated here.

[0217] After obtaining the fixed-point weights of each target network layer, the fixed-point weights can be used to replace the floating-point weights of the target network layer, so that all the replaced target network layers can be formed into a target decoding network. At this point, the target decoding network is successfully obtained, and the target decoding network can be deployed to the encoding end or the decoding end. For example, the target decoding network may include a probabilistic hyperparameter decoding network, a synthetic transformation network, or the target decoding network may include a mean prediction network, a probabilistic hyperparameter decoding network, and a synthetic transformation network. By deploying these target decoding networks on the encoding end, when the encoding end uses Example 2 to perform processing, the processing is performed based on the target decoding network. In addition, by deploying these target decoding networks on the decoding end, when the decoding end uses Example 3 to perform processing, the processing is performed based on the target decoding network.

[0218] Embodiment 8: When decoding a bitstream based on a neural network decoder, the sensitivity to quantization is different in each network layer of the neural network, that is, some network layers can still maintain quantization performance at a lower bit, while some network layers require a higher bit to ensure quantization accuracy. This is related to the parameters and feature distribution learned by the neural network. Therefore, under the use of a more friendly PTQ quantization, in order to achieve the maximum compression model bit width, while ensuring the performance of the compressed model, and at the same time the compressed model can be allowed on most devices, this embodiment proposes an efficient decoder compression and reasoning method, which mainly includes the following steps: 1. Collect several frames (such as 100-200 frames) of images in real scenes (application scenarios of image acquisition devices). 2. Input the image into the encoding network to extract the bitstream (i.e., the first bitstream and the second bitstream) to be transmitted to the decoding network, and store it. 3. Transmit the stored bitstream to the quantization analysis module to obtain the appropriate quantization bitwidth and quantization parameters (such as quantization hyperparameters) for each network layer of the decoding network. 4. According to the quantization bit width and quantization parameters of each network layer obtained by the quantization analysis module, each network layer of the decoding network is quantized to obtain the final quantization model, that is, each network layer of the sample decoding network is quantized to obtain the target decoding network.

[0219] Can be Figure 5A or Figure 5B On the basis of the decoding network shown in the figure, a code stream extraction module, a quantization analysis module and a decoding network quantization module are added. The functions of the code stream extraction module, the quantization analysis module and the decoding network quantization module are described below.

[0220] Code stream extraction module: used to collect several frames (such as 100-200 frames) of images in real scenes (application scenarios of image acquisition equipment), input the images into the encoding network for forward calculation, and extract the output of the encoding network (that is, the code stream used for transmission to the decoding network, that is, the first code stream and the second code stream), and store the code stream corresponding to each frame of the image.

[0221] Quantization analysis module: The bitstream information extracted by the bitstream extraction module will enter the quantization analysis module. The function of the quantization analysis module is to analyze the sensitivity of each network layer of the decoding network to quantization, and obtain the quantization bit width that should be allocated to each network layer of the decoding network while ensuring the decoding performance, that is, to obtain the quantization bit width of each network layer of the decoding network.

[0222] A forward calculation result of the decoding network is stored in the quantitative analysis module. For example, after the bitstream enters the decoding network, the forward decoding of the full floating point data type is first performed, and the input features of each network layer of the decoding network under floating point (f, i represents the i-th bitstream, l represents the l-th network layer) and the output features of the last layer (f i_o, i represents the i-th code stream) is stored, and the output features of the last layer are the reference output features of the above embodiment.

[0223] The quantization analysis module also supports the simulation quantization operation of the weights and input feature values ​​of each network layer of the decoding network, and can perform simulation quantization calculations to simulate the quantization reasoning of the decoding network. Among them, in the decoding network, the weights of the network layer (such as the convolution layer) are not sensitive to quantization under 8bit, while the input feature values ​​of the network layer (such as the convolution layer) are more sensitive to quantization. Therefore, using 8bit quantization in some network layers can meet the performance loss, but some network layers may be difficult to meet the quantization performance loss. Therefore, the quantization analysis module mainly analyzes the quantization of the input feature values ​​of the network layer (such as the convolution layer) of the decoding network, and selects the appropriate quantization bit width and quantization hyperparameters for the input of each network layer of the decoding network.

[0224] The quantization analysis module initializes the quantization hyperparameters of the input feature values ​​of each network layer of the decoding network. The initialization of the quantization hyperparameters depends on the data range of the floating-point features, which is determined by the input features of each network layer of the decoding network under floating point conditions (f i_l , i represents the i-th code stream, l represents the l-th network layer) statistics, that is, max l =max(abs(f 0_l ),abs(f 1_l ),…,abs(f n_l )). The initialized quantization hyperparameters are obtained by the calculation formula: param l =get_param(bw l ,max l ), bw l Indicates the bit width of the input feature of the lth layer, initialized to 8, indicating the quantization hyperparameter calculated based on the data range max under the bw quantization bit width. Since the way of calculating quantization hyperparameters is slightly different under different quantization algorithms, only one example is given here: param = bw-1-ceil(log2(max)). Considering that there are usually outliers (abnormal eigenvalues, the value range is usually very large) in the eigenvalues ​​of the neural network, these outliers are too large and will affect the quantization accuracy. Therefore, the candidate list of hyperparameters is set based on the initialized quantization hyperparameters, that is, param l_list =[param l -2,param l -1,param l ,param l +1,param l +2]. In addition, another example of calculating the quantization hyperparameter can also be: param = (max) / 2 bw-1, the candidate list of setting hyperparameters based on the initialized quantized hyperparameters is:

[0225]

[0226] The quantization analysis module will analyze the quantization of the decoding network layer by layer. The analysis method is to simulate and quantize the input feature values ​​layer by layer (such as 8-bit quantization initially), input the simulated quantized features into the network for forward calculation, save the output of the decoding network, and compare it with the floating-point output feature f saved in advance. i_o Perform error analysis. When the error is greater than a certain threshold, increase the quantization bit width of the layer to 16 bits. The calculation formula for simulation quantization can be: q =quant(data f ,bw,param), because the calculation formula of simulation quantization is different under different quantization algorithms, here is only an example of simulation quantization method: quant(data f ,bw,param)=2 -param ×clip(round(data f ×2 param ),-2 bw-1 ,2 bw-1 -1), and clip is used to represent the truncation function. In addition, for the simulation quantization process, another example of the simulation quantization method can also be:

[0227] quant(data f ,bw,param)=param×clip(round(data f ÷param),-2 bw-1 ,2 bw-1 -1).

[0228] In a possible implementation, the following steps may be used to calculate the quantization hyperparameter and quantization bit width of the eigenvalue:

[0229] Initialize the quantization bit width, such as setting the initial value of the quantization bit width to 8 bits.

[0230] Initialize the quantization hyperparameter, such as using the following formula to determine the initial value of the quantization hyperparameter: param = bw-1-ceil(log2(max)), the initial value of the quantization hyperparameter is the initial eigenvalue quantization hyperparameter of the above embodiment.

[0231] Initialize the quantization error, such as setting a preset threshold of the quantization error, which may be set based on experience.

[0232] Get the candidate list of quantization hyperparameters. For example, based on the initial value of the quantization hyperparameter, use the following formula to get the candidate list of quantization hyperparameters: param l_list =[param l -2,param l -1,param l ,param l +1,param l +2].

[0233] The input feature value is simulated and quantized, such as based on the quantization bit width and quantization hyperparameter, using the following formula for simulation quantization:

[0234] quant(data f ,bw,param)=2 -param ×clip(round(data f ×2 param ),-2 bw-1 ,2 bw-1 -1).

[0235] The simulated quantized feature values ​​are input into the decoding network for forward calculation.

[0236] Calculate the quantization error between the simulated quantized decoding network output and the floating-point decoding network output.

[0237] If the quantization error is less than the preset threshold, the quantization bit width of the output eigenvalue is 8 bits, and the quantization hyperparameter of the output eigenvalue is the quantization hyperparameter corresponding to the minimum quantization error (that is, a quantization hyperparameter in the quantization hyperparameter candidate list).

[0238] If the quantization error is not less than the preset threshold, the quantization bit width is increased to 16 bits.

[0239] Re-obtain the initial value of the quantization hyperparameter, that is, when using the above formula to obtain the initial value of the quantization hyperparameter, change the quantization bit width bw from 8 bits to 16 bits, and obtain the initial value of the quantization hyperparameter under the premise of 16 bits.

[0240] Re-obtain the quantization hyperparameter candidate list, such as obtaining the quantization hyperparameter candidate list based on the initial value of the quantization hyperparameter.

[0241] The input characteristic value is simulated and quantized, such as using the above formula to perform simulation and quantization.

[0242] The simulated quantized feature values ​​are input into the decoding network for forward calculation.

[0243] Calculate the quantization error between the simulated quantized decoding network output and the floating-point decoding network output.

[0244] The quantization bit width of the output eigenvalue is 16 bits, and the quantization hyperparameter of the output eigenvalue is the quantization hyperparameter corresponding to the minimum quantization error.

[0245] In the above process, when calculating the quantization error, the error loss function can be used to calculate the quantization error. The calculation method of the error loss function may include but is not limited to mse, cosine similarity, KL divergence, etc. After the above process calculation, the quantization bit width bw of each network layer input of the decoding network can be obtained. l And quantized hyperparameters param l .

[0246] Decoding network quantization module: According to the quantization bit width bw of each network layer input output by the quantization analysis module l And quantized hyperparameters param l , and the 8-bit quantization bit width of each network layer weight is fixed. The decoding network quantization module can quantize the decoding network. For example, the weight parameters of the decoding network will be quantized to 8 bits, which can greatly compress the storage space of the model. At the same time, the decoding network quantization module will mark the bit width information (8bit or 16bit) of the input of each network layer, and write the corresponding quantized hyperparameters into the calculation process of the network layer (usually expressed as a shift), so as to further compress the calculation amount and throughput bandwidth of the model while ensuring the decoding quality of the decoding network.

[0247] Embodiment 9: After obtaining the target decoding network, the target decoding network can be deployed to the encoding end or the decoding end. For example, the target decoding network includes a probabilistic hyperparameter decoding network and a synthetic transformation network, or the target decoding network includes a mean prediction network, a probabilistic hyperparameter decoding network, and a synthetic transformation network. By deploying these target decoding networks at the encoding end, when the encoding end uses Embodiment 2 to perform processing, the processing is performed based on the target decoding network. By deploying these target decoding networks at the decoding end, when the decoding end uses Embodiment 3 to perform processing, the processing is performed based on the target decoding network.

[0248] In a possible implementation, the encoding end or the decoding end may decode the target feature corresponding to the current image block from the code stream corresponding to the current image block (such as the first code stream and the second code stream), determine the input feature of the decoding network based on the target feature, process the input feature based on the decoding network to obtain the output feature of the decoding network, and after obtaining the output feature, determine the reconstructed image block corresponding to the current image block based on the output feature. This process is described below.

[0249] Exemplarily, the encoding end or the decoding end decodes the target feature corresponding to the current image block from the code stream corresponding to the current image block, and determines the first input feature of the target decoding network based on the target feature. Obtain the target feature value quantization bit width and the target feature value quantization hyperparameter of the target decoding network (such as obtaining the target feature value quantization bit width and the target feature value quantization hyperparameter from the configuration information of the target decoding network), and convert the first input feature into a second input feature based on the target feature value quantization bit width and the target feature value quantization hyperparameter. Process the second input feature based on the fixed-point weight of the target decoding network (the fixed-point weight can be determined based on the target weight quantization bit width and the target weight quantization hyperparameter) to obtain the output feature of the target decoding network. Determine the reconstructed image block corresponding to the current image block based on the output feature of the target decoding network.

[0250] Exemplarily, the target decoding network includes at least one target network layer, the target network layer is a network layer using fixed-point weights, and for each target network layer in the target decoding network: the first input feature of the target network layer can be converted into a second input feature based on the target feature value quantization bit width and the target feature value quantization hyperparameter of the target network layer; the second input feature is processed based on the fixed-point weight of the target network layer (the fixed-point weight of the target network layer is determined based on the target weight quantization bit width of the target network layer and the target weight quantization hyperparameter of the target network layer) to obtain the output feature of the target network layer. Among them, the target feature value quantization bit widths of different target network layers can be the same or different, the target feature value quantization hyperparameters of different target network layers can be the same or different, the target weight quantization bit widths of different target network layers can all be fixed quantization bit widths, and the target weight quantization hyperparameters of different target network layers can be the same or different.

[0251] Exemplarily, based on the target feature value quantization bit width and the target feature value quantization hyperparameter of the target network layer, the first input feature of the target network layer is converted into the second input feature, including but not limited to: using the following formula to convert the first input feature of the target network layer into the second input feature: quant = clip (round (data × 2 param ),-2 bw-1 ,2 bw-1 -1); or, quant = clip (round (data ÷ param), -2 bw-1 ,2 bw-1 -1); where quant is used to represent the second input feature, data is used to represent the first input feature, param is used to represent the target feature value quantization hyperparameter, bw is used to represent the target feature value quantization bit width, clip is used to represent the truncation function, and round is used to represent rounding.

[0252] See also Figure 5A As shown, after obtaining Bitstream#1 (i.e., the first bitstream) corresponding to the current image block, the encoding end or the decoding end decodes Bitstream#1 to obtain the hyperparameter quantization feature (i.e., the target feature of the current image block), dequantizes the hyperparameter quantization feature, and obtains the coefficient hyperparameter feature z_hat (i.e., the first input feature of the probability hyperparameter decoding network is determined based on the target feature). Alternatively, Bitstream#1 can be decoded to obtain the coefficient hyperparameter feature z_hat (i.e., the target feature of the current image block, which can be directly used as the first input feature of the probability hyperparameter decoding network).

[0253] After obtaining the coefficient hyperparameter feature z_hat (i.e., the first input feature of the probability hyperparameter decoding network), the coefficient hyperparameter feature z_hat is input to the probability hyperparameter decoding network (i.e., the target decoding network). The probability hyperparameter decoding network can use the target feature value quantization bit width and the target feature value quantization hyperparameter to convert the floating-point coefficient hyperparameter feature z_hat (i.e., the first input feature) into a fixed-point coefficient hyperparameter feature z_hat (i.e., the second input feature), and perform an inverse transformation of the coefficient hyperparameter feature of the fixed-point coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p corresponding to the current image block (i.e., the output feature of the probability hyperparameter decoding network). After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p.

[0254] For example, for each target network layer in the probabilistic hyperparameter decoding network, after obtaining the first floating-point input feature, the target network layer can convert the first floating-point input feature into a fixed-point second input feature based on the target feature value quantization bit width and the target feature value quantization hyperparameter of the target network layer. And because the target network layer uses fixed-point weights, the fixed-point second input feature can be processed based on the fixed-point weights of the target network layer to obtain the output feature of the target network layer. Among them, the output feature of the target network layer can be used as the input feature of the next network layer, and the output feature of the target network layer can be a floating-point feature, and so on.

[0255] See also Figure 5AAs shown, after obtaining Bitstream#2 (i.e., the second bitstream) corresponding to the current image block, the encoder or decoder decodes Bitstream#2 to obtain image quantization features (i.e., the target features of the current image block), dequantizes the image quantization features to obtain image features s', performs feature recovery on image features s', and obtains image features y_hat (i.e., determining the first input features of the synthetic transformation network based on the target features). Alternatively, Bitstream#2 can be decoded to obtain image features s' (i.e., the target features of the current image block), and features s' can be recovered to obtain image features y_hat (i.e., determining the first input features of the synthetic transformation network based on the target features). When decoding Bitstream#2, the encoder or decoder can decode Bitstream#2 based on a probability distribution model, and this process will not be described in detail.

[0256] After obtaining the image feature y_hat (i.e., the first input feature of the synthetic transformation network), the image feature y_hat is input to the synthetic transformation network (i.e., the target decoding network). The synthetic transformation network can use the target feature value quantization bit width and the target feature value quantization hyperparameter to convert the floating-point image feature y_hat (i.e., the first input feature) into a fixed-point image feature y_hat (i.e., the second input feature), and perform a synthetic transformation on the fixed-point image feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x (i.e., the output feature of the synthetic transformation network). In summary, it can be seen that the image feature y_hat can be synthetically transformed based on the synthetic transformation network to obtain the reconstructed image block x_hat corresponding to the current image block x.

[0257] For example, for each target network layer in the synthetic transformation network, after obtaining the first floating-point input feature, the target network layer can convert the first floating-point input feature into a fixed-point second input feature based on the target feature value quantization bit width and the target feature value quantization hyperparameter of the target network layer. And because the target network layer uses fixed-point weights, the fixed-point second input feature can be processed based on the fixed-point weights of the target network layer to obtain the output feature of the target network layer. Among them, the output feature of the target network layer can be used as the input feature of the next network layer, and the output feature of the target network layer can be a floating-point feature, and so on.

[0258] After obtaining the output features of the synthetic transformation network, the output features of the synthetic transformation network can be used as the reconstructed image block x_hat corresponding to the current image block x, that is, the reconstructed image block x_hat corresponding to the current image block x is determined based on the output features.

[0259] See also Figure 5BAs shown, after obtaining the Bitstream#1 corresponding to the current image block, the encoding end or the decoding end can decode Bitstream#1 to obtain the hyperparameter quantization feature (i.e., the target feature of the current image block), dequantize the hyperparameter quantization feature, and obtain the coefficient hyperparameter feature z_hat (i.e., determine the first input feature of the probability hyperparameter decoding network based on the target feature). Alternatively, Bitstream#1 can be decoded to obtain the coefficient hyperparameter feature z_hat (i.e., the target feature of the current image block, and the target feature is directly used as the first input feature of the probability hyperparameter decoding network).

[0260] After obtaining the coefficient hyperparameter feature z_hat (i.e., the first input feature of the probabilistic hyperparameter decoding network), the coefficient hyperparameter feature z_hat is input to the probabilistic hyperparameter decoding network (i.e., the target decoding network). The probabilistic hyperparameter decoding network can use the target feature value quantization bit width and the target feature value quantization hyperparameter to convert the floating-point coefficient hyperparameter feature z_hat (i.e., the first input feature) into a fixed-point coefficient hyperparameter feature z_hat (i.e., the second input feature), and perform a coefficient hyperparameter feature inverse transformation on the fixed-point coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p corresponding to the current image block (i.e., the output feature of the probabilistic hyperparameter decoding network).

[0261] For example, for each target network layer in the probabilistic hyperparameter decoding network, after obtaining the first floating-point input feature, the target network layer converts the first floating-point input feature into a fixed-point second input feature based on the target feature value quantization bit width and the target feature value quantization hyperparameter of the target network layer. The target network layer uses fixed-point weights, and the fixed-point second input feature is processed based on the fixed-point weights of the target network layer to obtain the output feature of the target network layer.

[0262] After obtaining the coefficient hyperparameter feature z_hat (i.e., the first input feature of the mean prediction network), the coefficient hyperparameter feature z_hat and the image feature y_hat of the previous image block are input to the mean prediction network (i.e., the target decoding network). The mean prediction network uses the target feature value quantization bit width and the target feature value quantization hyperparameter to convert the floating-point coefficient hyperparameter feature z_hat (first input feature) into the fixed-point coefficient hyperparameter feature z_hat (second input feature), and uses the target feature value quantization bit width and the target feature value quantization hyperparameter to convert the floating-point image feature y_hat (first input feature) into the fixed-point image feature y_hat (second input feature). The mean prediction network performs context-based prediction based on the fixed-point coefficient hyperparameter feature z_hat and the fixed-point image feature y_hat to obtain the prediction value mu corresponding to the current image block (i.e., the output feature of the mean prediction network).

[0263] For example, for each target network layer in the mean prediction network, after obtaining the first floating-point input feature, the target network layer converts the first floating-point input feature into a second fixed-point input feature based on the target feature value quantization bit width and the target feature value quantization hyperparameter of the target network layer. The target network layer uses fixed-point weights, and the fixed-point second input feature is processed based on the fixed-point weights of the target network layer to obtain the output feature of the target network layer.

[0264] After the encoding end or the decoding end obtains the Bitstream#2 corresponding to the current image block, the Bitstream#2 is decoded to obtain the image quantization feature (i.e., the target feature of the current image block), the image quantization feature is dequantized to obtain the image feature s', the image feature s' is feature restored (i.e., residual recovery) to obtain the residual feature r_hat, and the image feature y_hat is determined based on the residual feature r_hat and the prediction value mu, such as using the sum of the residual feature r_hat and the prediction value mu as the image feature y_hat (i.e., determining the first input feature of the synthesis transformation network based on the target feature). Alternatively, the Bitstream#2 is decoded to obtain the image feature s' (i.e., the target feature of the current image block), the image feature s' is feature restored to obtain the residual feature r_hat, and the image feature y_hat is determined based on the residual feature r_hat and the prediction value mu, such as using the sum of the residual feature r_hat and the prediction value mu as the image feature y_hat (i.e., determining the first input feature of the synthesis transformation network based on the target feature).

[0265] After obtaining the image feature y_hat (i.e., the first input feature of the synthetic transformation network), the image feature y_hat is input to the synthetic transformation network (i.e., the target decoding network). The synthetic transformation network can use the target feature value quantization bit width and the target feature value quantization hyperparameter to convert the floating-point image feature y_hat (i.e., the first input feature) into a fixed-point image feature y_hat (i.e., the second input feature), and perform a synthetic transformation on the fixed-point image feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x (i.e., the output feature of the synthetic transformation network). In summary, it can be seen that the image feature y_hat can be synthetically transformed based on the synthetic transformation network to obtain the reconstructed image block x_hat corresponding to the current image block x.

[0266] For example, for each target network layer in the synthetic transformation network, after obtaining the first floating-point input feature, the target network layer can convert the first floating-point input feature into a fixed-point second input feature based on the target feature value quantization bit width and the target feature value quantization hyperparameter of the target network layer. And because the target network layer uses fixed-point weights, the fixed-point second input feature can be processed based on the fixed-point weights of the target network layer to obtain the output feature of the target network layer. Among them, the output feature of the target network layer can be used as the input feature of the next network layer, and the output feature of the target network layer can be a floating-point feature, and so on.

[0267] After obtaining the output features of the synthetic transformation network, the output features of the synthetic transformation network can be used as the reconstructed image block x_hat corresponding to the current image block x, that is, the reconstructed image block x_hat corresponding to the current image block x is determined based on the output features.

[0268] As can be seen from the above technical solutions, in the embodiment of the present application, an end-to-end video image compression method is proposed, which can realize the decoding of video images based on the decoding network, so as to achieve the purpose of improving the encoding efficiency and decoding efficiency. The target weight quantization bit width and the target weight quantization hyperparameter are used to construct a decoding network with fixed-point weights, and the target eigenvalue quantization bit width and the target eigenvalue quantization hyperparameter are used to generate fixed-point input features, so as to realize adaptive decoding acceleration, and reduce the amount of decoding calculation under the premise of ensuring the decoding quality. A compression method for a decoding network in an image encoding and decoding framework based on a neural network is proposed, and the quantization bit width and quantization hyperparameter of each layer are determined by analyzing the quantization loss of each layer of the decoding network when the input code stream is decoded and calculated, and the decoding network is quantized according to the obtained quantization bit width and quantization hyperparameter, which can greatly compress the storage space, calculation amount, model throughput and bandwidth of the decoding network while ensuring the decoding quality of the decoding network, so that it is possible for a huge decoding network to run on a device with limited computing resources. By changing the reasoning of the decoding network from floating-point calculation to fixed-point calculation, it is helpful to achieve decoding consistency between different devices of the decoding network.

[0269] Illustratively, the above-mentioned embodiments can be implemented individually or in combination. For example, each embodiment in Embodiment 1 to Embodiment 9 can be implemented individually, and at least two embodiments in Embodiment 1 to Embodiment 9 can be implemented in combination.

[0270] Illustratively, in the above embodiments, the content of the encoding end can also be applied to the decoding end, that is, the decoding end can be processed in the same way, and the content of the decoding end can also be applied to the encoding end, that is, the encoding end can be processed in the same way.

[0271] Based on the same application concept as the above method, a decoding device is also proposed in the embodiment of the present application. The device is applied to the decoding end, and the device includes: a memory, which is configured to store video data; a decoder, which is configured to implement the decoding method in the above embodiments 1 to 9, that is, the processing flow of the decoding end.

[0272] For example, in one possible implementation, the decoder is configured to implement:

[0273] Decoding a target feature corresponding to the current image block from a bitstream corresponding to the current image block;

[0274] Determine a first input feature of a target decoding network based on the target feature;

[0275] Obtain a target eigenvalue quantization bit width and a target eigenvalue quantization hyperparameter of a target decoding network, and convert the first input feature into a second input feature based on the target eigenvalue quantization bit width and the target eigenvalue quantization hyperparameter;

[0276] Processing the second input feature based on the fixed-point weight of the target decoding network to obtain the output feature of the target decoding network; wherein the fixed-point weight is determined based on the target weight quantization bit width and the target weight quantization hyperparameter;

[0277] A reconstructed image block corresponding to the current image block is determined based on the output features of the target decoding network.

[0278] Based on the same application concept as the above method, the decoding end device (also referred to as a video decoder) provided in the embodiment of the present application, from the hardware level, its hardware architecture diagram can be specifically referred to as Figure 7 As shown. It includes: a processor 701 and a machine-readable storage medium 702, the machine-readable storage medium 702 stores machine-executable instructions that can be executed by the processor 701; the processor 701 is used to execute the machine-executable instructions to implement the decoding method of the above-mentioned embodiments 1-9 of the present application.

[0279] For example, in one possible implementation, the decoding end device is used to implement:

[0280] Decoding a target feature corresponding to the current image block from a bitstream corresponding to the current image block;

[0281] Determine a first input feature of a target decoding network based on the target feature;

[0282] Obtain a target eigenvalue quantization bit width and a target eigenvalue quantization hyperparameter of a target decoding network, and convert the first input feature into a second input feature based on the target eigenvalue quantization bit width and the target eigenvalue quantization hyperparameter;

[0283] Processing the second input feature based on the fixed-point weight of the target decoding network to obtain the output feature of the target decoding network; wherein the fixed-point weight is determined based on the target weight quantization bit width and the target weight quantization hyperparameter;

[0284] A reconstructed image block corresponding to the current image block is determined based on the output features of the target decoding network.

[0285] Based on the same application concept as the above method, an embodiment of the present application provides an electronic device, including: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the decoding method of the above embodiments 1-9 of the present application.

[0286] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the method disclosed in the above example of the present application can be implemented, such as the decoding method in the above embodiments.

[0287] Based on the same application concept as the above method, an embodiment of the present application further provides a computer application, which, when executed by a processor, can implement the decoding method disclosed in the above example of the present application.

[0288] A decoding device is also proposed in an embodiment of the present application. The decoding device can be applied to a decoding end, and the decoding device may include: a decoding module, used to decode a target feature corresponding to a current image block from a code stream corresponding to the current image block; a determination module, used to determine a first input feature of a target decoding network based on the target feature; a processing module, used to obtain a target feature value quantization bit width and a target feature value quantization hyperparameter of the target decoding network, and convert the first input feature into a second input feature based on the target feature value quantization bit width and the target feature value quantization hyperparameter; the second input feature is processed based on a fixed-point weight of the target decoding network to obtain an output feature of the target decoding network; wherein the fixed-point weight is determined based on the target weight quantization bit width and the target weight quantization hyperparameter; the determination module is used to determine a reconstructed image block corresponding to the current image block based on the output feature of the target decoding network.

[0289] Exemplarily, the target decoding network includes at least one target network layer, which is a network layer using fixed-point weights. For each target network layer in the target decoding network: the processing module is also used to convert the first input feature of the target network layer into a second input feature based on the target feature value quantization bit width and the target feature value quantization hyperparameter of the target network layer; the second input feature is processed based on the fixed-point weight of the target network layer to obtain the output feature of the target network layer; wherein the fixed-point weight of the target network layer is determined based on the target weight quantization bit width and the target weight quantization hyperparameter of the target network layer; wherein the target feature value quantization bit widths of different target network layers are the same or different, the target feature value quantization hyperparameters of different target network layers are the same or different, the target weight quantization bit widths of different target network layers are all fixed quantization bit widths, and the target weight quantization hyperparameters of different target network layers are the same or different.

[0290] Exemplarily, the processing module is specifically used to convert the first input feature of the target network layer into the second input feature based on the target feature value quantization bit width and the target feature value quantization hyperparameter of the target network layer: convert the first input feature into the second input feature using the following formula: quant = clip (round (data × 2 param ),-2 bw-1 ,2 bw-1 -1); or, quant = clip (round (data ÷ param), -2 bw-1 ,2 bw-1 -1); where quant is used to represent the second input feature, data is used to represent the first input feature, param is used to represent the target feature value quantization hyperparameter, bw is used to represent the target feature value quantization bit width, clip is used to represent the truncation function, and round is used to represent rounding.

[0291] Exemplarily, the sample decoding network includes multiple network layers using floating-point weights, the multiple network layers include a target network layer, and for each target network layer: the processing module is also used to obtain a weight quantization super-parameter candidate list corresponding to the target network layer, the weight quantization super-parameter candidate list includes multiple candidate weight quantization super-parameters; for each candidate weight quantization super-parameter, the floating-point weight of the target network layer is simulated and quantized using a fixed quantization bit width and the candidate weight quantization super-parameter to obtain a candidate simulation weight; based on the candidate simulation weight, the floating-point input feature of the target network layer is processed to obtain the floating-point output feature of the target network layer, and the quantization error corresponding to the candidate weight quantization super-parameter is determined based on the floating-point output feature of the target network layer; based on the quantization error corresponding to each candidate weight quantization super-parameter, the candidate weight quantization super-parameter corresponding to the minimum quantization error is determined as the target weight quantization super-parameter, and the fixed quantization bit width is determined as the target weight quantization bit width; based on the target weight quantization bit width and the target weight quantization super-parameter, the floating-point weight of the target network layer is fixed-point quantized to obtain the fixed-point weight of the target network layer; based on the fixed-point weight of each target network layer, a target decoding network is generated.

[0292] Exemplarily, when the processing module obtains the weight quantization hyperparameter candidate list corresponding to the target network layer, it is specifically used to: determine the maximum weight value based on all floating-point weights of the target network layer; each floating-point weight includes multiple weight values, and the maximum weight value is the maximum value of all weight values ​​of all floating-point weights; generate the initial weight quantization hyperparameter corresponding to the target network layer based on the maximum weight value; and construct the weight quantization hyperparameter candidate list based on the initial weight quantization hyperparameter.

[0293] Exemplarily, when the processing module generates the initial weight quantization hyperparameter corresponding to the target network layer based on the maximum weight value, it is specifically used to: generate the initial weight quantization hyperparameter using the following formula: param = bw-1-ceil(log2(max)); or, param = (max) / 2 bw-1 ; Wherein, param represents the initial weight quantization hyperparameter corresponding to the target network layer, bw represents the fixed quantization bit width, max represents the maximum weight value, and ceil represents the rounding up operation.

[0294] Exemplarily, the processing module uses a fixed quantization bit width and the candidate weight quantization hyperparameter to simulate and quantize the floating-point weight of the target network layer to obtain the candidate simulation weight: the following formula is used to simulate and quantize the floating-point weight to obtain the candidate simulation weight: quant = 2 -param ×clip(round(data×2 param ),-2 bw-1 ,2 bw -1-1); or, quant = param × clip (round (data ÷ param), -2 bw-1 ,2 bw-1 -1); where quant is used to represent the candidate simulation weight, param is used to represent the candidate weight quantization hyperparameter, data is used to represent the floating-point weight of the target network layer, bw is used to represent the fixed quantization bit width, clip represents the truncation function, and round represents rounding.

[0295] Exemplarily, when the processing module determines the quantization error corresponding to the candidate weight quantization hyperparameter based on the floating-point output feature of the target network layer, it is specifically used to: determine the sample output feature based on the floating-point output feature of the target network layer, the sample output feature is the output feature of the reference network layer in the sample decoding network, and the reference network layer is any network layer located after the target network layer; determine the quantization error corresponding to the candidate weight quantization hyperparameter based on the sample output feature and the reference output feature; wherein the method for obtaining the reference output feature includes: decoding the sample feature from the sample bitstream, determining the floating-point input feature corresponding to the sample decoding network based on the sample feature, and determining the reference output feature based on the floating-point input feature, and the reference output feature is the output feature of the reference network layer in the sample decoding network.

[0296] Exemplarily, the sample decoding network includes a target network layer using floating-point weights, and for each target network layer: the processing module is further used to obtain a candidate list of eigenvalue quantization super-parameters and a quantization bit width set corresponding to the target network layer, the candidate list of eigenvalue quantization super-parameters includes multiple candidate eigenvalue quantization super-parameters, and the quantization bit width set includes at least two quantization bit widths; traverse the current quantization bit width from the quantization bit width set, for each candidate eigenvalue quantization super-parameter, use the current quantization bit width and the candidate eigenvalue quantization super-parameter to simulate and quantize the floating-point input features of the target network layer to obtain a candidate input input feature; based on the pseudo weight of the target network layer, the candidate input feature is processed to obtain the floating-point output feature of the target network layer, and the quantization error corresponding to the candidate eigenvalue quantization hyperparameter is determined based on the floating-point output feature of the target network layer; based on the quantization error corresponding to each candidate eigenvalue quantization hyperparameter, if the minimum quantization error is less than a preset threshold, the candidate eigenvalue quantization hyperparameter corresponding to the minimum quantization error is determined as the target eigenvalue quantization hyperparameter, the current quantization bit width is determined as the target eigenvalue quantization bit width, and the target eigenvalue quantization hyperparameter and the target eigenvalue quantization bit width are recorded for the target network layer. Based on the quantization error corresponding to each candidate eigenvalue quantization hyperparameter, if the minimum quantization error is not less than the preset threshold, determine whether the current quantization bit width is the last quantization bit width in the quantization bit width set; if so, determine the candidate eigenvalue quantization hyperparameter corresponding to the minimum quantization error as the target eigenvalue quantization hyperparameter, determine the current quantization bit width as the target eigenvalue quantization bit width, and record the target eigenvalue quantization hyperparameter and the target eigenvalue quantization bit width for the target network layer; if not, traverse the next quantization bit width of the current quantization bit width as the current quantization bit width, and return to execute the operation of simulating and quantizing the floating-point input features of the target network layer using the current quantization bit width and the candidate eigenvalue quantization hyperparameter to obtain the candidate input features.

[0297] Exemplarily, when the processing module obtains the candidate list of eigenvalue quantization hyperparameters corresponding to the target network layer, it is specifically used to: determine the maximum eigenvalue based on all floating-point input features of the target network layer; each floating-point input feature includes multiple eigenvalues, and the maximum eigenvalue is the maximum value of all eigenvalues ​​of all floating-point input features; generate the initial eigenvalue quantization hyperparameters corresponding to the target network layer based on the maximum eigenvalue; construct the candidate list of eigenvalue quantization hyperparameters based on the initial eigenvalue quantization hyperparameters. When the processing module generates the initial eigenvalue quantization hyperparameters corresponding to the target network layer based on the maximum eigenvalue, it is specifically used to: generate the initial eigenvalue quantization hyperparameters using the following formula: get_param = bw-1-ceil(log2(max)); or, get_param = (max) / 2 bw-1; Wherein, get_param represents the initial eigenvalue quantization hyperparameter corresponding to the target network layer, bw represents the current quantization bit width, max represents the maximum eigenvalue, and ceil represents the rounding up operation.

[0298] The processing module uses the current quantization bit width and the candidate feature value quantization hyperparameter to simulate and quantize the floating-point input feature of the target network layer to obtain the candidate input feature. Specifically, the processing module uses the following formula to simulate and quantize the floating-point input feature to obtain the candidate input feature: quant=2 -param ×clip(round(data×2 param ),-2 bw-1 ,2 bw -1 -1); or, quant = param × clip (round (data ÷ param), -2 bw-1 ,2 bw-1 -1); where quant represents the candidate input feature, param represents the candidate feature value quantization hyperparameter, data represents the floating-point input feature, bw represents the current quantization bit width, clip represents the truncation function, and round represents rounding.

[0299] Exemplarily, the processing module is also used to use the target weight quantization hyperparameter of the target network layer and the target weight quantization bit width of the target network layer to simulate and quantize the floating-point weights of the target network layer to obtain the pseudo weights of the target network layer.

[0300] Exemplarily, when the processing module determines the quantization error corresponding to the candidate eigenvalue quantization hyperparameter based on the floating-point output feature of the target network layer, it is specifically used to: determine the sample output feature based on the floating-point output feature of the target network layer, the sample output feature is the output feature of the reference network layer in the sample decoding network, and the reference network layer is any network layer located after the target network layer; determine the quantization error corresponding to the candidate eigenvalue quantization hyperparameter based on the sample output feature and the reference output feature; wherein the reference output feature is obtained in a manner including: decoding the sample feature from a sample bitstream, determining the floating-point input feature corresponding to the sample decoding network based on the sample feature, and determining the reference output feature based on the floating-point input feature, and the reference output feature is the output feature of the reference network layer in the sample decoding network.

[0301] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. The present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. The embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The above is only an embodiment of the present application and is not intended to limit the present application.

[0302] For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An image decoding method, characterized in that: The method comprises: Obtain a target eigenvalue quantization bit width and a target eigenvalue quantization hyperparameter of a target decoding network, and convert a first input feature into a second input feature based on the target eigenvalue quantization bit width and the target eigenvalue quantization hyperparameter; Processing the second input feature based on the fixed-point weights of the target decoding network to obtain an output feature; The fixed-point weight is determined based on the target weight quantization bit width.

2. The method according to claim 1, characterized in that The converting the first input feature into the second input feature based on the target feature value quantization bit width and the target feature value quantization hyperparameter includes: If the target decoding network includes at least one target network layer, for each target network layer of the target decoding network, based on the target feature value quantization bit width of the target network layer and the target feature value quantization hyperparameter of the target network layer, the first input feature of the target network layer is converted into a second input feature; wherein the target feature value quantization bit widths of different target network layers are the same, and the target feature value quantization hyperparameters of different target network layers are the same.

3. The method according to claim 2, characterized in that The processing of the second input feature based on the fixed-point weight of the target decoding network to obtain the output feature includes: The second input feature of the target network layer is processed based on the fixed-point weight of the target network layer to obtain the output feature of the target network layer; wherein the fixed-point weight of the target network layer is determined based on the target weight quantization bit width of the target network layer; wherein the target weight quantization bit widths of different target network layers are all fixed quantization bit widths.

4. The method according to claim 2, characterized in that: The converting the first input feature of the target network layer into the second input feature based on the target feature value quantization bit width of the target network layer and the target feature value quantization hyperparameter of the target network layer includes: The first input feature of the target network layer is converted into the second input feature using the following formula: quant=clip(round(data×2 param ),-2 bw-1 ,2 bw-1 -1); Among them, quant represents the second input feature, data represents the first input feature, param represents the target feature value quantization hyperparameter, bw represents the target feature value quantization bit width, clip represents the truncation operation, and round represents the rounding operation.

5. The method according to any one of claims 1 to 4, characterized in that: If the target decoding network includes a probabilistic hyperparameter decoding network, the first input feature is a first coefficient hyperparameter feature, and the method further includes: The first coefficient hyperparameter feature is converted into a second coefficient hyperparameter feature through the target feature value quantization bit width and the target feature value quantization hyperparameter of the probabilistic hyperparameter decoding network; the second coefficient hyperparameter feature is inversely transformed based on the fixed-point weights of the probabilistic hyperparameter decoding network to obtain probability distribution parameters; wherein the probability distribution parameters are used to generate a probability distribution model, and the probability distribution model is used to decode the bitstream.

6. The method according to any one of claims 1 to 4, characterized in that: If the target decoding network includes a synthetic transformation network, the first input feature is a first image feature, and the method further includes: The first image feature is converted into a second image feature through the target eigenvalue quantization bit width and the target eigenvalue quantization hyperparameter of the synthetic transformation network; the second image feature is synthetically transformed based on the fixed-point weights of the synthetic transformation network to obtain a reconstructed image block corresponding to the current image block.

7. The method according to any one of claims 1 to 4, characterized in that: If the target decoding network includes a mean prediction network, the first input feature is a first coefficient hyperparameter feature, and the method further includes: The first coefficient hyperparameter feature is converted into a second coefficient hyperparameter feature through the target feature value quantization bit width and the target feature value quantization hyperparameter of the mean prediction network; the second coefficient hyperparameter feature is predicted based on the context based on the fixed-point weight of the mean prediction network to obtain the prediction value corresponding to the current image block; wherein the prediction value is used to determine the image feature, and the image feature is used to determine the reconstructed image block corresponding to the current image block.

8. An image decoding device, characterized in that: The device comprises: An acquisition module, used to obtain a target eigenvalue quantization bit width and a target eigenvalue quantization hyperparameter of a target decoding network; A processing module, used to convert the first input feature into a second input feature based on the target feature value quantization bit width and the target feature value quantization hyperparameter; the second input feature is processed based on the fixed-point weight of the target decoding network to obtain an output feature; wherein the fixed-point weight is determined based on the target weight quantization bit width.

9. An image decoding device, characterized in that: The image decoding device comprises: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; The processor is used to execute machine executable instructions to implement the method according to any one of claims 1 to 7.

10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores a plurality of computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images