A decoding method, apparatus and device

Abstract

The application provides a decoding method, a device and equipment thereof. The method comprises: decoding a target feature corresponding to a current image block from a code stream corresponding to the current image block; determining a first input feature of a target decoding network based on the target feature; obtaining a target feature value quantization bit width and a target feature value quantization parameter of the target decoding network, and converting the first input feature into a second input feature based on the target feature value quantization bit width and the target feature value quantization parameter; processing the second input feature based on a fixed-point type weight of the target decoding network to obtain an output feature of the target decoding network; wherein the fixed-point type weight is determined based on a target weight quantization bit width and a target weight quantization parameter; and determining a reconstructed image block corresponding to the current image block based on the output feature of the target decoding network. Through the technical scheme, the decoding calculation amount can be reduced while ensuring the decoding quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of coding and decoding technology, in particular to a decoding method, device and equipment thereof. BACKGROUND

[0002] In order to save space, video images are transmitted after being encoded. Complete video encoding can include prediction, transformation, quantization, entropy encoding, filtering and other processes. For the prediction process, the prediction process can include intra prediction and inter prediction. Inter prediction refers to using the correlation in the time domain of the video to predict the current pixel using the pixels of the adjacent coded image, so as to effectively remove the temporal redundancy of the video. Intra prediction refers to using the correlation in the spatial domain of the video to predict the current pixel using the pixels of the coded block of the current frame image, so as to remove the spatial redundancy of the video.

[0003] With the rapid development of deep learning technology, deep learning technology has achieved success in many high-level computer vision problems such as image classification and object detection. Deep learning technology has also gradually begun to be applied in the field of coding and decoding, that is, a neural network can be used to encode and decode images. Although the neural network-based coding and decoding method has shown great performance potential, the neural network-based coding and decoding method still has problems such as poor decoding performance and high complexity. SUMMARY

[0004] Therefore, the present application provides a decoding method, device and equipment thereof, which improves decoding performance and reduces decoding complexity.

[0005] The present application provides a decoding method, which comprises:

[0006] Decoding a target feature corresponding to a current image block from a code stream corresponding to the current image block;

[0007] Determining a first input feature of a target decoding network based on the target feature;

[0008] Obtaining a target feature value quantization bit width and a target feature value quantization hyperparameter of the target decoding network, and converting the first input feature into a second input feature based on the target feature value quantization bit width and the target feature value quantization hyperparameter;

[0009] Processing the second input feature based on a fixed-point type weight of the target decoding network to obtain an output feature of the target decoding network, wherein the fixed-point type weight is determined based on a target weight quantization bit width and a target weight quantization hyperparameter;

[0010] Determining a reconstructed image block corresponding to the current image block based on the output feature of the target decoding network.

[0011] The application provides a decoding device, which comprises:

[0012] A decoding module is configured to decode a target feature corresponding to a current image block from a code stream corresponding to the current image block.

[0013] A determining module is configured to determine a first input feature of a target decoding network based on the target feature.

[0014] A processing module is configured to acquire a target feature value quantization bit width and a target feature value quantization hyperparameter of the target decoding network, convert the first input feature into a second input feature based on the target feature value quantization bit width and the target feature value quantization hyperparameter, process the second input feature based on a fixed-point type weight of the target decoding network to obtain an output feature of the target decoding network, wherein the fixed-point type weight is determined based on a target weight quantization bit width and a target weight quantization hyperparameter.

[0015] The determining module is configured to determine a reconstructed image block corresponding to the current image block based on the output feature of the target decoding network.

[0016] The application provides a decoding end device, which comprises a processor and a machine readable storage medium, wherein the machine readable storage medium stores machine executable instructions which can be executed by the processor.

[0017] The processor is configured to execute the machine executable instructions to implement the decoding method.

[0018] The application provides a machine readable storage medium, wherein the machine readable storage medium stores a plurality of computer instructions, and the computer instructions are executed by a processor to implement the decoding method.

[0019] As can be seen from the above technical solutions, in the embodiments of the application, a target feature corresponding to a current image block is decoded from a code stream corresponding to the current image block, a first input feature of a target decoding network is determined based on the target feature, the first input feature is converted into a second input feature based on a target feature value quantization bit width and a target feature value quantization hyperparameter, the second input feature is processed based on a fixed-point type weight of the target decoding network to obtain an output feature of the target decoding network, and a reconstructed image block corresponding to the current image block is determined based on the output feature, thereby proposing an end-to-end video image compression method, which can realize the decoding of a video image based on a decoding network and achieve the purpose of improving the encoding efficiency and the decoding efficiency. The fixed-point type weight decoding network is constructed based on a target weight quantization bit width and a target weight quantization hyperparameter, and the fixed-point type input feature is generated based on a target feature value quantization bit width and a target feature value quantization hyperparameter, so that the adaptive decoding acceleration is realized, the decoding calculation amount is reduced, and the decoding quality is ensured. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 is a schematic diagram of a three-dimensional feature matrix in an embodiment of the present application;

[0021] Figure 2 is a flow chart of a decoding method in an embodiment of the present application;

[0022] Figure 3 is a schematic diagram of a processing procedure of an encoding end in an embodiment of the present application;

[0023] Figure 4 is a schematic diagram of a processing procedure of a decoding end in an embodiment of the present application;

[0024] Figure 5A and Figure 5B is a schematic diagram of a decoding network in an embodiment of the present application;

[0025] Figure 6A and Figure 6B is a schematic diagram of a decoding network in an embodiment of the present application;

[0026] Figure 7 is a hardware structure diagram of a decoding end device in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The terminology used in the embodiments of the present application is merely for the purpose of describing particular embodiments and is not intended to be limiting of the present application. As used in the description of the embodiments of the present application and the appended claims, the singular forms“a,”“an,” and“the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It also will be understood that the term“and / or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that, although the terms first, second, third, etc. can be used herein to describe various information, but those terms are not intended to denote a temporal sequence. The terms so used are only used to distinguish one information from another. For example, without departing from the scope of the embodiments of the present application, a first information can also be termed as a second information, and, similarly, a second information can also be termed as a first information, depending on the context. Furthermore, the word “if’ can be interpreted as meaning “when” or “upon” or “in response to determining” depending on the context.

[0028] A decoding method is proposed in the embodiments of the present application, which can involve the following technical terms:

[0029] JPEG (Joint Photographic Experts Group): JPEG is a standard for compressing continuous-tone still images, and the file suffix can be.jpg or.jpeg. It is a commonly used image file format. JPEG is a joint coding method using predictive coding (DPCM), discrete cosine transform (DCT), and entropy coding to remove redundant image and color data. It is a lossy compression format that can compress images into very small storage space, but will cause damage to image data to some extent. Especially when using too high a compression ratio, the quality of the image recovered after final decompression will be reduced. If high-quality images are pursued, JPEG should not use too high a compression ratio.

[0030] JPEG-AI (Joint Photographic Experts Group Artificial Intelligence): The scope of JPEG-AI is to create a learning-based image coding standard that provides a single-stream, compact, compression-domain representation, significantly improves compression efficiency over commonly used image coding standards at the same subjective quality, and effectively improves performance in image processing and computer vision tasks. JPEG-AI is aimed at a wide range of applications, such as cloud storage, visual management, autonomous driving cars and devices, image acquisition, storage and management, real-time management of visual data, and media distribution. The goal of JPEG-AI is to design a coding solution that significantly improves compression efficiency at the same subjective quality, providing efficient compression-domain processing for machine learning-based image processing and computer vision tasks. JPEG-AI requires hardware and software-friendly encoding and decoding, support for 8-bit and 10-bit depth, and efficient encoding and progressive decoding of images using text and graphics.

[0031] Entropy Encoding: Entropy encoding is a coding method that does not lose any information in the encoding process according to the entropy principle. The information entropy is the average amount of information of the source (a measure of uncertainty). The encoding method of entropy encoding can include but is not limited to: Shannon coding, Huffman coding, and arithmetic coding.

[0032] Neural Network (NN): Neural network refers to artificial neural network. The neural network is an operation model composed of a large number of nodes (or called neurons) connected with each other. In the neural network, the neuron processing unit can represent different objects, such as features, letters, concepts, or some meaningful abstract patterns. The types of processing units in the neural network can be divided into three categories: input units, output units and hidden units. The input unit accepts the signals and data of the external world; the output unit realizes the output of the processing result; the hidden unit is the unit between the input and the output unit, which cannot be observed from the outside of the system. The connection weight between neurons reflects the connection strength between units, and the information representation and processing are embodied in the connection relationship of processing units. Neural network is a non-programmed, brain-like information processing method. The essence of neural network is to obtain a parallel distributed information processing function through the transformation and dynamics of neural network, and to imitate the information processing function of the human brain neural system at different levels and levels. In the field of video processing, the commonly used neural network can include but is not limited to: convolutional neural network (CNN), recurrent neural network (RNN), fully connected network, etc.

[0033] Convolutional Neural Network (CNN): Convolutional neural network is a kind of feedforward neural network, which is one of the most representative network structures in deep learning technology. The artificial neuron of convolutional neural network can respond to a part of the surrounding units in the coverage range, and has excellent performance for large image processing. The basic structure of convolutional neural network can include two layers, one of which is the feature extraction layer (also called convolution layer), the input of each neuron is connected with the local receptive field of the previous layer, and the local feature is extracted. Once the local feature is extracted, the positional relationship between the local feature and other features is also determined. The second is the feature mapping layer (also called activation layer), each calculation layer of neural network is composed of multiple feature mappings, and each feature mapping is a plane. All the weights of the neurons on the plane are equal. The feature mapping structure can use Sigmoid function, ReLU function, Leaky-ReLU function, PReLU function, GDN function, etc. as the activation function of convolutional network. In addition, since the neurons on a mapping plane share weights, the number of free parameters of the network is reduced.

[0034] For example, one of the advantages of the convolutional neural network over the image processing algorithm is that it avoids complex preprocessing of the image (extracting artificial features, etc.), and can directly input the original image for end-to-end learning. One of the advantages of the convolutional neural network over the ordinary neural network is that the ordinary neural network adopts full connection, that is, all the neurons from the input layer to the hidden layer are connected, which will result in a large number of parameters, making the network training time-consuming or even difficult to train, while the convolutional neural network avoids this difficulty through local connection and weight sharing.

[0035] For example, the convolutional neural network can process large images, and generally includes convolutional layers, pooling layers, fully connected layers, etc. The convolutional neural network has made great progress in image classification, object detection, semantic segmentation, etc.

[0036] Deconvolution: Deconvolution is also called transposed convolution. The working process of deconvolution is similar to that of convolution, and the main difference is that deconvolution will pad the input to make the output larger than the input (or the same). If the stride is 1, the output size is equal to the input size. If the stride is N, the width of the output feature is N times the width of the input feature, and the height of the output feature is N times the height of the input feature.

[0037] Model quantization: Model quantization is a common neural network model acceleration algorithm. By quantizing the float type parameters of the model and the feature values of the input and output to a fixed number of bits, such as 8 bits or 4 bits, the calculation cost, data bandwidth and storage space of the network model can be reduced, so that the network model can be applied faster and better on hardware with limited memory and computing power. Model quantization includes quantization of model weights and feature values. The lower the number of bits of quantization, the more obvious the acceleration on hardware, and the performance will also decrease accordingly.

[0038] Image coding: The purpose of image coding technology is to compress the image to reduce the cost of transmission and storage of image data, and the decoded image can be restored. Higher compression ratio can result in lower transmission and storage consumption, but also brings great difficulty to image restoration. With the successful performance of artificial intelligence in various fields, artificial intelligence has also been applied to the field of image coding to achieve lower compression ratio and better image restoration effect.

[0039] Feature: The feature referred to in this application can be a three-dimensional feature matrix of C*W*H, as shown in Figure 1As shown, it is a schematic diagram of a three-dimensional feature matrix, in which C represents the number of channels, H represents the feature height, and W represents the feature width. The three-dimensional feature matrix can be an input of a neural network or an output of the neural network.

[0040] Rate-Distortion Optimized: There are two indicators for evaluating coding efficiency: code rate and PSNR (Peak Signal to Noise Ratio). The smaller the bit stream, the greater the compression rate, and the greater the PSNR, the better the quality of the reconstructed image. In mode selection, the evaluation formula is essentially a comprehensive evaluation of the two. For example, the cost of a mode: J (mode) = D + λ * R, where D represents Distortion (distortion), which can usually be measured using the SSE indicator, which is the sum of the squares of the differences between the reconstructed image block and the source image. In order to achieve cost consideration, the SAD indicator can also be used, which is the sum of the absolute values of the differences between the reconstructed image block and the source image. λ is the Lagrange multiplier, and R is the actual number of bits required for image block coding under the mode, including the total number of bits required for coding mode information, motion information, and residual. In mode selection, if the rate-distortion principle is used to compare and decide the coding mode, the best coding performance can usually be guaranteed.

[0041] A large number of encoding tools are proposed for each module of the encoding end, and each tool often has multiple modes. For different video sequences, the encoding tool that can obtain the optimal coding performance is often different. Therefore, in the encoding process, RDO (Rate-Distortion Opitimize) is usually used to compare the coding performance of different tools or modes to select the best mode. After determining the optimal tool or mode, the decision information of the tool or mode is transmitted by encoding the marker information in the bit stream. Although this method brings higher encoding complexity, it can adaptively select the optimal mode combination for different content to obtain the optimal coding performance. The decoding end can obtain the relevant mode information by directly parsing the marker information, and the complexity is less affected.

[0042] The decoding method in the embodiments of the present application will be described in detail below in combination with several specific embodiments.

[0043] Embodiment 1: A decoding method is proposed in the embodiments of the present application, as shown in Figure 2 The method can include:

[0044] Step 201, decoding the target feature corresponding to the current image block from the code stream corresponding to the current image block.

[0045] Step 202, determining a first input feature of a target decoding network based on the target feature.

[0046] Step 203, obtaining a target feature value quantization bit width and a target feature value quantization hyper parameter of the target decoding network, and converting the first input feature into a second input feature based on the target feature value quantization bit width and the target feature value quantization hyper parameter.

[0047] Step 204, processing the second input feature based on a fixed-point type weight of the target decoding network to obtain an output feature of the target decoding network; wherein the fixed-point type weight is determined based on a target weight quantization bit width and a target weight quantization hyper parameter.

[0048] Step 205, determining a reconstructed image block corresponding to the current image block based on the output feature of the target decoding network.

[0049] For example, the target decoding network includes at least one target network layer, and the target network layer can be a network layer using a fixed-point type weight. For each target network layer in the target decoding network: the first input feature of the target network layer is converted into a second input feature based on a target feature value quantization bit width and a target feature value quantization hyper parameter of the target network layer; the second input feature is processed based on a fixed-point type weight of the target network layer to obtain an output feature of the target network layer; wherein the fixed-point type weight of the target network layer is determined based on a target weight quantization bit width and a target weight quantization hyper parameter of the target network layer; wherein the target feature value quantization bit widths of different target network layers can be the same or different, the target feature value quantization hyper parameters of different target network layers can be the same or different, the target weight quantization bit widths of different target network layers can all be a fixed quantization bit width, and the target weight quantization hyper parameters of different target network layers can be the same or different.

[0050] For example, converting the first input feature of the target network layer into the second input feature based on the target feature value quantization bit width and the target feature value quantization hyper parameter of the target network layer includes but is not limited to: converting the first input feature of the target network layer into the second input feature using the following formula: quant = clip(round(data × 2 param ), -2 bw-1 , 2 bw-1 -1); or, quant = clip(round(data ÷ param), -2 bw-1 , 2 bw-1 -1); wherein quant is used to represent the second input feature, data is used to represent the first input feature, param is used to represent the target feature value quantization hyper parameter, bw is used to represent the target feature value quantization bit width, clip is used to represent a clipping function, and round is used to represent rounding.

[0051] wherein, by the operation of the clip function, if round(data x 2 param ) is less than -2 bw-1 , quant is -2 bw-1 , if round(data x 2 param ) is greater than 2 bw-1 -1, quant is 2 bw-1 -1, if round(data x 2 param ) is between -2 bw-1 and 2 bw-1 -1, quant is round(data x 2 param ).

[0052] wherein, by the operation of the clip function, if round(data ÷ param) is less than -2 bw-1 , quant is -2 bw-1 , if round(data ÷ param) is greater than 2 bw-1 -1, quant is 2 bw-1 -1, if round(data ÷ param) is between -2 bw-1 and 2 bw-1 -1, quant is round(data ÷ param).

[0053] In subsequent embodiments, the meaning of the clip operation is similar and will not be repeated hereinafter.

[0054] Exemplarily, the sample decoding network can include a plurality of network layers with floating-point weights, the plurality of network layers can include a target network layer, for each target network layer: a weight quantization hyperparameter candidate list corresponding to the target network layer is obtained, the weight quantization hyperparameter candidate list can include a plurality of candidate weight quantization hyperparameters; for each candidate weight quantization hyperparameter, a fixed quantization bit width and the candidate weight quantization hyperparameter are used to simulate and quantize the floating-point weights of the target network layer to obtain a candidate simulation weight; the floating-point input features of the target network layer are processed based on the candidate simulation weight to obtain floating-point output features of the target network layer, and a quantization error corresponding to the candidate weight quantization hyperparameter is determined based on the floating-point output features of the target network layer; based on the quantization error corresponding to each candidate weight quantization hyperparameter, the candidate weight quantization hyperparameter corresponding to the minimum quantization error is determined as a target weight quantization hyperparameter, and the fixed quantization bit width is determined as a target weight quantization bit width; the floating-point weights of the target network layer are fixed-point quantized based on the target weight quantization bit width and the target weight quantization hyperparameter to obtain fixed-point weights of the target network layer; and a target decoding network is generated based on the fixed-point weights of each target network layer.

[0055] For example, obtaining the candidate list of weight quantization hyperparameters corresponding to the target network layer may include, but is not limited to: determining the maximum weight value based on all floating-point weights of the target network layer; wherein each floating-point weight includes multiple weight values, and the maximum weight value is the maximum value among all weight values ​​of all floating-point weights; generating the initial weight quantization hyperparameters corresponding to the target network layer based on the maximum weight value; and constructing the candidate list of weight quantization hyperparameters based on the initial weight quantization hyperparameters.

[0056] For example, generating the initial weight quantization hyperparameters corresponding to the target network layer based on the maximum weight value can include, but is not limited to, generating the initial weight quantization hyperparameters using the following formula: param = bw-1 - ceil(log2(max)); or, param = (max) / 2 bw-1 Wherein, param represents the initial weight quantization hyperparameter corresponding to the target network layer, bw represents the fixed quantization bit width, max represents the maximum weight value, and ceil represents the round-up operation.

[0057] For example, using a fixed quantization bit width and the candidate weight quantization hyperparameter to simulate and quantize the floating-point weights of the target network layer to obtain candidate simulated weights can include: using the following formula to simulate and quantize the floating-point weights of the target network layer to obtain candidate simulated weights: quant = 2 -param ×clip(round(data×2 param ), -2 bw-1 ,2 bw -1 -1); or, quant = param × clip(round(data ÷ param), -2 bw-1 ,2 bw-1 -1); where quant is used to represent candidate simulation weights, param is used to represent the quantization hyperparameter of the candidate weights, data is used to represent the floating-point weights of the target network layer, bw is used to represent the fixed quantization bit width, clip is used to represent the truncation function, and round is used to represent rounding.

[0058] For example, determining the quantization error corresponding to the candidate weight quantization hyperparameter based on the floating-point output feature of the target network layer can include, but is not limited to: determining a sample output feature based on the floating-point output feature of the target network layer, the sample output feature being an output feature of a reference network layer in the sample decoding network, and the reference network layer being any network layer located after the target network layer; determining the quantization error corresponding to the candidate weight quantization hyperparameter based on the sample output feature and the reference output feature; and wherein the reference output feature can be obtained in the following manner: decoding a sample feature from a sample bitstream, determining a floating-point input feature corresponding to the sample decoding network based on the sample feature, and determining the reference output feature based on the floating-point input feature, the reference output feature being an output feature of a reference network layer in the sample decoding network.

[0059] For example, the sample decoding network can include a plurality of network layers with floating-point weights, the plurality of network layers including a target network layer, and for each target network layer: obtaining a feature value quantization hyperparameter candidate list and a quantization bit width set corresponding to the target network layer, the feature value quantization hyperparameter candidate list including a plurality of candidate feature value quantization hyperparameters, and the quantization bit width set including at least two quantization bit widths; traversing a current quantization bit width from the quantization bit width set, and for each candidate feature value quantization hyperparameter, performing simulated quantization on a floating-point input feature of the target network layer using the current quantization bit width and the candidate feature value quantization hyperparameter to obtain a candidate input feature; processing the candidate input feature based on a pseudo weight of the target network layer to obtain a floating-point output feature of the target network layer, and determining a quantization error corresponding to the candidate feature value quantization hyperparameter based on the floating-point output feature of the target network layer; and based on the quantization error corresponding to each candidate feature value quantization hyperparameter, if a minimum quantization error is less than a preset threshold, determining the candidate feature value quantization hyperparameter corresponding to the minimum quantization error as a target feature value quantization hyperparameter, determining the current quantization bit width as a target feature value quantization bit width, and recording the target feature value quantization hyperparameter and the target feature value quantization bit width for the target network layer.

[0060] For example, based on the quantization error corresponding to each candidate feature value quantization hyperparameter, if a minimum quantization error is not less than a preset threshold, determining whether the current quantization bit width is the last quantization bit width in the quantization bit width set; if yes, determining the candidate feature value quantization hyperparameter corresponding to the minimum quantization error as a target feature value quantization hyperparameter, determining the current quantization bit width as a target feature value quantization bit width, and recording the target feature value quantization hyperparameter and the target feature value quantization bit width for the target network layer; and if no, traversing a next quantization bit width of the current quantization bit width as the current quantization bit width, and returning to perform the operation of performing simulated quantization on the floating-point input feature of the target network layer using the current quantization bit width and the candidate feature value quantization hyperparameter to obtain a candidate input feature.

[0061] Exemplarily, the acquiring the feature value quantization hyperparameter candidate list corresponding to the target network layer comprises: determining a maximum feature value based on all floating-point input features of the target network layer; wherein each floating-point input feature comprises a plurality of feature values, and the maximum feature value is the maximum value among all feature values of all floating-point input features; generating an initial feature value quantization hyperparameter corresponding to the target network layer based on the maximum feature value; and constructing the feature value quantization hyperparameter candidate list based on the initial feature value quantization hyperparameter.

[0062] Exemplarily, the generating the initial feature value quantization hyperparameter corresponding to the target network layer based on the maximum feature value can comprise but is not limited to: generating the initial feature value quantization hyperparameter using the following formula: param=bw-1-ceil(log2(max)); or, param=(max) / 2 bw-1 ; wherein, param represents the initial feature value quantization hyperparameter corresponding to the target network layer, bw represents the current quantization bit width, max represents the maximum feature value, and ceil represents the rounding up operation.

[0063] Exemplarily, the simulating and quantizing the floating-point input feature of the target network layer using the current quantization bit width and the candidate feature value quantization hyperparameter to obtain the candidate input feature can comprise but is not limited to: simulating and quantizing the floating-point input feature of the target network layer using the following formula to obtain the candidate input feature: quant=2 -param ×clip(rnd(data×2 param ),-2 bw-1 ,2 bw-1 -1); or, quant=param×clip(round(data÷param),-2 bw-1 ,2 bw-1 -1); wherein, quant represents the candidate input feature, param represents the candidate feature value quantization hyperparameter, data represents the floating-point input feature, bw represents the current quantization bit width, clip represents the clipping function, and round represents the rounding.

[0064] Exemplarily, before the processing the candidate input feature based on the pseudo weight of the target network layer to obtain the floating-point output feature of the target network layer, the floating-point weight of the target network layer can be simulated and quantized using the target weight quantization hyperparameter of the target network layer and the target weight quantization bit width of the target network layer to obtain the pseudo weight of the target network layer.

[0065] For example, determining the quantization error corresponding to the candidate feature value quantization hyperparameter based on the floating-point output feature of the target network layer can include but is not limited to: determining a sample output feature based on the floating-point output feature of the target network layer, the sample output feature can be an output feature of a reference network layer in a sample decoding network, and the reference network layer is any network layer located behind the target network layer; determining the quantization error corresponding to the candidate feature value quantization hyperparameter based on the sample output feature and the reference output feature; wherein the reference output feature can be obtained in the following manner: decoding a sample feature from a sample bitstream, determining a floating-point input feature corresponding to the sample decoding network based on the sample feature, and determining the reference output feature based on the floating-point input feature, the reference output feature can be an output feature of a reference network layer in a sample decoding network.

[0066] For example, the above execution order is only an example given for convenience of description, and in actual application, the execution order between steps can also be changed, and the execution order is not limited. Moreover, in other embodiments, the steps of the corresponding method can not necessarily be executed in the order shown and described in the specification, and the steps included in the method can be more or less than those described in the specification. In addition, a single step described in the specification can be divided into multiple steps for description in other embodiments, and multiple steps described in the specification can also be combined into a single step for description.

[0067] From the above technical solutions, in the embodiments of the present application, the target feature corresponding to the current image block is decoded from the bitstream corresponding to the current image block, the first input feature of the target decoding network is determined based on the target feature, the first input feature is converted into the second input feature based on the target feature value quantization bit width and the target feature value quantization hyperparameter, the output feature of the target decoding network is obtained by processing the second input feature based on the fixed-point weight of the target decoding network, and the reconstructed image block corresponding to the current image block is determined based on the output feature. Thus, an end-to-end video image compression method is proposed, which can realize decoding of video images based on a decoding network to achieve the purpose of improving encoding efficiency and decoding efficiency. The fixed-point weight decoding network is constructed by using the target weight quantization bit width and the target weight quantization hyperparameter, and the fixed-point input feature is generated by using the target feature value quantization bit width and the target feature value quantization hyperparameter, thereby realizing adaptive decoding acceleration, reducing decoding calculation amount on the premise of ensuring decoding quality.

[0068] Embodiment 2: The processing process at the encoding end can be seen from Figure 3 Figure 3 This is only an example, and is not limited thereto.

[0069] ​After obtaining the current image block x (the current image block x can be the original image block x, i.e., the input image block), the encoding end can analyze and transform the current image block x through an analysis transformation network (i.e., a neural network) to obtain the image feature y corresponding to the current image block x. Wherein, the feature transformation of the current image block x through the analysis transformation network refers to: transforming the current image block x into the image feature y in the latent domain, so as to facilitate the operation of all subsequent processes in the latent domain.

[0070] The image can be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can also be the current image, i.e., the encoding and decoding process of the image block can also be directly used for the image.

[0071] After obtaining the image feature y, the encoding end performs coefficient hyperparameter feature transformation on the image feature y to obtain the coefficient hyperparameter feature z. For example, the image feature y can be input to a hyperparameter encoding network (i.e., a neural network) to perform coefficient hyperparameter feature transformation on the image feature y to obtain the coefficient hyperparameter feature z. Wherein, the hyperparameter encoding network can be a trained neural network, and the training process of the hyperparameter encoding network is not limited, as long as it can perform coefficient hyperparameter feature transformation on the image feature y. Wherein, the image feature y in the latent domain obtains the hyper-prior latent information z after passing through the hyperparameter encoding network.

[0072] After obtaining the coefficient hyperparameter feature z, the encoding end can quantize the coefficient hyperparameter feature z to obtain the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, i.e., Figure 3 The Q operation in the above formula is a quantization process. After obtaining the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, the hyperparameter quantization feature is encoded to obtain the Bitstream #1 (i.e., the first code stream, which belongs to the main code stream corresponding to the current image block) corresponding to the current image block, i.e., Figure 3 The AE operation in the above formula represents an encoding process, such as an entropy encoding process. Alternatively, the encoding end can directly encode the coefficient hyperparameter feature z after obtaining the coefficient hyperparameter feature z to obtain the Bitstream #1 corresponding to the current image block, without involving the quantization process. Wherein, the hyperparameter quantization feature or the coefficient hyperparameter feature z carried in the Bitstream #1 is mainly used to obtain the mean and the probability distribution parameter of the probability distribution model.

[0073] After obtaining the Bitstream #1 corresponding to the current image block, the encoding end can send the Bitstream #1 corresponding to the current image block to the decoding end. For the processing process of the decoding end for the Bitstream #1 corresponding to the current image block, see the subsequent embodiments.

[0074] The encoding end can also decode Bitstream#1 to obtain the hyper-quantized feature after obtaining Bitstream#1 corresponding to the current image block, that is, Figure 3 AD in the above formula represents the decoding process, and then the hyper-quantized feature is dequantized to obtain the coefficient hyper-parameter feature z_hat, which can be the same as or different from the coefficient hyper-parameter feature z, Figure 3 The IQ operation in the above formula is the dequantization process. Alternatively, the encoding end can also decode Bitstream#1 to obtain the coefficient hyper-parameter feature z_hat after obtaining Bitstream#1 corresponding to the current image block, without involving the dequantization process of the coefficient hyper-parameter feature z_hat.

[0075] For the encoding process of Bitstream#1, a fixed probability density model encoding method can be used, and for the decoding process of Bitstream#1, a fixed probability density model decoding method can be used, and the encoding and decoding processes are not limited.

[0076] After the encoding end obtains the coefficient hyper-parameter feature z_hat, context-based prediction can be performed based on the coefficient hyper-parameter feature z_hat of the current image block and the image feature y_hat of the previous image block (the determination process of the image feature y_hat is described in subsequent embodiments), to obtain the prediction value mu (i.e., the mean mu) of the current image block. For example, the coefficient hyper-parameter feature z_hat and the image feature y_hat are input into a mean prediction network, and the mean prediction network determines the prediction value mu based on the coefficient hyper-parameter feature z_hat and the image feature y_hat. The prediction process is not limited. For the context-based prediction process, the input includes the coefficient hyper-parameter feature z_hat and the decoded image feature y_hat, which are jointly input to obtain a more accurate prediction value mu. The prediction value mu is used to obtain a residual by subtracting the original feature, and the residual is added to the decoded feature to obtain the reconstruction y.

[0077] It should be noted that the mean prediction network is an optional neural network, that is, there can be no mean prediction network, that is, the prediction value mu does not need to be determined by the mean prediction network, Figure 3 The dashed box in the above formula indicates that the mean prediction network is optional.

[0078] After obtaining the image feature y, the encoding end can determine the residual feature r based on the image feature y and the prediction value mu, such as taking the difference between the image feature y and the prediction value mu as the residual feature r. Then, the residual feature r is processed to obtain the image feature s, and the feature processing process is not limited and can be any feature processing mode. In this case, a mean prediction network needs to be deployed to provide the prediction value mu. Alternatively, after obtaining the image feature y, the encoding end can process the image feature y to obtain the image feature s, and the feature processing process is not limited and can be any feature processing mode. In this case, the mean prediction network does not need to be deployed, and the residual process is optional, as indicated by the dashed box.

[0079] After obtaining the image feature s, the encoding end can quantize the image feature s to obtain the image quantization feature corresponding to the image feature s, that is, Figure 3 The Q operation in the above formula is a quantization process. After obtaining the image quantization feature corresponding to the image feature s, the encoding end can encode the image quantization feature to obtain the Bitstream#2 (i.e., the second code stream) corresponding to the current image block, that is, Figure 3 The AE operation in the above formula represents an encoding process, such as an entropy encoding process. Alternatively, the encoding end can also directly encode the image feature s to obtain the Bitstream#2 corresponding to the current image block, without involving the quantization process of the image feature s.

[0080] After obtaining the Bitstream#2 corresponding to the current image block, the encoding end can send the Bitstream#2 corresponding to the current image block to the decoding end. For the processing process of the decoding end for the Bitstream#2 corresponding to the current image block, see the subsequent embodiments.

[0081] After obtaining the Bitstream#2 corresponding to the current image block, the encoding end can also decode the Bitstream#2 to obtain the image quantization feature, that is, Figure 3 The AD in the above formula represents a decoding process, and then the encoding end can dequantize the image quantization feature to obtain the image feature s', which can be the same as or different from the image feature s, Figure 3 The IQ operation in the above formula is a dequantization process. Alternatively, after obtaining the Bitstream#2 corresponding to the current image block, the encoding end can also decode the Bitstream#2 to obtain the image feature s', without involving the dequantization process of the image quantization feature.

[0082] After obtaining image features s', the encoder can perform feature recovery (the inverse process of feature processing) on ​​s'. This feature recovery process is not restricted and can be any method, resulting in residual features r_hat. Residual features r_hat and r can be the same or different. After obtaining residual features r_hat, the encoder determines image features y_hat based on residual features r_hat and predicted values ​​mu. Image features y_hat and y can be the same or different; for example, the sum of residual features r_hat and predicted values ​​mu can be used as image features y_hat. In this case, a mean prediction network needs to be deployed to provide the predicted values ​​mu. Alternatively, after obtaining image features s', the encoder can perform feature recovery (the inverse process of feature processing) on ​​s' to obtain image features y_hat. Image features y_hat and y can be the same or different. In this case, a mean prediction network is not needed, and the residual process is indicated by a dashed box as optional.

[0083] After obtaining the image feature y_hat, the encoder can perform a synthetic transformation on the image feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the image feature y_hat can be input into the synthetic transformation network, which will perform a synthetic transformation on the image feature y_hat to obtain the reconstructed image block x_hat. Thus, the image reconstruction process is completed.

[0084] In one possible implementation, when the encoding end encodes the image quantization features or image features s to obtain Bitstream#2 corresponding to the current image block, the encoding end needs to first determine the probability distribution model, and then encode the image quantization features or image features s based on the probability distribution model. Furthermore, when the encoding end decodes Bitstream#2, it also needs to first determine the probability distribution model, and then decode Bitstream#2 based on the probability distribution model.

[0085] To obtain the probability distribution model, please refer to [link / reference]. Figure 3 As shown, after obtaining the coefficient hyperparameter feature z_hat, the encoder can perform an inverse hyperparameter feature transformation on z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat can be input into a probabilistic hyperparameter decoding network, which will then perform an inverse hyperparameter feature transformation on z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on p. The probabilistic hyperparameter decoding network can be a trained neural network; the training process of this network is not restricted, as long as it can perform the inverse hyperparameter feature transformation on z_hat.

[0086] In a possible implementation, the encoding process described above can be performed by a deep learning model or a neural network model, so as to implement an end-to-end image compression and encoding process, and the encoding process is not limited in this regard.

[0087] Embodiment 3: The decoding process can refer to Figure 4 as shown, Figure 4 which is only an example and is not limited in this regard.

[0088] After obtaining the Bitstream#1 corresponding to the current image block, the decoding end can further decode the Bitstream#1 to obtain the hyper-quantized feature, that is, Figure 4 AD in the above formula represents the decoding process, and then the hyper-quantized feature is dequantized to obtain the coefficient hyper-parameter feature z_hat, which can be the same as or different from the coefficient hyper-parameter feature z, Figure 4 the IQ operation in the above formula is a dequantization process. Alternatively, after obtaining the Bitstream#1 corresponding to the current image block, the decoding end can further decode the Bitstream#1 to obtain the coefficient hyper-parameter feature z_hat, without involving the dequantization process of the coefficient hyper-parameter feature z_hat.

[0089] The decoding process of the Bitstream#1 can adopt a decoding method of a fixed probability density model, and is not limited in this regard.

[0090] The image can be divided into one image block or multiple image blocks, and if the image is divided into one image block, the current image block x can also be the current image, that is, the decoding process of the image block can also be directly used for the image.

[0091] After obtaining the coefficient hyper-parameter feature z_hat, the decoding end can perform context-based prediction based on the coefficient hyper-parameter feature z_hat of the current image block and the image feature y_hat of the previous image block (the determination process of the image feature y_hat will be described in subsequent embodiments), to obtain the prediction value mu (that is, the mean mu) corresponding to the current image block, for example, inputting the coefficient hyper-parameter feature z_hat and the image feature y_hat into a mean prediction network, and determining the prediction value mu based on the coefficient hyper-parameter feature z_hat and the image feature y_hat by the mean prediction network, and the prediction process is not limited in this regard. For the context-based prediction process, the input includes the coefficient hyper-parameter feature z_hat and the decoded image feature y_hat, and the two are jointly input to obtain a more accurate prediction value mu.

[0092] It should be noted that the mean prediction network is an optional neural network, i.e., there can be no mean prediction network, i.e., the predicted value mu does not need to be determined by the mean prediction network, Figure 4 The dashed box in indicates that the mean prediction network is optional.

[0093] After obtaining the Bitstream#2 corresponding to the current image block, the decoding end can also decode the Bitstream#2 to obtain the image quantized feature, i.e., Figure 4 AD in indicates the decoding process, and then the decoding end can dequantize the image quantized feature to obtain the image feature s', which can be the same as or different from the image feature s, Figure 4 The IQ operation in is a dequantization process. Alternatively, after obtaining the Bitstream#2 corresponding to the current image block, the decoding end can also decode the Bitstream#2 to obtain the image feature s' without involving the dequantization process of the image quantized feature.

[0094] After obtaining the image feature s', the decoding end can perform feature restoration (i.e., the inverse process of feature processing) on the image feature s' to obtain the residual feature r_hat, which can be the same as or different from the residual feature r. After obtaining the residual feature r_hat, the decoding end determines the image feature y_hat based on the residual feature r_hat and the predicted value mu, which can be the same as or different from the image feature y, such as taking the sum of the residual feature r_hat and the predicted value mu as the image feature y_hat. In this case, the mean prediction network needs to be deployed to provide the predicted value mu. Alternatively, after obtaining the image feature s', the decoding end can perform feature restoration on the image feature s' to obtain the image feature y_hat, which can be the same as or different from the image feature y. In this case, the mean prediction network does not need to be deployed, and the residual process is indicated by the dashed box as an optional process.

[0095] After obtaining the image feature y_hat, the decoding end can perform synthesis transformation on the image feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x, such as inputting the image feature y_hat to the synthesis transformation network to perform synthesis transformation on the image feature y_hat by the synthesis transformation network to obtain the reconstructed image block x_hat, thereby completing the image reconstruction process.

[0096] In one possible implementation, when decoding the Bitstream#2, the decoding end needs to first determine the probability distribution model, and then decode the Bitstream#2 based on the probability distribution model. In order to obtain the probability distribution model, continue to refer to Figure 4As shown, after obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat is input into the probability hyperparameter decoding network, and the probability hyperparameter decoding network performs coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, the probability distribution model can be generated based on the probability distribution parameter p. The probability hyperparameter decoding network can be a trained neural network, and the training process of the probability hyperparameter decoding network is not limited. The probability hyperparameter decoding network can perform coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p.

[0097] In a possible implementation, the processing process of the decoding end described above can be performed by a deep learning model or a neural network model, so as to realize an end-to-end image compression and encoding process, and the decoding process is not limited.

[0098] In embodiment 4, for embodiments 2 and 3, the encoding end involves an analysis transformation network, a hyperparameter encoding network, a mean prediction network, a probability hyperparameter decoding network, and a synthesis transformation network, and the decoding end involves a mean prediction network, a probability hyperparameter decoding network, and a synthesis transformation network. All of the above networks can be neural networks. All neural networks involved in the decoding end can be used as decoding networks, or part of the neural networks involved in the decoding end can be used as decoding networks, and the decoding process is not limited.

[0099] In a possible implementation, all neural networks involved in the decoding end are taken as decoding networks, for example, referring to FIG. 6, Figure 5A As shown, the decoding network can include a probability hyperparameter decoding network and a synthesis transformation network. Referring to FIG. 7, Figure 5B As shown, the decoding network can include a mean prediction network, a probability hyperparameter decoding network, and a synthesis transformation network. Of course, when Figure 5A and Figure 5B the decoding network can only include the synthesis transformation network and does not include other networks, and the decoding process is not limited.

[0100] In a possible implementation, the encoding end or the decoding end can decode the target feature corresponding to the current image block from the code stream (such as the first code stream and the second code stream) corresponding to the current image block, determine the input feature of the decoding network based on the target feature, process the input feature based on the decoding network to obtain the output feature of the decoding network, and determine the reconstructed image block corresponding to the current image block based on the output feature after the output feature is obtained. The process is described below.

[0101] Referring to Figure 5A As shown in the figure, after the encoding end or the decoding end obtains the Bitstream#1 (that is, the first code stream) corresponding to the current image block, the encoding end or the decoding end can decode the Bitstream#1 to obtain the hyperparameter quantized feature (that is, the target feature of the current image block), and perform inverse quantization on the hyperparameter quantized feature to obtain the coefficient hyperparameter feature z_hat (that is, the input feature of the probability hyperparameter decoding network is determined based on the target feature). Alternatively, after the encoding end or the decoding end obtains the Bitstream#1 corresponding to the current image block, the encoding end or the decoding end can decode the Bitstream#1 to obtain the coefficient hyperparameter feature z_hat (that is, the target feature of the current image block, which is directly used as the input feature of the probability hyperparameter decoding network), without involving the inverse quantization process of the coefficient hyperparameter feature z_hat.

[0102] After the coefficient hyperparameter feature z_hat (that is, the input feature of the probability hyperparameter decoding network) is obtained, the coefficient hyperparameter feature z_hat is input into the probability hyperparameter decoding network (that is, the decoding network), and the coefficient hyperparameter feature z_hat is inversely transformed by the probability hyperparameter decoding network to obtain the probability distribution parameter p (that is, the output feature of the probability hyperparameter decoding network) corresponding to the current image block. After the probability distribution parameter p is obtained, the probability distribution model can be generated based on the probability distribution parameter p.

[0103] After obtaining Bitstream#2 (the second bitstream) corresponding to the current image block, the encoding or decoding end can decode Bitstream#2 to obtain image quantization features (i.e., the target features of the current image block), inverse quantize the image quantization features to obtain image feature s', and perform feature recovery on image feature s' to obtain image feature y_hat (i.e., determine the input features of the synthetic transform network based on this target feature). Alternatively, after obtaining Bitstream#2 corresponding to the current image block, the encoding or decoding end can decode Bitstream#2 to obtain image feature s' (i.e., the target features of the current image block), without involving the inverse quantization process of image quantization features, and perform feature recovery on image feature s' to obtain image feature y_hat (i.e., determine the input features of the synthetic transform network based on this target feature). When decoding Bitstream#2, the encoding or decoding end can decode Bitstream#2 based on a probability distribution model; this process will not be elaborated further.

[0104] After obtaining the image feature y_hat, the encoder or decoder inputs the image feature y_hat into the synthesis transform network (i.e., the decoding network). The synthesis transform network performs a synthesis transform on the image feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x (i.e., the output feature of the synthesis transform network). In summary, it can be seen that the image feature y_hat can be synthesized and transformed based on the synthesis transform network to obtain the reconstructed image block x_hat corresponding to the current image block x.

[0105] After obtaining the output features of the synthetic transform network, the output features of the synthetic transform network can be used as the reconstructed image block x_hat corresponding to the current image block x. That is, the reconstructed image block x_hat corresponding to the current image block x is determined based on the output features.

[0106] See Figure 5B As shown, after obtaining Bitstream#1 corresponding to the current image patch, the encoder or decoder can decode Bitstream#1 to obtain the hyperparameter quantization feature (i.e., the target feature of the current image patch), and then dequantize the hyperparameter quantization feature to obtain the coefficient hyperparameter feature z_hat (i.e., the input feature of the probabilistic hyperparameter decoding network based on this target feature). Alternatively, after obtaining Bitstream#1 corresponding to the current image patch, the encoder or decoder can decode Bitstream#1 to obtain the coefficient hyperparameter feature z_hat (i.e., the target feature of the current image patch, which is then directly used as the input feature of the probabilistic hyperparameter decoding network).

[0107] After obtaining the coefficient hyperparameter feature z_hat (i.e., the input feature of the probability hyperparameter decoding network), the coefficient hyperparameter feature z_hat is input into the probability hyperparameter decoding network (i.e., the decoding network), and the coefficient hyperparameter feature z_hat is inversely transformed by the probability hyperparameter decoding network to obtain the probability distribution parameter p corresponding to the current image block (i.e., the output feature of the probability hyperparameter decoding network). After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p.

[0108] After obtaining the coefficient hyperparameter feature z_hat (i.e., the input feature of the mean prediction network), the coefficient hyperparameter feature z_hat and the image feature y_hat of the previous image block are input into the mean prediction network (i.e., the decoding network), and the predicted value mu (i.e., the output feature of the mean prediction network) is determined by the mean prediction network based on the coefficient hyperparameter feature z_hat and the image feature y_hat, i.e., the mean prediction network performs context-based prediction to obtain the predicted value mu (i.e., the mean mu) corresponding to the current image block.

[0109] After obtaining the Bitstream#2 corresponding to the current image block, the encoding end or the decoding end can further decode the Bitstream#2 to obtain the image quantization feature (i.e., the target feature of the current image block), inverse quantize the image quantization feature to obtain the image feature s', restore the image feature s' to obtain the residual feature r_hat, and determine the image feature y_hat based on the residual feature r_hat and the predicted value mu, such as taking the sum of the residual feature r_hat and the predicted value mu as the image feature y_hat (i.e., determining the input feature of the synthesis transform network based on the target feature).

[0110] Alternatively, after obtaining the Bitstream#2 corresponding to the current image block, the encoding end or the decoding end can further decode the Bitstream#2 to obtain the image feature s' (i.e., the target feature of the current image block), restore the image feature s' to obtain the residual feature r_hat, and determine the image feature y_hat based on the residual feature r_hat and the predicted value mu, such as taking the sum of the residual feature r_hat and the predicted value mu as the image feature y_hat (i.e., determining the input feature of the synthesis transform network based on the target feature).

[0111] After obtaining the image feature y_hat, the encoding end or the decoding end inputs the image feature y_hat to the synthesis transformation network (i.e., the decoding network), and performs synthesis transformation on the image feature y_hat by the synthesis transformation network to obtain the reconstructed image block x_hat corresponding to the current image block x (i.e., the output feature of the synthesis transformation network). As can be seen from the above, the synthesis transformation network can be used to perform synthesis transformation on the image feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x.

[0112] After obtaining the output feature of the synthesis transformation network, the output feature of the synthesis transformation network can be taken as the reconstructed image block x_hat corresponding to the current image block x, i.e., the output feature is used to determine the reconstructed image block x_hat corresponding to the current image block x.

[0113] In embodiment 5, the decoding network can be a floating-point decoding network, i.e., the decoding network can use floating-point weights (i.e., floating-point parameters), i.e., the weights of the decoding network are all float-type weights, the input feature values of the decoding network are float-type feature values, and the output feature values of the decoding network are float-type feature values. For example, for each network layer of the decoding network, the network layer can use floating-point weights, i.e., the weights of the network layer are all float-type weights, the input feature values of the network layer are float-type feature values, and the output feature values of the network layer are float-type feature values.

[0114] However, in order to realize the floating-point decoding network, the decoding end needs to store float-type weights, and a large amount of storage resources are required when storing float-type weights. Moreover, when the decoding end processes float-type feature values using float-type weights, a large amount of computing resources are required, and the processing speed of the decoding end is relatively slow.

[0115] In view of the above finding, the present embodiment proposes an efficient inference method for image coding and decoding, which can use a fixed-point decoding network to process image features. The decoding network involved in embodiment 4 is a fixed-point decoding network, i.e., the decoding network can use fixed-point weights (i.e., fixed-point parameters), i.e., the weights of the decoding network are all fixed-bit weights, the input feature values of the decoding network are fixed-bit feature values, and the output feature values of the decoding network are fixed-bit feature values. For example, for a network layer of the decoding network, the network layer can use fixed-point weights, i.e., the weights of the network layer are fixed-bit weights, the input feature values of the network layer are fixed-bit feature values, and the output feature values of the network layer are fixed-bit feature values.

[0116] Obviously, in order to realize the decoding network of fixed-point weight, the decoding end needs to store the weight of fixed bits (such as 8 bits, 16 bits, etc.), and only a small amount of storage resources are occupied when storing the weight of fixed bits, thereby saving storage resources. Moreover, when the decoding end processes the feature value of fixed bits by using the weight of fixed bits, only a small amount of calculation resources are occupied, thereby saving calculation resources. Moreover, when the decoding end processes the feature value of fixed bits by using the weight of fixed bits, the processing speed can be improved, and adaptive acceleration of the decoding process is realized.

[0117] The embodiment is a training-free scheme, which can be quickly integrated into any floating-point decoding network to obtain a decoding network of fixed-point weight. The decoding network of fixed-point weight can be directly used for inference on a hardware platform. When the decoding network of fixed-point weight is obtained, the sensitivity of each network layer of the decoding network to quantization is evaluated to determine a more suitable quantization bit width of each network layer, thereby ensuring the quantization compression rate of the decoding network and the quantization performance of the decoding network.

[0118] For example, when the floating-point parameters of the decoding network are quantized into fixed-point parameters, the storage consumption of the decoding network can be reduced. When the floating-point feature values of the decoding network are quantized into fixed-point feature values, the bandwidth during forward calculation can also be reduced. In addition, when the floating-point operation in the decoding network is converted into a fixed-point operation, the calculation amount of the decoding network can be reduced, and the calculation under the fixed-point type also helps the decoding consistency between different devices. However, the quantization of the decoding network itself is realized by reducing the data type and the accuracy of calculation, thereby affecting the performance of the decoding network, which is unacceptable in image coding. In view of the above finding, considering that the decoder based on the neural network has different sensitivities to quantization in each network layer of the neural network when decoding the code stream, that is, some network layers can maintain the quantization performance at a lower bit, and some network layers need a higher bit to ensure the quantization accuracy, which is related to the parameters and feature distribution learned by the neural network. In the embodiment, in order to realize the maximum compression bit width while ensuring the performance after compression, an efficient compression and inference method is proposed, which can construct a decoding network of fixed-point weight by using the target weight quantization bit width and the target weight quantization hyperparameter, and generate fixed-point input features by using the target feature value quantization bit width and the target feature value quantization hyperparameter, realize adaptive decoding acceleration, and reduce the decoding calculation amount on the premise of ensuring the decoding quality.

[0119] Embodiment 6: Referring to Figure 6A As shown in Figure 5A The decoding network (such as the probability hyperparameter decoding network and the synthetic transform network) shown in the embodiment is based on three modules: a code stream extraction module, a quantization analysis module, and a decoding network quantization module; referring to Figure 6BAs shown in Figure 5B Three modules are added on the basis of the decoding network (such as the mean prediction network, the probability hyperparameter decoding network, and the synthesis transform network) shown, including a code stream extraction module, a quantization analysis module, and a decoding network quantization module.

[0120] Exemplarily, after the decoding network is trained (the decoding network is trained by using floating-point weights), the floating-point weights of the decoding network are fixed. For convenience of distinction, the decoding network using floating-point weights is referred to as a sample decoding network. The sample decoding network can include a plurality of network layers using floating-point weights, and the plurality of network layers can include a target network layer, which is a network layer that needs to use fixed-point weights (i.e., the floating-point weights of the target network layer need to be converted into fixed-point weights). Assuming that the sample decoding network includes network layer 1 and network layer 2, network layer 1 needs to use fixed-point weights, and network layer 2 does not need to use fixed-point weights, then network layer 1 is the target network layer, and network layer 2 is not the target network layer.

[0121] In a possible implementation, all network layers in the sample decoding network can be target network layers, part of the network layers in the sample decoding network can be target network layers, or all convolutional layers in the sample decoding network can be target network layers. Of course, the above are only examples, and the present disclosure is not limited in this regard. Any one or more network layers in the sample decoding network can be target network layers, and all network layers are taken as an example.

[0122] Exemplarily, to convert the sample decoding network into a target decoding network with fixed-point weights, the following steps can be used:

[0123] In step S11, the code stream extraction module obtains the sample code stream and inputs the sample code stream to the quantization analysis module.

[0124] For example, a plurality of images in a real scene (such as an application scene of an image acquisition device) can be collected, such as 100 images, 200 images, etc. The sample decoding network is used to process the collected images to obtain the first code stream and the second code stream corresponding to each image. Figure 3 The flow shown is used to process each image to obtain the first code stream (Bitstream#1) and the second code stream (Bitstream#2) corresponding to the image. The first code stream and the second code stream can be referred to as sample code streams, or the first code stream can be referred to as a sample code stream, or the second code stream can be referred to as a sample code stream. For convenience of description, the first code stream and the second code stream are taken as sample code streams in the subsequent process. The code stream extraction module can obtain the sample code stream corresponding to each image and input the sample code streams to the quantization analysis module.

[0125] In step S12, the quantization analysis module receives the sample code stream corresponding to each image and stores the sample code stream corresponding to each image.

[0126] In step S13, the quantization analysis module decodes the sample feature from the sample code stream, determines the floating-point input feature corresponding to the sample decoding network based on the sample feature, and determines the reference output feature corresponding to the sample feature based on the floating-point input feature. The reference output feature can be the output feature of a reference network layer in the sample decoding network. The reference network layer can be any network layer in the sample decoding network, for example, the last network layer in the sample decoding network, or the second last network layer in the sample decoding network, or the last target network layer in the sample decoding network, or any network layer after the last target network layer in the sample decoding network. The position of the reference network layer is not limited.

[0127] For example, for each sample code stream, the sample feature can be obtained by decoding the sample code stream. The process of obtaining the sample feature is described in Embodiment 4 and will not be repeated here. After obtaining the sample feature, the floating-point input feature corresponding to the sample decoding network is determined based on the sample feature. The process of determining the input feature of the decoding network based on the target feature is described in Embodiment 4 and will not be repeated here. After obtaining the floating-point input feature corresponding to the sample decoding network, the floating-point input feature is input into the sample decoding network, and each network layer of the sample decoding network is sequentially passed through. After passing through the reference network layer of the sample decoding network, the output feature of the reference network layer is taken as the reference output feature corresponding to the sample feature.

[0128] Obviously, for each sample code stream, the reference output feature corresponding to the sample code stream can be obtained. Based on this, the quantization analysis module can store the average value of the reference output features corresponding to all sample code streams, or the maximum value of the reference output features corresponding to all sample code streams, or the minimum value of the reference output features corresponding to all sample code streams. Taking the average value of the reference output features corresponding to all sample code streams as an example, the average value is stored as the reference output feature.

[0129] In step S14, for each target network layer in the sample decoding network, the quantization analysis module obtains the weight quantization hyperparameter candidate list corresponding to the target network layer. The weight quantization hyperparameter candidate list can include multiple candidate weight quantization hyperparameters.

[0130] Firstly, the quantization analysis module determines a maximum weight value based on all floating-point weights of the target network layer; wherein each floating-point weight comprises a plurality of weight values, and the maximum weight value is the maximum value among all weight values of all floating-point weights. For example, the target network layer can comprise a plurality of floating-point weights, each floating-point weight comprises a plurality of weight values, for each floating-point weight, the maximum value can be selected from all weight values corresponding to the floating-point weight, and then the maximum weight value can be selected from the maximum values corresponding to all floating-point weights. For example, assuming that the target network layer is network layer l, the maximum weight value can be denoted as max l = max(abs(w 0_l ), abs(w 1_l ), …, abs(w n_l )), and abs(w 0_l ) represents the maximum value corresponding to the first floating-point weight of network layer l, and so on, and max l represents the maximum weight value among all floating-point weights.

[0131] The target network layer can comprise a plurality of channels, if all channels correspond to the same target weight quantization hyperparameter, the maximum weight value is the maximum value among all weight values of the floating-point weights of all channels. If each channel corresponds to a target weight quantization hyperparameter independently, each channel corresponds to a maximum weight value, that is, the maximum weight value is the maximum value among all weight values of the floating-point weights of the channel. For convenience of description, the following takes all channels corresponding to the same target weight quantization hyperparameter as an example for description, the implementation mode of each channel corresponding to a target weight quantization hyperparameter independently is similar, and the same mode can be adopted to determine each channel corresponding to a target weight quantization hyperparameter independently, and the determination mode is the same as that of the target weight quantization hyperparameter corresponding to all channels.

[0132] Then, the quantization analysis module generates the initial weight quantization hyperparameter corresponding to the target network layer based on the maximum weight value. For example, the initial weight quantization hyperparameter corresponding to the target network layer can be generated by using the following formula: param = bw-1-ceil(log2(max)); wherein param is used to represent the initial weight quantization hyperparameter corresponding to the target network layer, bw is used to represent the fixed quantization bit width, max is used to represent the maximum weight value, and ceil is used to represent the rounding up operation.

[0133] For example, the initial weight quantization parameter corresponding to the target network layer is param, and param = get-param(bw, max). Here, bw represents the quantization bit width corresponding to the target network layer, which can be a fixed quantization bit width. The fixed quantization bit width can be configured according to experience, such as 4 bits, 8 bits, 16 bits, and the like. Take 8 bits as an example for illustration. param = get_param(bw, max) represents the initial weight quantization parameter calculated according to the data range max under the quantization bit width bw. Since the calculation method of the initial weight quantization parameter is different under different quantization algorithms, an example of the initial weight quantization parameter is given as follows: param = bw-1-ceil(log2(max)).

[0134] Then, the quantization analysis module constructs a weight quantization parameter candidate list based on the initial weight quantization parameter. The weight quantization parameter candidate list can include multiple candidate weight quantization parameters, i.e., multiple candidate weight quantization parameters corresponding to the target network layer.

[0135] For example, considering that there are usually outliers (abnormal weight values, the value range of which is usually particularly large) in the weight values of the neural network, these outliers are too large to affect the quantization accuracy. Therefore, the weight quantization parameter candidate list can be constructed based on the initial weight quantization parameter. For example, an example of the weight quantization parameter candidate list can be as follows: param = [param-2, param-1, param, param+1, param+2]. l_list l l l l l Of course, the above is only an example of the weight quantization parameter candidate list, and the candidate weight quantization parameters in the weight quantization parameter candidate list are not limited. Take the target network layer as network layer l for example. param represents the initial weight quantization parameter corresponding to network layer l, param-2, param-1, param, param+1, and param+2 are five candidate weight quantization parameters corresponding to network layer l. l l l l l l

[0136] Step S15: For each candidate weight quantization parameter, the quantization analysis module performs simulation quantization on the floating-point weights of the target network layer using the fixed quantization bit width and the candidate weight quantization parameter to obtain candidate simulation weights.

[0137] ​​​​​​​​​​​Exemplarily, the quantization bit width corresponding to the target network layer can be a fixed quantization bit width, which can be configured according to experience, such as 4 bits, 8 bits, 16 bits, and the like. Taking 8 bits as an example, the fixed quantization bit width is described.

[0138] Exemplarily, each candidate weight quantization parameter in the candidate weight quantization parameter list is traversed in sequence. For the currently traversed candidate weight quantization parameter, the fixed quantization bit width and the candidate weight quantization parameter can be used to simulate and quantize the floating-point weight of the target network layer to obtain the candidate simulation weight corresponding to the candidate weight quantization parameter. For example, the floating-point weight of the target network layer can be simulated and quantized to obtain the candidate simulation weight by using the following formula: quant = 2 -param × clip(round(data × 2 param ), -2 bw-1 , 2 bw-1 -1). Wherein, quant is used to represent the candidate simulation weight, param is used to represent the candidate weight quantization parameter, data is used to represent the floating-point weight of the target network layer, bw is used to represent the fixed quantization bit width, clip is used to represent the clipping function, and round is used to represent the rounding operation. Since the simulation and quantization manner is different under different quantization algorithms, the above formula only gives an example of simulation and quantization.

[0139] In summary, for each candidate weight quantization parameter, after the candidate weight quantization parameter param is substituted into the above formula, the candidate simulation weight quant corresponding to the candidate weight quantization parameter can be obtained.

[0140] After obtaining the candidate simulation weight, the quantization analysis module processes the floating-point input features of the target network layer based on the candidate simulation weight to obtain the floating-point output features of the target network layer. The quantization analysis module determines the quantization error corresponding to the candidate weight quantization parameter based on the floating-point output features of the target network layer.

[0141] Firstly, the floating-point input feature of the target network layer can be acquired. For example, if the target network layer is the first network layer of the sample decoding network, the input feature of the sample decoding network is the floating-point input feature of the target network layer. Or, if the target network layer is not the first network layer of the sample decoding network, the output feature of the previous network layer of the target network layer is taken as the floating-point input feature of the target network layer. For the network layer (denoted as network layer A) in front of the target network layer, if the network layer A is not a network layer requiring fixed-point weight, the network layer A directly processes the input feature to obtain a floating-point output feature, which is taken as the floating-point input feature of the target network layer. If the network layer A is a network layer requiring fixed-point weight, after obtaining the target weight quantization hyperparameter and the target weight quantization bit width of the network layer A, the floating-point weight of the network layer A can be simulated and quantized to obtain a pseudo weight of the network layer A by using the target weight quantization hyperparameter and the target weight quantization bit width, and the floating-point input feature of the network layer A is processed by using the pseudo weight to obtain the floating-point output feature of the network layer A, which is taken as the floating-point input feature of the target network layer.

[0142] Then, the floating-point input feature of the target network layer is processed based on the candidate simulation weight to obtain the floating-point output feature of the target network layer. For example, if the target network layer is a convolution layer, the floating-point input feature of the target network layer is processed based on the candidate simulation weight to obtain the floating-point output feature of the target network layer. If the target network layer is a pooling layer, the floating-point input feature of the target network layer is processed based on the candidate simulation weight to obtain the floating-point output feature of the target network layer. Of course, the above is only an example, and the processing mode is not limited, and is related to the function supported by the target network layer.

[0143] Then, the quantization error corresponding to the candidate weight quantization hyperparameter is determined based on the floating-point output feature of the target network layer. For example, the sample output feature is determined based on the floating-point output feature of the target network layer. The sample output feature can be the output feature of a reference network layer in the sample decoding network, and the reference network layer is any network layer located after the target network layer.

[0144] For example, the floating-point output feature of the target network layer is taken as the input feature of the next network layer (the next network layer still uses floating-point weight), the input feature is processed by the next network layer to obtain the floating-point output feature of the next network layer, and so on, until the reference network layer in the sample decoding network, the input feature is processed by the reference network layer to obtain the output feature of the reference network layer, and the output feature of the reference network layer is taken as the sample output feature.

[0145] Then, a quantization error corresponding to the candidate weight quantization hyperparameter is determined based on the sample output feature and the reference output feature. For example, based on the sample output feature and the reference output feature, the quantization error can be determined by using an error loss function, which can include but is not limited to an mse, a cosine similarity, a KL divergence, and the like, without limitation.

[0146] The sample output feature can be an average of sample output features corresponding to all sample code streams, and the reference output feature can be an average of reference output features corresponding to all sample code streams. Alternatively, the sample output feature can be a maximum of sample output features corresponding to all sample code streams, and the reference output feature can be a maximum of reference output features corresponding to all sample code streams. Alternatively, the sample output feature can be a minimum of sample output features corresponding to all sample code streams, and the reference output feature can be a minimum of reference output features corresponding to all sample code streams, without limitation.

[0147] In step S17, based on the quantization error corresponding to each candidate weight quantization hyperparameter, the quantization analysis module determines the candidate weight quantization hyperparameter corresponding to the minimum quantization error as the target weight quantization hyperparameter corresponding to the target network layer, and determines the fixed quantization bit width as the target weight quantization bit width corresponding to the target network layer. At this point, the quantization analysis module can obtain the target weight quantization hyperparameter corresponding to the target network layer and the target weight quantization bit width corresponding to the target network layer.

[0148] In step S18, the quantization analysis module uses the target weight quantization hyperparameter of the target network layer and the target weight quantization bit width of the target network layer to perform simulation quantization on the floating-point weight of the target network layer to obtain pseudo-weights of the target network layer.

[0149] For example, after obtaining the target weight quantization hyperparameter and the target weight quantization bit width, the target weight quantization hyperparameter and the target weight quantization bit width can also be used to perform simulation quantization on the floating-point weight of the target network layer to obtain pseudo-weights of the target network layer, i.e., the pseudo-weights of the target network layer are used to replace the floating-point weight of the target network layer. For the floating-point input feature of the target network layer, the target network layer processes the floating-point input feature of the target network layer through the pseudo-weights to obtain a floating-point output feature of the target network layer, which is used as a floating-point input feature of a subsequent network layer.

[0150] For example, based on the target weight quantization hyperparameter and the target weight quantization bit width, the pseudo-weights can be obtained by simulation quantization on the floating-point weight using the following formula: quant = 2 -param × clip(round(data × 2 param ), -2 bw-1 , 2bw-1 -1). Wherein, quant is used to represent the pseudo weight of the target network layer, param is used to represent the target weight quantization parameter of the target network layer, data is used to represent the floating-point weight of the target network layer, bw is used to represent the target weight quantization bit width of the target network layer, clip is used to represent the clipping function, and round is used to represent the rounding operation. Wherein, since the simulation quantization manner is different under different quantization algorithms, the above formula only gives an example of simulation quantization.

[0151] For example, for each target network layer, after obtaining the target weight quantization parameter and the target weight quantization bit width of the target network layer, the target feature value quantization bit width and the target feature value quantization parameter of the target network layer can also be obtained, for example, the target feature value quantization bit width and the target feature value quantization parameter of the target network layer can be obtained by the following steps:

[0152] Step S19, for each target network layer in the sample decoding network: the quantization analysis module obtains the feature value quantization parameter candidate list and the quantization bit width set corresponding to the target network layer, the feature value quantization parameter candidate list can include multiple candidate feature value quantization parameters, and the quantization bit width set can include at least two quantization bit widths.

[0153] Firstly, the quantization analysis module can determine the maximum feature value based on all floating-point input features of the target network layer; wherein each floating-point input feature includes multiple feature values, and the maximum feature value is the maximum value among all feature values of all floating-point input features. For example, the target network layer can correspond to multiple floating-point input features (such as 100 sample code streams corresponding to 100 floating-point input features), each floating-point input feature includes multiple feature values, for each floating-point input feature, the maximum value can be selected from all feature values corresponding to the floating-point input feature, and then the maximum feature value can be selected from the maximum values corresponding to all floating-point input features. For example, assuming that the target network layer is network layer l, the maximum feature value can be denoted as max l = max(abs(f 0_l ), abs(f 1_l ), …, abs(f n_l )), abs(f 0_l ) represents the maximum value of the first floating-point input feature of network layer l, and so on, max l represents the maximum feature value among all floating-point input features.

[0154] Then, the quantization analysis module generates an initial eigenvalue quantization hyperparameter corresponding to the target network layer based on the maximum eigenvalue. For example, the initial eigenvalue quantization hyperparameter corresponding to the target network layer can be generated by using the following formula: param = bw-1-ceil(log2(max)); where param is used to represent the initial eigenvalue quantization hyperparameter corresponding to the target network layer, bw is used to represent the current quantization bit width, max is used to represent the maximum eigenvalue, and ceil is used to represent the rounding up operation.

[0155] For example, the initial eigenvalue quantization hyperparameter corresponding to the target network layer is param, and param = het_param(bw, max), where bw is used to represent the quantization bit width corresponding to the target network layer, and the quantization bit width corresponding to the target network layer is the current quantization bit width (i.e., the quantization bit width currently traversed from the quantization bit width set). param = get_param(bw, max) represents the initial eigenvalue quantization hyperparameter calculated according to the data range max under the bw quantization bit width. Since the way of calculating the initial eigenvalue quantization hyperparameter is different under different quantization algorithms, only one example of the initial eigenvalue quantization hyperparameter is given here, and this is not limited.

[0156] Then, the quantization analysis module constructs an eigenvalue quantization hyperparameter candidate list based on the initial eigenvalue quantization hyperparameter. The eigenvalue quantization hyperparameter candidate list can include multiple candidate eigenvalue quantization hyperparameters, i.e., multiple candidate eigenvalue quantization hyperparameters corresponding to the target network layer. For example, considering that there are usually outliers (abnormal eigenvalues, the numerical range of abnormal eigenvalues is usually particularly large) in the eigenvalues of the neural network, these outliers are too large to affect the quantization accuracy. Therefore, the eigenvalue quantization hyperparameter candidate list can be constructed based on the initial eigenvalue quantization hyperparameter. For example, one example of the eigenvalue quantization hyperparameter candidate list can be: param l_list = [param l -2, param l -1, param l , param l +1, param l +2]; Of course, the above is only an example of the eigenvalue quantization hyperparameter candidate list, and the candidate eigenvalue quantization hyperparameters in the eigenvalue quantization hyperparameter candidate list are not limited. Taking the target network layer as network layer l for example, param l represents the initial eigenvalue quantization hyperparameter corresponding to network layer l, and the eigenvalue quantization hyperparameter candidate list is five candidate eigenvalue quantization hyperparameters corresponding to network layer l.

[0157] The quantization analysis module can further generate a quantization bit width set, which can include at least two quantization bit widths, such as two quantization bit widths, e.g., 8bit, 16bit, or 8bit, 32bit, or 16bit, 32bit, or 4bit, 8bit. For another example, the quantization bit width set can include three quantization bit widths, e.g., 8bit, 16bit, 32bit, or 4bit, 8bit, 16bit, or 8bit, 16bit, 24bit. For another example, the quantization bit width set can include four quantization bit widths. Of course, the above are only a few examples of the quantization bit width set, and are not limited thereto.

[0158] In step S20, the quantization analysis module traverses the current quantization bit width from the quantization bit width set in ascending order. For each candidate feature value quantization parameter, the quantization analysis module simulates and quantizes the floating-point input features of the target network layer using the current quantization bit width and the candidate feature value quantization parameter to obtain candidate input features.

[0159] For example, based on the ascending order of the quantization bit width, the quantization analysis module first traverses the first quantization bit width as the current quantization bit width. For example, assuming that the quantization bit width set includes two quantization bit widths, 8bit and 16bit, the quantization analysis module first traverses the quantization bit width of 8bit as the current quantization bit width, and executes step S20 based on the current quantization bit width.

[0160] For example, the candidate feature value quantization parameters in the candidate list are traversed in turn. For the currently traversed candidate feature value quantization parameter, the candidate input features corresponding to the candidate feature value quantization parameter can be obtained by simulating and quantizing the floating-point input features of the target network layer using the current quantization bit width and the candidate feature value quantization parameter. For example, the candidate input features can be obtained by simulating and quantizing the floating-point input features of the target network layer using the following formula: quant = 2 -param × clip(round(data × 2 param ), -2 bw-1 , 2 bw-1 -1); where quant represents the candidate input features, param represents the candidate feature value quantization parameter, data represents the floating-point input features, bw represents the current quantization bit width, clip represents a clipping function, and round represents a rounding operation. Since the simulation and quantization method is different under different quantization algorithms, the above formula is only an example of simulation and quantization.

[0161] Based on the above, for each candidate feature value quantization hyperparameter, after substituting the candidate feature value quantization hyperparameter param into the above formula, the candidate input feature corresponding to the candidate feature value quantization hyperparameter can be obtained.

[0162] In step S21, after obtaining the candidate input feature, the quantization analysis module processes the candidate input feature of the target network layer based on the pseudo weight of the target network layer to obtain the floating-point output feature of the target network layer. The quantization analysis module determines the quantization error corresponding to the candidate feature value quantization hyperparameter based on the floating-point output feature of the target network layer.

[0163] First, for the floating-point input feature of the target network layer (the content of the floating-point input feature can be referred to step S16), the floating-point input feature has been simulated and quantized by the current quantization bit width and the candidate feature value quantization hyperparameter to obtain the candidate input feature corresponding to the candidate feature value quantization hyperparameter, and this candidate input feature is taken as the input feature of the target network layer.

[0164] Then, the candidate input feature is processed based on the pseudo weight of the target network layer to obtain the floating-point output feature of the target network layer. For example, if the target network layer is a convolution layer, the candidate input feature is convolved based on the pseudo weight to obtain the floating-point output feature of the target network layer. If the target network layer is a pooling layer, the candidate input feature is pooled based on the pseudo weight to obtain the floating-point output feature of the target network layer. Of course, the above is only an example, and the processing method is not limited, which is related to the function supported by the target network layer. Among them, since the target weight quantization hyperparameter and the target weight quantization bit width corresponding to the target network layer are obtained, the pseudo weight is obtained by simulating and quantizing the floating-point weight of the target network layer, and therefore, the floating-point output feature is obtained by processing the candidate input feature based on the pseudo weight of the target network layer.

[0165] Then, the quantization error corresponding to the candidate feature value quantization hyperparameter is determined based on the floating-point output feature of the target network layer. For example, the sample output feature is determined based on the floating-point output feature of the target network layer, and the sample output feature can be the output feature of the reference network layer in the sample decoding network, and the reference network layer is any network layer located behind the target network layer.

[0166] For example, the floating-point output feature of the target network layer is taken as the input feature of the next network layer (the next network layer still uses floating-point weight), the input feature is processed by the next network layer to obtain the floating-point output feature of the next network layer, and so on, until the reference network layer in the sample decoding network, the input feature is processed by the reference network layer to obtain the output feature of the reference network layer, and the output feature of the reference network layer is taken as the sample output feature.

[0167] Then, a quantization error corresponding to the candidate feature value quantization hyperparameter is determined based on the sample output feature and the reference output feature; for example, based on the sample output feature and the reference output feature, the quantization error can be determined by using an error loss function, which can include but is not limited to mse, cosine similarity, KL divergence, etc. loss function, which is not limited.

[0168] The sample output feature can be the average of the sample output features corresponding to all sample code streams, and the reference output feature can be the average of the reference output features corresponding to all sample code streams; or the sample output feature can be the maximum of the sample output features corresponding to all sample code streams, and the reference output feature can be the maximum of the reference output features corresponding to all sample code streams; or the sample output feature can be the minimum of the sample output features corresponding to all sample code streams, and the reference output feature can be the minimum of the reference output features corresponding to all sample code streams, which is not limited.

[0169] Step S22, based on the quantization error corresponding to each candidate feature value quantization hyperparameter, the quantization analysis module determines whether the minimum quantization error is less than a preset threshold (which can be configured according to experience), if yes, execute step S23, if not, execute step S24.

[0170] Step S23, the quantization analysis module determines the candidate feature value quantization hyperparameter corresponding to the minimum quantization error as the target feature value quantization hyperparameter corresponding to the target network layer, and determines the current quantization bit width as the target feature value quantization bit width corresponding to the target network layer, and records the target feature value quantization hyperparameter and the target feature value quantization bit width for the target network layer.

[0171] At this point, the quantization analysis module can obtain the target feature value quantization hyperparameter and the target feature value quantization bit width corresponding to the target network layer, and record the target feature value quantization hyperparameter and the target feature value quantization bit width for the target network layer.

[0172] Step S24, the quantization analysis module determines whether the current quantization bit width is the last quantization bit width of the quantization bit width set.

[0173] If yes, the candidate feature value quantization hyperparameter corresponding to the minimum quantization error is determined as the target feature value quantization hyperparameter corresponding to the target network layer, and the current quantization bit width is determined as the target feature value quantization bit width corresponding to the target network layer, and the target feature value quantization hyperparameter and the target feature value quantization bit width are recorded for the target network layer, which can be referred to step S23.

[0174] If not, the next quantization bit-width of the current quantization bit-width is traversed as the current quantization bit-width, and step S20 is returned. Based on the current quantization bit-width, the quantization analysis module uses the current quantization bit-width and the candidate feature value quantization hyperparameter to simulate and quantize the floating-point input features of the target network layer to obtain candidate input features.

[0175] For example, the first traversal is 8bit quantization bit-width as the current quantization bit-width, if the minimum quantization error is not less than the preset threshold, the second traversal is 16bit quantization bit-width as the current quantization bit-width, and the above steps are repeated, and so on.

[0176] For example, after recording the target feature value quantization hyperparameter and the target feature value quantization bit-width for the target network layer, the quantization analysis module successfully obtains the target weight quantization hyperparameter, the target weight quantization bit-width, the target feature value quantization hyperparameter and the target feature value quantization bit-width corresponding to the target network layer. At this point, the processing process of the target network layer is completed, and the above steps are repeated for the next target network layer of the target network layer, and so on, until the target weight quantization hyperparameter, the target weight quantization bit-width, the target feature value quantization hyperparameter and the target feature value quantization bit-width corresponding to each target network layer are obtained.

[0177] Step S25, after obtaining the target weight quantization hyperparameter, the target weight quantization bit-width, the target feature value quantization hyperparameter and the target feature value quantization bit-width corresponding to each target network layer, the decoding network quantization module generates a target decoding network based on the sample decoding network. For example, for each target network layer in the sample decoding network: the decoding network quantization module performs fixed-point quantization on the floating-point weights of the target network layer based on the target weight quantization bit-width and the target weight quantization hyperparameter corresponding to the target network layer to obtain the fixed-point weights of the target network layer. After obtaining the fixed-point weights of each target network layer, the decoding network quantization module can generate a target decoding network based on the fixed-point weights of each target network layer.

[0178] For example, for each target network layer, the quantization analysis module can output the target weight quantization bit-width and the target weight quantization hyperparameter corresponding to the target network layer to the decoding network quantization module, and the decoding network quantization module performs fixed-point quantization on the floating-point weights of the target network layer based on the target weight quantization bit-width and the target weight quantization hyperparameter corresponding to the target network layer to obtain the fixed-point weights of the target network layer. The fixed-point quantization process of this floating-point weight is not limited.

[0179] For each target network layer, the quantization analysis module can also output the target feature value quantization parameter and the target feature value quantization bit width corresponding to the target network layer to the decoding network quantization module, and the decoding network quantization module records the target feature value quantization parameter and the target feature value quantization bit width for the target network layer. For example, the decoding network quantization module can record the target feature value quantization parameter and the target feature value quantization bit width in the configuration information of the target network layer. When the target network layer performs related processing, the target feature value quantization parameter and the target feature value quantization bit width can be read from the configuration information. Of course, the target feature value quantization parameter and the target feature value quantization bit width can also be recorded in other storage areas, as long as the target network layer can obtain the target feature value quantization parameter and the target feature value quantization bit width.

[0180] After obtaining the fixed-point weight of each target network layer, the fixed-point weight can be used to replace the floating-point weight of the target network layer, so that all the target network layers after replacement can form a target decoding network. At this point, the target decoding network is successfully obtained, and the target decoding network can be deployed to the encoding end or the decoding end. For example, the target decoding network can include a probability parameter decoding network and a synthesis transformation network, or the target decoding network can include a mean prediction network, a probability parameter decoding network, and a synthesis transformation network. By deploying these target decoding networks to the encoding end, the encoding end performs processing based on the target decoding network when the encoding end performs processing according to embodiment 2. In addition, by deploying these target decoding networks to the decoding end, the decoding end performs processing based on the target decoding network when the decoding end performs processing according to embodiment 3.

[0181] Embodiment 7: Referring to Figure 6A Based on the decoding network shown in Figure 5A Based on the decoding network shown in Figure 6B Based on the decoding network shown in Figure 5B Based on the decoding network shown in

[0182] For example, in order to convert the sample decoding network into a target decoding network with fixed-point weights, the following steps can be taken:

[0183] In step S31, the code stream extraction module acquires the sample code stream and inputs the sample code stream to the quantization analysis module.

[0184] In step S32, the quantization analysis module receives the sample code stream corresponding to each frame of image and stores the sample code stream corresponding to each frame of image.

[0185] In step S33, the quantization analysis module decodes the sample feature from the sample code stream, determines the floating-point input feature corresponding to the sample decoding network based on the sample feature, and determines the reference output feature corresponding to the sample feature based on the floating-point input feature. The reference output feature can be the output feature of a reference network layer in the sample decoding network. The reference network layer can be any network layer in the sample decoding network, for example, the last network layer in the sample decoding network, or the second last network layer in the sample decoding network, or the last target network layer in the sample decoding network, or any network layer after the last target network layer in the sample decoding network. The position of the reference network layer is not limited.

[0186] For example, steps S31-S33 are similar to steps S11-S13, which will not be repeated here.

[0187] In step S34, for each target network layer in the sample decoding network, the quantization analysis module acquires the weight quantization hyperparameter candidate list corresponding to the target network layer, which can include multiple candidate weight quantization hyperparameters.

[0188] First, the quantization analysis module determines the maximum weight value based on all floating-point weights of the target network layer. Each floating-point weight can include multiple weight values, and the maximum weight value can be the maximum value of all weight values of all floating-point weights. For example, the target network layer can include multiple floating-point weights, each of which can include multiple weight values. For each floating-point weight, the maximum value can be selected from all weight values corresponding to the floating-point weight, and then the maximum weight value can be selected from the maximum values corresponding to all floating-point weights.

[0189] Then, the quantization analysis module generates the initial weight quantization hyperparameter corresponding to the target network layer based on the maximum weight value. For example, the initial weight quantization hyperparameter corresponding to the target network layer is generated by the following formula: param = (max) / 2 bw -1; wherein, param is used to represent the initial weight quantization parameter corresponding to the target network layer, bw is used to represent the fixed quantization bit width, and max is used to represent the maximum weight value. The initial weight quantization parameter corresponding to the target network layer is param, param = get_param(bw, max), and bw is used to represent the quantization bit width corresponding to the target network layer, which can be a fixed quantization bit width, such as 4 bits, 8 bits, 16 bits, etc. param = get_param(bw, max) represents that the initial weight quantization parameter is calculated according to the data range max under the bw quantization bit width. Since the way of calculating the initial weight quantization parameter is different under different quantization algorithms, only an example of the initial weight quantization parameter is given here.

[0190] Then, the quantization analysis module constructs a weight quantization parameter candidate list based on the initial weight quantization parameter, which can include multiple candidate weight quantization parameters, i.e., multiple candidate weight quantization parameters corresponding to the target network layer. For example, considering that there are usually outliers (abnormal weight values) in the weight values of the neural network, a weight quantization parameter candidate list can be constructed based on the initial weight quantization parameter, and an example of the weight quantization parameter candidate list can be as follows: Of course, the above is only an example of the weight quantization parameter candidate list, and the candidate weight quantization parameters in the weight quantization parameter candidate list are not limited. Taking the target network layer as network layer l for example, param l represents the initial weight quantization parameter corresponding to network layer l, and the weight quantization parameter candidate list shows 7 candidate weight quantization parameters corresponding to network layer l.

[0191] Step S35, for each candidate weight quantization parameter, the quantization analysis module uses the fixed quantization bit width and the candidate weight quantization parameter to simulate and quantize the floating-point weight of the target network layer to obtain a candidate simulation weight.

[0192] For example, the quantization bit width corresponding to the target network layer can be a fixed quantization bit width, which can be configured according to experience, and is not limited. Each candidate weight quantization parameter in the weight quantization parameter candidate list can be traversed in turn, and for the currently traversed candidate weight quantization parameter, the fixed quantization bit width and the candidate weight quantization parameter can be used to simulate and quantize the floating-point weight of the target network layer to obtain the candidate simulation weight corresponding to the candidate weight quantization parameter. For example, the floating-point weight of the target network layer can be simulated and quantized to obtain the candidate simulation weight using the following formula: quant = param x clip(round*data ÷ param), -2 bw-1 , 2 bw-1-1), where quant represents the candidate simulation weight, param represents the candidate weight quantization parameter, data represents the floating-point weight of the target network layer, bw represents the fixed quantization bit width, clip represents a clipping function, and round represents a rounding operation. Since the simulation quantization method varies under different quantization algorithms, the above formula only gives an example of simulation quantization.

[0193] As described above, for each candidate weight quantization parameter, after substituting the candidate weight quantization parameter param into the above formula, the candidate simulation weight corresponding to the candidate weight quantization parameter can be obtained.

[0194] Step S36, after obtaining the candidate simulation weight, the quantization analysis module processes the floating-point input features of the target network layer based on the candidate simulation weight to obtain the floating-point output features of the target network layer. The quantization analysis module determines the quantization error corresponding to the candidate weight quantization parameter based on the floating-point output features of the target network layer.

[0195] Step S37, based on the quantization error corresponding to each candidate weight quantization parameter, the quantization analysis module determines the candidate weight quantization parameter corresponding to the minimum quantization error as the target weight quantization parameter corresponding to the target network layer, and determines the fixed quantization bit width as the target weight quantization bit width corresponding to the target network layer. At this point, the quantization analysis module can obtain the target weight quantization parameter corresponding to the target network layer and the target weight quantization bit width corresponding to the target network layer.

[0196] For example, steps S36-S37 are similar to steps S16-S17, which will not be repeated here.

[0197] Step S38, the quantization analysis module uses the target weight quantization parameter of the target network layer and the target weight quantization bit width of the target network layer to simulate and quantize the floating-point weight of the target network layer to obtain the pseudo weight of the target network layer.

[0198] For example, after obtaining the target weight quantization parameter and the target weight quantization bit width, the target weight quantization parameter and the target weight quantization bit width can also be used to simulate and quantize the floating-point weight of the target network layer to obtain the pseudo weight of the target network layer. For the floating-point input features of the target network layer, the target network layer processes the floating-point input features of the target network layer through the pseudo weight to obtain the floating-point output features of the target network layer, which are used as the floating-point input features of the subsequent network layer. For example, the following formula can be used to simulate and quantize the floating-point weight to obtain the pseudo weight: quant = param x clip(round(data ÷ param), -2 bw-1,2 bw-1 -1).

[0199] wherein quant denotes the pseudo weight of the target network layer, param denotes the target weight quantization parameter of the target network layer, data denotes the floating point weight of the target network layer, bw denotes the target weight quantization bit width of the target network layer, clip denotes the clipping function, and round denotes the rounding operation. Since the simulation quantization manner is different under different quantization algorithms, the above formula only gives an example of simulation quantization.

[0200] For example, for each target network layer, after obtaining the target weight quantization parameter and the target weight quantization bit width of the target network layer, the target feature value quantization bit width and the target feature value quantization parameter of the target network layer can also be obtained. For example, the target feature value quantization bit width and the target feature value quantization parameter of the target network layer can be obtained by the following steps:

[0201] In step S39, for each target network layer in the sample decoding network, the quantization analysis module obtains the feature value quantization parameter candidate list and the quantization bit width set corresponding to the target network layer. The feature value quantization parameter candidate list can include multiple candidate feature value quantization parameters, and the quantization bit width set can include at least two quantization bit widths.

[0202] Firstly, the quantization analysis module can determine the maximum feature value based on all floating point input features of the target network layer. Each floating point input feature includes multiple feature values, and the maximum feature value is the maximum value among all feature values of all floating point input features. For example, the target network layer can correspond to multiple floating point input features, each floating point input feature includes multiple feature values, for each floating point input feature, the maximum value can be selected from all feature values corresponding to the floating point input feature, and then the maximum feature value can be selected from the maximum values corresponding to all floating point input features.

[0203] Then, the quantization analysis module generates the initial feature value quantization parameter corresponding to the target network layer based on the maximum feature value. For example, the initial feature value quantization parameter corresponding to the target network layer can be generated by the following formula: param = (max) / 2 bw -1; wherein, param is used to represent the initial eigenvalue quantization parameter, bw is used to represent the fixed quantization bit width, and max is used to represent the maximum eigenvalue. The initial eigenvalue quantization parameter is param, param = get_param(bw, max), and bw is used to represent the quantization bit width corresponding to the target network layer. The quantization bit width corresponding to the target network layer is the current quantization bit width (i.e., the quantization bit width currently traversed in the quantization bit width set). param = get_param(bw, max) represents that the initial eigenvalue quantization parameter is calculated according to the data range max under the quantization bit width bw. Since the way of calculating the initial eigenvalue quantization parameter is different under different quantization algorithms, only an example of the initial eigenvalue quantization parameter is given here, and this is not limited.

[0204] Then, the quantization analysis module can construct an eigenvalue quantization parameter candidate list based on the initial eigenvalue quantization parameter. The eigenvalue quantization parameter candidate list can include a plurality of candidate eigenvalue quantization parameters, i.e., a plurality of candidate eigenvalue quantization parameters corresponding to the target network layer. For example, considering that there are usually outliers in the eigenvalues of the neural network, these outliers are too large to affect the quantization precision. Therefore, the quantization analysis module can construct an eigenvalue quantization parameter candidate list based on the initial eigenvalue quantization parameter. For example, an example of the eigenvalue quantization parameter candidate list can be: Of course, the above is only an example of the eigenvalue quantization parameter candidate list, and the eigenvalue quantization parameter candidate list is not limited.

[0205] The quantization analysis module can also generate a quantization bit width set, which can include at least two quantization bit widths.

[0206] In step S40, the quantization analysis module traverses the current quantization bit width from the quantization bit width set in the order from small to large. For each candidate eigenvalue quantization parameter, the quantization analysis module uses the current quantization bit width and the candidate eigenvalue quantization parameter to simulate and quantize the floating-point input features of the target network layer to obtain candidate input features.

[0207] For example, the quantization analysis module can traverse the first quantization bit width as the current quantization bit width in the order of quantization bit widths from small to large. The quantization analysis module traverses each candidate feature value quantization parameter in the candidate feature value quantization parameter list in turn. For the candidate feature value quantization parameter currently traversed, the candidate input feature corresponding to the candidate feature value quantization parameter can be obtained by using the current quantization bit width and the candidate feature value quantization parameter to simulate and quantize the floating-point input feature of the target network layer. For example, the candidate input feature can be obtained by using the following formula to simulate and quantize the floating-point input feature of the target network layer: quant = param x clip(round(data ÷ param), -2 bw-1 , 2 bw-1 -1); where quant represents the candidate input feature, param represents the candidate feature value quantization parameter, data represents the floating-point input feature, bw represents the current quantization bit width, clip represents a clipping function, and round represents a rounding operation. Since the simulation and quantization manner is different under different quantization algorithms, the above formula is only an example of simulation and quantization.

[0208] As described above, for each candidate feature value quantization parameter, the candidate input feature corresponding to the candidate feature value quantization parameter can be obtained after the candidate feature value quantization parameter param is substituted into the above formula.

[0209] Step S41, after obtaining the candidate input feature, the quantization analysis module processes the candidate input feature of the target network layer based on the pseudo weight of the target network layer to obtain the floating-point output feature of the target network layer. The quantization analysis module determines the quantization error corresponding to the candidate feature value quantization parameter based on the floating-point output feature of the target network layer.

[0210] Step S42, based on the quantization error corresponding to each candidate feature value quantization parameter, the quantization analysis module determines whether the minimum quantization error is less than a preset threshold (which can be configured according to experience). If yes, step S43 is performed, and if no, step S44 is performed.

[0211] Step S43, the quantization analysis module determines the candidate feature value quantization parameter corresponding to the minimum quantization error as the target feature value quantization parameter corresponding to the target network layer, and determines the current quantization bit width as the target feature value quantization bit width corresponding to the target network layer. The target feature value quantization parameter and the target feature value quantization bit width are recorded for the target network layer.

[0212] Step S44, the quantization analysis module determines whether the current quantization bit width is the last quantization bit width in the set of quantization bit widths.

[0213] If yes, the candidate feature value quantization hyper-parameter corresponding to the minimum quantization error is determined as the target feature value quantization hyper-parameter corresponding to the target network layer, the current quantization bit-width is determined as the target feature value quantization bit-width corresponding to the target network layer, and the target feature value quantization hyper-parameter and the target feature value quantization bit-width are recorded for the target network layer. The process can be referred to step S43.

[0214] If no, the next quantization bit-width of the current quantization bit-width is taken as the current quantization bit-width, and step S40 is returned. Based on the current quantization bit-width, for each candidate feature value quantization hyper-parameter, the candidate input feature is obtained by using the current quantization bit-width and the candidate feature value quantization hyper-parameter to simulate and quantize the floating-point input feature of the target network layer.

[0215] Step S45, after obtaining the target weight quantization hyper-parameter, the target weight quantization bit-width, the target feature value quantization hyper-parameter and the target feature value quantization bit-width corresponding to each target network layer, the decoding network quantization module generates a target decoding network based on the sample decoding network. For example, for each target network layer in the sample decoding network: the decoding network quantization module performs fixed-point quantization on the floating-point weight of the target network layer based on the target weight quantization bit-width and the target weight quantization hyper-parameter corresponding to the target network layer to obtain the fixed-point weight of the target network layer. After obtaining the fixed-point weight of each target network layer, the decoding network quantization module can generate a target decoding network based on the fixed-point weight of each target network layer.

[0216] For example, steps S41-S45 are similar to steps S21-S25, which will not be repeated here.

[0217] After obtaining the fixed-point weight of each target network layer, the floating-point weight of the target network layer can be replaced with the fixed-point weight. In this way, all the target network layers after replacement can form a target decoding network. At this point, the target decoding network is successfully obtained, and the target decoding network can be deployed to the encoding end or the decoding end. For example, the target decoding network can include a probability hyper-parameter decoding network and a synthesis transformation network, or the target decoding network can include a mean prediction network, a probability hyper-parameter decoding network and a synthesis transformation network. By deploying these target decoding networks to the encoding end, the encoding end performs processing based on the target decoding network when the encoding end performs processing according to embodiment 2. In addition, by deploying these target decoding networks to the decoding end, the decoding end performs processing based on the target decoding network when the decoding end performs processing according to embodiment 3.

[0218] Embodiment 8: The sensitivity of the neural network-based decoder to quantization is different at each network layer in the neural network, that is, some network layers can still maintain quantization performance at a lower bit, and some network layers need a higher bit to ensure quantization accuracy. This is related to the parameters and feature distribution learned by the neural network. Therefore, in order to achieve the maximum compression of the model bit width while ensuring the performance of the compressed model, and at the same time allowing the compressed model to be allowed on most devices, an efficient decoder compression and inference method is proposed in this embodiment, which mainly includes the following steps: 1. Collect a number of images (such as 100-200 frames) in a real scene (application scene of an image acquisition device). 2. Input the images into the encoding network to extract the code stream to be transmitted to the decoding network (i.e. the first code stream and the second code stream), and store it. 3. Transmit the stored code stream to the quantization analysis module to obtain the appropriate quantization bit width and quantization parameter (such as quantization hyperparameter) of each network layer of the decoding network. 4. Quantize each network layer of the decoding network according to the quantization bit width and quantization parameter of each network layer obtained by the quantization analysis module to obtain the final quantization model, that is, quantize each network layer of the sample decoding network to obtain the target decoding network.

[0219] The code stream extraction module, the quantization analysis module, and the decoding network quantization module can be added to the decoding network as shown in Figure 5A Or Figure 5B The functions of the code stream extraction module, the quantization analysis module, and the decoding network quantization module are described below.

[0220] Code stream extraction module: used to collect a number of images (such as 100-200 frames) in a real scene (application scene of an image acquisition device), input the images into the encoding network for forward calculation, and extract the output of the encoding network (i.e. the code stream for transmission to the decoding network, i.e. the first code stream and the second code stream), and store the code stream corresponding to each frame of image.

[0221] Quantization analysis module: the code stream information extracted by the code stream extraction module enters the quantization analysis module. The function of the quantization analysis module is to analyze the sensitivity of each network layer of the decoding network to quantization, and to obtain the quantization bit width that should be allocated to each network layer of the decoding network under the condition of ensuring decoding performance, that is, to obtain the quantization bit width of each network layer of the decoding network.

[0222] In the quantization analysis module, a copy of the forward calculation result of the decoding network is stored. For example, after the code stream enters the decoding network, it first performs a decoding forward calculation of the full floating-point data type, and the input features (f, i represents the i-th code stream, and l represents the l-th network layer) of each network layer of the decoding network under the floating-point, and the output features (f i_o, i represents the i-th code stream) are stored down, and the output feature of the last layer is the reference output feature of the above embodiment.

[0223] The quantization analysis module also supports simulating quantization operations on the weight values and input feature values of each network layer of the decoding network, and can perform simulation quantization calculation to simulate the quantization inference of the decoding network. In the decoding network, the weight values of the network layers (such as convolution layers) are not sensitive to quantization at 8 bits, while the input feature values of the network layers (such as convolution layers) are more sensitive to quantization. Therefore, using 8-bit quantization for part of the network layers can meet the performance loss, but part of the network layers may be difficult to meet the quantization performance loss. Therefore, the quantization analysis module mainly analyzes the quantization of the input feature values of the network layers (such as convolution layers) of the decoding network, and selects appropriate quantization bit width and quantization hyperparameters for the input of each network layer of the decoding network.

[0224] The quantization analysis module initializes the quantization hyperparameters of the input feature values of each network layer of the decoding network. The initialization of the quantization hyperparameters depends on the data range of the floating-point features, and the data range is obtained by calculating the absolute value of the input feature (f i_l , i represents the i-th code stream, and l represents the l-th network layer) of each network layer of the decoding network under floating point, that is, max l = max(abs(f 0_l ), abs(f 1_l ), …, abs(f n_l )). The initialized quantization hyperparameters are obtained by the calculation formula: param l = get_param(bw l , max l ), bw l represents the bit width of the l-th layer input feature, which is initialized to 8, and represents the quantization hyperparameters calculated according to the data range max under the bw quantization bit width. Since the way of calculating the quantization hyperparameters is slightly different under different quantization algorithms, only one example is given here: param = bw-1-ceil(log2(max)). Considering that there are usually outliers (abnormal feature values, usually with a particularly large numerical range) in the feature values of the neural network, these outliers are too large to affect the quantization accuracy. Therefore, based on the initialized quantization hyperparameters, a candidate list of hyperparameters is set, that is, param l_list = [param l -2, param l -1, param l , param l +1, param l +2]. In addition, another example of calculating the quantization hyperparameters can also be: param = (max) / 2 bw-1, the candidate list of the quantization parameter is set based on the initialized quantization parameter:

[0225] The quantization analysis module analyzes the quantization of the decoding network layer by layer. The analysis method is to simulate quantization (such as initial 8-bit quantization) of input feature values layer by layer, input the simulated quantized features into the network for forward calculation, save the output of the decoding network, and compare it with the saved floating-point output features f i_o Perform error analysis. When the error is greater than a certain threshold, increase the quantization bit width of the layer to 16 bits. The calculation formula of the simulated quantization can be: data q = quant(data f , bw, param). Since the calculation formula of the simulated quantization is different under different quantization algorithms, only one example of the simulated quantization method is given here: quant(data f , bw, param) = 2 -param × clip(round(data f × 2 param ), -2 bw-1 , 2 bw-1 -1), clip represents a clipping function. In addition, another example of the simulated quantization method for the simulated quantization process can also be: quant(data f , bw, param) = param × clip(round(data f ÷param), -2 bw-1 , 2 bw-1 -1).

[0226] In one possible implementation, the quantization parameter and quantization bit width of the feature value can be calculated by the following steps:

[0227] Initialize the quantization bit width, such as setting the initial value of the quantization bit width to 8 bits.

[0228] Initialize the quantization parameter, such as determining the initial value of the quantization parameter using the following formula: param = bw-1-ceil(log2(max)), where the initial value of the quantization parameter is the initial feature value quantization parameter in the above embodiment.

[0229] Initialize the quantization error, such as setting a preset threshold for the quantization error, which can be set according to experience.

[0230] Obtain the candidate list of the quantization parameter, such as obtaining the candidate list of the quantization parameter based on the initial value of the quantization parameter using the following formula: param l_list =[param l -2, paraml -1, param l , param l +1, param l +2].

[0231] The input feature value is simulated and quantized, such as based on the quantization bit width and the quantization parameter, and the simulated quantization is performed using the following formula: quant(data f , bw, param) = 2 -param * clip(round(data f * 2 param ), -2 bw-1 , 2 bw-1 -1).

[0232] The simulated and quantized feature value is input to the decoding network for forward calculation.

[0233] The quantization error between the output of the simulated and quantized decoding network and the output of the floating-point decoding network is calculated.

[0234] If the quantization error is less than a preset threshold, the quantization bit width of the output feature value is 8 bits, and the quantization parameter of the output feature value is the quantization parameter corresponding to the minimum quantization error (i.e., a quantization parameter in the quantization parameter candidate list).

[0235] If the quantization error is not less than the preset threshold, the quantization bit width is increased to 16 bits.

[0236] The initial value of the quantization parameter is reacquired, that is, when the initial value of the quantization parameter is acquired using the above formula, the quantization bit width bw is modified from 8 bits to 16 bits, and the initial value of the quantization parameter is acquired under the premise of 16 bits.

[0237] The quantization parameter candidate list is reacquired, such as based on the initial value of the quantization parameter to acquire the quantization parameter candidate list.

[0238] The input feature value is simulated and quantized, such as using the above formula for simulated quantization.

[0239] The simulated and quantized feature value is input to the decoding network for forward calculation.

[0240] The quantization error between the output of the simulated and quantized decoding network and the output of the floating-point decoding network is calculated.

[0241] The quantization bit width of the output feature value is 16 bits, and the quantization parameter of the output feature value is the quantization parameter corresponding to the minimum quantization error.

[0242] In the above process, when calculating the quantization error, an error loss function can be used to calculate the quantization error, and the calculation method of the error loss function can include but is not limited to mse, cosine similarity, KL divergence, etc. After the above process is calculated, the quantization bit width bw of each network layer input of the decoding network can be obtained l and quantization hyperparameters param l .

[0243] The decoding network quantization module: according to the quantization bit width bw of each network layer input output by the quantization analysis module l and quantization hyperparameters param l , and the fixed 8-bit quantization bit width of each network layer weight, the decoding network quantization module can quantize the decoding network. For example, the weight parameters of the decoding network will be quantized to 8 bits, which can greatly compress the storage space of the model. At the same time, the decoding network quantization module will mark the bit width information (8 bits or 16 bits) of each network layer input, and write the corresponding quantization hyperparameters into the calculation process of the network layer (usually represented as shift). In this way, the decoding quality of the decoding network can be guaranteed, and the calculation amount and throughput bandwidth of the model can be further compressed.

[0244] Embodiment 9: After obtaining the target decoding network, the target decoding network can be deployed to the encoding end or the decoding end. For example, the target decoding network includes a probability hyperparameter decoding network and a synthesis transformation network, or the target decoding network includes a mean prediction network, a probability hyperparameter decoding network, and a synthesis transformation network. By deploying these target decoding networks to the encoding end, when the encoding end performs processing according to embodiment 2, the processing is performed based on the target decoding network. By deploying these target decoding networks to the decoding end, when the decoding end performs processing according to embodiment 3, the processing is performed based on the target decoding network.

[0245] In one possible implementation, the encoding end or the decoding end can decode the target feature corresponding to the current image block from the code stream (such as the first code stream and the second code stream) corresponding to the current image block, determine the input feature of the decoding network based on the target feature, process the input feature based on the decoding network, and obtain the output feature of the decoding network. After obtaining the output feature, the reconstructed image block corresponding to the current image block can be determined based on the output feature, and the process is described below.

[0246] For example, the encoding end or the decoding end decodes a target feature corresponding to a current image block from a bitstream corresponding to the current image block, and determines a first input feature of a target decoding network based on the target feature. The target feature value quantization bit width and the target feature value quantization parameter of the target decoding network are obtained (for example, from the configuration information of the target decoding network), and the first input feature is converted into a second input feature based on the target feature value quantization bit width and the target feature value quantization parameter. The second input feature is processed based on the fixed-point type weight of the target decoding network (the fixed-point type weight can be determined based on the target weight quantization bit width and the target weight quantization parameter) to obtain an output feature of the target decoding network. The reconstructed image block corresponding to the current image block is determined based on the output feature of the target decoding network.

[0247] For example, the target decoding network includes at least one target network layer, and the target network layer is a network layer using a fixed-point type weight. For each target network layer in the target decoding network: the first input feature of the target network layer can be converted into a second input feature based on the target feature value quantization bit width and the target feature value quantization parameter of the target network layer; and the second input feature is processed based on the fixed-point type weight of the target network layer (the fixed-point type weight of the target network layer is determined based on the target weight quantization bit width and the target weight quantization parameter of the target network layer) to obtain an output feature of the target network layer. The target feature value quantization bit width of different target network layers can be the same or different, the target feature value quantization parameter of different target network layers can be the same or different, the target weight quantization bit width of different target network layers can all be a fixed quantization bit width, and the target weight quantization parameter of different target network layers can be the same or different.

[0248] For example, the first input feature of the target network layer is converted into the second input feature based on the target feature value quantization bit width and the target feature value quantization parameter of the target network layer, including but not limited to: the first input feature of the target network layer is converted into the second input feature by using the following formula: quant=clip(round(data×2 param ),-2 bw-1 ,2 bw-1 -1); or quant=clip(round(data÷param),-2 bw-1 ,2 bw-1 -1); wherein quant is used to represent the second input feature, data is used to represent the first input feature, param is used to represent the target feature value quantization parameter, bw is used to represent the target feature value quantization bit width, clip is used to represent a clipping function, and round is used to represent rounding.

[0249] Referring toFigure 5A As shown, the encoding end or the decoding end decodes Bitstream#1 (i.e., the first code stream) to obtain the hyperparameter quantized feature (i.e., the target feature of the current image block) after obtaining the Bitstream#1 corresponding to the current image block. The hyperparameter quantized feature is dequantized to obtain the coefficient hyperparameter feature z_hat (i.e., the first input feature of the probability hyperparameter decoding network based on the target feature). Alternatively, Bitstream#1 can be decoded to obtain the coefficient hyperparameter feature z_hat (i.e., the target feature of the current image block, which can be directly used as the first input feature of the probability hyperparameter decoding network).

[0250] After obtaining the coefficient hyperparameter feature z_hat (i.e., the first input feature of the probability hyperparameter decoding network), the coefficient hyperparameter feature z_hat is input into the probability hyperparameter decoding network (i.e., the target decoding network). The probability hyperparameter decoding network can convert the floating-point coefficient hyperparameter feature z_hat (i.e., the first input feature) into a fixed-point coefficient hyperparameter feature z_hat (i.e., the second input feature) by using the target feature value quantization bit width and the target feature value quantization hyperparameter, and perform coefficient hyperparameter feature inverse transformation on the fixed-point coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p (i.e., the output feature of the probability hyperparameter decoding network) corresponding to the current image block. After obtaining the probability distribution parameter p, the probability distribution model can be generated based on the probability distribution parameter p.

[0251] For example, for each target network layer in the probability hyperparameter decoding network, the target network layer can convert the floating-point first input feature into a fixed-point second input feature based on the target feature value quantization bit width and the target feature value quantization hyperparameter of the target network layer after obtaining the floating-point first input feature. Since the target network layer uses fixed-point weights, the fixed-point second input feature can be processed based on the fixed-point weights of the target network layer to obtain the output feature of the target network layer. The output feature of the target network layer can be used as the input feature of the next network layer, and the output feature of the target network layer can be a floating-point feature, and so on.

[0252] Referring to Figure 5AAs shown, after obtaining the Bitstream#2 (i.e., the second code stream) corresponding to the current image block, the encoding end or the decoding end decodes the Bitstream#2 to obtain the image quantization feature (i.e., the target feature of the current image block), dequantizes the image quantization feature to obtain the image feature s', and restores the image feature s' to obtain the image feature y_hat (i.e., the first input feature of the synthesis transform network determined based on the target feature). Alternatively, the Bitstream#2 can be decoded to obtain the image feature s' (i.e., the target feature of the current image block), and the image feature s' is restored to obtain the image feature y_hat (i.e., the first input feature of the synthesis transform network determined based on the target feature). When decoding the Bitstream#2, the encoding end or the decoding end can decode the Bitstream#2 based on the probability distribution model, and the process will not be described again.

[0253] After obtaining the image feature y_hat (i.e., the first input feature of the synthesis transform network), the image feature y_hat is input into the synthesis transform network (i.e., the target decoding network). The synthesis transform network can convert the floating-point image feature y_hat (i.e., the first input feature) into a fixed-point image feature y_hat (i.e., the second input feature) by using the target feature value quantization bit width and the target feature value quantization hyperparameter, and perform synthesis transform on the fixed-point image feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x (i.e., the output feature of the synthesis transform network). As can be seen from the above, the synthesis transform network can be used to perform synthesis transform on the image feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x.

[0254] For example, for each target network layer in the synthesis transform network, after obtaining the floating-point first input feature, the target network layer can convert the floating-point first input feature into a fixed-point second input feature based on the target feature value quantization bit width and the target feature value quantization hyperparameter of the target network layer. Since the target network layer uses fixed-point weights, the fixed-point second input feature can be processed based on the fixed-point weights of the target network layer to obtain the output feature of the target network layer. The output feature of the target network layer can be used as the input feature of the next network layer, and the output feature of the target network layer can be a floating-point feature, and so on.

[0255] After obtaining the output feature of the synthesis transform network, the output feature of the synthesis transform network can be used as the reconstructed image block x_hat corresponding to the current image block x, i.e., the reconstructed image block x_hat corresponding to the current image block x is determined based on the output feature.

[0256] Referring to Figure 5BAs shown, after obtaining the Bitstream#1 corresponding to the current image block, the encoding end or the decoding end can decode the Bitstream#1 to obtain the hyperparameter quantized feature (i.e., the target feature of the current image block), and dequantize the hyperparameter quantized feature to obtain the coefficient hyperparameter feature z_hat (i.e., the first input feature of the probability hyperparameter decoding network determined based on the target feature). Alternatively, the Bitstream#1 can be decoded to obtain the coefficient hyperparameter feature z_hat (i.e., the target feature of the current image block, which is directly used as the first input feature of the probability hyperparameter decoding network).

[0257] After obtaining the coefficient hyperparameter feature z_hat (i.e., the first input feature of the probability hyperparameter decoding network), the coefficient hyperparameter feature z_hat is input into the probability hyperparameter decoding network (i.e., the target decoding network). The probability hyperparameter decoding network can convert the floating-point coefficient hyperparameter feature z_hat (i.e., the first input feature) into a fixed-point coefficient hyperparameter feature z_hat (i.e., the second input feature) by using the target feature value quantization bit width and the target feature value quantization hyperparameter, and perform coefficient hyperparameter feature inverse transformation on the fixed-point coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p (i.e., the output feature of the probability hyperparameter decoding network) corresponding to the current image block.

[0258] For example, for each target network layer in the probability hyperparameter decoding network, after obtaining the floating-point first input feature, the target network layer converts the floating-point first input feature into a fixed-point second input feature based on the target feature value quantization bit width and the target feature value quantization hyperparameter of the target network layer. The target network layer uses fixed-point weights, and processes the fixed-point second input feature based on the fixed-point weights of the target network layer to obtain the output feature of the target network layer.

[0259] After obtaining the coefficient hyperparameter feature z_hat (i.e., the first input feature of the mean prediction network), the coefficient hyperparameter feature z_hat and the image feature y_hat of the previous image block are input into the mean prediction network (i.e., the target decoding network). The mean prediction network converts the floating-point coefficient hyperparameter feature z_hat (the first input feature) into a fixed-point coefficient hyperparameter feature z_hat (the second input feature) by using the target feature value quantization bit width and the target feature value quantization hyperparameter, and converts the floating-point image feature y_hat (the first input feature) into a fixed-point image feature y_hat (the second input feature) by using the target feature value quantization bit width and the target feature value quantization hyperparameter. The mean prediction network performs context-based prediction based on the fixed-point coefficient hyperparameter feature z_hat and the fixed-point image feature y_hat to obtain the prediction value mu (i.e., the output feature of the mean prediction network) corresponding to the current image block.

[0260] For example, for each target network layer in the mean prediction network, after obtaining the first input feature of the floating point type, the target network layer converts the first input feature of the floating point type into the second input feature of the fixed point type based on the target feature value quantization bit width and the target feature value quantization hyperparameter of the target network layer. The target network layer adopts the fixed point type weight, and processes the second input feature of the fixed point type based on the fixed point type weight of the target network layer to obtain the output feature of the target network layer.

[0261] After the encoding end or the decoding end obtains the Bitstream#2 corresponding to the current image block, the Bitstream#2 is decoded to obtain the image quantized feature (i.e., the target feature of the current image block), the image quantized feature is dequantized to obtain the image feature s', the image feature s' is feature restored (i.e., residual error is restored) to obtain the residual error feature r_hat, and the image feature y_hat is determined based on the residual error feature r_hat and the prediction value mu, such as the sum of the residual error feature r_hat and the prediction value mu as the image feature y_hat (i.e., the first input feature of the synthesis transform network is determined based on the target feature). Or, the Bitstream#2 is decoded to obtain the image feature s' (i.e., the target feature of the current image block), the image feature s' is feature restored to obtain the residual error feature r_hat, and the image feature y_hat is determined based on the residual error feature r_hat and the prediction value mu, such as the sum of the residual error feature r_hat and the prediction value mu as the image feature y_hat (i.e., the first input feature of the synthesis transform network is determined based on the target feature).

[0262] After obtaining the image feature y_hat (i.e., the first input feature of the synthesis transform network), the image feature y_hat is input to the synthesis transform network (i.e., the target decoding network). The synthesis transform network can convert the image feature y_hat of the floating point type (i.e., the first input feature) into the image feature y_hat of the fixed point type (i.e., the second input feature) based on the target feature value quantization bit width and the target feature value quantization hyperparameter, and perform synthesis transform on the image feature y_hat of the fixed point type to obtain the reconstructed image block x_hat corresponding to the current image block x (i.e., the output feature of the synthesis transform network). As can be seen from the above, the image feature y_hat can be synthesized and transformed based on the synthesis transform network to obtain the reconstructed image block x_hat corresponding to the current image block x.

[0263] For example, for each target network layer in the synthesis conversion network, after obtaining the first input feature of the floating point type, the target network layer can convert the first input feature of the floating point type into the second input feature of the fixed point type based on the target feature value quantization bit width and the target feature value quantization hyperparameter of the target network layer. Since the target network layer uses fixed point weights, the fixed point second input feature can be processed based on the fixed point weights of the target network layer to obtain the output feature of the target network layer. The output feature of the target network layer can be used as the input feature of the next network layer, and the output feature of the target network layer can be a feature of the floating point type, and so on.

[0264] After obtaining the output feature of the synthesis conversion network, the output feature of the synthesis conversion network can be used as the reconstructed image block x_hat corresponding to the current image block x, that is, the output feature is used to determine the reconstructed image block x_hat corresponding to the current image block x.

[0265] As can be seen from the above technical solutions, in the embodiments of the present application, an end-to-end video image compression method is proposed, which can realize video image decoding based on a decoding network to improve the encoding efficiency and decoding efficiency. The decoding network with fixed point weights is constructed by using the target weight quantization bit width and the target weight quantization hyperparameter, and the input feature of the fixed point type is generated by using the target feature value quantization bit width and the target feature value quantization hyperparameter, so as to realize adaptive decoding acceleration, reduce the decoding calculation amount, and ensure the decoding quality. A compression method for the decoding network in the image coding and decoding framework based on the neural network is proposed, the quantization bit width and the quantization hyperparameter of each layer are determined by analyzing the quantization loss of each layer when the decoding network decodes the input code stream, and the decoding network is quantized according to the obtained quantization bit width and quantization hyperparameter. The storage space, calculation amount, model throughput and bandwidth of the decoding network can be greatly compressed while ensuring the decoding quality of the decoding network, so that the huge decoding network can run on devices with limited computing resources. By changing the floating point calculation of the decoding network to fixed point calculation, the decoding consistency of the decoding network between different devices can be improved.

[0266] For example, each of the above embodiments can be implemented alone or in combination, for example, each of embodiments 1-9 can be implemented alone, and at least two of embodiments 1-9 can be implemented in combination.

[0267] For example, in each of the above embodiments, the content of the encoding end can also be applied to the decoding end, that is, the decoding end can be processed in the same way, and the content of the decoding end can also be applied to the encoding end, that is, the encoding end can be processed in the same way.

[0268] Based on the same application concept as the above method, the embodiment of the present application also proposes a decoding device. The device is applied to a decoding end. The device comprises a memory configured to store video data; and a decoder configured to implement the decoding method in the above embodiments 1-9, i.e., the processing flow of the decoding end.

[0269] For example, in a possible implementation, the decoder is configured to implement:

[0270] decode the target feature corresponding to the current image block from the code stream corresponding to the current image block;

[0271] determine the first input feature of the target decoding network based on the target feature;

[0272] obtain the target feature value quantization bit width and the target feature value quantization hyperparameter of the target decoding network, and convert the first input feature into a second input feature based on the target feature value quantization bit width and the target feature value quantization hyperparameter;

[0273] process the second input feature based on the fixed-point type weight of the target decoding network to obtain the output feature of the target decoding network; wherein the fixed-point type weight is determined based on the target weight quantization bit width and the target weight quantization hyperparameter;

[0274] determine the reconstructed image block corresponding to the current image block based on the output feature of the target decoding network.

[0275] Based on the same application concept as the above method, the decoding end device (which can also be referred to as a video decoder) provided by the embodiment of the present application can be specifically referred to from the hardware layer as shown in the hardware architecture diagram of the decoding end device. Figure 7 The decoding end device comprises a processor 701 and a machine readable storage medium 702. The machine readable storage medium 702 stores machine executable instructions that can be executed by the processor 701. The processor 701 is configured to execute the machine executable instructions to implement the decoding method in the above embodiments 1-9 of the present application.

[0276] For example, in a possible implementation, the decoding end device is configured to implement:

[0277] decode the target feature corresponding to the current image block from the code stream corresponding to the current image block;

[0278] determine the first input feature of the target decoding network based on the target feature;

[0279] obtain the target feature value quantization bit width and the target feature value quantization hyperparameter of the target decoding network, and convert the first input feature into a second input feature based on the target feature value quantization bit width and the target feature value quantization hyperparameter;

[0280] The fixed-point type weight of the target decoding network processes the second input feature to obtain an output feature of the target decoding network; wherein the fixed-point type weight is determined based on a target weight quantization bit width and a target weight quantization hyperparameter;

[0281] The output feature of the target decoding network is used to determine a reconstructed image block corresponding to the current image block.

[0282] Based on the same application concept as the above method, an electronic device is provided. It includes a processor and a machine readable storage medium, the machine readable storage medium stores machine executable instructions that can be executed by the processor; the processor is used to execute the machine executable instructions to implement the decoding method of embodiments 1-9 of the present application.

[0283] Based on the same application concept as the above method, the present application embodiment further provides a machine readable storage medium, the machine readable storage medium stores a plurality of computer instructions, the computer instructions are executed by the processor, and the decoding method disclosed in the above examples can be realized.

[0284] Based on the same application concept as the above method, the present application embodiment further provides a computer application program, which can realize the decoding method disclosed in the above examples when executed by the processor.

[0285] In the present application embodiment, a decoding device is also proposed, which can be applied to a decoding end. The decoding device can include: a decoding module, configured to decode a target feature corresponding to a current image block from a code stream corresponding to the current image block; a determination module, configured to determine a first input feature of a target decoding network based on the target feature; a processing module, configured to obtain a target feature value quantization bit width and a target feature value quantization hyperparameter of the target decoding network, convert the first input feature into a second input feature based on the target feature value quantization bit width and the target feature value quantization hyperparameter, and process the second input feature based on a fixed-point type weight of the target decoding network to obtain an output feature of the target decoding network; wherein the fixed-point type weight is determined based on a target weight quantization bit width and a target weight quantization hyperparameter; and the determination module, configured to determine a reconstructed image block corresponding to the current image block based on the output feature of the target decoding network.

[0286] Exemplarily, the target decoding network comprises at least one target network layer, the target network layer is a network layer with fixed-point type weights, and for each target network layer in the target decoding network: the processing module is further configured to convert the first input feature of the target network layer into a second input feature based on a target feature value quantization bit width and a target feature value quantization hyperparameter of the target network layer; process the second input feature based on the fixed-point type weights of the target network layer to obtain an output feature of the target network layer; wherein the fixed-point type weights of the target network layer are determined based on a target weight quantization bit width and a target weight quantization hyperparameter of the target network layer; wherein the target feature value quantization bit widths of different target network layers are the same or different, the target feature value quantization hyperparameters of different target network layers are the same or different, the target weight quantization bit widths of different target network layers are all fixed quantization bit widths, and the target weight quantization hyperparameters of different target network layers are the same or different.

[0287] Exemplarily, when the processing module converts the first input feature of the target network layer into the second input feature based on the target feature value quantization bit width and the target feature value quantization hyperparameter of the target network layer, the processing module is specifically configured to convert the first input feature into the second input feature by using the following formula: quant = clip(round(data × 2 param ), -2 bw-1 , 2 bw-1 -1); or quant = clip(round(data ÷ param), -2 bw-1 , 2 bw-1 -1); wherein quant represents the second input feature, data represents the first input feature, param represents the target feature value quantization hyperparameter, bw represents the target feature value quantization bit width, clip represents a clipping function, and round represents rounding.

[0288] Exemplarily, the sample decoding network comprises a plurality of network layers adopting floating-point weights, the plurality of network layers comprising a target network layer, for each target network layer: the processing module is further configured to acquire a weight quantization hyperparameter candidate list corresponding to the target network layer, the weight quantization hyperparameter candidate list comprising a plurality of candidate weight quantization hyperparameters; for each candidate weight quantization hyperparameter, the candidate weight quantization hyperparameter is used to simulate quantization of the floating-point weight of the target network layer to obtain a candidate simulated weight by using a fixed quantization bit width; the floating-point input feature of the target network layer is processed based on the candidate simulated weight to obtain a floating-point output feature of the target network layer, and a quantization error corresponding to the candidate weight quantization hyperparameter is determined based on the floating-point output feature of the target network layer; the candidate weight quantization hyperparameter corresponding to the minimum quantization error is determined as a target weight quantization hyperparameter, and the fixed quantization bit width is determined as a target weight quantization bit width, based on the quantization error corresponding to each candidate weight quantization hyperparameter; the floating-point weight of the target network layer is fixed-point quantized based on the target weight quantization bit width and the target weight quantization hyperparameter to obtain a fixed-point weight of the target network layer; and the target decoding network is generated based on the fixed-point weight of each target network layer.

[0289] Exemplarily, when the processing module acquires the weight quantization hyperparameter candidate list corresponding to the target network layer, the processing module is specifically configured to: determine a maximum weight value based on all floating-point weights of the target network layer; each floating-point weight comprises a plurality of weight values, and the maximum weight value is the maximum value in all weight values of all floating-point weights; generate an initial weight quantization hyperparameter corresponding to the target network layer based on the maximum weight value; and construct the weight quantization hyperparameter candidate list based on the initial weight quantization hyperparameter.

[0290] Exemplarily, when the processing module generates the initial weight quantization hyperparameter corresponding to the target network layer based on the maximum weight value, the processing module is specifically configured to generate the initial weight quantization hyperparameter by using the following formula: param=bw-1-ceil(log2(max));or, param=(max) / 2 bw-1 ; wherein, param represents the initial weight quantization hyperparameter corresponding to the target network layer, bw represents the fixed quantization bit width, max represents the maximum weight value, and ceil represents a rounding-up operation.

[0291] Exemplarily, when the processing module simulates quantization of the floating-point weight of the target network layer to obtain the candidate simulated weight by using the candidate weight quantization hyperparameter and the fixed quantization bit width, the processing module is specifically configured to simulate quantization of the floating-point weight to obtain the candidate simulated weight by using the following formula: quant=2 -param ×clip(round(data×2 param ),-2 bw-1 ,2 bw -1-1); or, quant = param x clip(round(data ÷ param), -2 bw-1 , 2 bw-1 -1); wherein, quant represents the candidate weight quantization, param represents the candidate weight quantization parameter, data represents the floating-point weight of the target network layer, bw represents the fixed quantization bit width, clip represents the clipping function, and round represents rounding.

[0292] For example, the processing module determines the quantization error corresponding to the candidate weight quantization parameter based on the floating-point output feature of the target network layer, specifically by: determining a sample output feature based on the floating-point output feature of the target network layer, the sample output feature being an output feature of a reference network layer in a sample decoding network, the reference network layer being any network layer located after the target network layer; determining the quantization error corresponding to the candidate weight quantization parameter based on the sample output feature and a reference output feature; wherein the reference output feature is obtained by: decoding a sample feature from a sample bitstream, determining a floating-point input feature corresponding to the sample decoding network based on the sample feature, and determining the reference output feature based on the floating-point input feature, the reference output feature being an output feature of the reference network layer in the sample decoding network.

[0293] For example, the sample decoding network includes a target network layer using floating-point weights, and for each target network layer: the processing module is further configured to obtain a feature value quantization hyperparameter candidate list and a quantization bit width set corresponding to the target network layer, the feature value quantization hyperparameter candidate list includes a plurality of candidate feature value quantization hyperparameters, and the quantization bit width set includes at least two quantization bit widths; traverse a current quantization bit width from the quantization bit width set, for each candidate feature value quantization hyperparameter, simulate and quantize floating-point input features of the target network layer using the current quantization bit width and the candidate feature value quantization hyperparameter to obtain candidate input features; process the candidate input features based on pseudo weights of the target network layer to obtain floating-point output features of the target network layer, and determine a quantization error corresponding to the candidate feature value quantization hyperparameter based on the floating-point output features of the target network layer; based on the quantization error corresponding to each candidate feature value quantization hyperparameter, if a minimum quantization error is less than a preset threshold, the candidate feature value quantization hyperparameter corresponding to the minimum quantization error is determined as a target feature value quantization hyperparameter, the current quantization bit width is determined as a target feature value quantization bit width, and the target feature value quantization hyperparameter and the target feature value quantization bit width are recorded for the target network layer. Based on the quantization error corresponding to each candidate feature value quantization hyperparameter, if the minimum quantization error is not less than the preset threshold, it is determined whether the current quantization bit width is the last quantization bit width in the quantization bit width set; if yes, the candidate feature value quantization hyperparameter corresponding to the minimum quantization error is determined as the target feature value quantization hyperparameter, the current quantization bit width is determined as the target feature value quantization bit width, and the target feature value quantization hyperparameter and the target feature value quantization bit width are recorded for the target network layer; if no, a next quantization bit width of the current quantization bit width is traversed as the current quantization bit width, and the operation of simulating and quantizing the floating-point input features of the target network layer using the current quantization bit width and the candidate feature value quantization hyperparameter to obtain the candidate input features is performed.

[0294] For example, when the processing module obtains the feature value quantization hyperparameter candidate list corresponding to the target network layer, the processing module is specifically configured to: determine a maximum feature value based on all floating-point input features of the target network layer; each floating-point input feature includes a plurality of feature values, and the maximum feature value is the maximum value among all feature values of all floating-point input features; generate an initial feature value quantization hyperparameter corresponding to the target network layer based on the maximum feature value; and construct the feature value quantization hyperparameter candidate list based on the initial feature value quantization hyperparameter. When the processing module generates the initial feature value quantization hyperparameter corresponding to the target network layer based on the maximum feature value, the processing module is specifically configured to: generate the initial feature value quantization hyperparameter using the following formula: get_param=bw-1-ceil(log2(max));or, get_param=(max) / 2 bw-1; wherein get_param represents an initial eigenvalue quantization parameter corresponding to the target network layer, bw represents the current quantization bit width, max represents the maximum eigenvalue, and ceil represents an upward rounding operation.

[0295] The processing module is specifically configured to simulate and quantize the floating-point input feature of the target network layer to obtain the candidate input feature using the current quantization bit width and the candidate eigenvalue quantization parameter. -param param bw-1 bw -1 bw-1 bw-1 The processing module is specifically configured to simulate and quantize the floating-point input feature of the target network layer to obtain the candidate input feature using the current quantization bit width and the candidate eigenvalue quantization parameter.

[0296] The processing module is specifically configured to simulate and quantize the floating-point input feature of the target network layer to obtain the candidate input feature using the current quantization bit width and the candidate eigenvalue quantization parameter.

[0297] The processing module is specifically configured to simulate and quantize the floating-point input feature of the target network layer to obtain the candidate input feature using the current quantization bit width and the candidate eigenvalue quantization parameter.

[0298] ​​​​​Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Embodiments of the application can adopt the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Embodiments of the application can adopt the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk memory, CD-ROMs, optical storage media, etc.) having computer-usable program code embodied thereon. The above description is only for the embodiments of the application and is not intended to limit the application.

[0299] Various modifications and changes can be made as would be obvious to a person skilled in the art having the benefit of this disclosure. It is intended that the application embrace all such modifications and changes and, accordingly, there is included within this application's scope all equivalents that are within the spirit and scope of the application.

Claims

1. A decoding method, comprising: The method comprises: decoding a target feature corresponding to a current image block from a bitstream corresponding to the current image block; determining a first input feature of a target decoding network based on the target feature; obtaining a target feature value quantization bit width and a target feature value quantization parameter of the target decoding network, and converting the first input feature into a second input feature based on the target feature value quantization bit width and the target feature value quantization parameter; processing the second input feature based on fixed-point type weights of the target decoding network to obtain an output feature of the target decoding network; wherein the fixed-point type weights are determined based on a target weight quantization bit width and a target weight quantization parameter; determining a reconstructed image block corresponding to the current image block based on the output feature of the target decoding network.

2. The method of claim 1, wherein, The target decoding network comprises at least one target network layer, and the target network layer is a network layer using fixed-point type weights. For each target network layer in the target decoding network: convert a first input feature of the target network layer into a second input feature based on a target feature value quantization bit width and a target feature value quantization parameter of the target network layer; process the second input feature based on fixed-point type weights of the target network layer to obtain an output feature of the target network layer; wherein the fixed-point type weights of the target network layer are determined based on a target weight quantization bit width and a target weight quantization parameter of the target network layer; wherein the target feature value quantization bit widths of different target network layers are the same or different, the target feature value quantization parameters of different target network layers are the same or different, the target weight quantization bit widths of different target network layers are all fixed quantization bit widths, and the target weight quantization parameters of different target network layers are the same or different.

3. The method of claim 2, wherein, The conversion of the first input feature of the target network layer into the second input feature based on the target feature value quantization bit width and the target feature value quantization parameter of the target network layer comprises: convert the first input feature of the target network layer into the second input feature using the following formula: quant = clip(round(data x 2 param ), -2 bw-1 , 2 bw-1 - 1); or, quant = clip( round( data ÷ param ), -2 bw-1 , 2 bw-1 - 1 ); wherein quant represents the second input feature, data represents the first input feature, param represents the target feature value quantization parameter, bw represents the target feature value quantization bit width, clip represents a clipping function, and round represents rounding.

4. The method according to any one of claims 1 to 3, characterized in that, The sample decoding network comprises a plurality of network layers using floating-point type weights, and the plurality of network layers comprise target network layers. For each target network layer: obtain a weight quantization parameter candidate list corresponding to the target network layer, wherein the weight quantization parameter candidate list comprises a plurality of candidate weight quantization parameters; for each candidate weight quantization parameter, simulate quantize the floating-point type weights of the target network layer using a fixed quantization bit width and the candidate weight quantization parameter to obtain candidate simulated weights; process floating-point type input features of the target network layer based on the candidate simulated weights to obtain floating-point type output features of the target network layer, and determine a quantization error corresponding to the candidate weight quantization parameter based on the floating-point type output features of the target network layer. determining the candidate weight quantization parameter corresponding to the minimum quantization error as the target weight quantization parameter, and determining the fixed quantization bit width as the target weight quantization bit width based on each candidate weight quantization parameter and a quantization error corresponding to the candidate weight quantization parameter; performing fixed-point quantization on the floating-point weight of the target network layer based on the target weight quantization bit width and the target weight quantization parameter to obtain a fixed-point weight of the target network layer; generating a target decoding network based on the fixed-point weight of each target network layer.

5. The method of claim 4, wherein the obtaining a candidate list of weight quantization parameters corresponding to the target network layer comprises: determining a maximum weight value based on all floating-point weights of the target network layer, wherein each floating-point weight comprises a plurality of weight values, and the maximum weight value is a maximum value among all weight values of all floating-point weights; generating an initial weight quantization parameter corresponding to the target network layer based on the maximum weight value; and constructing the candidate list of weight quantization parameters based on the initial weight quantization parameter.

6. The method of claim 5, wherein the generating an initial weight quantization parameter corresponding to the target network layer based on the maximum weight value comprises: generating the initial weight quantization parameter using the following formula: param = bw - 1 - ceil (log2(max)) ; or wherein param represents the initial weight quantization parameter corresponding to the target network layer, bw represents the fixed quantization bit width, max represents the maximum weight value, and ceil represents a rounding-up operation. The simulating quantization of the floating-point weight of the target network layer using the fixed quantization bit width and the candidate weight quantization parameter to obtain a candidate simulation weight comprises: param = (max) / 2 bw-1 ; simulating quantization of the floating-point weight of the target network layer using the following formula to obtain a candidate simulation weight:

7. The method of claim 4, wherein, wherein quant represents the candidate simulation weight, param represents the candidate weight quantization parameter, data represents the floating-point weight of the target network layer, bw represents the fixed quantization bit width, clip represents a clipping function, and round represents rounding. The determining a quantization error corresponding to the candidate weight quantization parameter based on the floating-point output feature of the target network layer comprises: quant = 2 -param × clip(round(data × 2 param ), -2 bw-1 , 2 bw-1 - 1); or, quant = param x clip(round(data ÷ param), -2 bw-1 , 2 bw-1 - 1); determining a sample output feature based on the floating-point output feature of the target network layer, wherein the sample output feature is an output feature of a reference network layer in a sample decoding network, and the reference network layer is any network layer located after the target network layer; 8. The method of claim 4, wherein, determining a quantization error corresponding to the candidate weight quantization parameter based on the sample output feature and a reference output feature; wherein the reference output feature is obtained by: decoding a sample feature from a sample bitstream, determining a floating-point input feature corresponding to the sample decoding network based on the sample feature, and determining the reference output feature based on the floating-point input feature, wherein the reference output feature is an output feature of the reference network layer in the sample decoding network. The sample decoding network comprises a plurality of network layers using floating-point weights, and the plurality of network layers comprises target network layers, and for each target network layer: ​ 9. The method according to any one of claims 1 to 3, characterized in that, ​ obtain a feature value quantization hyperparameter candidate list and a quantization bit width set corresponding to the target network layer, the feature value quantization hyperparameter candidate list including a plurality of candidate feature value quantization hyperparameters, and the quantization bit width set including at least two quantization bit widths; simulate and quantize, for each candidate feature value quantization hyperparameter, a floating-point input feature of the target network layer to obtain a candidate input feature by using a current quantization bit width and the candidate feature value quantization hyperparameter; process the candidate input feature based on pseudo weights of the target network layer to obtain a floating-point output feature of the target network layer, and determine a quantization error corresponding to the candidate feature value quantization hyperparameter based on the floating-point output feature of the target network layer; if the minimum quantization error is smaller than a preset threshold, determine the candidate feature value quantization hyperparameter corresponding to the minimum quantization error as a target feature value quantization hyperparameter, determine the current quantization bit width as a target feature value quantization bit width, and record the target feature value quantization hyperparameter and the target feature value quantization bit width for the target network layer.

10. The method of claim 9, wherein, The method further includes: if the minimum quantization error is not smaller than the preset threshold, determine whether the current quantization bit width is the last quantization bit width in the quantization bit width set; if yes, determine the candidate feature value quantization hyperparameter corresponding to the minimum quantization error as the target feature value quantization hyperparameter, determine the current quantization bit width as the target feature value quantization bit width, and record the target feature value quantization hyperparameter and the target feature value quantization bit width for the target network layer; if no, traverse a next quantization bit width of the current quantization bit width as the current quantization bit width, and return to perform the operation of simulating and quantizing the floating-point input feature of the target network layer to obtain the candidate input feature by using the current quantization bit width and the candidate feature value quantization hyperparameter.

11. The method of claim 9, wherein the obtaining of the feature value quantization hyperparameter candidate list corresponding to the target network layer includes: determining a maximum feature value based on all floating-point input features of the target network layer, wherein each floating-point input feature includes a plurality of feature values, and the maximum feature value is a maximum value among all feature values of all floating-point input features; generating an initial feature value quantization hyperparameter corresponding to the target network layer based on the maximum feature value; constructing the feature value quantization hyperparameter candidate list based on the initial feature value quantization hyperparameter.

12. The method of claim 11, wherein the generating of the initial feature value quantization hyperparameter corresponding to the target network layer based on the maximum feature value includes: generating the initial feature value quantization hyperparameter by using the following formula: param = bw - 1 - ceil (log2 (max)) ; or, param = (max) / 2 bw-1 ; wherein param represents the initial feature value quantization hyperparameter corresponding to the target network layer, bw represents the current quantization bit width, max represents the maximum feature value, and ceil represents a rounding-up operation.

13. The method of claim 9, wherein, The candidate input feature is obtained by simulating quantization of the floating-point input feature of the target network layer using the current quantization bit width and the candidate feature value quantization parameter, and the simulating quantization includes: The candidate input feature is obtained by simulating quantization of the floating-point input feature of the target network layer using the following formula: quant = 2 -param × clip(round9data × 2 param ), -2 bw-1 , 2 bw-1 -1); or, quant = param x clip(round(data ÷ param), -2 bw-1 , 2 bw-1 - 1); Wherein, quant represents the candidate input feature, param represents the candidate feature value quantization parameter, data represents the floating-point input feature, bw represents the current quantization bit width, clip represents a clipping function, and round represents rounding.

14. The method of claim 9, wherein, Before the processing of the candidate input feature based on the pseudo weight of the target network layer to obtain the floating-point output feature of the target network layer, the method further includes: The pseudo weight of the target network layer is obtained by simulating quantization of the floating-point weight of the target network layer using the target weight quantization parameter of the target network layer and the target weight quantization bit width of the target network layer.

15. The method of claim 9, wherein, The quantization error corresponding to the candidate feature value quantization parameter is determined based on the floating-point output feature of the target network layer, and the method includes: The sample output feature is determined based on the floating-point output feature of the target network layer, the sample output feature being an output feature of a reference network layer in a sample decoding network, the reference network layer being any network layer located behind the target network layer; The quantization error corresponding to the candidate feature value quantization parameter is determined based on the sample output feature and a reference output feature; Wherein, the reference output feature is obtained in the following manner: decoding a sample feature from a sample bit stream, determining a floating-point input feature corresponding to the sample decoding network based on the sample feature, and determining the reference output feature based on the floating-point input feature, the reference output feature being an output feature of a reference network layer in the sample decoding network.

16. A decoding apparatus, comprising: The device includes: A decoding module configured to decode a target feature corresponding to a current image block from a bit stream corresponding to the current image block; A determining module configured to determine a first input feature of a target decoding network based on the target feature; A processing module configured to obtain a target feature value quantization bit width and a target feature value quantization parameter of the target decoding network, convert the first input feature into a second input feature based on the target feature value quantization bit width and the target feature value quantization parameter, and process the second input feature based on a fixed-point weight of the target decoding network to obtain an output feature of the target decoding network, wherein the fixed-point weight is determined based on a target weight quantization bit width and a target weight quantization parameter; The determining module is configured to determine a reconstructed image block corresponding to the current image block based on the output feature of the target decoding network.

17. A decoding-side device, comprising: It includes: A processor and a machine readable storage medium, the machine readable storage medium stores machine executable instructions that can be executed by the processor; The processor is configured to execute the machine executable instructions to implement the method of any one of claims 1-15.

18. A machine-readable storage medium, characterized in that, The machine readable storage medium stores a plurality of computer instructions, and the computer instructions are executed by the processor to implement the method of any one of claims 1-15.

Images