Decoding network model, quantization method of decoding network model and related device

Abstract

The invention discloses a decoding network model, a quantization method of the decoding network model and a related device, and belongs to the technical field of coding and decoding. The decoding network model comprises a target network, the target network is a quantized network, and the target network comprises a super decoding network and / or a synthesis transformation network; under the condition that the target network comprises the super-decoding network, the quantization precision of each network layer in the super-decoding network is the same; under the condition that the target network comprises the synthetic transformation network, the quantization precision of at least part of network layers in the synthetic transformation network is the same. By quantifying the target network, the model volume of the decoding network model is greatly reduced, so that the memory consumption in the operation process of the decoding network model is reduced, the operation speed of the decoding network model on hardware is improved, and the decoding network model can be deployed on end-side equipment with limited resources, such as a mobile phone and a robot.

Description

[0001] This application claims priority to Chinese patent application No. 202310852069.6, filed on July 11, 2023, and entitled “Decoding network model quantization method and encoding network model quantization method”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of coding and decoding technology, and in particular to a decoding network model, a quantization method of the decoding network model and related devices. Background Art

[0003] With the widespread application of deep learning technology in image recognition, target detection and other fields, deep learning technology has also been applied to image compression tasks, that is, using codec network models for image compression. Although using codec network models for image compression can greatly improve codec performance and image compression effects, it also introduces a huge amount of parameters and calculations, which leads to various problems when the codec network model runs on resource-limited end-side devices. For example, when the codec network model runs on a low-performance mobile device or a low-power embedded device, the efficiency of model reasoning will be reduced. For another example, some end-side devices do not support floating-point calculations of the codec network model, which limits the deployment of the codec network model on the end-side devices. Therefore, it is necessary to quantize the codec network model to reduce the model calculation amount, parameter size and memory consumption, so that the quantized codec network model can be deployed on resource-limited end-side devices such as mobile phones and robots. Summary of the invention

[0004] The present application provides a decoding network model, a decoding network model quantization method and related devices, which reduces the model volume of the decoding network model and the amount of model calculation by quantizing the target network in the decoding network model. The technical solution is as follows:

[0005] In a first aspect, a decoding network model is provided, wherein the decoding network model includes a target network, wherein the target network is a quantized network, and wherein the target network includes a super-decoding network and / or a synthetic transformation network; when the target network includes the super-decoding network, the quantization accuracy of each network layer in the super-decoding network is the same; when the target network includes the synthetic transformation network, the quantization accuracy of at least some network layers in the synthetic transformation network is the same.

[0006] Since the target network in the decoding network model is a quantized network, the target network is smaller and has a higher data processing rate. By quantizing the target network, the size of the decoding network model is greatly reduced, thereby reducing the memory consumption during the operation of the decoding network model and improving the operation speed of the decoding network model on hardware, so that the decoding network model can be deployed on end-side devices with limited resources such as mobile phones and robots.

[0007] Optionally, the quantization accuracy includes weight quantization accuracy and / or activation quantization accuracy; the super-decoding network includes multiple network layers, the weight quantization accuracy of each network layer in the super-decoding network is the same and less than or equal to M bits, the activation quantization accuracy of each network layer in the super-decoding network is the same and less than or equal to N bits, the M is less than the N, and the M and the N are both less than the accuracy of the super-decoding network before quantization; the synthetic transformation network includes multiple network layers, the weight quantization accuracy of each network layer in the synthetic transformation network is the same and less than or equal to P bits, the activation quantization accuracy of each network layer in the synthetic transformation network is the same and less than or equal to Q bits, the P is less than the Q, and the P and the Q are both less than the accuracy of the synthetic transformation network before quantization.

[0008] That is, for each network layer in the target network, quantization is performed from two dimensions: weight and activation, so that the target network after quantization is smaller in size and has a higher data processing rate. Moreover, for each network layer in the target network, a unified quantization accuracy is used to quantize the weights of each network layer, and a unified quantization accuracy is used to quantize the activations of each network layer, so that the quantization degree of each network layer is the same, thereby ensuring the overall quantization performance of the target network.

[0009] Optionally, the synthetic transformation network includes a residual module, a first deconvolution layer, a first cropping layer, a first residual activation unit, a second deconvolution layer, a second cropping layer, a second residual activation unit, a first convolution layer, a third residual activation unit, a second convolution layer, a pixel shifting layer and a third cropping layer connected in sequence; the super decoding network includes a third deconvolution layer, a fourth cropping layer, a first activation layer, a third convolution layer, a fourth deconvolution layer, a fifth cropping layer, a second activation layer, a fourth convolution layer and a third activation layer connected in sequence.

[0010] Optionally, the residual module includes a fifth convolutional layer and a fourth activation layer connected in sequence, and the sum of the input tensor of the fifth convolutional layer and the output tensor of the fourth activation layer is the output of the residual module.

[0011] Optionally, the first residual activation unit includes a fifth activation layer and a sixth convolutional layer connected in sequence, and the product of the input tensor of the fifth activation layer and the output tensor of the sixth convolutional layer is the output of the first residual activation unit; the second residual activation unit includes a sixth activation layer and a seventh convolutional layer connected in sequence, and the product of the input tensor of the sixth activation layer and the output tensor of the seventh convolutional layer is the output of the second residual activation unit; the third residual activation unit includes a seventh activation layer and an eighth convolutional layer connected in sequence, and the product of the input tensor of the seventh activation layer and the output tensor of the eighth convolutional layer is the output of the third residual activation unit.

[0012] It should be noted that in the hierarchical structure of the target network, "connected in sequence" refers to a logical connection, not a physical connection. In other words, "connected in sequence" means that the output of the previous network layer is the input of the next network layer.

[0013] Optionally, the quantization accuracy includes weight quantization accuracy and / or activation quantization accuracy; the super-decoding network includes multiple network layers, the weight quantization accuracy of each network layer in the super-decoding network is the same and less than or equal to M bits, the activation quantization accuracy of each network layer in the super-decoding network is the same and less than or equal to N bits, M is less than N, and both M and N are less than the accuracy of the super-decoding network before quantization; the synthetic transformation network includes multiple network layers, the weight quantization accuracy of each network layer in the synthetic transformation network is the same and less than or equal to P bits, the activation quantization accuracy of at least one network layer in the synthetic transformation network is the same and less than or equal to P bits, the activation quantization accuracy of other network layers in the synthetic transformation network except the at least one network layer is the same and less than or equal to Q bits, P is less than Q, and both P and Q are less than the accuracy of the synthetic transformation network before quantization.

[0014] That is, for each network layer in the target network, quantization is performed from the two dimensions of weight and activation, so that the target network after quantization is smaller in size and has a higher data processing rate. Moreover, for each network layer in the target network, the corresponding weight quantization accuracy and / or activation quantization accuracy are set according to the characteristics of each network layer, and each network layer in the target network is quantized with mixed quantization accuracy, making the quantization of the target network more personalized.

[0015] Optionally, the synthetic transformation network includes a residual module, a first deconvolution layer, a first cropping layer, a first residual activation unit, a second deconvolution layer, a second cropping layer, a second residual activation unit, a first convolution layer, a third residual activation unit, a second convolution layer, a pixel shifting layer and a third cropping layer connected in sequence; the super decoding network includes a third deconvolution layer, a fourth cropping layer, a first activation layer, a third convolution layer, a fourth deconvolution layer, a fifth cropping layer, a second activation layer, a fourth convolution layer and a third activation layer connected in sequence.

[0016] Optionally, the residual module includes a fifth convolutional layer and a fourth activation layer connected in sequence, and the sum of the input tensor of the fifth convolutional layer and the output tensor of the fourth activation layer is the output of the residual module.

[0017] Optionally, the first residual activation unit includes a fifth activation layer and a sixth convolutional layer connected in sequence, and the product of the input tensor of the fifth activation layer and the output tensor of the sixth convolutional layer is the output of the first residual activation unit; the second residual activation unit includes a sixth activation layer and a seventh convolutional layer connected in sequence, and the product of the input tensor of the sixth activation layer and the output tensor of the seventh convolutional layer is the output of the second residual activation unit; the third residual activation unit includes a seventh activation layer and an eighth convolutional layer connected in sequence, and the product of the input tensor of the seventh activation layer and the output tensor of the eighth convolutional layer is the output of the third residual activation unit.

[0018] Optionally, at least one network layer in the synthetic transformation network whose activation quantization accuracy is the same and less than or equal to P bits includes but is not limited to: the second deconvolution layer and the second residual activation unit; or, the first convolution layer and the third residual activation unit; or, the second deconvolution layer, the second residual activation unit, the first convolution layer and the third residual activation unit; or, the first deconvolution layer, the first residual activation unit, the second deconvolution layer, the second residual activation unit, the first convolution layer and the third residual activation unit.

[0019] Since quantization is to reduce the size of the model while increasing its data processing rate, when mixed precision is used to quantize each network layer in the target network, for the residual activation units of non-minimum hierarchical structures, a smaller number of bits can be used to quantize their activations, which can minimize the size of the target network after quantization.

[0020] Optionally, the values ​​of M and P are both 8, and the values ​​of N and Q are both 16.

[0021] Optionally, the decoding network model includes the synthetic transform network, the super decoding network and the super variance decoding network, and the super decoding network and the super variance decoding network are both connected to the synthetic transform network.

[0022] That is, in addition to the above-mentioned target network, the hypervariance decoding network can also be quantized to further reduce the size of the decoding network model so that it can be deployed on end-side devices with limited resources such as mobile phones and robots.

[0023] Optionally, the decoding network model includes two synthetic transform networks, two super decoding networks, two super variance decoding networks and an inter-channel correction information filtering network; wherein each synthetic transform network is respectively connected to one super decoding network and one super variance decoding network, and two synthetic transform networks are both connected to the inter-channel correction information filtering network.

[0024] That is, in addition to the above-mentioned target network, the hypervariance decoding network and / or the inter-channel correction information filtering network can also be quantized to further reduce the size of the decoding network model so that it can be deployed on end-side devices with limited resources such as mobile phones and robots.

[0025] In a second aspect, a quantization method for a decoding network model is provided, the method comprising: obtaining quantization configuration information of an unquantized decoding network model, the quantization configuration information indicating the quantization accuracy of each network layer in a target network in the decoding network model, the target network comprising a super-decoding network and / or a synthetic transformation network; wherein, in the case where the target network comprises the super-decoding network, the quantization accuracy of each network layer in the super-decoding network is the same; in the case where the target network comprises the synthetic transformation network, the quantization accuracy of at least some network layers in the synthetic transformation network is the same; based on the quantization configuration information, performing a quantization-aware training process to obtain a quantization parameter of the target network; based on the quantization parameter of the target network, quantizing the target network in the decoding network model to obtain a quantized decoding network model.

[0026] It can be seen that when the decoding network model is quantized, the present application first obtains the quantization configuration information of the unquantized decoding network model, and the quantization configuration information indicates the quantization accuracy of each network layer in the target network in the decoding network model, and the target network includes a super decoding network and / or a synthetic transformation network. Then, based on the quantization configuration information, a quantization-aware training process is performed to obtain the quantization parameters of the target network, and then the target network in the decoding network model is quantized according to the quantization parameters of the target network to obtain a quantized decoding network model. Since quantizing the target network can reduce the volume of the target network and increase the calculation speed of the target network, by quantizing the target network, the model volume of the decoding network model can be greatly reduced, the memory consumption during the model operation can be reduced, and the running speed of the decoding network model on the hardware can be improved.

[0027] Optionally, the quantization accuracy includes weight quantization accuracy and / or activation quantization accuracy; the super-decoding network includes multiple network layers, the weight quantization accuracy of each network layer in the super-decoding network is the same and less than or equal to M bits, the activation quantization accuracy of each network layer in the super-decoding network is the same and less than or equal to N bits, the M is less than the N, and the M and the N are both less than the accuracy of the super-decoding network before quantization; the synthetic transformation network includes multiple network layers, the weight quantization accuracy of each network layer in the synthetic transformation network is the same and less than or equal to P bits, the activation quantization accuracy of each network layer in the synthetic transformation network is the same and less than or equal to Q bits, the P is less than the Q, and the P and the Q are both less than the accuracy of the synthetic transformation network before quantization.

[0028] Optionally, the synthetic transformation network includes a residual module, a first deconvolution layer, a first cropping layer, a first residual activation unit, a second deconvolution layer, a second cropping layer, a second residual activation unit, a first convolution layer, a third residual activation unit, a second convolution layer, a pixel shifting layer and a third cropping layer connected in sequence; the super decoding network includes a third deconvolution layer, a fourth cropping layer, a first activation layer, a third convolution layer, a fourth deconvolution layer, a fifth cropping layer, a second activation layer, a fourth convolution layer and a third activation layer connected in sequence.

[0029] Optionally, the residual module includes a fifth convolutional layer and a fourth activation layer connected in sequence, and the sum of the input tensor of the fifth convolutional layer and the output tensor of the fourth activation layer is the output of the residual module.

[0030] Optionally, the first residual activation unit includes a fifth activation layer and a sixth convolutional layer connected in sequence, and the product of the input tensor of the fifth activation layer and the output tensor of the sixth convolutional layer is the output of the first residual activation unit; the second residual activation unit includes a sixth activation layer and a seventh convolutional layer connected in sequence, and the product of the input tensor of the sixth activation layer and the output tensor of the seventh convolutional layer is the output of the second residual activation unit; the third residual activation unit includes a seventh activation layer and an eighth convolutional layer connected in sequence, and the product of the input tensor of the seventh activation layer and the output tensor of the eighth convolutional layer is the output of the third residual activation unit.

[0031] Optionally, the quantization accuracy includes weight quantization accuracy and / or activation quantization accuracy; the super-decoding network includes multiple network layers, the weight quantization accuracy of each network layer in the super-decoding network is the same and less than or equal to M bits, the activation quantization accuracy of each network layer in the super-decoding network is the same and less than or equal to N bits, M is less than N, and both M and N are less than the accuracy of the super-decoding network before quantization; the synthetic transformation network includes multiple network layers, the weight quantization accuracy of each network layer in the synthetic transformation network is the same and less than or equal to P bits, the activation quantization accuracy of at least one network layer in the synthetic transformation network is the same and less than or equal to P bits, the activation quantization accuracy of other network layers in the synthetic transformation network except the at least one network layer is the same and less than or equal to Q bits, P is less than Q, and both P and Q are less than the accuracy of the synthetic transformation network before quantization.

[0032] Optionally, the synthetic transformation network includes a residual module, a first deconvolution layer, a first cropping layer, a first residual activation unit, a second deconvolution layer, a second cropping layer, a second residual activation unit, a first convolution layer, a third residual activation unit, a second convolution layer, a pixel shifting layer and a third cropping layer connected in sequence; the super decoding network includes a third deconvolution layer, a fourth cropping layer, a first activation layer, a third convolution layer, a fourth deconvolution layer, a fifth cropping layer, a second activation layer, a fourth convolution layer and a third activation layer connected in sequence.

[0033] Optionally, the residual module includes a fifth convolutional layer and a fourth activation layer connected in sequence, and the sum of the input tensor of the fifth convolutional layer and the output tensor of the fourth activation layer is the output of the residual module.

[0034] Optionally, the first residual activation unit includes a fifth activation layer and a sixth convolutional layer connected in sequence, and the product of the input tensor of the fifth activation layer and the output tensor of the sixth convolutional layer is the output of the first residual activation unit; the second residual activation unit includes a sixth activation layer and a seventh convolutional layer connected in sequence, and the product of the input tensor of the sixth activation layer and the output tensor of the seventh convolutional layer is the output of the second residual activation unit; the third residual activation unit includes a seventh activation layer and an eighth convolutional layer connected in sequence, and the product of the input tensor of the seventh activation layer and the output tensor of the eighth convolutional layer is the output of the third residual activation unit.

[0035] Optionally, at least one network layer in the synthetic transformation network whose activation quantization accuracy is the same and less than or equal to P bits includes but is not limited to: the second deconvolution layer and the second residual activation unit; or, the first convolution layer and the third residual activation unit; or, the second deconvolution layer, the second residual activation unit, the first convolution layer and the third residual activation unit; or, the first deconvolution layer, the first residual activation unit, the second deconvolution layer, the second residual activation unit, the first convolution layer and the third residual activation unit.

[0036] Optionally, the values ​​of M and P are both 8, and the values ​​of N and Q are both 16.

[0037] Optionally, the decoding network model includes the synthetic transform network, the super decoding network and the super variance decoding network, and the super decoding network and the super variance decoding network are both connected to the synthetic transform network.

[0038] Optionally, the decoding network model includes two synthetic transform networks, two super decoding networks, two super variance decoding networks and an inter-channel correction information filtering network; wherein each synthetic transform network is respectively connected to one super decoding network and one super variance decoding network, and two synthetic transform networks are both connected to the inter-channel correction information filtering network.

[0039] In a third aspect, a quantization device for a decoding network model is provided, wherein the quantization device for a decoding network model has the function of implementing the quantization method behavior of the decoding network model in the second aspect. The quantization device for a decoding network model includes at least one module, and the at least one module is used to implement the quantization method of the decoding network model provided in the second aspect.

[0040] In a fourth aspect, a computer device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the quantization method of the decoding network model described in the second aspect when calling and executing the computer program.

[0041] Optionally, the computer device may further include a communication bus, which is used to establish a connection between the processor and the memory.

[0042] In a fifth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium. When the computer program is executed on the computer device, the computer device executes the steps of the quantization method of the decoding network model described in the second aspect.

[0043] In a sixth aspect, a computer program product is provided, which includes a computer program. When the computer program is executed on the computer device, the computer device executes the steps of the quantization method of the decoding network model described in the second aspect.

[0044] The technical effects obtained in the above-mentioned second to sixth aspects are similar to the technical effects obtained by the corresponding technical means in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a schematic diagram of an image compression framework provided in an embodiment of the present application;

[0046] Figure 2 is a schematic diagram of a network model provided in an embodiment of the present application;

[0047] Figure 3 is a structural schematic diagram of a computer device provided in an embodiment of the present application;

[0048] Figure 4 is a schematic diagram of another image compression framework provided in an embodiment of the present application;

[0049] Figure 5 It is a schematic diagram of a coding and decoding network model based on super-prior information provided in an embodiment of the present application;

[0050] Figure 6 is a schematic diagram of a coding network model based on the JPEG AI standard provided in an embodiment of the present application;

[0051] Figure 7 is a schematic diagram of a decoding network model based on the JPEG AI standard provided in an embodiment of the present application;

[0052] Figure 8is a schematic diagram of another coding network model based on the JPEG AI standard provided in an embodiment of the present application;

[0053] Fig. 9 is a schematic diagram of another decoding network model based on the JPEG AI standard provided in an embodiment of the present application;

[0054] Fig.10 It is a flowchart of a decoding network model quantization method provided in an embodiment of the present application;

[0055] Fig.11 is a structural diagram of a synthetic transformation network provided in an embodiment of the present application;

[0056] Fig.12 is a schematic diagram of the structure of a super decoding network provided in an embodiment of the present application;

[0057] Fig.13 is a schematic diagram of the structure of a residual module provided in an embodiment of the present application;

[0058] Fig.14 is a structural schematic diagram of a residual activation unit provided in an embodiment of the present application;

[0059] Fig.15 It is a flowchart of a coding network model quantization method provided in an embodiment of the present application;

[0060] Fig.16 It is a structural diagram of a decoding network model quantization device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0062] For ease of understanding, before explaining in detail the quantization method of the encoding and decoding network model provided in the embodiment of the present application, the nouns, implementation environment and application scenarios involved in the embodiment of the present application are first introduced.

[0063] First, the terms involved in the embodiments of the present application are explained.

[0064] Artificial intelligence (AI) is a technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.

[0065] Convolutional neural network (CNN): A feedforward neural network with deep structure and convolutional calculations. It is one of the representative algorithms of deep learning. CNN may also contain activation layers (such as ReLU, etc.), pooling layers, batch normalization layers, fully connected layers and other modules. Typical convolutional neural networks include LeNet, AlexNet, VGGNet, ResNet, etc. Basic CNN can be composed of a backbone network and a head network, while complex CNN is composed of a backbone, neck and head network.

[0066] Feature map: refers to the three-dimensional data output by the convolution layer, activation layer, pooling layer, batch normalization layer and other network layers in CNN. The three dimensions of this three-dimensional data are width, height and channel.

[0067] As an example, an image can output a feature map after passing through a network layer. For any network layer, if the network layer contains 3 convolution kernels, the 3 convolution kernels can respectively convolve the feature map input by the network layer to obtain the feature map output by the network layer. The number of channels of the feature map output by the network layer is 3, that is, the number of convolution kernels contained in the network layer corresponds one to one to the number of channels of the feature map output by the network layer.

[0068] Rectified linear unit (ReLU): is a commonly used activation function in artificial neural networks, usually referring to nonlinear functions represented by ramp functions and their variants.

[0069] Variational autoencoder (VAE): An AI-based image codec used for data compression or noise removal. VAE-based image compression frameworks such as Figure 1 As shown in the figure, during the encoding process, the original image to be compressed is input into the encoding network, and the original image is feature extracted through the encoding network to obtain the features to be encoded. Then, the features to be encoded are quantized to obtain the quantized features to be encoded. For the quantized features to be encoded, the probability distribution of the features to be encoded is estimated through the entropy estimation network, and the quantized features to be encoded are entropy encoded based on the estimated probability distribution to obtain the image code stream. The decoding process corresponds to the above encoding process.

[0070] AI compression algorithm (AICodec): refers to a data compression method based on deep learning technology.

[0071] Entropy estimation: The process of predicting the distribution of quantized data.

[0072] BD-Rate: full name -delta-rate is used to evaluate the rate-distortion (RD) performance of different video encoders. In the performance measurement between compression methods, the BD-Rate of method A compared to method B indicates the difference in bitrate between method A and method B under the same objective indicators. BD-Rate is usually expressed as a percentage. If it is -x%, it means that method A can save x% of space compared to method B; if it is x%, it means that x% of space is increased.

[0073] Rate-distortion curve (RD-Curve): The horizontal axis is the bit rate, and the vertical axis is the image quality. Generally, the closer the curve is to the upper left, the better the encoding and decoding performance.

[0074] Bit rate: In image compression, it refers to the code length required to encode a unit pixel. The higher the bit rate, the better the image reconstruction quality.

[0075] Bits per pixel (BPP): Also known as bits / pixel, BPP is the number of bits used to store each pixel. It is calculated as the total image bitstream size / image size. The smaller the BPP, the lower the compression bitrate, and the larger the BPP, the higher the compression bitrate.

[0076] Peak signal to noise ratio (PSNR): is an objective standard for evaluating image quality. The PSNR value range is greater than zero. The larger the PSNR value, the better the image quality is, and the more similar the reconstructed image is to the original image.

[0077] Multi-scale structural similarity (MS-SSIM): It is a structural similarity (SSIM) index based on multiple scales (i.e., the images are scaled from large to small according to certain rules). The value range of MS-SSIM is 0-1. The larger the MS-SSIM value, the better the image quality is, and the more similar the reconstructed image is to the original image.

[0078] In some embodiments, PSNR and MS-SSIM are two indicators used to measure image reconstruction quality.

[0079] Quantization: refers to converting a continuous signal into a discrete signal. In the image compression process, it refers to converting continuous features into discrete features. In entropy coding, the probability value of the probability distribution is usually converted from a continuous value to a discrete value.

[0080] Model quantization: refers to the method of converting a floating-point neural network model into a fixed-point model that can be represented by a finite number of bits. Model quantization can establish a mapping relationship between floating-point data and fixed-point data. Its principle can be understood as using integers in the range of n (e.g. 8, 10, 16) bits to represent the floating-point weights in the network model and the tensors in the calculation process. For ease of understanding, we will now use Figure 2 Taking the network model shown in the figure as an example, the process of quantizing tensors in the model calculation process is introduced. Figure 2 In the figure, x represents the input of the network model, y represents the output of the network model, a1 is the feature map output by the convolutional layer (Conv), a2 is the feature map output by the activation layer (Relu), qx represents the fixed-point value of x after quantization, qa1 represents the feature map of a1 after quantization, qa2 represents the feature map of a2 after quantization, and qy represents the data output by the fully connected layers (FC).

[0081] In model quantization, based on sample data, the value ranges of x, a1, a2, and y can be statistically analyzed to obtain the maximum and minimum values ​​in the value ranges of x, a1, a2, and y. Based on the maximum and minimum values ​​in the value ranges of x, a1, a2, and y, the quantization formulas corresponding to x, a1, a2, and y are determined. In this way, during the inference process of the model, the input of the model and the tensor in the calculation process can be quantized based on the quantization formulas corresponding to x, a1, a2, and y to obtain qy, and qy can be dequantized to obtain the inferred floating-point value y.

[0082] As an example, quantization from floating point values ​​to fixed point values ​​can be determined by the following formula (1).

[0083]

[0084] In the formula, x quant represents the quantized fixed-point value, x represents the unquantized floating-point value, scale represents the minimum scale that the quantized fixed-point value can represent, i.e., the scale, zero_point represents the fixed-point value corresponding to the floating-point value 0, i.e., the zero point position, quant min Represents x quant The minimum value in the range of quant max Represents x quantThe round() function is used to round the numbers in the brackets, and the clamp() function is used to clamp x quant Restricted to quant max and quant min That is, if the calculated fixed-point value exceeds x quant The value range of x is truncated. quant is equal to quant max and quant min The value in that is closest to the calculated fixed-point value.

[0085] The main model quantization methods include post training quantization (PTQ) and quantization aware training (QAT).

[0086] Among them, PTQ refers to directly converting a floating-point network model into a fixed-point network model without retraining the network model, that is, without updating the weights of the network model. In other words, PTQ can directly convert a trained 32-bit floating-point network into a fixed-point calculation network without any training of the original model in the process, and the quantization process can be completed by adjusting a few hyperparameters. QAT refers to inserting pseudo-quantization (simulated quantization) operations into the network model during network model training or fine-tuning to simulate the quantization operation, so that the parameters of the network model can better adapt to the information loss caused by quantization. In other words, QAT models the quantization noise source during training, so that the trained network model can be more robust to quantization errors, and the state of the network model after training and the quantization parameters are more consistent.

[0087] Conditional color separation (CCS): A coding method that encodes the Y color component and the UV color component separately. When encoding and decoding different color components, other color components can be used as conditions.

[0088] JPEG AI: An image coding standard based on AI image coding methods, which aims to create a learning-based image coding standard that provides a single-stream, compact compressed domain representation. Compared with commonly used image coding standards for human visualization at the same subjective quality, it significantly improves image compression efficiency and the performance of image processing and computer vision tasks.

[0089] Secondly, the implementation environment involved in the embodiments of the present application is introduced.

[0090] Please refer to Figure 3 , Figure 330 is a schematic diagram of a computer device provided in an embodiment of the present application. The computer device can perform quantization processing on a coding and decoding network model. The computer device includes one or more processors 301, a communication bus 302, a memory 303, and one or more communication interfaces 304.

[0091] The processor 301 is a general-purpose central processing unit (CPU), a network processing (NP), a microprocessor, or one or more integrated circuits for implementing the solution of the present application, such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. Optionally, the PLD is a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0092] The memory 303 is a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), an optical disc (including a compact disc read-only memory (CD-ROM), a compressed optical disc, a laser disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. Optionally, the memory 303 exists independently and is connected to the processor 301 via the communication bus 302, or the memory 303 is integrated with the processor 301.

[0093] The communication interface 304 uses any transceiver-like device for communicating with other devices or communication networks. The communication interface 304 includes a wired communication interface, and optionally, also includes a wireless communication interface. Among them, the wired communication interface is, for example, an Ethernet interface. Optionally, the Ethernet interface is an optical interface, an electrical interface, or a combination thereof. The wireless communication interface is a wireless local area network (WLAN) interface, a cellular network communication interface, or a combination thereof.

[0094] The communication bus 302 is used to transmit information between the above components. Optionally, the communication bus 302 is divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0095] Optionally, in some embodiments, the computer device may include multiple processors, such as Figure 3 The processor 301 and the processor 305 shown in FIG. Each of these processors is a single-core processor or a multi-core processor. Optionally, the processor here refers to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0096] In some embodiments, the computer device further includes an output device 306 and an input device 307. The output device 306 communicates with the processor 301 and can display information in a variety of ways. For example, the output device 306 is a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device or a projector. The input device 307 communicates with the processor 301 and can receive user input in a variety of ways. For example, the input device 307 is a mouse, a keyboard, a touch screen device or a sensor device.

[0097] In a specific implementation, the memory 303 can be used to store program codes for executing the solution of the present application, and the processor 301 can execute the program codes stored in the memory 303. The program codes include one or more software modules, and the computer device can implement the following by means of the processor 301 and the program codes in the memory 303. Fig.10 The quantization method of the decoding network model provided in the embodiment, and / or, Fig.13 The embodiment provides a quantization method for a coding network model.

[0098] Finally, the application scenarios involved in the embodiments of the present application are introduced.

[0099] Coding and decoding technology refers to a technology that uses data characteristics such as spatial redundancy, visual redundancy and statistical redundancy to represent the original signal data with fewer bits in a lossy or lossless manner. It can achieve effective transmission and storage of information such as images / videos / audio, and plays an important role in the current media era where the types and amounts of information transmitted or stored are increasing.

[0100] Taking image compression as an example, image compression is divided into lossy compression and lossless compression. Lossy compression achieves a larger compression ratio at the cost of a certain degree of image quality degradation, while lossless compression does not cause loss of image details. Traditional image / video compression algorithms have undergone decades of development and have formed mature compression standards such as high efficiency video coding (HEVC) and versatile video coding (VVC). Traditional image / video compression algorithms have the advantages of comprehensive support functions, good versatility, and good hardware support, and have been widely used, but they have also encountered bottlenecks in improving coding efficiency.

[0101] As convolutional neural networks outperform traditional algorithms in many computer vision tasks such as image recognition and target detection, more and more researchers are beginning to explore image / video compression methods based on deep learning. Unlike traditional image algorithms that use artificial design to encode, quantize, entropy encode, etc., and optimize them separately, the modules in the AI ​​image compression algorithm (encoding network, entropy estimation network, decoding network, etc.) are optimized as a whole, so the AI ​​image compression scheme has better compression effect. VAE methods (such as Figure 1 (as shown in the figure) is the mainstream technical solution for the current AI image lossy compression technology. Its compression effects on the two image evaluation indicators, PSNR and MS-SSIM, far exceed those of traditional image lossy compression technologies such as the Joint Photographic Experts Group (JPEG).

[0102] However, neural networks are resource-intensive algorithms that require not only a lot of computational cost but also consume a lot of memory. Although computing resources are increasing, optimizing the training and reasoning of deep neural networks is still very important for the implementation of models. In particular, more and more models will be moved from the server side to resource-limited devices such as smartphones and embedded devices. How to deploy complex models on resource-limited devices is a problem that must be solved by current deep learning technology.

[0103] As one of the general neural network model optimization methods, model quantization can reduce the size of deep neural network models and model inference time. It is applicable to most models and different hardware devices. Among them, the model quantization of neural network is the process of converting the weight values, activation values, etc. of the network model from high precision to low precision, for example, converting float32 data to int 8 or int 16 data; at the same time, it is expected that the accuracy of the neural network model after quantization is similar to that before quantization. Since model quantization usually maps model parameters and data generated by calculations from a space with a wider expression range to a space with a narrower expression range, quantization is generally accompanied by a loss of model accuracy.

[0104] In order to popularize AI codec technology in the codec field, the codec network model also needs to be quantized. On the one hand, the quantized codec network model will have a smaller model size and lower power consumption, which makes it more feasible for end-side applications; on the other hand, the quantized codec network model will have a faster computing speed and can better support real-time decoding and other features. However, if the general network model quantization scheme is simply implemented in the codec network model, the rate-distortion (BD-Rate) performance of the codec model when processing images will be greatly reduced.

[0105] Based on this, an embodiment of the present application provides a quantization method for a codec network model, so as to quantize the target network in the coding network model and / or decoding network model through determined quantization configuration information, so as to minimize the degree of reduction in the rate-distortion performance of the codec algorithm after model quantization, and at the same time, reduce the model size and improve the operating efficiency of the codec algorithm on hardware with limited computing power.

[0106] After the codec network model is quantized by the method provided in the embodiment of the present application, the quantized codec network model can be applied to a variety of scenarios, such as cloud services, video surveillance, live broadcast, storage, transmission and other business scenarios, and can be specifically applied to terminal recording, video albums, cloud storage, etc. That is to say, in these scenarios, the image can be encoded and decoded by the quantized codec network model.

[0107] The images encoded and decoded in various scenarios may be images included in image files or images included in video files.

[0108] See also Figure 4 , Figure 4 An image compression framework based on a codec network model is provided in an embodiment of the present application, and the image compression framework includes: an encoding end and a decoding end, the encoding end includes an AI encoding module, an entropy encoding module and a file saving module, and the decoding end includes a file loading module, an entropy decoding module and an AI decoding module.

[0109] During the compression process, the encoder obtains the image to be compressed, such as videos and pictures captured by the camera. After that, the AI ​​encoding module obtains the features to be encoded and the corresponding probability distribution. Based on the probability distribution, the entropy encoding module performs entropy encoding on the features to be encoded to obtain a bitstream file, which is saved by the file saving module to obtain a compressed file of the image. The compressed file is input to the decoder, which loads the compressed file through the file loading module and obtains the reconstructed image through the entropy decoding module and the AI ​​decoding module.

[0110] The above encoding and decoding process can be implemented by an encoding and decoding network model, and the quantization method of the encoding and decoding network model provided in the embodiment of the present application can quantize the encoding and decoding network model. Optionally, the encoding and decoding network model includes at least one network model of an encoding network model, a decoding network model and an entropy estimation network model.

[0111] It should be noted that Figure 4 and Figure 1 The image encoding and decoding principles of the image compression framework shown are similar. Figure 4 The AI ​​coding module in Figure 1 The encoding network and entropy estimation network in the AI ​​decoding module are equivalent to the Figure 1 The decoding network and entropy estimation network in .

[0112] Optionally, the data processing process of the AI ​​encoding module and the AI ​​decoding module is implemented on an embedded neural network processing unit (NPU) to improve data processing efficiency, and the processes of entropy coding, saving files, and loading files are implemented on a central processing unit (CPU).

[0113] Furthermore, based on the above-mentioned image compression framework, the quantization method of the codec network model provided in the embodiment of the present application can be applied to any codec network model, such as a general codec network model based on super-prior information, a codec network model based on the learning JPEG AI standard, etc., and the embodiment of the present application does not impose any restrictions on this.

[0114] To facilitate understanding, we first briefly introduce the encoding and decoding process of these two encoding and decoding network models.

[0115] (1) A general encoding and decoding network model based on super-prior information.

[0116] Please refer to Figure 5At the encoding end, the original image is input into the encoding network to extract features to obtain image features y to be quantified of multiple feature points, and the image features y to be quantified of the multiple feature points are quantized to obtain the first image features of the multiple feature points The first image feature of the plurality of feature points Input to the super coding network to obtain the super prior features z of the multiple feature points to be quantized, and quantize the super prior features z of the multiple feature points to be quantized to obtain the first super prior features of the multiple feature points The first super prior features of the multiple feature points are calculated according to the specified probability distribution. Entropy coding is performed to convert Encoded into the bitstream. Figure 5 As shown, The bit sequence obtained by entropy coding is the partial bit sequence included in the bit stream. This partial bit sequence (such as Figure 5 The black and white bars on the right side of the middle) can be called a super-prior bit stream.

[0117] In addition, the first super prior features of the multiple feature points Input the super decoding network to obtain the prior features ψ of the multiple feature points. The context model (CM) is input to obtain the context features φ of the multiple feature points. The probability distribution N(μ, σ) of the multiple feature points is estimated through a probability distribution estimation network model (shown as a gather model, GM) based on the probability distribution N(μ, σ) of the multiple feature points. The first image feature of each feature point in the multiple feature points is sequentially converted into the image feature vector. to be encoded into the bitstream. Figure 5 As shown, The bit sequence obtained by entropy coding is the partial bit sequence included in the bit stream. This partial bit sequence (such as Figure 5 The black and white bars on the left side of the image (shown in the figure) can be called an image bit stream.

[0118] At the decoding end, first, the first super-prior feature of the plurality of feature points is obtained by entropy decoding from the super-prior bit stream included in the bit stream according to the specified probability distribution. The first super prior features of the multiple feature points Input the super decoding network to obtain the prior features ψ of the multiple feature points. For the first feature point among the multiple feature points, estimate the probability distribution of the first feature point based on the prior features of the first feature point, and parse the first image feature of the first feature point from the image bit stream included in the code stream based on the probability distribution of the first feature point. For the non-first feature point among the multiple feature points, such as the first feature point, determine the surrounding information of the first feature point from the first image features of each decoded feature point, input the surrounding information of the first feature point into the context model CM to obtain the context feature of the first feature point, combine the prior features and context features of the first feature point, estimate the probability distribution of the first feature point through the probability distribution estimation network model GM, and parse the first image feature of the first feature point from the image bit stream included in the code stream based on the probability distribution of the first feature point. Entropy decode the first image feature of the multiple feature points from the code stream Afterwards, Input the decoding network to get the reconstructed image.

[0119] It should be noted that if any of the above-mentioned probability distribution estimation network models uses a Gaussian model (such as a single Gaussian model or a mixed Gaussian model) to model, the estimated probability distribution parameters include mean and variance. If any of the above-mentioned probability distribution estimation network models uses a Laplace distribution model to model, the estimated probability distribution parameters include location parameters and scale parameters. If any of the above-mentioned probability distribution estimation network models uses a logistic distribution model to model, the estimated probability distribution parameters include mean and scale parameters. In addition, the probability distribution estimation network model in the embodiment of the present application can also be referred to as a factor entropy model, and the probability distribution estimation network model is a part of the entropy estimation network model, and the entropy estimation network model also includes the above-mentioned super encoding network and super decoding network.

[0120] (2) Codec network model based on JPEG AI standard.

[0121] The coding and decoding scheme based on the JPEG AI codec standard is based on a color component separation autoencoder structure, and realizes the coding and decoding of the original image by encoding and decoding the main component and the secondary component in the original image to be encoded respectively.

[0122] On the image coding side, the networks that perform coding operations in the coding network model based on the JPEG AI standard include: Analysis Transform network, entropy estimation network based on hyper-prior information or entropy coding network that combines hyper-prior information and context information, Hyper Decoder network, Hyper Scale Decoder network, Gain Unit, etc.

[0123] In some embodiments, the coding network model based on the JPEG AI standard can encode each single component contained in the original image to be encoded in sequence to obtain the encoded code stream of the original image.

[0124] See also Figure 6 The process of encoding a single component in the original image by the coding network model is as follows: extract the features of the single component x through the analysis transformation network to obtain the corresponding image feature y; then process the image feature y in turn through the super coding network, entropy coding network, entropy decoding network, super variance decoding network, super decoding network, etc. to obtain the coded code stream corresponding to the single component x (i.e., stream-y in the figure).

[0125] For an image in YUV format, the encoded single component may be a Y component or a UV component.

[0126] For an image separated into Y and UV components, Figure 6 The coding network model shown can first encode the Y component to obtain the coded bit stream of the Y component, and then encode the UV component to obtain the coded bit stream of the UV component; or, Figure 6 The coding network model shown can first encode the UV component to obtain the coding stream of the UV component, and then encode the Y component to obtain the coding stream of the Y component. The embodiment of the present application does not limit the coding order of a single component in the original image.

[0127] That is, when encoding the original image separated into Y component and UV component, Figure 6 The encoding network model shown needs to perform two encoding processes to complete the encoding of the original image.

[0128] In some embodiments, the coding network model based on the JPEG AI standard can also directly encode the original image to be encoded, that is, encode multiple single components in parallel to obtain the encoded code stream of the original image.

[0129] See also Figure 7 The encoding network model encodes the original image in the following process: first, the original image in RGB (a color mode, where "R" represents red, G represents green, and B represents blue) format is converted into YUV (a color encoding method, where "Y" represents brightness, that is, grayscale value, and "U" and "V" represent chrominance) format through the color space, and then the Y component and UV component are encoded in parallel through the analysis transformation network, super encoding network and super decoding network, and finally the encoded code stream y corresponding to the Y component is output. Y , and the encoded bitstream y corresponding to the UV component UV .

[0130] That is, when encoding the original image separated into Y component and UV component, Figure 7 The encoding network model shown can complete the encoding of the original image by performing the encoding process only once.

[0131] On the image decoding side, the networks that perform decoding operations based on the JPEG AI standard encoding network model include: Synthesis Transform network, entropy estimation network based on hyper-prior information or entropy decoding network that combines hyper-prior information and context information, hyper-decoding (Hyper Decoder) network, hyper-variance decoding (Hyper Scale Decoder) network, inverse gain unit (Inv Gain Unit), etc.

[0132] In some embodiments, a decoding network model based on the JPEG AI standard can decode the encoded bitstream of a single component corresponding to the original image in sequence to obtain a reconstructed image.

[0133] See also Figure 8 The decoding network model decodes the coded bitstream of a single component as follows: decode the coded bitstream of a single component (i.e., stream-y in the figure) through networks such as entropy decoding network, super decoding network, and super variance decoding network to obtain the decoded image features. The decoded image features are then transformed through the synthetic transformation network Processing is performed to obtain a single component reconstructed

[0134] For an image in YUV format, the encoded code stream of a single component to be decoded may be an encoded code stream of a Y component or an encoded code stream of a UV component.

[0135] For an image separated into Y and UV components, Figure 8 The decoding network model shown first decodes the coded bitstream of the Y component to obtain the reconstructed Y component, and then decodes the coded bitstream of the UV component to obtain the reconstructed UV component.

[0136] That is, when decoding the coded bitstream of the original image separated into the Y component and the UV component, Figure 8 The decoding network model shown needs to perform two decoding processes to obtain the reconstructed image.

[0137] In some embodiments, the coding network model based on the JPEG AI standard can also directly decode the coded code stream of the original image, that is, decode multiple single-component coded code streams in parallel to obtain a reconstructed image.

[0138] See also Fig. 9The decoding network model decodes the coded bitstream of the original image in the following process: through the synthesis transformation network and the super decoding network, the coded bitstreams of the Y component and the UV component are decoded in parallel to obtain the reconstructed Y component and UV component. Then, the reconstructed Y component and UV component are processed through the Inter Channel Correlation Information (ICCI) network to obtain the reconstructed image in YUV format, and finally, the reconstructed image in YUV format is converted into RGB format through the inverse conversion of the color space.

[0139] That is, when decoding the coded bitstream of the original image separated into the Y component and the UV component, Fig. 9 The decoding network model shown only performs one decoding process to obtain the reconstructed image.

[0140] Based on the image compression framework and encoding and decoding network model introduced above, when quantizing the decoding network model, the embodiment of the present application can quantize the target network in the decoding network model to speed up the running rate of the decoding network model on the hardware, while ensuring the rate-distortion performance of the decoding algorithm as much as possible; similarly, when quantizing the encoding network model, the target network in the encoding network model can be quantized to speed up the running efficiency of the encoding network model on the hardware, while ensuring the rate-distortion performance of the encoding algorithm as much as possible.

[0141] Among them, for the decoding network model, the target network to be quantized can be one network or multiple networks in the decoding network model; for the encoding network model, the target network to be quantized can be one network or multiple networks in the encoding network model. That is, the embodiment of the present application does not limit the number of networks to be quantized in the encoding and decoding network model, which can be one network in the network model or multiple networks.

[0142] It should be noted that the decoding logic of the decoding network model based on the JPEG AI standard and the general decoding network model based on super-prior information are similar. The synthesis transformation network in the decoding network model based on the JPEG AI standard and the decoding network in the general decoding network model based on super-prior information are both used to perform decoding operations, and the only difference is that the names of the networks performing decoding operations are different in the two decoding network models. Similarly, the encoding logic of the encoding network model based on the JPEG AI standard and the general encoding network model based on super-prior information are similar. The analysis transformation network in the encoding network model based on the JPEG AI standard and the encoding network in the general encoding network model based on super-prior information are both used to perform encoding operations, and the names of the networks performing encoding operations are different in the two encoding network models.

[0143] Based on this, although the following embodiments use the codec network model based on the JPEG AI standard as an example to explain the quantization of the target network, in actual application, the quantization method below can also be referred to to quantize the corresponding network in the general codec network model based on super-prior information, and the specific quantization logic is similar. That is, the following uses the codec network model based on the JPEG AI standard as an example to explain the quantization method of the codec network model provided in the embodiment of the present application, but it does not constitute a limitation on the codec network model to which the quantization method is applicable.

[0144] Next, taking the coding network model based on the JPEG AI standard as an example, the quantization method of the coding and decoding network model provided in the embodiment of the present application is explained.

[0145] Fig.10 is a flowchart of a quantization method for a decoding network model provided in an embodiment of the present application, which can be applied to Figure 3 For the computer equipment shown, please refer to Fig.10 , the method comprises the following steps.

[0146] Step 1001: Obtain quantization configuration information of an unquantized decoding network model, where the quantization configuration information indicates the quantization accuracy of each network layer in a target network in the decoding network model, where the target network includes a super decoding network and / or a synthetic transformation network.

[0147] Among them, when the target network includes a super-decoding network, the quantization accuracy of each network layer in the super-decoding network is the same; when the target network includes a synthetic transformation network, the quantization accuracy of at least some network layers in the synthetic transformation network is the same.

[0148] The weight is used to represent the connection coefficient between the neurons of each network layer in the network model, and the activation is used to represent the input and / or output of each network layer in the network model. Therefore, when quantizing the network model, the weight of the network layer can be quantized to reduce the memory usage of the network model (i.e., the model size), and the input and / or output of the network layer can also be quantized to achieve a significant model acceleration effect. Therefore, the quantization accuracy of the target network can include weight quantization accuracy and / or activation quantization accuracy. Among them, the weight quantization accuracy is used to quantize the weights of each network layer in the target network, and the activation quantization accuracy is used to quantize the input and / or output of each network layer in the target network.

[0149] In some embodiments, the quantization configuration information of the decoding network model can be determined through multiple implementation methods, three of which are introduced below.

[0150] In a first implementation manner, the preset quantization configuration information is determined as the quantization configuration information of the decoding network model.

[0151] That is, the quantization configuration information determined in advance based on experience is directly used as the quantization configuration information of the decoding network model.

[0152] In some embodiments, when the target network includes a super decoding network, the preset quantization configuration information is: all network layers in the super decoding network uniformly use M bits of quantization accuracy to quantize weights, and use N bits of quantization accuracy to quantize inputs and / or outputs of each network layer during calculation. When the target network includes a synthetic transformation network, the preset quantization configuration information is: all network layers in the synthetic transformation network uniformly use P bits of quantization accuracy to quantize weights, and use Q bits of quantization accuracy to quantize outputs and / or outputs of each network layer during calculation. Among them, the value of M is less than the value of N, and the value of P is less than the value of Q.

[0153] As an example, taking a 32-bit floating-point decoding network model as an example, the quantization configuration information of the target network can be shown in Table 1 below.

[0154] Table 1

[0155] Weight quantization accuracy Activate quantization accuracy Super decoding network ≤8bit ≤16bit Synthetic Transformation Network ≤8bit ≤16bit

[0156] In the preset quantization configuration information, when quantizing the target network in the decoding network model, the weight quantization accuracy of each network layer in the target network is the same, and the activation quantization accuracy of each network layer is also the same.

[0157] In other embodiments, when the target network includes a super-decoding network, the preset quantization configuration information may be: all network layers in the super-decoding network uniformly use M-bit quantization accuracy to quantize weights, and the input and / or output of a certain middle layer and several network layers before and after it are quantized with M-bit quantization accuracy, and the input and / or output of the calculation of the remaining network layers are quantized with N-bit quantization accuracy. When the target network includes a synthetic transformation network, all network layers in the synthetic transformation network uniformly use P-bit quantization accuracy to quantize weights, and the input and / or output of a certain middle layer and several network layers before and after it are quantized with P-bit quantization accuracy, and the input and / or output of the calculation of the remaining layers are quantized with Q-bit quantization accuracy.

[0158] Among them, the value of M is smaller than the value of Q, and the values ​​of M and N are both smaller than the accuracy of the super decoding network before quantization. The value of P is smaller than the value of Q, and the values ​​of P and Q are both smaller than the accuracy of the synthesis transform network before quantization.

[0159] It should be noted that, for the super decoding network and the synthetic transform network, the weight quantization accuracy of each network layer can be the same or different, that is, the value of M can be the same as or different from the value of P; similarly, for the super decoding network and the synthetic transform network, the activation quantization accuracy of each network layer can be the same or different, that is, the value of N can be the same as or different from the value of Q.

[0160] As an example, taking a 32-bit floating-point decoding network model as an example, the preset quantization configuration information of the target network may be as shown in Table 2 below.

[0161] Table 2

[0162] Weight quantization accuracy Activate quantization accuracy Super decoding network ≤8bit Some network layers are ≤8 bits, and other network layers are 16 bits Synthetic Transformation Network ≤8bit Some network layers are ≤8 bits, and other network layers are 16 bits

[0163] In the preset quantization configuration information, when quantizing the target network in the decoding network model, the weight quantization precision of each network layer in the target network is the same, but the activation quantization precision of each network layer is not exactly the same. That is, when quantizing the input and / or output of the calculation of each network layer in the target network, the activation of each network layer in the target network is quantized using mixed quantization precision.

[0164] It should be noted that when hybrid quantization precision is used to quantize the calculated input and / or output of each network layer in the target network, it can be determined by statistical or quantization-aware training methods which network layers' activation quantization precision is set to P bits and which network layers' activation quantization precision is set to Q bits. The embodiment of the present application does not limit the value of the activation quantization precision of the specific configuration of any network layer.

[0165] In the first implementation method mentioned above, since the quantization accuracy of the target network layer in the preset quantization configuration information is less than or equal to the progress of the target network of the decoding network model before quantization, quantizing the target network in the decoding network model using the quantization accuracy determined by the implementation method can effectively reduce the model size of the decoding network model and speed up its operating efficiency on the hardware device.

[0166] The second implementation method is to obtain the characteristic distribution of each network layer in the target network; based on the characteristic distribution of each network layer in the target network, determine the quantitative configuration information of the decoding network model.

[0167] Among them, for setting the weight quantization accuracy and activation quantization accuracy of the target network, the feature distributions required for statistical analysis are different. Therefore, determining the feature distributions of each network layer in the target network includes the following two situations:

[0168] (1) When the quantization accuracy of the target network includes the weight quantization accuracy, the feature distribution of each network layer in the target network includes the weight distribution of multiple neurons in the corresponding network layer.

[0169] That is, after the decoding network model is trained, the weight distribution of multiple neurons in each network layer of the target network is counted. When the target network includes a super decoding network, the weight distribution of multiple neurons in each network layer of the super decoding network is counted; when the target network includes a synthetic transformation network, the weight distribution of multiple neurons in each network layer of the synthetic transformation network is counted.

[0170] Based on this, based on the weight distribution of each network layer in the target network, the implementation process of determining the weight quantization accuracy of the decoding network model can be as follows: based on the weight distribution of each network layer in the target network layer, determine the weight distribution concentration of each network layer, and set the weight quantization accuracy of the network layer in the middle of the target network and with the largest weight distribution concentration to M bits; based on the network layer with the largest weight distribution concentration, search forward and backward in turn, set the weight quantization accuracy of the network layer whose weight distribution concentration is greater than the weight concentration threshold to M bits, and set the activation quantization accuracy of the remaining network layers to N bits.

[0171] Among them, the methods for calculating the weight distribution concentration of the network layer include but are not limited to: 3-sigma rule, the difference between the maximum and minimum values, Chauville criterion, Dixon criterion, Grubbs criterion, Peirce criterion, chi-square test, T test, etc., so that the calculated weight distribution concentration can reflect the weight distribution of multiple neurons in the network layer as realistically as possible. The embodiment of the present application does not limit the specific algorithm type used when calculating the weight distribution concentration.

[0172] In some embodiments, when the target network includes a super-decoding network, the weight distribution concentration of each network layer in the super-decoding network can be determined according to the weight distribution of multiple neurons in each network layer in the super-decoding network, thereby determining the weight quantization accuracy of each network layer in the super-decoding network based on the weight distribution concentration of each network layer in the super-decoding network.

[0173] Among them, the weight quantization accuracy of each network layer in the super decoding network may be the same or different, and the embodiment of the present application does not impose any limitation on this.

[0174] As an example, taking a 32-bit floating-point decoding network model as an example, based on a statistical analysis of the weight distribution of each network layer in the super decoding network, the weight quantization accuracy of each network layer in the super decoding network is set to 8 bits.

[0175] In some embodiments, when the target network includes a synthetic transformation network, the weight distribution concentration of each network layer in the synthetic transformation network can be determined according to the weight distribution of multiple neurons in each network layer in the synthetic transformation network, thereby determining the weight quantization accuracy of each network layer in the synthetic transformation network based on the weight distribution concentration of each network layer in the synthetic transformation network.

[0176] Among them, the weight quantization accuracy of each network layer in the synthetic transformation network may be the same or different, and the embodiment of the present application does not impose any limitation on this.

[0177] As an example, taking a 32-bit floating-point decoding network model as an example, based on the statistical analysis of the weight distribution of each network layer in the synthetic transformation network, the weight quantization accuracy of each network layer in the synthetic transformation network is also set to 8 bits.

[0178] (2) When the quantization accuracy of the target network includes the activation quantization accuracy, the feature distribution of each network layer in the target network includes the input feature distribution and / or output feature distribution of the corresponding network layer.

[0179] In a possible implementation, the process of obtaining the feature distribution of each network layer in the target network can be as follows: multiple reference feature maps are input into the decoding network model to obtain multiple input feature maps and multiple output feature maps corresponding to each network layer in the target network, and the multiple input feature maps and the multiple output feature maps are respectively compared with the multiple reference feature maps. Figure 1 One-to-one correspondence; based on multiple input feature maps corresponding to each network layer, statistics are made on the input feature distribution of each network layer, and / or, based on multiple output feature maps corresponding to each network layer, statistics are made on the output feature distribution of each network layer.

[0180] That is, for the decoding network model, its input is multiple reference feature maps, and for a network layer in the target network, there is an input feature map and an output feature map for each reference feature map. When multiple reference feature maps are input to the decoding network model, there are multiple input feature maps and multiple output feature maps for any network layer in the target network. Therefore, for each network layer in the target network, the input feature distribution of the network layer can be statistically analyzed based on the multiple input feature maps flowing into the network layer; similarly, for each network layer in the target network, the output feature distribution of the network layer can be statistically analyzed based on the multiple output feature maps output by the network layer.

[0181] It should be understood that for any network layer in the target network, there are multiple input feature maps and multiple output feature maps, and the output feature map of the previous network layer is the same as the input feature map of the adjacent subsequent network layer. Therefore, when counting the input feature distribution and the output feature distribution, only the input feature distribution of each network layer can be counted. For two adjacent network layers, the output feature distribution of the previous network layer can reuse the input feature distribution of the subsequent network layer.

[0182] The implementation process of determining the activation quantization accuracy of the decoding network model based on the input feature distribution and / or output feature distribution of each network layer in the target network can be as follows: based on the input feature distribution and / or output feature distribution of each network layer in the target network, determine the feature distribution concentration of each network layer, the feature distribution concentration includes the input feature distribution concentration and / or the output feature distribution concentration; set the activation quantization accuracy of the network layer that is in the middle of the target network and has the largest feature distribution concentration to P bits; based on the network layer with the largest feature distribution concentration, search forward and backward in turn, set the activation quantization accuracy of the network layer whose feature distribution concentration is greater than the feature concentration threshold to P bits, and set the activation quantization accuracy of the remaining network layers to Q bits.

[0183] Taking a 32-bit floating-point decoding network model as an example, by counting the input feature distribution and / or output feature distribution of each network layer in the target network, the implementation process of determining the activation quantization accuracy of each network layer can be: according to the input feature distribution and / or output feature distribution of each network layer obtained by statistics, determine the first network layer with the highest feature distribution concentration, and set the activation quantization accuracy of the first network layer to 8 bits or lower bits. Then, starting from the first network layer, search forward in sequence to determine whether the feature distribution concentration of the second network layer adjacent to the first network layer and located before the first network layer is greater than the feature concentration threshold. If the feature distribution concentration of the second network layer is greater than the feature concentration threshold, the activation quantization accuracy of the second network layer is also set to 8 bits or lower, and continue to determine whether the feature distribution concentration of the third network layer adjacent to the second network layer and located before the second network layer is greater than the feature concentration threshold, and so on, until the feature distribution concentration of the adjacent network layer is less than or equal to the feature concentration threshold, then stop searching forward; if the feature distribution concentration of the second network layer is less than or equal to the feature concentration threshold, then stop searching and set the activation quantization accuracy of all network layers before the first network layer to 16 bits. Similarly, starting from the first network layer, search backward in sequence to determine whether the feature distribution concentration of the fourth network layer adjacent to the first network layer and located after the first network layer is greater than the feature concentration threshold. If the feature distribution concentration of the fourth network layer is greater than the feature concentration threshold, the activation quantization accuracy of the fourth network layer is also set to 8 bits or lower bits, and continue to judge whether the feature distribution concentration of the fifth network layer adjacent to the fourth network layer and located after the fourth network layer is greater than the feature concentration threshold, and so on, until the feature distribution concentration of the adjacent network layer is less than or equal to the feature concentration threshold, then stop the backward search; if the feature distribution concentration of the fourth network layer is less than or equal to the feature concentration threshold, stop the search, and set the activation quantization accuracy of all network layers after the first network layer to 16 bits.

[0184] Among them, the method of calculating the feature distribution concentration of the network layer includes but is not limited to: 3-sigma rule, the difference between the maximum and minimum values, Chauville criterion, Dixon criterion, Grubbs criterion, Peirce criterion, chi-square test, T test, etc., so that the calculated feature distribution concentration can reflect the input feature distribution and / or output feature distribution of the network layer as realistically as possible. The embodiment of the present application does not limit the specific algorithm type used when calculating the feature distribution concentration.

[0185] Optionally, after determining the quantization configuration information of the decoding network model based on the characteristic distribution of each network layer in the target network, the quantization configuration information preset in the previous text can also be optimized based on the determined quantization configuration information. For example, the quantization configuration information determined in the second implementation method mentioned above can be used to replace the preset quantization configuration information to improve the reliability of the model quantization result when the decoding network model is quantized based on the preset quantization configuration information.

[0186] In the above-mentioned second implementation, the weight distribution concentration of each network layer of the target network in the decoding network model is statistically analyzed, and the encoding and decoding process is performed through a certain amount of reference feature graphs, and the input feature distribution and / or output feature distribution of each network layer of the target network in the decoding network model are statistically analyzed to analyze the feature distribution concentration of each network layer in the target network, thereby determining the weight quantization accuracy of each network layer in the target network based on the weight distribution concentration of each network layer in the target network, and determining the activation quantization accuracy of each network layer in the target network based on the feature distribution concentration of each network layer in the target network. Since the weight distribution of the network layer, as well as the input feature distribution and / or output feature distribution can reflect the encoding and decoding performance of the network layer, the weight quantization accuracy determined based on the weight distribution of each network layer is more reliable, and the activation quantization accuracy determined based on the input feature distribution and / or output feature distribution of each network layer is more reliable. In this way, quantizing the target network in the decoding network model using the quantization accuracy determined by this implementation can effectively reduce the model volume of the decoding network model and speed up its operation efficiency on the hardware device.

[0187] The third implementation method is to perform quantization-aware training on the decoding network model based on multiple reference quantization configuration information to obtain multiple quantized decoding network models, and the multiple quantized decoding network models correspond one-to-one to the multiple reference quantization configuration information; based on multiple reference feature maps, the quantization configuration information of the decoding network model is determined from the multiple reference quantization configuration information through the multiple quantized decoding network models.

[0188] Among them, each reference quantization configuration information includes a reference quantization accuracy of a target network in a decoding network model, and the reference quantization accuracy includes a weight reference quantization accuracy and / or an activation reference quantization accuracy.

[0189] In some embodiments, multiple reference quantization configuration information are used in sequence to quantize the target network in the decoding network model, and quantization-aware training is performed on the quantized decoding network model to obtain the quantized decoding network model corresponding to each reference quantization configuration information.

[0190] In one possible implementation, the implementation process of determining the quantization configuration information of the decoding network model from multiple reference quantization configuration information can be: inputting multiple reference feature maps into each quantized decoding network model, obtaining the quantization evaluation results corresponding to each quantized decoding network model, and obtaining multiple quantization evaluation results, the quantization evaluation results including rate-distortion performance and / or running speed; based on the multiple quantization evaluation results, determining the quantization configuration information of the decoding network model from multiple reference quantization configuration information.

[0191] Among them, the quantitative evaluation result is used to describe the decoding performance of the quantized decoding network model when processing the reference feature map under the action of different reference quantization configuration information. The above-mentioned rate-distortion performance includes bit rate overhead and image reconstruction quality. The evaluation indicators describing the image reconstruction quality may include multiple-scale structural similarity (MSSSIM), visual quality fidelity (VIF), feature similarity (fsim), normalization laplacian pyramid distance (NLPD), information content weighted structural similarity (IW-SSIM), video multi-method assessment fusion (VMAF), color space (hue, saturation, value, referred to as HSV) peak signal-to-noise ratio (PSNR-HSV) and one or more, and the embodiments of the present application are not limited to this.

[0192] After determining the quantitative evaluation results of the quantized decoding network model corresponding to each reference quantization configuration information processing multiple reference feature maps, the reference quantization configuration information used by the quantized decoding network model with the best quantitative evaluation result can be determined as the quantization configuration information of the decoding network model. Of course, other selection strategies can also be used to determine one of the reference quantization configuration information as the quantization configuration information of the decoding network model based on the quantitative evaluation results of multiple quantized decoding network models, and the embodiments of the present application are not limited to this.

[0193] As an example, based on a 32-bit floating-point decoding network model, the activation quantization accuracy of a certain layer or multiple consecutive layers in the target network is first set to 8 bits or lower, and the activation quantization accuracy of the remaining layers is set to 16 bits. Then, based on multiple reference feature maps, the 32-bit floating-point decoding network model is trained with quantization awareness, and the quantization configuration information of the decoding network model is determined based on the rate-distortion performance of the quantized decoding network model and the acceleration benefit of the quantized decoding network model on hardware.

[0194] In the third implementation method described above, based on multiple reference quantization configuration information, multiple reference feature maps are used to perform quantization-aware training on the decoding network model that has been quantized based on the reference quantization configuration information to determine the activation quantization accuracy of each network layer in the target network. Since quantization-aware training uses quantization accuracy to simulate the quantization of the decoding network model, the quantization effect of the decoding network model in each reference quantization configuration information can be accurately obtained, and the selected quantization configuration information is more reliable. In this way, by quantizing the decoding network model based on the quantization configuration information determined by the perceptual training, the model volume of the decoding network model can be effectively reduced, and its operating efficiency on the hardware device can be accelerated.

[0195] Based on the quantization configuration information obtained by the above three implementation methods, for the decoding network model of the JPEG AI standard, the quantization accuracy of the target network can include the following two cases:

[0196] (1) When the target network includes a super decoding network, the weight quantization accuracy of each network layer in the super decoding network is the same and less than or equal to M bits, and the activation quantization accuracy of each network layer in the super decoding network is the same and less than or equal to N bits; when the target network includes a synthetic transform network, the weight quantization accuracy of each network layer in the synthetic transform network is the same and less than or equal to P bits, and the activation quantization accuracy of each network layer in the synthetic transform network is the same and less than or equal to Q bits.

[0197] In some embodiments, see Fig.11 The synthetic transformation network includes a residual module, a first deconvolution layer, a first cropping layer, a first residual activation unit, a second deconvolution layer, a second cropping layer, a second residual activation unit, a first convolution layer, a third residual activation unit, a second convolution layer, a pixel shifting layer, and a third cropping layer, which are connected in sequence. Fig.12 The super decoding network includes a third deconvolution layer, a fourth cropping layer, a first activation layer, a third convolution layer, a fourth deconvolution layer, a fifth cropping layer, a second activation layer, a fourth convolution layer, and a third activation layer, which are connected in sequence.

[0198] It should be understood that the “connected in sequence” here refers to a logical connection, not a physical connection. That is, for multiple network layers connected in sequence, the output of the previous network layer is the input of the next network layer. For the synthetic transformation network, the output tensor of the previous network layer (for example, the residual module) is used as the input tensor of the next network layer connected to it (that is, the first deconvolution layer), and the output tensor of the first deconvolution layer is used as the input tensor of the next network layer connected to it (that is, the first cropping layer). By analogy, the output of the synthetic transformation network is the output tensor of the last network layer (that is, the third cropping layer). Similarly, the output tensor of the first deconvolution layer in the super-decoding network is the input tensor of the fourth cropping layer, and by analogy, the output of the super-decoding network is the output tensor of the last network layer (that is, the third activation layer).

[0199] Among them, tensors are a kind of data container that can include multi-dimensional data. Vectors can be understood as one-dimensional tensors, and matrices can be understood as two-dimensional tensors. Since the amount of data processed by each network layer in a neural network is very large, for the sake of convenience, its calculation data can be represented by tensors.

[0200] It should be noted that, in actual application, the super-decoding network may also include a ninth convolutional layer, which may be independently deployed before the third deconvolutional layer or integrated into the third deconvolutional layer. The embodiment of the present application does not impose any restrictions on this.

[0201] In some embodiments, see Fig.13 , the residual module includes a fifth convolutional layer and a fourth activation layer connected in sequence, and the sum of the input tensor of the fifth convolutional layer and the output tensor of the fourth activation layer is the output of the residual module. In other words, for the residual module, its output is the sum of the input tensor of the residual module and the output tensor of the fourth activation layer.

[0202] In some embodiments, see Fig.14 , the residual activation unit includes an activation layer and a convolution layer connected in sequence, and the product of the input tensor of the activation layer and the output tensor of the convolution layer is the output of the residual activation unit. For the first residual activation unit, the second residual activation unit, and the third residual activation unit, the hierarchical structures in each residual activation unit are similar, but the parameters of the activation layer and the convolution layer in each residual activation unit may be different.

[0203] In a possible implementation, the first residual activation unit includes a fifth activation layer and a sixth convolutional layer connected in sequence, and the product of the input tensor of the fifth activation layer and the output tensor of the sixth convolutional layer is the output of the first residual activation unit; the second residual activation unit includes a sixth activation layer and a seventh convolutional layer connected in sequence, and the product of the input tensor of the sixth activation layer and the output tensor of the seventh convolutional layer is the output of the second residual activation unit; the third residual activation unit includes a seventh activation layer and an eighth convolutional layer connected in sequence, and the product of the input tensor of the seventh activation layer and the output tensor of the eighth convolutional layer is the output of the third residual activation unit.

[0204] Taking the first residual activation unit as an example, the input tensor of the first residual activation unit first enters the fifth activation layer for activation processing, and the output tensor of the fifth activation layer is used as the input tensor of the sixth convolutional layer. Finally, the product of the input tensor of the fifth activation layer (that is, the input tensor of the first residual activation unit) and the output tensor of the sixth convolutional layer is used as the output of the first residual activation unit.

[0205] It should be noted that although Fig.13 and Fig.14 The internal hierarchical structure of the residual module and the residual activation unit is given, but when the residual module and the residual activation unit are quantized, the quantization accuracy of each internal level is the same.

[0206] As an example, if the weight quantization accuracy of the residual module is P bits and the activation quantization accuracy is Q bits, it means that the weight quantization accuracy of the fifth convolutional layer inside the residual module is P bits, and the activation quantization accuracy is Q bits; the weight quantization accuracy of the fourth activation layer is also P bits, and the activation quantization accuracy is also Q bits.

[0207] In practical applications, each residual activation unit is also used to perform product operations and sum operations on the input tensor and the convolutional layer output tensor.

[0208] In a possible implementation, the summation operation can be integrated into the convolution layer. Taking the first residual activation unit as an example, after the sixth convolutional layer performs convolution processing on the output tensor of the fifth activation layer, it is also necessary to perform a point-to-point (i.e., data alignment in each dimension) summation operation on the input tensor of the fifth activation layer and the tensor after convolution processing, and the result of the summation operation is used as the output tensor of the sixth convolutional layer. Then, the output tensor of the sixth convolutional layer and the input tensor of the fifth activation layer are multiplied to obtain the final output of the first residual activation unit.

[0209] In another possible implementation, the summation operation can be performed separately. Taking the first residual activation unit as an example, after the sixth convolutional layer completes the convolution operation, a point-to-point (i.e., data alignment in each dimension) product operation is performed on the output tensor of the sixth convolutional layer and the input tensor of the fifth activation layer. Then, the tensor obtained after the product operation and the input tensor of the fifth activation layer are summed to finally obtain the output tensor of the first residual activation unit.

[0210] Based on the hierarchical structure of the above-mentioned synthetic transformation network, when the target network includes the synthetic transformation network, the synthetic transformation network is quantized by quantizing the number of weight bits of the residual module, the first deconvolution layer, the first cropping layer, the first activation unit, the second deconvolution layer, the second cropping layer, the second activation unit, the first convolution layer, the third activation unit, the second convolution layer, the pixel shifting layer and the third cropping layer to P bits, and the number of activation bits to Q bits.

[0211] Based on the hierarchical structure of the above-mentioned super-decoding network, when the target network includes the super-decoding network, the super-decoding network is quantized by quantizing the number of weight bits of the third deconvolution layer, the fourth cropping layer, the first activation layer, the fourth convolution layer, the fourth deconvolution layer, the fifth cropping layer, the second activation layer, the fifth convolution layer and the third activation layer to M bits, and the number of activation bits to N bits.

[0212] Assume that the decoding network model is a 32-bit floating-point model, and the target networks that need to be quantized in the decoding network model include the super-decoding network and the synthetic transformation network, the values ​​of M and P are both 8, and the values ​​of N and Q are both 16. That is, the weight quantization accuracy of each network layer in the super-decoding network is 8 bits, the activation quantization accuracy of each network layer in the super-decoding network is 16 bits, the weight quantization accuracy of each network layer in the synthetic transformation network is 8 bits, and the activation quantization accuracy of each network layer in the synthetic transformation network is 16 bits. The comparison of the 32-bit floating-point decoding network model before and after quantization is shown in Tables 3 and 4 below.

[0213] Among them, Table 3 shows the comparison of the BD-Rate of 50 standard test images processed by the decoding network model based on the JPEG AI standard before and after quantization at 4 bitrate points for the versatile video coding (VVC) coding standard; Table 4 shows the comparison of the BD-Rate of 50 standard test images processed by the decoding network model based on the JPEG AI standard before and after quantization at 5 bitrate points for the decoding of the VVC coding standard.

[0214] Table 3

[0215]

[0216] Table 4

[0217]

[0218] Among them, the average distortion rate is determined based on the values ​​of the above-mentioned MSSSIM, VIF, FSIM, IW-SSIM, VMAF and PSNR-HSV indicators for evaluating the image reconstruction quality, which can reflect the effect of the reconstructed image of the decoding network model before and after quantization compared with the VVC method.

[0219] It can be seen from Tables 3 and 4 that based on the quantization accuracy of the target network in the above example, the BD-Rate of the decoding network model based on the JPEG AI standard on 50 standard test images is only reduced by 0.2% after quantization (the difference in average distortion rate before and after quantization), and the volume of the target network in the decoding network model after quantization is reduced to 25% before quantization (the 32-bit floating-point decoding network model is quantized using 8-bit quantization accuracy, and the model volume quantization ratio is 32 / 8), and the decoding speed on hardware is faster than that of the floating-point decoding network model.

[0220] In this way, the unified quantization accuracy can take into account the characteristics of each network layer in the target network. When each network layer in the target network is quantized with a unified quantization accuracy, the overall quantization performance of the target network can be guaranteed.

[0221] (2) When the target network includes a super-decoding network, the weight quantization accuracy of each network layer in the super-decoding network is the same and less than or equal to M bits, and the activation quantization accuracy of each network layer in the super-decoding network is the same and less than or equal to N bits; when the target network includes a synthetic transform network, the weight quantization accuracy of each network layer in the synthetic transform network is the same and less than or equal to P bits, the activation quantization accuracy of at least one network layer in the synthetic transform network is the same and less than or equal to P bits, and the activation quantization accuracy of other network layers except at least one network layer in the synthetic transform network is the same and less than or equal to Q bits.

[0222] When the target network includes a super-decoding network, the super-decoding network is quantized by quantizing the number of weight bits of the third deconvolution layer, the fourth cropping layer, the first activation layer, the fourth convolution layer, the fourth deconvolution layer, the fifth cropping layer, the second activation layer, the fifth convolution layer, and the third activation layer to M bits, and the number of activation bits to N bits.

[0223] When the target network includes a synthetic transformation network, the synthetic transformation network is quantized, and the quantization methods may include but are not limited to the following:

[0224] The first quantization method: the weights of each network layer in the synthetic transformation network are quantized with P-bit precision, the activations of the second deconvolution layer and the second activation unit in the synthetic transformation network are quantized with P-bit precision, and the activations of the remaining network layers (the first deconvolution layer, the first cropping layer, the first activation unit, the second cropping layer, the first convolution layer, the third activation unit, the second convolution layer, the pixel shifting layer, and the third cropping layer) are quantized with Q-bit precision.

[0225] The second quantization method: the weights of each network layer in the synthetic transformation network are quantized with P-bit precision, the activations of the first convolutional layer and the third activation unit in the synthetic transformation network are quantized with P-bit precision, and the activations of the remaining network layers (residual module, first deconvolution layer, first cropping layer, first activation unit, second deconvolution layer, second cropping layer, second activation unit, second convolution layer, pixel shifting layer and third cropping layer) are quantized with Q-bit precision.

[0226] The third quantization method: The weights of each network layer in the synthetic transformation network are quantized with P-bit precision, the activations of the second deconvolution layer, the second activation unit, the first convolution layer and the third activation unit in the synthetic transformation network are quantized with P-bit precision, and the activations of the remaining network layers (residual module, first deconvolution layer, first cropping layer, first activation unit, second cropping layer, second convolution layer, pixel shifting layer and third cropping layer) are quantized with Q-bit precision.

[0227] The fourth quantization method: The weights of each network layer in the synthetic transformation network are quantized with P-bit precision. The activations of the first deconvolution layer, the first activation unit, the second deconvolution layer, the second activation unit, the first convolution layer and the third activation unit in the synthetic transformation network are quantized with P-bit precision. The activations of the remaining network layers (residual module, first cropping layer, second cropping layer, second convolution layer, pixel shifting layer and third cropping layer) are quantized with Q-bit precision.

[0228] It should be noted that when quantizing the calculated input and / or output of each network layer in the synthetic transformation network based on any of the above-mentioned quantization methods, it is also possible to quantize only the activation of one of the network layers. The embodiment of the present application does not limit the number of specific quantized network layers, which can be one layer or multiple layers.

[0229] Taking the first quantization method mentioned above as an example, in actual quantization, only the activation of the second deconvolution layer can be quantized with P-bit precision, and the activations of the remaining network layers can be quantized with Q-bit precision; or, only the activation of the second activation unit can be quantized with P-bit precision, and the activations of the remaining network layers can be quantized with Q-bit precision.

[0230] That is, when the activations of each network layer in the synthetic transformation network are quantized using mixed precision in the above four methods, only one of the network layers can be quantized using a lower P-bit precision, and the activations of the remaining network layers can be quantized using Q-bit precision.

[0231] Assume that the decoding network model is a 32-bit floating-point model, and the target networks that need to be quantized in the decoding network model include the super-decoding network and the synthetic transformation network, the values ​​of M and P are both 8, and the values ​​of N and Q are both 16. That is, the weight quantization accuracy of each network layer in the super-decoding network is 8 bits, the activation quantization accuracy of each network layer in the super-decoding network is 16 bits, the weight quantization accuracy of each network layer in the synthetic transformation network is 8 bits, the activation quantization accuracy of some network layers in the synthetic transformation network is 8 bits, and the activation quantization accuracy of some network layers is 16 bits. The comparison of the 32-bit floating-point decoding network model before and after quantization is shown in Tables 5 and 6 below.

[0232] Among them, Table 5 shows the comparison of the BD-Rate of 50 standard test images processed by the decoding network model based on the JPEG AI standard before and after quantization at 4 bitrate points compared with the VVC encoding standard; Table 6 shows the comparison of the BD-Rate of 50 standard test images processed by the decoding network model based on the JPEG AI standard before and after quantization at 5 bitrate points compared with the VVC encoding standard.

[0233] Table 5

[0234]

[0235] Table 6

[0236]

[0237] Among them, the average distortion rate is determined based on the values ​​of the above-mentioned MSSSIM, VIF, FSIM, IW-SSIM, VMAF and PSNR-HSV indicators for evaluating the image reconstruction quality, which can reflect the effect of the reconstructed image of the decoding network model before and after quantization compared with the VVC method.

[0238] It can be seen from Tables 5 and 6 that based on the quantization accuracy of the target network in the above example, the BD-Rate of the decoding network model based on the JPEG AI standard on 50 standard test images is only reduced by 0.65%-1.7% (the difference in average distortion rate before and after quantization), and the volume of the super decoding network and the synthetic transformation network after quantization is reduced to 25% before quantization (the 32-bit floating-point decoding network model is quantized using 8-bit quantization accuracy, and the model volume quantization ratio is 32 / 8), and the decoding speed on hardware is faster than that of the floating-point decoding network model.

[0239] In this way, the corresponding weight quantization accuracy and / or activation quantization accuracy can be set according to the characteristics of each network layer in the target network. Each network layer in the target network is quantized using mixed quantization accuracy, making the quantization of the target network more personalized.

[0240] Step 1002: Based on the quantization configuration information, a quantization-aware training process is performed to obtain quantization parameters of the target network.

[0241] Among them, the quantization parameters of the target network may include an offset (zero_point) and a scaling factor (scale), the offset represents the quantized fixed-point value corresponding to the floating-point value of 0, and the scaling factor indicates the minimum scale that can be represented after fixed-point quantization. Therefore, based on the weight quantization accuracy of each network layer in the target network, by executing a quantization-aware training process, the weight offset and weight scaling factor of each network layer when quantized according to the corresponding weight quantization accuracy can be determined. Similarly, based on the activation quantization accuracy of each network layer in the target network, by executing a quantization-aware training process, the activation offset and activation scaling factor of each network layer when quantizing the calculated input and / or output according to the corresponding activation quantization accuracy can be determined.

[0242] In other words, for each network layer in the target network, after determining the weight quantization accuracy and activation quantization accuracy of each network layer based on the quantization configuration information, the weight offset and weight scaling factor of each network layer when quantizing the weights, as well as the activation offset and activation scaling factor of each network layer when quantizing the input and / or output can be determined through the quantization-aware training process.

[0243] In a possible implementation, the quantization-aware training process may include: for an unquantized decoding network model, statistically analyzing the weight distribution of multiple neurons in each network layer of the target network in the decoding network model; based on the weight distribution of multiple neurons in each network layer of the target network, determining the weight scaling coefficient of each network layer in the target network at the corresponding weight quantization accuracy. At the same time, based on the weight quantization accuracy of each network layer in the target network, determining the weight offset corresponding to each network layer in the target network.

[0244] Similarly, multiple reference feature maps are input into the decoding network model, and the input feature distribution and / or output feature distribution of each network layer in the target network are counted; based on the input feature distribution and / or output feature distribution of each network layer, the activation scaling factor of each network layer in the target network under the corresponding activation quantization accuracy is determined. At the same time, based on the activation quantization accuracy of each network layer in the target network, the activation offset of each network layer in the target network is determined.

[0245] As an example, if the weight quantization precision of each network layer in the target network defined in the quantization configuration information is 8 bits, the number of bits of the quantized weights of each network layer in the target network should be 2 s , that is, 256 bits. In the symmetric case, the number of weight bits after quantization is 0bit to 255bit; in the asymmetric case, the number of weight bits after quantization is -127bit to +128bit. When calculating the weight scaling factor, the weight difference between the maximum weight and the minimum weight can be determined based on the weight distribution of multiple neurons in each network layer in the target network, and then the weight scaling factor of each network layer can be determined based on the ratio between the weight difference and 256.

[0246] It should be noted that the above-mentioned maximum weight value and minimum weight value can be the maximum weight value and minimum weight value of multiple neurons in the corresponding network layer; they can also be the maximum weight value and minimum weight value determined from the remaining weight values ​​after removing some weight values ​​greater than the weight upper limit threshold and some weight values ​​less than the weight lower limit threshold; they can also be the maximum weight value and minimum weight value selected from the weight values ​​of multiple neurons in the corresponding network layer based on the 3-sigma rule, and the embodiments of the present application do not impose any restrictions on this.

[0247] Further, based on the weight offset value, weight scaling factor, activation offset value and activation scaling factor of each network layer in the target network calculated by the statistical distribution method, other networks in the decoding network model (e.g., hypervariance decoding network, gain unit, all encoding networks, etc.) can also be used to train the quantized target network, thereby determining the final quantization parameter according to the training result of the target network. The embodiment of the present application is not limited to this.

[0248] Step 1003: quantize the target network in the decoding network model based on the quantization parameters of the target network to obtain a quantized decoding network model.

[0249] Referring to the above formula (1), based on the floating-point weight floating-point values ​​of each network layer in the target network in the floating-point decoding network model, and the weight offset and weight scaling coefficient determined based on the weight quantization accuracy, the quantized weight fixed-point values ​​of each network layer in the target network are calculated.

[0250] Similarly, referring to the above formula (1), based on the activation floating-point values ​​of each network layer in the target network in the floating-point decoding network, and the activation offset value and activation scaling factor determined based on the activation quantization accuracy, the quantized activation weight fixed-point value of each network layer in the target network is calculated.

[0251] In some embodiments, based on the quantization parameters of the target network, PTQ can be used to quantize the target network in the decoding network model, or QAT can be used to quantize the target network in the decoding network model. Of course, other methods can also be used to quantize the target network in the decoding network model, and the embodiments of the present application are not limited to this.

[0252] In an embodiment of the present application, when the decoding network model is quantized, the quantization configuration information of the unquantized decoding network model is first obtained, and the quantization configuration information indicates the quantization accuracy of each network layer in the target network in the decoding network model, and the target network includes a super decoding network and / or a synthetic transformation network. Then, based on the quantization configuration information, a quantization-aware training process is performed to obtain the quantization parameters of the target network, and then the target network in the decoding network model is quantized according to the quantization parameters of the target network to obtain a quantized decoding network model. Since quantizing the target network can reduce the volume of the target network and improve the efficiency of the target network in processing data, by quantizing the target network, the model volume of the decoding network model can be greatly reduced, the memory consumption during the model operation can be reduced, and the running speed of the decoding network model on the hardware can be improved.

[0253] Based on the quantization method of the above decoding network model, the target network in the decoding network model can be quantized to obtain a quantized decoding network model, so that the quantized decoding network model can be deployed on terminal devices with limited resources such as mobile phones and robots.

[0254] Based on this, the embodiment of the present application also provides a decoding network model, which is a decoding network model for the above Figure 7 and Fig. 9 The target network in the decoding network model based on the JPEG AI standard is quantized as shown.

[0255] It should be understood that quantization does not change the hierarchical architecture of the decoding network model before and after, but it will reduce the number of weight bits and activation bits in each network layer of the target network in the decoding network model, thereby compressing the volume of the target network in the decoding network model and improving the calculation speed of the target network. Therefore, the decoding network model containing the quantized target network has a smaller model size, consumes less memory during model operation, and improves the running speed of the decoding network model on hardware.

[0256] In one embodiment, the decoding network model includes a target network, the target network is a quantized network, and the target network includes a super decoding network and / or a synthetic transformation network.

[0257] Among them, when the target network includes a super-decoding network, the quantization accuracy of each network layer in the super-decoding network is the same; when the target network includes a synthetic transformation network, the quantization accuracy of at least some network layers in the synthetic transformation network is the same.

[0258] Optionally, the quantization accuracy includes weight quantization accuracy and / or activation quantization accuracy; the super-decoding network includes multiple network layers, the weight quantization accuracy of each network layer in the super-decoding network is the same and less than or equal to M bits, the activation quantization accuracy of each network layer in the super-decoding network is the same and less than or equal to N bits, M is less than N, and M and N are both less than the accuracy of the super-decoding network before quantization; the synthetic transformation network includes multiple network layers, the weight quantization accuracy of each network layer in the synthetic transformation network is the same and less than or equal to P bits, the activation quantization accuracy of each network layer in the synthetic transformation network is the same and less than or equal to Q bits, P is less than Q, and P and Q are both less than the accuracy of the synthetic transformation network before quantization.

[0259] Optionally, the synthetic transformation network includes a residual module, a first deconvolution layer, a first cropping layer, a first residual activation unit, a second deconvolution layer, a second cropping layer, a second residual activation unit, a first convolution layer, a third residual activation unit, a second convolution layer, a pixel shifting layer and a third cropping layer connected in sequence; the super decoding network includes a third deconvolution layer, a fourth cropping layer, a first activation layer, a third convolution layer, a fourth deconvolution layer, a fifth cropping layer, a second activation layer, a fourth convolution layer and a third activation layer connected in sequence.

[0260] Optionally, the residual module includes a fifth convolutional layer and a fourth activation layer connected in sequence, and the sum of the input tensor of the fifth convolutional layer and the output tensor of the fourth activation layer is the output of the residual module.

[0261] Optionally, the first residual activation unit includes a fifth activation layer and a sixth convolutional layer connected in sequence, and the product of the input tensor of the fifth activation layer and the output tensor of the sixth convolutional layer is the output of the first residual activation unit; the second residual activation unit includes a sixth activation layer and a seventh convolutional layer connected in sequence, and the product of the input tensor of the sixth activation layer and the output tensor of the seventh convolutional layer is the output of the second residual activation unit; the third residual activation unit includes a seventh activation layer and an eighth convolutional layer connected in sequence, and the product of the input tensor of the seventh activation layer and the output tensor of the eighth convolutional layer is the output of the third residual activation unit.

[0262] Optionally, the quantization accuracy includes weight quantization accuracy and / or activation quantization accuracy; the super-decoding network includes multiple network layers, the weight quantization accuracy of each network layer in the super-decoding network is the same and less than or equal to M bits, the activation quantization accuracy of each network layer in the super-decoding network is the same and less than or equal to N bits, M is less than N, and M and N are both less than the accuracy of the super-decoding network before quantization; the synthetic transformation network includes multiple network layers, the weight quantization accuracy of each network layer in the synthetic transformation network is the same and less than or equal to P bits, the activation quantization accuracy of at least one network layer in the synthetic transformation network is the same and less than or equal to P bits, the activation quantization accuracy of other network layers except at least one network layer in the synthetic transformation network is the same and less than or equal to Q bits, P is less than Q, and P and Q are both less than the accuracy of the synthetic transformation network before quantization.

[0263] Optionally, the synthetic transformation network includes a residual module, a first deconvolution layer, a first cropping layer, a first residual activation unit, a second deconvolution layer, a second cropping layer, a second residual activation unit, a first convolution layer, a third residual activation unit, a second convolution layer, a pixel shifting layer and a third cropping layer connected in sequence; the super decoding network includes a third deconvolution layer, a fourth cropping layer, a first activation layer, a third convolution layer, a fourth deconvolution layer, a fifth cropping layer, a second activation layer, a fourth convolution layer and a third activation layer connected in sequence.

[0264] Optionally, the residual module includes a fifth convolutional layer and a fourth activation layer connected in sequence, and the sum of the input tensor of the fifth convolutional layer and the output tensor of the fourth activation layer is the output of the residual module.

[0265] Optionally, the first residual activation unit includes a fifth activation layer and a sixth convolutional layer connected in sequence, and the product of the input tensor of the fifth activation layer and the output tensor of the sixth convolutional layer is the output of the first residual activation unit; the second residual activation unit includes a sixth activation layer and a seventh convolutional layer connected in sequence, and the product of the input tensor of the sixth activation layer and the output tensor of the seventh convolutional layer is the output of the second residual activation unit; the third residual activation unit includes a seventh activation layer and an eighth convolutional layer connected in sequence, and the product of the input tensor of the seventh activation layer and the output tensor of the eighth convolutional layer is the output of the third residual activation unit.

[0266] Optionally, at least one network layer in the synthetic transformation network whose activation quantization precision is the same and less than or equal to P bits includes but is not limited to:

[0267] A second deconvolution layer and a second residual activation unit; or,

[0268] The first convolutional layer and the third residual activation unit; or,

[0269] a second deconvolution layer, a second residual activation unit, a first convolution layer, and a third residual activation unit; or,

[0270] The first deconvolution layer, the first residual activation unit, the second deconvolution layer, the second residual activation unit, the first convolution layer, and the third residual activation unit.

[0271] Optionally, the values ​​of M and P are both 8, and the values ​​of N and Q are both 16.

[0272] Optionally, the decoding network model includes a synthetic transform network, a super decoding network and a super variance decoding network, and the super decoding network and the super variance decoding network are both connected to the synthetic transform network.

[0273] It should be noted that, with the development of model quantization technology, the weights and activations of the supervariance decoding network in the super decoding network model can also be quantized to further reduce the size of the decoding network model, reduce the memory consumption during the model operation, and improve the operation speed of the decoding network model on the hardware. The embodiments of the present application are not limited to this.

[0274] Optionally, the decoding network model includes two synthetic transform networks, two super decoding networks, two super variance decoding networks and an inter-channel correction information filtering network; wherein each synthetic transform network is respectively connected to a super decoding network and a super variance decoding network, and the two synthetic transform networks are both connected to the inter-channel correction information filtering network.

[0275] It should be noted that, with the development of model quantization technology, the weights and activations of the inter-channel correction information filter network in the super decoding network model can also be quantized to further reduce the size of the decoding network model, reduce the memory consumption during the model operation, and improve the operation speed of the decoding network model on the hardware. The embodiments of the present application are not limited to this.

[0276] Referring to the quantization method of the above-mentioned decoding network model, the same or similar model quantization logic can also be used to quantize the encoding network model to reduce the model volume of the encoding network model and improve the computational efficiency of the model when processing data.

[0277] See also Fig.15 , Fig.15 is a flowchart of a quantization method for a coding network model provided in an embodiment of the present application, which can be applied to Figure 3 In the computer device shown, the method includes the following steps.

[0278] Step 1501: Obtain quantization configuration information of an unquantized coding network model, where the quantization configuration information indicates the quantization accuracy of each network layer in a target network in the coding network model, where the target network includes at least one of a super coding network, an analysis transformation network, and a super decoding network.

[0279] In some embodiments, the quantization accuracy of the target network includes the weight quantization accuracy and / or activation quantization accuracy of each network layer in the target network.

[0280] That is, when the coding network model is quantized, the quantization configuration information may include the weight quantization accuracy and / or activation quantization accuracy of each network layer in the super-coding network, and may also include the weight quantization accuracy and / or activation quantization accuracy of each network layer in the analysis transformation network. The embodiment of the present application does not limit the number of quantization configuration objects (i.e., target networks) contained in the quantization configuration information, and the number of quantization configuration items (i.e., weight quantization accuracy and / or activation quantization accuracy).

[0281] Referring to the above-mentioned method for setting the quantization configuration information of the decoding network model, determining the quantization configuration information of the encoding network model includes the following three implementation methods:

[0282] In a first implementation manner, preset quantization configuration information is determined as quantization configuration information of the encoding network model.

[0283] The second implementation method is to obtain the characteristic distribution of each network layer in the target network; based on the characteristic distribution of each network layer in the target network, determine the quantization configuration information of the encoding network model.

[0284] In some embodiments, the quantization accuracy of the target network includes the weight quantization accuracy, and the feature distribution includes the weight distribution of multiple neurons in the corresponding network layer.

[0285] In some embodiments, the quantization accuracy of the target network includes the activation quantization accuracy, and the feature distribution includes the input feature distribution and / or output feature distribution of the corresponding network layer. Then, the implementation process of obtaining the feature distribution of each network layer in the target network can be: inputting multiple reference feature maps into the encoding network model, obtaining multiple input feature maps and multiple output feature maps corresponding to each network layer in the target network, and the multiple input feature maps and multiple output feature maps are respectively compared with the multiple reference feature maps. Figure 1 One-to-one correspondence; based on multiple input feature maps corresponding to each network layer, statistics are made on the input feature distribution of each network layer, and / or, based on multiple output feature maps corresponding to each network layer, statistics are made on the output feature distribution of each network layer.

[0286] The third implementation method is to perform quantization perception training on the coding network model based on multiple reference quantization configuration information to obtain multiple quantized coding network models, and the multiple quantized coding network models correspond one to one to the multiple reference quantization configuration information; based on multiple reference feature maps, the quantization configuration information of the coding network model is determined from the multiple reference quantization configuration information through the multiple quantized coding network models.

[0287] In one possible implementation, multiple reference feature maps are input into each quantized decoding network model, and a quantization evaluation result corresponding to each quantized decoding network model is obtained to obtain multiple quantization evaluation results, which include rate-distortion performance and / or running speed; based on the multiple quantization evaluation results, the quantization configuration information of the encoding network model is determined from multiple reference quantization configuration information.

[0288] It should be noted that, based on the above-determined quantization configuration information, a unified quantization accuracy can be used to quantize the weights and / or activations of each network layer in the target network, or a mixed quantization accuracy can be used to quantize the activations of each network layer in the target network, and the embodiments of the present application do not limit this. In addition, for the detailed implementation process of the above three implementation methods, please refer to the determination process of the quantization configuration information of the decoding network model, which will not be repeated here.

[0289] Step 1502: Based on the quantization configuration information, a quantization-aware training process is performed to obtain quantization parameters of the target network.

[0290] In some embodiments, the quantization parameters of the target network may include an offset (zero_point) and a scaling factor (scale), wherein the offset represents a quantized fixed-point value corresponding to a 0 floating-point value, and the scaling factor indicates a minimum scale that can be represented after fixed-point quantization.

[0291] It should be noted that the quantization parameter can be determined by statistical distribution or by quantization-aware training, which is not limited in the present embodiment. The specific implementation process can be found in the explanation of step 1002 above, which will not be repeated here.

[0292] Step 1503: quantize the target network in the coding network model based on the quantization parameters of the target network to obtain a quantized coding network model.

[0293] Referring to the above formula (1), based on the floating-point weight floating-point values ​​of each network layer in the target network in the floating-point encoding network model and the weight quantization accuracy in the quantization configuration information, the quantized weight fixed-point values ​​of each network layer in the target network are calculated.

[0294] Similarly, referring to the above formula (1), based on the activation floating-point values ​​of each network layer in the target network in the floating-point encoding network and the activation quantization accuracy in the quantization configuration information, the quantized activation weight fixed-point values ​​of each network layer in the target network are calculated.

[0295] In some embodiments, based on the quantization parameters of the target network, PTQ can be used to quantize the target network in the coding network model, or QAT can be used to quantize the target network in the coding network model. Of course, other methods can also be used to quantize the target network in the coding network model, and the embodiments of the present application are not limited to this.

[0296] In an embodiment of the present application, when quantizing the coding network model, the quantization configuration information of the unquantized coding network model is first obtained, and the quantization configuration information indicates the quantization accuracy of each network layer in the target network in the coding network model, wherein the target network includes at least one of a super-coding network, an analysis transformation network, and a super-decoding network. Then, based on the quantization configuration information, a quantization-aware training process is performed to obtain the quantization parameters of the target network, and then the target network in the coding network model is quantized according to the quantization parameters of the target network to obtain a quantized coding network model. Since quantizing the target network can reduce the volume of the target network and improve the efficiency of the target network in processing data, by quantizing the target network, the model volume of the coding network model can be greatly reduced, the memory consumption during the model operation can be reduced, and the running speed of the coding network model on the hardware can be improved.

[0297] Furthermore, based on the quantization method of the above-mentioned coding network model, the target network in the coding network model can be quantized to obtain a quantized coding network model, so that the quantized coding network model can be deployed on terminal devices with limited resources such as mobile phones and robots.

[0298] Based on this, the quantization method of the above coding network model can be used to Figure 6 and Figure 8 The target network in the coding network model based on the JPEG AI standard is quantized to obtain a quantized coding network model.

[0299] That is, the hierarchical structure of the encoding network model will not be changed before and after quantization, but the number of weight bits and activation bits in each network layer of the target network in the encoding network model will be reduced, thereby compressing the volume of the target network in the encoding network model, reducing the complexity of its data processing, and improving the efficiency of data processing by the target network in the encoding network model.

[0300] Fig.16is a schematic diagram of the structure of a quantization device for a decoding network model provided in an embodiment of the present application. The quantization device for a decoding network model can be implemented by software, hardware, or a combination of both to form part or all of a terminal. Fig.16 The quantization device 1600 of the decoding network model includes: a configuration acquisition module 1601, a quantization training module 1602 and a model quantization module 1603.

[0301] The configuration acquisition module 1601 is used to acquire quantization configuration information of an unquantized decoding network model, where the quantization configuration information indicates the quantization accuracy of each network layer in a target network in the decoding network model, where the target network includes a super decoding network and / or a synthetic transformation network;

[0302] Wherein, when the target network includes a super decoding network, the quantization accuracy of each network layer in the super decoding network is the same; when the target network includes a synthetic transformation network, the quantization accuracy of at least some network layers in the synthetic transformation network is the same;

[0303] A quantization training module 1602 is used to perform a quantization-aware training process based on the quantization configuration information to obtain quantization parameters of the target network;

[0304] The model quantization module 1603 is used to quantize the target network in the decoding network model based on the quantization parameters of the target network to obtain a quantized decoding network model.

[0305] Optionally, the quantization accuracy includes weight quantization accuracy and / or activation quantization accuracy;

[0306] The super decoding network includes multiple network layers, the weight quantization precision of each network layer in the super decoding network is the same and is less than or equal to M bits, the activation quantization precision of each network layer in the super decoding network is the same and is less than or equal to N bits, M is less than N, and both M and N are less than the precision of the super decoding network before quantization;

[0307] The synthetic transform network includes multiple network layers, the weight quantization accuracy of each network layer in the synthetic transform network is the same and is less than or equal to P bits, the activation quantization accuracy of each network layer in the synthetic transform network is the same and is less than or equal to Q bits, P is less than Q, and both P and Q are less than the accuracy of the synthetic transform network before quantization.

[0308] Optionally, the synthetic transformation network includes a residual module, a first deconvolution layer, a first cropping layer, a first residual activation unit, a second deconvolution layer, a second cropping layer, a second residual activation unit, a first convolution layer, a third residual activation unit, a second convolution layer, a pixel shifting layer and a third cropping layer connected in sequence;

[0309] The super decoding network includes a third deconvolution layer, a fourth cropping layer, a first activation layer, a third convolution layer, a fourth deconvolution layer, a fifth cropping layer, a second activation layer, a fourth convolution layer and a third activation layer, which are sequentially connected.

[0310] Optionally, the residual module includes a fifth convolutional layer and a fourth activation layer connected in sequence, and the sum of the input tensor of the fifth convolutional layer and the output tensor of the fourth activation layer is the output of the residual module.

[0311] Optionally, the first residual activation unit includes a fifth activation layer and a sixth convolutional layer connected in sequence, and the product of the input tensor of the fifth activation layer and the output tensor of the sixth convolutional layer is the output of the first residual activation unit; the second residual activation unit includes a sixth activation layer and a seventh convolutional layer connected in sequence, and the product of the input tensor of the sixth activation layer and the output tensor of the seventh convolutional layer is the output of the second residual activation unit; the third residual activation unit includes a seventh activation layer and an eighth convolutional layer connected in sequence, and the product of the input tensor of the seventh activation layer and the output tensor of the eighth convolutional layer is the output of the third residual activation unit.

[0312] Optionally, the quantization accuracy includes weight quantization accuracy and / or activation quantization accuracy;

[0313] The super decoding network includes multiple network layers, the weight quantization accuracy of each network layer in the super decoding network is the same and is less than or equal to M bits, the activation quantization accuracy of each network layer in the super decoding network is the same and is less than or equal to N bits, M is less than N, and both M and N are less than the accuracy of the super decoding network before quantization;

[0314] The synthetic transform network includes multiple network layers, the weight quantization accuracy of each network layer in the synthetic transform network is the same and less than or equal to P bits, the activation quantization accuracy of at least one network layer in the synthetic transform network is the same and less than or equal to P bits, the activation quantization accuracy of other network layers except at least one network layer in the synthetic transform network is the same and less than or equal to Q bits, P is less than Q, and both P and Q are less than the accuracy of the synthetic transform network before quantization.

[0315] Optionally, the synthetic transformation network includes a residual module, a first deconvolution layer, a first cropping layer, a first residual activation unit, a second deconvolution layer, a second cropping layer, a second residual activation unit, a first convolution layer, a third residual activation unit, a second convolution layer, a pixel shifting layer and a third cropping layer connected in sequence;

[0316] The super decoding network includes a third deconvolution layer, a fourth cropping layer, a first activation layer, a third convolution layer, a fourth deconvolution layer, a fifth cropping layer, a second activation layer, a fourth convolution layer and a third activation layer, which are sequentially connected.

[0317] Optionally, the residual module includes a fifth convolutional layer and a fourth activation layer connected in sequence, and the sum of the input tensor of the fifth convolutional layer and the output tensor of the fourth activation layer is the output of the residual module.

[0318] Optionally, the first residual activation unit includes a fifth activation layer and a sixth convolutional layer connected in sequence, and the product of the input tensor of the fifth activation layer and the output tensor of the sixth convolutional layer is the output of the first residual activation unit; the second residual activation unit includes a sixth activation layer and a seventh convolutional layer connected in sequence, and the product of the input tensor of the sixth activation layer and the output tensor of the seventh convolutional layer is the output of the second residual activation unit; the third residual activation unit includes a seventh activation layer and an eighth convolutional layer connected in sequence, and the product of the input tensor of the seventh activation layer and the output tensor of the eighth convolutional layer is the output of the third residual activation unit.

[0319] Optionally, at least one network layer in the synthetic transformation network whose activation quantization precision is the same and less than or equal to P bits includes but is not limited to:

[0320] A second deconvolution layer and a second residual activation unit; or,

[0321] The first convolutional layer and the third residual activation unit; or,

[0322] a second deconvolution layer, a second residual activation unit, a first convolution layer, and a third residual activation unit; or,

[0323] The first deconvolution layer, the first residual activation unit, the second deconvolution layer, the second residual activation unit, the first convolution layer, and the third residual activation unit.

[0324] Optionally, the values ​​of M and P are both 8, and the values ​​of N and Q are both 16.

[0325] Optionally, the decoding network model includes a synthetic transform network, a super decoding network and a super variance decoding network, and both the super decoding network and the super variance decoding network are connected to the synthetic transform network.

[0326] Optionally, the decoding network model includes two synthetic transformation networks, two super decoding networks, two super variance decoding networks and an inter-channel correction information filtering network;

[0327] Among them, each synthetic transformation network is connected to a super decoding network and a super variance decoding network respectively, and the two synthetic transformation networks are connected to the inter-channel correction information filtering network.

[0328] In an embodiment of the present application, when the decoding network model is quantized, the quantization device of the decoding network model first obtains the quantization configuration information of the unquantized decoding network model, and the quantization configuration information indicates the quantization accuracy of each network layer in the target network in the decoding network model, and the target network includes a super decoding network and / or a synthetic transformation network. Then, based on the quantization configuration information, a quantization-aware training process is performed to obtain the quantization parameters of the target network, and then the target network in the decoding network model is quantized according to the quantization parameters of the target network to obtain a quantized decoding network model. Since quantizing the target network can reduce the volume of the target network and improve the efficiency of the target network in processing data, by quantizing the target network, the model volume of the decoding network model can be greatly reduced, the memory consumption during the model operation can be reduced, and the running speed of the decoding network model on the hardware can be improved.

[0329] It should be noted that: when the quantization device of the decoding network model provided in the above embodiment quantizes the target network in the unquantized decoding network model, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the quantization device of the decoding network model provided in the above embodiment and the quantization method embodiment of the decoding network model belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0330] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer device, the computer device executes the steps of the quantization method of the decoding network model shown in the above embodiment.

[0331] An embodiment of the present application also provides a computer program product containing instructions, which includes a computer program. When the computer program is run on a computer device, the computer device executes the steps of the quantization method of the decoding network model shown in the above embodiment.

[0332] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on the computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (for example: coaxial cable, optical fiber, data user line (digital subscriber line, DSL)) or wireless (for example: infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that can be accessed by the computer, or a data storage device such as a server or data center that contains one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc. It is worth noting that the computer-readable storage medium mentioned in the embodiment of the present application may be a non-volatile storage medium, in other words, a non-transient storage medium.

[0333] It should be understood that the "multiple" mentioned herein refers to two or more. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not limit them to be necessarily different.

[0334] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of the present application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0335] The above are embodiments provided for the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A decoding network model, characterized in that: The decoding network model includes a target network, the target network is a quantized network, and the target network includes a super decoding network and / or a synthetic transformation network; When the target network includes the super-decoding network, the quantization precision of each network layer in the super-decoding network is the same; when the target network includes the synthetic transformation network, the quantization precision of at least some network layers in the synthetic transformation network is the same.

2. The decoding network model according to claim 1, characterized in that The quantization accuracy includes weight quantization accuracy and / or activation quantization accuracy; The super decoding network includes multiple network layers, the weight quantization precision of each network layer in the super decoding network is the same and is less than or equal to M bits, the activation quantization precision of each network layer in the super decoding network is the same and is less than or equal to N bits, the M is less than the N, and the M and the N are both less than the precision of the super decoding network before quantization; The synthetic transformation network includes multiple network layers, the weight quantization accuracy of each network layer in the synthetic transformation network is the same and is less than or equal to P bits, the activation quantization accuracy of each network layer in the synthetic transformation network is the same and is less than or equal to Q bits, the P is less than the Q, and both the P and the Q are less than the accuracy of the synthetic transformation network before quantization.

3. The super decoding network model according to claim 2, characterized in that: The synthetic transformation network includes a residual module, a first deconvolution layer, a first cropping layer, a first residual activation unit, a second deconvolution layer, a second cropping layer, a second residual activation unit, a first convolution layer, a third residual activation unit, a second convolution layer, a pixel shifting layer and a third cropping layer connected in sequence; The super decoding network includes a third deconvolution layer, a fourth cropping layer, a first activation layer, a third convolution layer, a fourth deconvolution layer, a fifth cropping layer, a second activation layer, a fourth convolution layer and a third activation layer which are connected in sequence.

4. The super decoding network model according to claim 3, characterized in that: The residual module includes a fifth convolutional layer and a fourth activation layer connected in sequence, and the sum of an input tensor of the fifth convolutional layer and an output tensor of the fourth activation layer is the output of the residual module.

5. The super decoding network model according to claim 3, characterized in that: The first residual activation unit includes a fifth activation layer and a sixth convolutional layer connected in sequence, and the product of the input tensor of the fifth activation layer and the output tensor of the sixth convolutional layer is the output of the first residual activation unit; the second residual activation unit includes a sixth activation layer and a seventh convolutional layer connected in sequence, and the product of the input tensor of the sixth activation layer and the output tensor of the seventh convolutional layer is the output of the second residual activation unit; the third residual activation unit includes a seventh activation layer and an eighth convolutional layer connected in sequence, and the product of the input tensor of the seventh activation layer and the output tensor of the eighth convolutional layer is the output of the third residual activation unit.

6. The decoding network model according to claim 1, characterized in that: The quantization accuracy includes weight quantization accuracy and / or activation quantization accuracy; The super decoding network includes multiple network layers, the weight quantization precision of each network layer in the super decoding network is the same and is less than or equal to M bits, the activation quantization precision of each network layer in the super decoding network is the same and is less than or equal to N bits, the M is less than the N, and the M and the N are both less than the precision of the super decoding network before quantization; The synthetic transformation network includes multiple network layers, the weight quantization accuracy of each network layer in the synthetic transformation network is the same and less than or equal to P bits, the activation quantization accuracy of at least one network layer in the synthetic transformation network is the same and less than or equal to P bits, the activation quantization accuracy of other network layers in the synthetic transformation network except the at least one network layer is the same and less than or equal to Q bits, the P is less than the Q, and both the P and the Q are less than the accuracy of the synthetic transformation network before quantization.

7. The super decoding network model according to claim 6, characterized in that: The synthetic transformation network includes a residual module, a first deconvolution layer, a first cropping layer, a first residual activation unit, a second deconvolution layer, a second cropping layer, a second residual activation unit, a first convolution layer, a third residual activation unit, a second convolution layer, a pixel shifting layer and a third cropping layer connected in sequence; The super decoding network includes a third deconvolution layer, a fourth cropping layer, a first activation layer, a third convolution layer, a fourth deconvolution layer, a fifth cropping layer, a second activation layer, a fourth convolution layer and a third activation layer which are connected in sequence.

8. The super decoding network model according to claim 7, characterized in that: The residual module includes a fifth convolutional layer and a fourth activation layer connected in sequence, and the sum of an input tensor of the fifth convolutional layer and an output tensor of the fourth activation layer is the output of the residual module.

9. The super decoding network model according to claim 7, characterized in that: The first residual activation unit includes a fifth activation layer and a sixth convolutional layer connected in sequence, and the product of the input tensor of the fifth activation layer and the output tensor of the sixth convolutional layer is the output of the first residual activation unit; the second residual activation unit includes a sixth activation layer and a seventh convolutional layer connected in sequence, and the product of the input tensor of the sixth activation layer and the output tensor of the seventh convolutional layer is the output of the second residual activation unit; the third residual activation unit includes a seventh activation layer and an eighth convolutional layer connected in sequence, and the product of the input tensor of the seventh activation layer and the output tensor of the eighth convolutional layer is the output of the third residual activation unit.

10. The decoding network model according to any one of claims 7 to 9, characterized in that: At least one network layer in the synthetic transform network whose activation quantization precision is the same and less than or equal to P bits includes but is not limited to: the second deconvolution layer and the second residual activation unit; or, The first convolutional layer and the third residual activation unit; or, The second deconvolution layer, the second residual activation unit, the first convolution layer and the third residual activation unit; or, The first deconvolution layer, the first residual activation unit, the second deconvolution layer, the second residual activation unit, the first convolution layer and the third residual activation unit.

11. The decoding network model according to claim 2 or 6, characterized in that: The values ​​of M and P are both 8, and the values ​​of N and Q are both 16.

12. The decoding network model according to any one of claims 1 to 11, characterized in that: The decoding network model includes the synthetic transform network, the super decoding network and the super variance decoding network, and both the super decoding network and the super variance decoding network are connected to the synthetic transform network.

13. The decoding network model according to any one of claims 1 to 11, characterized in that: The decoding network model includes two synthetic transformation networks, two super decoding networks, two super variance decoding networks and an inter-channel correction information filtering network; Each of the synthetic transform networks is connected to one of the super decoding networks and one of the super variance decoding networks respectively, and both of the synthetic transform networks are connected to the inter-channel correction information filtering network.

14. A quantization method for a decoding network model, characterized in that: The method comprises: Acquire quantization configuration information of an unquantized decoding network model, wherein the quantization configuration information indicates quantization accuracy of each network layer in a target network in the decoding network model, wherein the target network includes a super decoding network and / or a synthetic transformation network; Wherein, when the target network includes the super decoding network, the quantization precision of each network layer in the super decoding network is the same; when the target network includes the synthetic transformation network, the quantization precision of at least some network layers in the synthetic transformation network is the same; Based on the quantization configuration information, a quantization-aware training process is performed to obtain quantization parameters of the target network; Based on the quantization parameters of the target network, the target network in the decoding network model is quantized to obtain a quantized decoding network model.

15. The method according to claim 14, characterized in that The quantization accuracy includes weight quantization accuracy and / or activation quantization accuracy; The super decoding network includes multiple network layers, the weight quantization precision of each network layer in the super decoding network is the same and is less than or equal to M bits, the activation quantization precision of each network layer in the super decoding network is the same and is less than or equal to N bits, the M is less than the N, and the M and the N are both less than the precision of the super decoding network before quantization; The synthetic transformation network includes multiple network layers, the weight quantization accuracy of each network layer in the synthetic transformation network is the same and is less than or equal to P bits, the activation quantization accuracy of each network layer in the synthetic transformation network is the same and is less than or equal to Q bits, the P is less than the Q, and both the P and the Q are less than the accuracy of the synthetic transformation network before quantization.

16. The method according to claim 15, characterized in that The synthetic transformation network includes a residual module, a first deconvolution layer, a first cropping layer, a first residual activation unit, a second deconvolution layer, a second cropping layer, a second residual activation unit, a first convolution layer, a third residual activation unit, a second convolution layer, a pixel shifting layer and a third cropping layer connected in sequence; The super decoding network includes a third deconvolution layer, a fourth cropping layer, a first activation layer, a third convolution layer, a fourth deconvolution layer, a fifth cropping layer, a second activation layer, a fourth convolution layer and a third activation layer which are connected in sequence.

17. The method according to claim 16, characterized in that The residual module includes a fifth convolutional layer and a fourth activation layer connected in sequence, and the sum of an input tensor of the fifth convolutional layer and an output tensor of the fourth activation layer is the output of the residual module.

18. The method according to claim 16, characterized in that The first residual activation unit includes a fifth activation layer and a sixth convolutional layer connected in sequence, and the product of the input tensor of the fifth activation layer and the output tensor of the sixth convolutional layer is the output of the first residual activation unit; the second residual activation unit includes a sixth activation layer and a seventh convolutional layer connected in sequence, and the product of the input tensor of the sixth activation layer and the output tensor of the seventh convolutional layer is the output of the second residual activation unit; the third residual activation unit includes a seventh activation layer and an eighth convolutional layer connected in sequence, and the product of the input tensor of the seventh activation layer and the output tensor of the eighth convolutional layer is the output of the third residual activation unit.

19. The method of claim 14, wherein: The quantization accuracy includes weight quantization accuracy and / or activation quantization accuracy; The super decoding network includes multiple network layers, the weight quantization precision of each network layer in the super decoding network is the same and is less than or equal to M bits, the activation quantization precision of each network layer in the super decoding network is the same and is less than or equal to N bits, the M is less than the N, and the M and the N are both less than the precision of the super decoding network before quantization; The synthetic transformation network includes multiple network layers, the weight quantization accuracy of each network layer in the synthetic transformation network is the same and less than or equal to P bits, the activation quantization accuracy of at least one network layer in the synthetic transformation network is the same and less than or equal to P bits, the activation quantization accuracy of other network layers in the synthetic transformation network except the at least one network layer is the same and less than or equal to Q bits, the P is less than the Q, and both the P and the Q are less than the accuracy of the synthetic transformation network before quantization.

20. The method of claim 19, wherein: The synthetic transformation network includes a residual module, a first deconvolution layer, a first cropping layer, a first residual activation unit, a second deconvolution layer, a second cropping layer, a second residual activation unit, a first convolution layer, a third residual activation unit, a second convolution layer, a pixel shifting layer and a third cropping layer connected in sequence; The super decoding network includes a third deconvolution layer, a fourth cropping layer, a first activation layer, a third convolution layer, a fourth deconvolution layer, a fifth cropping layer, a second activation layer, a fourth convolution layer and a third activation layer which are connected in sequence.

21. The method of claim 20, wherein: The residual module includes a fifth convolutional layer and a fourth activation layer connected in sequence, and the sum of an input tensor of the fifth convolutional layer and an output tensor of the fourth activation layer is the output of the residual module.

22. The method of claim 20, wherein: The first residual activation unit includes a fifth activation layer and a sixth convolutional layer connected in sequence, and the product of the input tensor of the fifth activation layer and the output tensor of the sixth convolutional layer is the output of the first residual activation unit; the second residual activation unit includes a sixth activation layer and a seventh convolutional layer connected in sequence, and the product of the input tensor of the sixth activation layer and the output tensor of the seventh convolutional layer is the output of the second residual activation unit; the third residual activation unit includes a seventh activation layer and an eighth convolutional layer connected in sequence, and the product of the input tensor of the seventh activation layer and the output tensor of the eighth convolutional layer is the output of the third residual activation unit.

23. The method according to any one of claims 20 to 22, characterized in that At least one network layer in the synthetic transform network whose activation quantization precision is the same and less than or equal to P bits includes but is not limited to: the second deconvolution layer and the second residual activation unit; or, The first convolutional layer and the third residual activation unit; or, The second deconvolution layer, the second residual activation unit, the first convolution layer and the third residual activation unit; or, The first deconvolution layer, the first residual activation unit, the second deconvolution layer, the second residual activation unit, the first convolution layer and the third residual activation unit.

24. The method according to claim 15 or 19, characterized in that The values ​​of M and P are both 8, and the values ​​of N and Q are both 16.

25. The method according to any one of claims 14 to 24, characterized in that The decoding network model includes the synthetic transform network, the super decoding network and the super variance decoding network, and both the super decoding network and the super variance decoding network are connected to the synthetic transform network.

26. The method according to any one of claims 14 to 24, characterized in that The decoding network model includes two synthetic transformation networks, two super decoding networks, two super variance decoding networks and an inter-channel correction information filtering network; Each of the synthetic transform networks is connected to one of the super decoding networks and one of the super variance decoding networks respectively, and both of the synthetic transform networks are connected to the inter-channel correction information filtering network.

27. A quantization device for a decoding network model, characterized in that: The device comprises: A configuration acquisition module, used to acquire quantization configuration information of an unquantized decoding network model, wherein the quantization configuration information indicates the quantization accuracy of each network layer in a target network in the decoding network model, wherein the target network includes a super decoding network and / or a synthetic transformation network; Wherein, when the target network includes the super decoding network, the quantization precision of each network layer in the super decoding network is the same; when the target network includes the synthetic transformation network, the quantization precision of at least some network layers in the synthetic transformation network is the same; A quantization training module, used to perform a quantization-aware training process based on the quantization configuration information to obtain quantization parameters of the target network; A model quantization module is used to quantize the target network in the decoding network model based on the quantization parameters of the target network to obtain a quantized decoding network model.

28. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in any one of claims 14 to 26 when calling and executing the computer program.

29. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on the computer device, the computer device executes the steps of any one of the methods of claims 14 to 26.

30. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed on the computer device, the computer device is caused to execute the steps of the method according to any one of claims 14 to 26.

Images