Image encoding method, image decompression method and device

Abstract

The present application provides an image encoding method, an image decompression method and a device related to computer vision in the field of artificial intelligence, which are used to combine the output of an autoregressive model and an autoencoder model for encoding, reduce the size of the required model, and improve the encoding and decoding efficiency. The image encoding method includes: taking an input image as an input of an autoregressive model, outputting a first image; obtaining the residual between the first image and the input image, and obtaining a first residual image; and also taking the input image as an input of an autoencoder model, outputting a hidden variable and a first residual distribution, the hidden variable including features extracted from the input image, and the first residual distribution including residual values ​​corresponding to each pixel point in the input image output by the autoencoder model; encoding the first residual image and the first residual distribution to obtain residual encoded data; encoding the hidden variable to obtain hidden variable encoded data, and the hidden variable encoded data and the residual encoded data are used to obtain the input image after decompression.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image encoding method, an image decompression method and a device. Background Art

[0002] Images are widely used in various fields, and may involve the transmission or storage of images in a large number of scenarios. As the resolution of an image increases, more storage space is consumed when saving the image, and the bandwidth required for transmitting the image is higher, and the transmission efficiency is lower. Therefore, in order to facilitate the transmission or storage of images, images can usually be compressed to reduce the number of bits occupied by the image, thereby reducing the storage space required to save the image and the bandwidth required to transmit the image.

[0003] For example, in some commonly used image compression methods, entropy coding can be used for image compression, such as Huffman coding, arithmetic coding, ANS coding, etc. Common entropy coding algorithms are used for image compression. However, the compression rates of various entropy coding methods have reached the optimal level, and it is difficult to further improve the compression rate. Therefore, how to improve the encoding and decoding efficiency has become an urgent problem to be solved. Summary of the invention

[0004] The present application provides an image encoding method, an image decompression method and a device for encoding by combining the outputs of an autoregressive model and an autoencoding model, thereby reducing the size of the required model and improving encoding and decoding efficiency.

[0005] In view of this, in a first aspect, the present application provides an image encoding method, comprising: taking an input image as an input of an autoregressive model, and outputting a first image; obtaining a residual between the first image and the input image, and obtaining a first residual image; taking the input image as an input of an autoencoding model, and outputting latent variables and a first residual distribution, the latent variables comprising features extracted from the input image, and the first residual distribution comprising residual values ​​corresponding to each pixel point in the input image and each pixel point in the first residual image predicted by the autoencoding model; encoding the first residual image and the first residual distribution to obtain residual encoded data; encoding the latent variables to obtain latent variable encoded data, and the latent variable encoded data and the residual encoded data are used to obtain the input image after decompression.

[0006] Therefore, in this application, the output results of the autoregressive model and the autoencoder model are combined for encoding, which can control both the autoencoder and the autoregressive model to be very small, avoiding the problem of long inference time caused by the large network of the autoencoder model, and achieving efficient image compression. In addition, in the method provided by this application, the entire process can be implemented based on AI lossless compression of AI chips, including AI models and entropy coding, avoiding the transmission problem between system memory and AI chip memory, and improving coding efficiency.

[0007] In a possible implementation, the aforementioned encoding of the first residual image and the first residual distribution to obtain residual coded data includes: taking the first residual image and the first residual distribution as input of a semi-dynamic entropy encoder, and outputting residual coded data, the semi-dynamic entropy encoder being used to perform entropy coding using a first preset type of coding operation, the first preset type of coding operation including addition, subtraction or bit operation, and the semi-dynamic entropy encoder does not include a second preset type of coding operation, the second preset type including at least one of multiplication, division or remainder operation, that is, the semi-dynamic entropy encoder does not include time-consuming operations such as multiplication, division or remainder operation, that is, the semi-dynamic entropy encoder may only include simple addition and subtraction operations, thereby achieving efficient coding.

[0008] Therefore, in the implementation of the present application, the residual image can be semi-dynamically entropy encoded and encoded in a finite distribution manner. Compared with dynamic entropy coding, this reduces time-consuming operations such as multiplication, division, and remainder operations, greatly improving the encoding efficiency.

[0009] In a possible implementation, the semi-dynamic entropy encoder may be obtained by converting a dynamic upper encoder. Specifically, the operation of the dynamic entropy encoder may be approximated, such as replacing the operation of the dynamic entropy encoder with an approximate operation, reducing or removing operations such as multiplication, division, and remainder, and then performing a transformation process to transform the operation, thereby converting all operations that take more than a certain time (such as remaining remainder, multiplication, and division losses) into table access, and lightweight operations such as addition, subtraction, and bit operations to obtain the semi-dynamic entropy encoder provided by the present application. It can be understood that the semi-dynamic entropy encoder may be an entropy encoder obtained by replacing or converting some operations in the dynamic entropy encoder. When the semi-dynamic entropy encoder is used for entropy encoding, simple operations such as addition, subtraction, bit operations, and other efficient encoding operations can be used, thereby achieving efficient encoding.

[0010] In a possible implementation, the aforementioned encoding of the latent variable to obtain residual coded data may include: using the latent variable as an input of a static entropy encoder to obtain latent variable coded data.

[0011] Therefore, in the embodiments of the present application, static entropy coding can be performed on the features extracted from the input image, and the coding can be achieved efficiently.

[0012] In a possible implementation, the autoencoder model may include an encoding model and a decoding model, taking an input image as the input of the autoencoder model, and outputting latent variables and a first residual distribution, including: taking the input image as the input of the encoding model, outputting latent variables, and the encoding model is used to extract features from the input image; taking the latent variables as the input of the decoding model to obtain the first residual distribution, and the decoding model is used to predict the residual value between the input image and the corresponding pixel distribution.

[0013] In the implementation manner of the present application, a trained autoencoder model can be used to extract important features from an input image and predict a corresponding residual image, so as to combine the output of the autoregressive model to obtain residual coded data that can represent the data in the input image.

[0014] In one possible implementation, the autoregressive model is used to predict the values ​​of pixels on the same line using the pixel values ​​of predicted pixels, so that in the subsequent decoding process, for pixels on the same line, there is no need to wait for other pixels to be decoded before decoding the current pixel, thereby achieving decoding efficiency for pixels on the same line and improving decoding efficiency for the input image.

[0015] In a second aspect, the present application provides an image decompression method, comprising: obtaining latent variable coded data and residual coded data, the latent variable coded data comprising encoding the features extracted from the input image by the encoding end, and the residual coded data comprising encoding the residual between the image output by the forward propagation of the autoregressive model and the input image; decoding the latent variable coded data to obtain a latent variable, the latent variable comprising the features extracted from the input image by the encoding end; using the latent variable as the input of the autoencoder model, and outputting a second residual distribution; decoding in combination with the second residual distribution and the residual coded data to obtain a second residual image; using the second residual image as the input of the back propagation of the autoregressive model, and outputting a decompressed image.

[0016] Therefore, in the implementation mode of the present application, the autoencoder model usually has poor fitting ability and requires a deeper network to achieve a better compression rate, while the present application combines the output results of the autoregressive model, thereby reducing the size of the autoencoder model. Therefore, in the present application, the autoregressive model and the autoencoder model are combined for decoding, and both the autoencoder and the autoregressive model can be controlled to be very small, avoiding the problem of long reasoning time caused by the large network of the autoencoder model, and achieving efficient image decompression. Moreover, in the method provided in the present application, the entire process can be implemented based on AI lossless compression of the AI ​​chip, including AI model and entropy coding, avoiding the transmission problem between system memory and AI chip memory, and improving coding efficiency.

[0017] In a possible implementation, the aforementioned decoding of the latent variable encoded data to obtain the latent variable includes: using the latent variable encoded data as input of a static entropy encoder and outputting the latent variable. The decoding can be understood as the inverse operation of the static entropy encoding performed by the encoder, thereby losslessly restoring the important features in the image.

[0018] In a possible implementation, the above-mentioned decoding in combination with the second residual distribution and the residual coding data to obtain the second residual image includes: using the second residual distribution and the residual coding data as inputs of a semi-dynamic entropy encoder, and outputting the second residual image, the semi-dynamic entropy encoder is used to perform entropy coding using a first preset type of coding operation, the first preset type of coding operation includes addition, subtraction or bit operation, and the semi-dynamic entropy encoder does not include a second preset type of coding operation, the second preset type includes at least one of multiplication, division or remainder operation, that is, the semi-dynamic entropy encoder does not include multiplication, division or remainder operation and other time-consuming operations, that is, the semi-dynamic entropy encoder can only include simple addition and subtraction operations, so that efficient coding can be achieved. Therefore, the residual image can be decoded based on semi-dynamic entropy coding, and decoded in a limited distribution mode, which reduces the time-consuming operation losses such as multiplication, division and remainder operation compared to dynamic entropy coding, and greatly improves the decoding efficiency.

[0019] In a possible implementation, the semi-dynamic entropy encoder may be obtained by converting a dynamic upper encoder. Specifically, the operation of the dynamic entropy encoder may be approximated, such as replacing the operation of the dynamic entropy encoder with an approximate operation, reducing or removing operations such as multiplication, division, and remainder, and then performing a transformation process to transform the operation, thereby converting all operations that take more than a certain time (such as remaining remainder, multiplication, and division losses) into table access, and lightweight operations such as addition, subtraction, and bit operations to obtain the semi-dynamic entropy encoder provided by the present application. It can be understood that the semi-dynamic entropy encoder may be an entropy encoder obtained by replacing or converting some operations in the dynamic entropy encoder. When the semi-dynamic entropy encoder is used for entropy encoding, simple operations such as addition, subtraction, bit operations, and other efficient encoding operations can be used, thereby achieving efficient encoding.

[0020] In a possible implementation, the aforementioned use of the second residual image as the input of the back propagation of the autoregressive model and outputting the decompressed image includes: decoding the pixels on the same line in the second residual image in parallel through the autoregressive model to obtain the decompressed image. Therefore, for the pixels on the same line, there is no need to wait for other pixels to be decoded before decoding the current pixel, thereby achieving the decoding efficiency of the pixels on the same line and improving the decoding efficiency of the input image.

[0021] In a third aspect, the present application provides an image encoding device, including:

[0022] An autoregressive module, used to take an input image as an input of an autoregressive model, and output a first image, an autoregressive model;

[0023] A residual calculation module, used to obtain a residual between the first image and the input image to obtain a first residual image;

[0024] An autoencoder module, used to take an input image as an input of an autoencoder model, and output a latent variable and a first residual distribution, wherein the latent variable includes features extracted from the input image, and the first residual distribution includes a residual value output by the autoencoder model and corresponding to each pixel point in the input image and each pixel point in the first residual image;

[0025] A residual coding module, used for encoding the first residual image and the first residual distribution to obtain residual coding data;

[0026] The latent variable encoding module is used to encode the latent variable to obtain latent variable encoded data. The latent variable encoded data and the residual encoded data are used to obtain the input image after decompression.

[0027] In one possible implementation, a residual coding module is specifically used to use a first residual image and a first residual distribution as inputs of a semi-dynamic entropy encoder, and output residual coding data. The semi-dynamic entropy encoder is used to perform entropy coding using a first preset type of coding operation. The first preset type of coding operation includes addition, subtraction or bit operation, and the semi-dynamic entropy encoder does not include a second preset type of coding operation. The second preset type includes at least one of multiplication, division or remainder operation. That is, the semi-dynamic entropy encoder does not include time-consuming operations such as multiplication, division or remainder operation. That is, the semi-dynamic entropy encoder may only include simple addition and subtraction operations, thereby achieving efficient coding.

[0028] In a possible implementation, the semi-dynamic entropy encoder may be obtained by converting a dynamic upper encoder. Specifically, the operation of the dynamic entropy encoder may be approximated, such as replacing the operation of the dynamic entropy encoder with an approximate operation, reducing or removing operations such as multiplication, division, and remainder, and then performing a transformation process to transform the operation, thereby converting all operations that take more than a certain time (such as remaining remainder, multiplication, and division losses) into table access, and lightweight operations such as addition, subtraction, and bit operations to obtain the semi-dynamic entropy encoder provided by the present application. It can be understood that the semi-dynamic entropy encoder may be an entropy encoder obtained by replacing or converting some operations in the dynamic entropy encoder. When the semi-dynamic entropy encoder is used for entropy encoding, simple operations such as addition, subtraction, bit operations, and other efficient encoding operations can be used, thereby achieving efficient encoding.

[0029] In a possible implementation, the latent variable encoding module is specifically configured to use the latent variable as an input of a static entropy encoder to obtain latent variable encoded data.

[0030] In one possible implementation, the autoencoder model includes an encoding model and a decoding model, and the autoencoder module is specifically used to: use the input image as the input of the encoding model and output latent variables, and the encoding model is used to extract features from the input image; use the latent variables as the input of the decoding model to obtain a first residual distribution, and the decoding model is used to predict the residual value between the input image and the corresponding pixel distribution.

[0031] In a possible implementation, the autoregressive model is used to predict the values ​​of pixels on the same line using the pixel values ​​of predicted pixels.

[0032] In a fourth aspect, the present application provides an image decompression device, comprising:

[0033] A transceiver module, used to obtain latent variable coded data and residual coded data, wherein the latent variable coded data includes data obtained by encoding features extracted from an input image by the encoder, and the residual coded data includes data obtained by encoding a residual between an image output by an autoregressive model and the input image;

[0034] A latent variable decoding module, used to decode latent variable encoded data to obtain latent variables, where the latent variables include features extracted from the input image by the encoder;

[0035] An autoencoder module, used for taking the latent variable as the input of the autoencoder model and outputting a second residual distribution;

[0036] A residual decoding module, used for decoding in combination with the second residual distribution and the residual coding data to obtain a second residual image;

[0037] The autoregressive module is used to use the second residual image as an input of the back propagation of the autoregressive model and output a decompressed image.

[0038] In a possible implementation, the latent variable decoding module is specifically configured to use the latent variable encoded data as input of a static entropy encoder to output the latent variable.

[0039] In one possible implementation, the residual decoding module is specifically used to use the second residual distribution and the residual encoding data as input of a semi-dynamic entropy encoder to output a second residual image. The semi-dynamic entropy encoder is used to perform entropy encoding using a first preset type of encoding operation. The first preset type of encoding operation includes addition, subtraction or bit operation, and the semi-dynamic entropy encoder does not include a second preset type of encoding operation. The second preset type includes at least one of multiplication, division or remainder operation, that is, the semi-dynamic entropy encoder does not include time-consuming operations such as multiplication, division or remainder operation, that is, the semi-dynamic entropy encoder may only include simple addition and subtraction operations, thereby achieving efficient encoding.

[0040] In a possible implementation, the semi-dynamic entropy encoder may be obtained by converting a dynamic upper encoder. Specifically, the operation of the dynamic entropy encoder may be approximated, such as replacing the operation of the dynamic entropy encoder with an approximate operation, reducing or removing operations such as multiplication, division, and remainder, and then performing a transformation process to transform the operation, thereby converting all operations that take more than a certain time (such as remaining remainder, multiplication, and division losses) into table access, and lightweight operations such as addition, subtraction, and bit operations to obtain the semi-dynamic entropy encoder provided by the present application. It can be understood that the semi-dynamic entropy encoder may be an entropy encoder obtained by replacing or converting some operations in the dynamic entropy encoder. When the semi-dynamic entropy encoder is used for entropy encoding, simple operations such as addition, subtraction, bit operations, and other efficient encoding operations can be used, thereby achieving efficient encoding.

[0041] In a possible implementation, the autoregressive module is specifically configured to perform parallel decoding on pixel points on the same line in the second residual image through an autoregressive model to obtain a decompressed image.

[0042] In a fifth aspect, an embodiment of the present application provides an image coding device, which has the function of implementing the image processing method of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0043] In a sixth aspect, an embodiment of the present application provides an image decompression device, which has the function of implementing the image processing method of the second aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0044] In a seventh aspect, an embodiment of the present application provides an image encoding device, comprising: a processor and a memory, wherein the processor and the memory are interconnected via a line, and the processor calls a program code in the memory to execute the processing-related functions in the image encoding method shown in any one of the first aspects above. Optionally, the image encoding device may be a chip.

[0045] In an eighth aspect, an embodiment of the present application provides an image decompression device, comprising: a processor and a memory, wherein the processor and the memory are interconnected via a line, and the processor calls a program code in the memory to execute the processing-related functions in the image decompression method shown in any one of the second aspects above. Optionally, the image decompression device may be a chip.

[0046] In the ninth aspect, an embodiment of the present application provides an image encoding device, which may also be referred to as a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to perform functions related to processing in the above-mentioned first aspect or any optional embodiment of the first aspect.

[0047] In the tenth aspect, an embodiment of the present application provides an image decompression device, which may also be referred to as a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to perform functions related to processing as described in the second aspect or any optional embodiment of the second aspect.

[0048] In the eleventh aspect, an embodiment of the present application provides an image processing system, characterized in that it includes an image encoding device and an image decompression device, wherein the image encoding device is used to perform functions related to processing as in the above-mentioned first aspect or any optional implementation scheme of the first aspect, and the image decompression device is used to perform functions related to processing as in the above-mentioned second aspect or any optional implementation scheme of the second aspect.

[0049] In the twelfth aspect, an embodiment of the present application provides a computer-readable storage medium, including instructions, which, when executed on a computer, enables the computer to execute a method in any optional implementation of the first aspect or the second aspect above.

[0050] In a thirteenth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute a method in any optional implementation of the first aspect or the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of an artificial intelligence main framework for this application;

[0052] Figure 2 A schematic diagram of a system architecture provided for an embodiment of the present application;

[0053] Figure 3 A schematic diagram of an application scenario of an embodiment of the present application;

[0054] Figure 4 A schematic diagram of another application scenario of an embodiment of the present application;

[0055] Figure 5 A schematic diagram of another application scenario of an embodiment of the present application;

[0056] Figure 6 A schematic diagram of a flow chart of an image encoding method provided in an embodiment of the present application;

[0057] Figure 7 A schematic diagram of a flow chart of another image encoding method provided in an embodiment of the present application;

[0058] Figure 8 A schematic diagram of a prediction method of an autoregressive model provided in an embodiment of the present application;

[0059] Fig. 9 A schematic diagram of the prediction sequence of an autoregressive model provided in an embodiment of the present application;

[0060] Fig.10 A schematic diagram of a residual calculation method provided in an embodiment of the present application;

[0061] Fig.11 A schematic diagram of a data structure provided for an embodiment of the present application;

[0062] Fig.12 A schematic diagram of a flow chart of an image decompression method provided in an embodiment of the present application;

[0063] Fig.13 A flowchart of another image decompression method provided in an embodiment of the present application;

[0064] Fig.14 A schematic diagram of the structure of an image encoding device provided by the present application;

[0065] Fig.15 A schematic diagram of the structure of an image decoding device provided by the present application;

[0066] Fig.16 A schematic diagram of the structure of another image encoding device provided by the present application;

[0067] Fig.17 A schematic diagram of the structure of another image decoding device provided by the present application;

[0068] Fig.18 A schematic diagram of a chip structure provided for this application. DETAILED DESCRIPTION

[0069] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0070] First, the overall workflow of the artificial intelligence system is described. Figure 1 , Figure 1 The figure shows a structural diagram of the main framework of artificial intelligence. The following is an explanation of the above artificial intelligence theme framework from the two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecology process of the system.

[0071] (1) Infrastructure

[0072] The infrastructure provides computing power support for the artificial intelligence system, enables communication with the outside world, and is supported by the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips, such as central processing units (CPU), neural-network processing units (NPU), graphics processing units (GPU), application specific integrated circuits (ASIC) or field programmable gate arrays (FPGA) and other hardware acceleration chips; the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0073] (2) Data

[0074] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0075] (3) Data processing

[0076] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0077] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0078] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0079] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0080] (4) General ability

[0081] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0082] (5) Smart products and industry applications

[0083] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.

[0084] The embodiments of the present application involve a large number of neural network and image related applications. In order to better understand the solutions of the embodiments of the present application, the relevant terms and concepts in the field of neural networks and images that may be involved in the embodiments of the present application are first introduced below.

[0085] (1) Neural Network

[0086] A neural network may be composed of neural units, and a neural unit may refer to an operation unit having xs and an intercept 1 as input, and the output of the operation unit may be shown as follows:

[0087]

[0088] Where s=1, 2, ...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.

[0089] (2) Deep Neural Networks

[0090] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple intermediate layers. According to the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, intermediate layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all intermediate layers, or hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.

[0091] Although DNN looks complicated, each layer can be expressed as a linear relationship expression: in, is the input vector, is the output vector, is the offset vector or bias parameter, w is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Taking the coefficient w as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0092] In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as

[0093] It should be noted that the input layer does not have a W parameter. In a deep neural network, more intermediate layers allow the network to better characterize complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (a weight matrix formed by many layers of vectors W).

[0094] (3) Convolutional Neural Network

[0095] Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer, which can be regarded as a filter. Convolutional layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolutional layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connection between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0096] (4) Loss Function

[0097] In the process of training deep neural networks, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make the prediction lower, and continue to adjust until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which is an important equation used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible. The loss function can usually include loss functions such as squared error, cross entropy, logarithm, exponential, etc. For example, the squared error can be used as a loss function, defined as The specific loss function can be selected according to the actual application scenario.

[0098] (5) Back propagation algorithm

[0099] Neural networks can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0100] (6) Entropy Coding

[0101] Entropy coding is a coding process that does not lose any information according to the entropy principle. Information entropy is the average amount of information in the source (a measure of uncertainty). Common entropy coding methods include Shannon coding, Huffman coding, and arithmetic coding.

[0102] For example, if the distribution of pixel values ​​of each pixel in the predicted image is known, the optimal compression scheme can be obtained using entropy coding technology. Using entropy coding technology, an image with probability p can be represented by -log2 p bits. For example, an image with probability 1 / 8 needs to be represented by 3 bits, and an image with probability 1 / 256 needs to be represented by 8 bits.

[0103] To determine the number of bits per letter, the algorithm needs to know the probability of each letter appearing as accurately as possible. The model's task is to provide this data. The better the model's predictions, the better the compression results. In addition, the model must come up with the same data when compressing and restoring.

[0104] The static model (or static entropy coding) analyzes the entire text before compression and calculates the probability of each letter. This calculation result is used for the entire text. The coding table only needs to be calculated once, so the encoding speed is high, and the result is definitely not longer than the original text except for the probability value required for decoding. In the method provided by the present application, the entropy coding used can include static entropy coding methods such as tANS or fse.

[0105] Dynamic model In this model, the probability changes continuously as the encoding process progresses. This can be achieved through a variety of algorithms, such as:

[0106] Forward dynamics: The probability is calculated based on the letters that have been encoded, and each time a letter is encoded its probability increases.

[0107] Reverse dynamics: Before encoding, calculate the probability of each letter in the remaining part that has not been encoded. As the encoding progresses, more and more letters no longer appear, and their probability becomes 0, while the probability of the remaining letters increases, and the number of bits to encode them decreases. The compression rate continues to increase, until the last letter only needs 0 bits to encode.

[0108] Therefore, the model is optimized according to the specificity of different parts; probability data does not need to be fed into the forward model.

[0109] In this application, entropy coding is divided into multiple types, such as static entropy coding, semi-dynamic entropy coding and dynamic entropy coding. No matter which encoder is used, the purpose is to encode data with a probability of p with a length close to log2 p. The difference is that static entropy coding uses a single probability distribution for encoding, semi-dynamic uses multiple (i.e., finite) probability distributions for encoding, and dynamic entropy coding uses any infinite probability distributions for encoding.

[0110] (7) Autoregressive model

[0111] It is a method of processing time series, which uses previous historical data of the same variable to predict current data.

[0112] For example, using the same variable, say x, in previous periods, i.e. x1 to x t-1 To predict this period x t , and assume that they are in a linear relationship. Because this is developed from linear regression in regression analysis, it is not used to predict y, but to predict x with x; so it is called autoregression.

[0113] (8) Autoencoder Model

[0114] The autoencoder model is a neural network that uses the back-propagation algorithm to make the output value equal to the input value. The input data is first compressed into a latent space representation, and then the output is reconstructed through this representation.

[0115] The autoencoder model usually includes an encoder model and a decoder model. In the present application, the trained encoder model is used to extract features from the input image to obtain latent variables, which are then input into the trained decoder model to output the residual corresponding to the predicted input image.

[0116] (9) Lossless compression

[0117] The technology of compressing data. The compressed data occupies less space than the uncompressed data, and the compressed data can be restored to the original data by decompression. The decompressed data is completely consistent with the data before compression.

[0118] Generally, the greater the probability of each pixel in the image appearing (i.e., the probability value obtained by predicting the pixel value of the current pixel by the pixel values ​​of other pixels), the shorter the length after compression. The probability of a real image is much higher than that of a randomly generated image, so the number of bits per pixel (bpd) required to compress is much smaller than the latter. In practical applications, the BPD of most images is significantly smaller than before compression, and only a very small probability is higher than before compression, thereby reducing the average bpd of each image.

[0119] (10) Compression rate

[0120] The ratio of the original data size to the compressed data size. If there is no compression, the value is 1. The larger the value, the better.

[0121] (11) Throughput

[0122] The size of raw data that can be compressed / decompressed per second.

[0123] (12) Receptive field

[0124] When predicting a pixel, we need to know the points in advance. Changing the points outside the receptive field will not change the prediction of the pixel.

[0125] The encoding method and decoding method provided in the embodiment of the present application can be executed on a server or on a terminal device. Accordingly, the neural network mentioned below in the present application can be deployed on a server or on a terminal, and can be adjusted according to the actual application scenario. For example, the encoding method and decoding method provided in the present application can be deployed in a terminal in the form of a plug-in. The terminal device can be a mobile phone with image processing function, a tablet personal computer (tablet personal computer, TPC), a media player, a smart TV, a laptop computer (laptop computer, LC), a personal digital assistant (personal digital assistant, PDA), a personal computer (personal computer, PC), a camera, a video camera, a smart watch, a wearable device (wearable device, WD) or an autonomous driving vehicle, etc., and the embodiment of the present application does not limit this. The following is exemplarily described by taking the encoding method and decoding method provided in the present application as an example of being deployed in a terminal.

[0126] All or part of the processes in the encoding method and decoding method provided in the present application can be implemented by a neural network, such as the autoregressive model and the autoencoding model, which can be implemented by a neural network. Usually, the neural network needs to be deployed on the terminal after training, such as Figure 2 As shown, the embodiment of the present application provides a system architecture 100. Figure 2 In some optional implementations, in the present application, for the autoregressive model and the autoencoder model, the training data may include a large number of high-definition images.

[0127] After the training data is collected, the data collection device 160 stores the training data in the database 130, and the training device 120 obtains the target model / rule 101 through training based on the training data maintained in the database 130. Optionally, the training set mentioned in the following embodiments of the present application can be obtained from the database 130 or through user input data.

[0128] Among them, the target model / rule 101 can be a neural network trained in the embodiment of the present application, and the neural network can include one or more networks, such as an autoregressive model or an autoencoding model.

[0129] The following describes how the training device 120 obtains the target model / rule 101 based on the training data. The training device 120 processes the input three-dimensional model and compares the output image with the high-quality rendered image corresponding to the input three-dimensional model until the difference between the image output by the training device 120 and the high-quality rendered image is less than a certain threshold, thereby completing the training of the target model / rule 101.

[0130] The above-mentioned target model / rule 101 can be used to implement the neural network mentioned in the encoding method and the decoding method of the embodiment of the present application, that is, the data to be processed (such as the image to be compressed) is input into the target model / rule 101 after relevant pre-processing, and the processing result can be obtained. The target model / rule 101 in the embodiment of the present application can specifically be the neural network mentioned below in the present application, and the neural network can be the aforementioned CNN, DNN or RNN type of neural network. It should be noted that in actual applications, the training data maintained in the database 130 does not necessarily come from the collection of the data acquisition device 160, and may also be received from other devices. It should also be noted that the training device 120 does not necessarily train the target model / rule 101 completely based on the training data maintained by the database 130, and it is also possible to obtain training data from the cloud or other places for model training, and the present application does not limit this.

[0131] The target model / rule 101 trained by the training device 120 can be applied to different systems or devices, such as Figure 2 The execution device 110 shown in the figure may also be called a computing device. The execution device 110 may be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR), a vehicle terminal, etc., or a server or a cloud device, etc. Figure 2 In the embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. The user can input data to the I / O interface 112 through the client device 140. The input data in this embodiment of the application may include: data to be processed input by the client device. The client may be other hardware devices, such as a terminal or a server, etc., and the client may also be software deployed on the terminal, such as an APP, a web page, etc.

[0132] Preprocessing module 113 and preprocessing module 114 are used to perform preprocessing according to input data (such as data to be processed) received by I / O interface 112. In an embodiment of the present application, preprocessing module 113 and preprocessing module 114 may not be provided (or only one of the preprocessing modules may be provided), and computing module 111 may be used directly to process the input data.

[0133] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculations and other related processing, the execution device 110 can call the data, code, etc. in the data storage system 150 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0134] Finally, the I / O interface 112 returns the processing result to the client device 140 for providing it to the user. For example, if the first neural network is used for image classification and the processing result is a classification result, the I / O interface 112 returns the classification result obtained above to the client device 140 for providing it to the user.

[0135] It should be noted that the training device 120 can generate corresponding target models / rules 101 based on different training data for different goals or different tasks, and the corresponding target models / rules 101 can be used to achieve the above goals or complete the above tasks, thereby providing the user with the desired results. In some scenarios, the execution device 110 and the training device 120 can be the same device, or located in the same computing device. For ease of understanding, this application will introduce the execution device and the training device separately, without limitation.

[0136] exist Figure 2 In the case shown, the user can manually give input data, and the manual giving can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 140 can also be used as a data acquisition terminal to collect the input data input to the I / O interface 112 as shown in the figure and the predicted label output from the I / O interface 112 as new sample data, and store it in the database 130. Of course, it is also possible not to collect through the client device 140, but the I / O interface 112 directly stores the input data input to the I / O interface 112 as shown in the figure and the predicted label output from the I / O interface 112 as new sample data in the database 130.

[0137] It should be noted that Figure 2 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 2 In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.

[0138] like Figure 2 As shown, the target model / rule 101 is obtained by training according to the training device 120. The target model / rule 101 in the embodiment of the present application can be the neural network in the present application. Specifically, the neural network provided in the embodiment of the present application can include CNN, deep convolutional neural networks (DCNN), recurrent neural networks (RNN) or constructed neural networks, etc.

[0139] The encoding method and decoding method in the embodiment of the present application can be executed by an electronic device, which is the aforementioned execution device. The electronic device includes a CPU and a GPU, which can compress the image. Of course, other devices such as NPU or ASIC, etc. can also be included, which are only exemplary descriptions here and will not be repeated one by one. Exemplarily, the electronic device can be, for example, a mobile phone, a tablet computer, a laptop computer, a PC, a mobile internet device (mobile internet device, MID), a wearable device, a virtual reality (virtual reality, VR) device, an augmented reality (augmented reality, AR) device, a wireless electronic device in industrial control (industrial control), a wireless electronic device in self-driving, a wireless electronic device in remote medical surgery, a wireless electronic device in smart grid (smart grid), a wireless electronic device in transportation safety (transportation safety), a wireless electronic device in a smart city (smart city), a wireless electronic device in a smart home (smart home), etc. The electronic device can be a device running an Android system, an IOS system, a windows system, and other systems. An application program that needs to compress an image to obtain a compressed image, such as communication software, photo album, or camera application, may be run in the electronic device.

[0140] Usually, in some image compression scenarios, entropy coding can be used for compression. However, the distribution of the image is unknown, so the original distribution needs to be estimated and the estimated distribution is input into the entropy encoder for encoding. Generally, the more accurate the estimation, the higher the compression rate. Traditional image lossless compression algorithms mostly adopt the principle of "similar pixel values ​​are usually closer" and use a fixed prediction method. However, this method has low encoding efficiency.

[0141] In some scenarios, AI image lossless compression can also be used for compression. Compared with traditional encoding algorithms, AI algorithms can achieve significantly higher compression rates, but the compression / decompression efficiency is very low.

[0142] For example, an autoregressive model can be used for image compression. If an autoregressive model is constructed and the values ​​of all previous pixels are input, the distribution parameters of the predicted point can be output. If the distribution is a Gaussian distribution, the output is two parameters, the mean and the variance. When using an autoregressive model for compression, all pixels can be input into the autoregressive model to obtain the distribution prediction of the pixels, and the distribution prediction of the pixels and the pixel values ​​are input into the entropy encoder to obtain the encoded data. When decompressing, all pixels are input into the autoregressive model to obtain the distribution prediction of the pixels, and the distribution prediction and its encoded data are input into the entropy encoder to obtain the decoded data. However, during the encoding and decoding process, the prediction of each pixel depends on all previous pixels, and the operation efficiency is low. When decompressing, all pixels before the current pixel must be decompressed before the current pixel can be decompressed. Only one pixel can be decompressed at a time, the number of network inferences is large, and the decompression efficiency is low.

[0143] For another example, an autoencoder model can be used for image compression. When encoding, the original data is input into the encoding network (Encoder) to obtain hidden variables, and the hidden variables are input into the decoding network (Decoder) to obtain the distribution prediction of the image; the manually designed distribution and the value of the hidden variable are input into the entropy coding to encode the hidden variable; the distribution prediction of the image and the original image are input into the entropy coding to encode the image. When decoding, the manually designed distribution and the encoding of the hidden variable are input into the entropy coding to decode the hidden variable; the hidden variable is input into the decoding network (Decoder) to obtain the distribution prediction of the image; the distribution prediction of the image and the encoding of the image are input into the entropy coding to decode the image. Compared with the autoregressive model, the autoencoder model has poor fitting ability. If the compression rate exceeds that of the traditional compression algorithm, a deeper network is required, and the latency of a single network inference is high.

[0144] Therefore, the present application provides an encoding method and a decoding method, which utilizes an autoregressive model and an autoencoder model for lossless compression, and provides an efficient semi-dynamic entropy encoder, so that both the model reasoning and the encoding process are run on the AI ​​chip, reducing the transmission between the system memory and the AI ​​chip memory, and achieving high-bandwidth compression and decompression.

[0145] First, to facilitate understanding, some application scenarios of the encoding method and decoding method provided in the present application are exemplarily introduced.

[0146] Scenario 1: Saving captured images locally

[0147] Taking the method provided in this application as an example, the terminal may include a mobile phone, a camera, a monitoring device, or other equipment with a shooting function or connected to a camera. Figure 3 As shown, after the image is captured, in order to reduce the storage space occupied by the image, the image can be losslessly compressed by the encoding method provided by the present application, thereby obtaining compressed encoded data. When the image needs to be read, such as when the image is displayed in an album, it can be decoded by the decoding method provided by the present application to obtain a high-definition image. The method provided by the present application can efficiently and losslessly compress the image, reduce the content required to save the image, and losslessly restore the image to obtain a high-definition image.

[0148] Scenario 2: Image Transmission

[0149] In some communication scenarios, image transmission may be involved. For example, Figure 4 As shown, when users use communication software to communicate, they can transmit images through wired or wireless networks. In order to increase the transmission rate and reduce the network resources occupied by transmitting images, the image can be losslessly compressed by the encoding method provided by this application to obtain compressed encoded data, and then the encoded data can be transmitted. After receiving the encoded data, the receiving end can decode the encoded data by the decoding method provided by this application to obtain the restored image.

[0150] Scenario 3: The server saves a large number of images

[0151] In some platforms or databases that provide services to users, it is usually necessary to save a large number of high-definition images. If they are saved directly according to the pixels of each frame, it will take up a very large storage space. For example, Figure 5 As shown, some shopping software or public data sets need to save a large number of high-definition images in the server, and users can read the required images from the server. The encoding method provided by this application can be used to efficiently and losslessly compress the images to be saved to obtain compressed data. When the image needs to be read, the saved encoded data can be decoded by the decoding method provided by this application to obtain a high-definition image.

[0152] For ease of understanding, the encoding method and decoding method provided in this application are respectively introduced below.

[0153] See also Figure 6 , a flow chart of an encoding method provided in this application is as described below.

[0154] 601. Use the input image as input of the autoregressive model and output a first image.

[0155] Among them, the input image can be an image to be compressed, and the autoregressive model can be used to use the values ​​of other pixels in the input image except the current pixel to predict the pixel value of the current pixel, and obtain the predicted pixel distribution of each pixel, that is, the first image.

[0156] The input image may include multiple images, and the sources of the input image may be different depending on the scene. For example, the input image may be a photographed image, or a received image.

[0157] Optionally, in the process of prediction by the autoregressive model, for the pixels on the same line, the pixel values ​​of the predicted pixels can be used for prediction, so that in the subsequent decoding process, for the pixels on the same line, there is no need to wait for other pixels to be decoded before decoding the current pixel, thereby achieving the decoding efficiency of the pixels on the same line and improving the decoding efficiency of the input image. The same line can be the same row, the same column, or the same diagonal line, etc., which can be determined according to the actual application scenario.

[0158] 602. Obtain a residual between the first image and the input image to obtain a first residual image.

[0159] After the first image is obtained, the residual value between each pixel point in the first image and the corresponding pixel point in the input image may be calculated to obtain a first residual image.

[0160] Among them, the resolution between the first image and the input image is usually the same, that is, the pixels in the first image and the input image correspond one to one, so that when calculating the residual value, the residual value between each pair of pixels can be calculated, and the obtained residual values ​​can form an image, that is, the first residual image.

[0161] Optionally, when calculating the residual, the residual value is usually an integer type in the range of [-255, 255]. The residual value can be converted to a low-precision numerical type for representation, such as converting the integer to the uint8 numerical type, thereby reducing the value to [0, 255]. The offset can be set so that the residual value of each pixel is distributed around 128, thereby making the data more concentrated, and the residual distribution between the input image and the output image of the autoregressive model can be represented by less data.

[0162] 603. Use the input image as the input of the autoencoder model, and output the latent variable and the first residual distribution.

[0163] After obtaining the input image, the input image can also be used as the input of the autoencoder model to output the corresponding latent variable and the first residual distribution.

[0164] The latent variable may include features extracted from the input image, and the first residual distribution may include residual values ​​between each pixel of the input image predicted by the autoencoder model and the corresponding pixel in the first residual image.

[0165] Specifically, the autoencoder model may include an encoding model and a decoding model, wherein the encoding model may be used to extract features from an input image, and the decoding model may be used to predict the residual between the input image and the image output by the autoregressive model. That is, features may be extracted from the input image through the encoding model to obtain latent variables for representing important features of the input image, and the latent variables are used as inputs of the decoding model to output a first residual distribution.

[0166] It should be noted that the present application does not limit the execution order of step 601 and step 603. Step 601 may be executed first, or step 603 may be executed first, or step 601 and step 603 may be executed simultaneously. The specific execution order may be adjusted according to the actual application scenario.

[0167] 604. Encode the first residual image and the first residual distribution to obtain residual encoding data.

[0168] After obtaining the first residual image and the first residual distribution, the first residual image and the first residual distribution may be encoded to obtain residual encoding data.

[0169] Specifically, when encoding the first residual image and the first residual distribution, semi-dynamic entropy coding can be used, that is, a finite number of probability distributions are used for encoding to obtain the encoded data of the residual image, that is, the residual encoded data. The semi-dynamic entropy encoder is used to perform entropy coding using a first preset type of encoding operation, the first preset type of encoding operation includes addition, subtraction or bit operation, and the semi-dynamic entropy encoder does not include a second preset type of encoding operation, the second preset type includes at least one of multiplication, division or remainder operation that takes a long time to improve the encoding efficiency. Therefore, in the implementation of the present application, a finite number of probability distributions can be used for encoding to obtain the encoding of the residual image. Compared with dynamic entropy coding, more instructions are required to decompress a character; division and exponentiation take a long time, and each instruction takes dozens of times the time of addition. The semi-dynamic entropy coding of a finite number of probability distributions provided in the present application can achieve efficient coding and improve coding efficiency.

[0170] In a possible implementation, the semi-dynamic entropy encoder may be obtained by converting a dynamic upper encoder. Specifically, the operation of the dynamic entropy encoder may be approximated, such as replacing the operation of the dynamic entropy encoder with an approximate operation, reducing or removing operations such as multiplication, division, and remainder, and then performing a transformation process to transform the operation, thereby converting all operations that take more than a certain time (such as remaining remainder, multiplication, and division losses) into table access, and lightweight operations such as addition, subtraction, and bit operations to obtain the semi-dynamic entropy encoder provided by the present application. It can be understood that the semi-dynamic entropy encoder may be an entropy encoder obtained by replacing or converting some operations in the dynamic entropy encoder. When the semi-dynamic entropy encoder is used for entropy encoding, simple operations such as addition, subtraction, bit operations, and other efficient encoding operations can be used, thereby achieving efficient encoding.

[0171] 605. Encode the latent variables to obtain residual coded data.

[0172] The latent variable may include important features extracted from the input image. Therefore, when performing image compression, the extracted important features may be encoded to obtain residual coding data, so as to facilitate subsequent image restoration to obtain a lossless image.

[0173] Optionally, when encoding the latent variable, static entropy coding may be used to encode the latent variable, and the latent variable is used as an input of a static entropy encoder to output an encoded bit stream of the latent variable.

[0174] The latent variable coded data and the residual coded data can be used for lossless restoration of the image at the decoding end, thereby achieving lossless compression and restoration of the image.

[0175] Usually, the autoencoder model usually has poor fitting ability and requires a deeper network to achieve a better compression rate. The present application combines the output results of the autoregressive model, thereby reducing the size of the autoencoder model. Therefore, in the present application, the output results of the autoregressive model and the autoencoder model are combined for encoding, and both the autoencoder and the autoregressive model can be controlled to be very small, avoiding the problem of long reasoning time caused by the large network of the autoencoder model, and achieving efficient image compression. In addition, in the method provided in the present application, the entire process can be implemented based on AI lossless compression of the AI ​​chip, including AI model and entropy coding, avoiding the transmission problem between system memory and AI chip memory, and improving coding efficiency.

[0176] The above is an introduction to the process of the encoding method provided by this application. The following is a more detailed introduction to the process of the encoding method provided by this application in combination with specific application scenarios. Figure 7 , a flowchart of another encoding method provided by the present application.

[0177] First, obtain the input image 701.

[0178] The input image 701 may include an image collected by itself or an image received. For example, if the method provided in this application is deployed on a terminal, the input image may include an image collected by the terminal or an image received by the terminal from other servers or terminals.

[0179] Subsequently, use the input image 701 as the input of the autoregressive model 702 to output a predicted image 703.

[0180] Among them, the autoregressive model can be used to predict the pixel probability distribution of each pixel point using the pixel points adjacent to each pixel point to obtain the predicted image 703, that is, the aforementioned first image.

[0181] It can be understood that the autoregressive model can use the pixel values of adjacent pixel points to predict the pixel value of the current pixel point.

[0182] In the embodiments of this application, in order to improve the decoding efficiency of the decoding end, when the autoregressive model makes a prediction, for the pixel points on the same line, the pixel values of the pixel points adjacent to it can be used to perform parallel predictions. Taking a specific autoregressive model as an example, as Figure 8 shown, given an m×n image and a hyperparameter h (0≤h<n), if for any pixel point (i,j), all points (i′,j′) in the autoregressive model that predict (i,j) satisfy: h×i′+j′<h×i+j, then this image can be predicted in parallel n+(m - 1)×h times, as Figure 8 shown, when h = 1, for the pixel points on the same diagonal line, the pixel values of multiple pixel points selected on its left by 1 unit can be used as the receptive field to predict the pixel probability distribution of the current pixel point, that is, the probability of the pixel point being each pixel value. As Figure 8 shown, when h = 2, the pixel values of multiple pixel points selected on its left by 2 units can be used as the receptive field to predict the pixel probability distribution of the current pixel point. So that when decompressing later, the pixel points on the same diagonal line can be decompressed in parallel.

[0183] In addition, the prediction order for each pixel point can be as Fig. 9 shown, where the smaller the number, the higher the prediction priority, and the pixel points with the same number are predicted simultaneously. Therefore, for the pixel points on the same diagonal line, parallel predictions can be performed to improve the prediction efficiency of the autoregressive model.

[0184] Subsequently, calculate the residual between the predicted image and the input image to obtain an image residual 704.

[0185] After obtaining the predicted image 703 output by the autoregression, the residual between the predicted image and each pixel in the input image can be calculated to obtain the image residual 704, that is, the aforementioned first residual image.

[0186] Given the original image x, that is, the input image, the autoregressive model is used to predict the original image to obtain the predicted reconstructed image The image residual between each pixel of the reconstructed image and the original image can be calculated

[0187] For example, Fig.10 As shown, after obtaining the input image and the predicted image, the difference between the corresponding pixels of the input image and the predicted image can be calculated to obtain the residual value between each pixel to form a residual image.

[0188] Optionally, when calculating the residual, the integer type in the range of [-255, 255] can be used, and the residual value can be converted to a low-precision numerical type to represent it, such as converting the integer to the uint8 numerical type, thereby reducing the value to [0, 255], and the offset can be set so that the residual value of each pixel point is distributed around 128, thereby making the data more concentrated, and the residual distribution between the input image and the output image of the autoregressive model can be represented by less data.

[0189] For example, if we use the autoregressive model to input the original image x and output y, then the predicted image x′ = round(clip(y,0,M-1)), and the residual is calculated as The value range of each pixel in x′ is an integer from 0 to M-1. Using model 2 to predict r, we get the distribution N(μ,σ). Then we use Distributed Encoding Where N is a Gaussian distribution or a logistic distribution.

[0190] In addition, the input image is input to the autoencoder model 705, and the prediction residual 707 and latent variable 706 are output.

[0191] For example, the original image x may be input into the autoencoder model, and the autoencoder model may be used to estimate the probability distribution p(r|x) of the image residual r, that is, the prediction residual 707 .

[0192] Specifically, the self-encoding model may include an encoding model (encoder) and a decoding model (decoder). The input image is used as the input of the encoding model, and important features can be extracted from the input image to obtain latent variables 706. The latent variables are then used as the input of the decoding model to output the prediction residual 707.

[0193] Typically, the autoencoder model can be a pre-trained model, and specifically can adopt an autoencoder (AutoEncoder, AE), a variational autoencoder (Variational AutoEncoder, VAE) or VQ-VAE (Vector Quantised-Variational AutoEncoder), etc., which can be adjusted according to the actual application scenario, and this application does not limit this.

[0194] Subsequently, the latent variable 706 may be encoded to obtain a latent variable encoding 708 .

[0195] Specifically, the hidden variables can be encoded by static entropy coding, that is, the data with high probability is represented by a shorter number of bits and the data with low probability is represented by a longer number of bits by using the number structure.

[0196] For example, the tree structure can be Fig.11 As shown, the corresponding bits can be expressed as shown in Table 1.

[0197] character Probability coding <![CDATA[a1]]> 0.4 0 <![CDATA[a2]]> 0.35 10 <![CDATA[a3]]> 0.2 110 <![CDATA[a4]]> 0.05 111

[0198] Table 1

[0199] Therefore, the data a1a2a1a4 is encoded as 0100110.

[0200] In addition, the image residual 704 and the prediction residual 707 may also be encoded to obtain residual coding 709 .

[0201] Specifically, semi-dynamic entropy coding may be performed on the image residual 704 and the prediction residual 707 to obtain residual coding.

[0202] For ease of understanding, the difference between dynamic entropy coding and semi-dynamic entropy coding provided in this application is explained.

[0203] First, taking rANS coding as an example, dynamic coding is to use the state (usually a large integer, or a large and small fraction) to represent data, and use the probability information of the data to change the state value. The final encoding value is 0 or 1 for the state. In rANS coding, we first need to set an M value, which represents the number of bits required to represent a probability. For a character a i , and its corresponding PMF i Proportional to their probability, and the sum is 2 M ; Its corresponding CDF i It is the sum of all previous PMF values, that is, PMF1+PMF2+…+PMF i-1 In the above table, if M=4, the PMF and CDF corresponding to the probability value are shown in Table 2:

[0204] character Probability PMF CDF <![CDATA[a1]]> 0.4 6 0 <![CDATA[a2]]> 0.35 6 6 <![CDATA[a3]]> 0.2 3 12 <![CDATA[a4]]> 0.05 1 15

[0205] Table 2

[0206] If the states before and after compressing a character x are S and S' respectively, then

[0207] S'=S / PMF(x)*2 M +CDF(x)+S%PMF(x)

[0208] Dynamic entropy coding can also be used as static entropy coding. When the value in the table is a fixed value, it is static entropy coding; when the tables of different symbols are not exactly the same, dynamic entropy coding is required.

[0209] The speed bottlenecks in dynamic entropy coding include: symbol search and operations in decompression: among which division and remainder operations are the most time-consuming, followed by multiplication. Therefore, in view of the reduced efficiency caused by the probability distribution method in wireless in dynamic entropy coding, the present application provides a semi-dynamic entropy coding. Based on the aforementioned dynamic entropy coding, that is, the coding formula of rANS, first do approximation processing, such as replacing the multiplication, division, remainder and other operations in dynamic entropy coding with approximate lightweight operations such as addition, subtraction, and bit, and greatly reduce or remove multiplication, division, remainder and other operations under the premise of a small loss of compression rate; then through a series of transformation processing, all operations that take more than a certain time (such as the remaining remainder, multiplication and division, etc.) are converted into table access, and lightweight operations such as addition, subtraction, and bit. It can be understood that the semi-dynamic entropy coding provided by the present application can remove all time-consuming operations such as symbol search, multiplication and division, and remainder through algorithm transformation and tabular processing, and achieve a throughput rate equivalent to that of static entropy coding.

[0210] For example, similar to the commonly used rANS implementation, the state value S is truncated and approximated, but the differences include:

[0211] Unlike the usual rANS, which truncates S to [2 M ,2 2M ) of 2 2M -2 M states; this scheme truncates them to [2 M ,2 M +1 ) of 2 M To achieve a smaller state space, which is convenient for subsequent tabular processing;

[0212] Unlike the usual rANS calculation using division and remainder, this solution changes it to an approximate solution method of loop + bit operation, which is convenient for further reducing the storage space required for tabulation. The loop in this calculation takes a long time, so after this processing, the time consumption usually exceeds the original rANS, but in subsequent processing, the number of loops will be tabulated to achieve efficient compression and decompression.

[0213] During the compression process, for each distribution and symbol, the number of cycles (i.e., the number of right-shifted bits of the state) and the difference between the next state and the current state under this distribution and symbol are pre-calculated and stored using a table. For example, when performing compression, for each input distribution index and symbol, the corresponding δ is obtained by looking up the table, and the number of right-shifted bits of the state b = (δ + S) >> M is calculated; the rightmost b bits of the state are pushed into the memory, and the state value is right-shifted by b bits; the difference between the next state and the current state is obtained by looking up the table through the distribution index and symbol, and this difference is added to the current state value to obtain the updated state value.

[0214] Compared to directly storing the number of loops, this solution stores the intermediate result δ, and the number of loops can be calculated as (δ+S)>>M. The encoding method provided by the present application can reduce the memory space required for the storage table. Compared to directly storing the difference between two states, the semi-dynamic entropy encoding method provided by the present application stores the difference between two states after the state is shifted right. This method can be stored with unsigned numbers, reducing the memory space by half with the same number of bits.

[0215] After obtaining the residual code 709 and the latent variable code 708, subsequent operations may be performed, such as saving the residual code 709 and the latent variable code 708, or transmitting the residual code 709 and the latent variable code 708 to a receiving end, etc., which may be determined according to actual application scenarios.

[0216] Therefore, the method provided in the embodiment of the present application can be applied to image lossless compression to achieve efficient image lossless compression. It also provides an efficient semi-dynamic entropy encoder, so that both the model reasoning and the encoding process are run on the AI ​​chip, reducing the transmission between the system memory and the AI ​​chip memory, and achieving high-bandwidth compression and decompression.

[0217] The above describes the process of the encoding method provided by the present application. The following describes the process of the corresponding decoding method, which is the inverse operation of the above encoding process. Fig.12 , a flowchart of a decoding method provided in the present application is as follows.

[0218] 1201. Obtain latent variable encoding data and residual encoding data.

[0219] Among them, the decoding end can read the latent variable coding data and residual coding data locally, or receive the latent variable coding data and residual coding data sent by the encoding end. The source of the latent variable coding data and residual coding data can be determined according to the actual application scenario, and this application does not limit this.

[0220] Specifically, the latent variable encoded data can be obtained by encoding the features extracted from the input image by the encoder. The residual encoded data can be obtained by encoding the aforementioned image residual and prediction residual by the encoder, and the image residual can include the residual between the input image of the encoder and the image output by the autoregressive model. The latent variable encoded data and the residual encoded data can refer to the aforementioned Figure 6-Figure 11 The relevant introduction in will not be repeated here.

[0221] 1202. Decode the latent variable encoded data to obtain the latent variable.

[0222] The method of decoding the latent variable encoded data may correspond to the encoding end. For example, if the encoding end uses a static entropy encoder for encoding, the static entropy encoder may be used for decoding during decoding. For example, the latent variable encoded data is used as the input of the static entropy encoder, and the latent variable is output. The latent variable may include features extracted from the input image, and for the decompression end, the latent variable represents the features in the decompressed image.

[0223] 1203. Use the latent variable as the input of the autoencoder model and output the second residual distribution.

[0224] After decoding the latent variable encoded data to obtain the latent variable, the latent variable can be used as the input of the autoencoder model to output the corresponding second residual distribution, that is, the image corresponding to the first residual distribution of the aforementioned encoding end, which can be understood as the residual distribution between the image output by the autoregressive model in the encoding end and the input image.

[0225] Specifically, the autoencoder model may include a decoding model, and the latent variables are used as inputs of the decoding model to output a predicted residual image. The decoding model may be a trained model for outputting a residual image corresponding to an input image, and the residual image may be understood as a residual value between a residual image predicted by an autoregressive model and an input image.

[0226] It should be noted that both the encoding end and the decoding end are deployed with an autoregressive model and an autoencoding model, and the autoregressive model on the encoding end is the same as the autoregressive model on the decoding end. If the encoding end and the decoding end are deployed in the same device, the autoencoding models of the encoding end and the decoding end are the same. If the encoding end and the decoding end are deployed in different devices, the encoding end and the decoding end can deploy the same autoencoding model, or deploy a complete autoencoding model on the encoding end and deploy a decoding model in the autoencoding model on the decoding end. The specific adjustment can be made according to the actual application scenario, and this application is not limited to this.

[0227] 1204. Decode the second residual distribution and the residual coding data to obtain a second residual image.

[0228] After the second residual distribution and the residual coding data are obtained, the second residual distribution and the residual coding data can be combined for decoding to obtain a second residual image.

[0229] Specifically, if the encoding end adopts semi-dynamic entropy coding for encoding, the decoding end can also decode based on the semi-dynamic entropy coding to output a second residual image, that is, an image corresponding to the first residual image of the encoding end. The semi-dynamic entropy encoder is used to perform entropy encoding using a first preset type of encoding operation, the first preset type of encoding operation includes addition, subtraction or bit operation, and the semi-dynamic entropy encoder does not include a second preset type of encoding operation, the second preset type includes at least one of multiplication, division or remainder operation, that is, the semi-dynamic entropy encoder does not include multiplication, division or remainder operation and other time-consuming operations, that is, the semi-dynamic entropy encoder can only include simple addition and subtraction operations, so that efficient encoding can be achieved.

[0230] More specifically, the semi-dynamic entropy encoder can participate in the aforementioned Figure 6-Figure 11 The relevant description in will not be repeated here.

[0231] It can be understood that, for the process of encoding the first residual image and the first residual distribution at the aforementioned encoding end to obtain residual coding data, after the decoding end obtains the second residual distribution and residual coding data, an inverse operation can be performed to infer the second residual distribution, which is equivalent to obtaining the residual between the first image output by the autoregressive model of the aforementioned encoding end and the input image, that is, the first residual distribution.

[0232] 1205. Use the second residual image as the input of the back propagation of the autoregressive model, and output the decompressed image.

[0233] After the second residual image is obtained, the second residual distribution can be used as the input of the autoregressive model for back propagation to infer the decompressed image, that is, to achieve lossless restoration of the input image at the encoding end.

[0234] In addition, when the second residual image is used as the input of the autoregressive model for back propagation, if the autoregressive model at the encoding end uses the pixel values ​​of the predicted pixels to predict the values ​​of the pixels on the same line, when the decoding end performs the decoding operation, the values ​​of the pixels on the same line can be decoded in parallel to achieve efficient decoding. The same line can be the same row, the same column, or the same diagonal line, etc., which can be determined according to the actual application scenario.

[0235] Therefore, in the implementation mode of the present application, the autoencoder model usually has poor fitting ability and requires a deeper network to achieve a better compression rate, while the present application combines the output results of the autoregressive model, thereby reducing the size of the autoencoder model. Therefore, in the present application, the autoregressive model and the autoencoder model are combined for decoding, and both the autoencoder and the autoregressive model can be controlled to be very small, avoiding the problem of long reasoning time caused by the large network of the autoencoder model, and achieving efficient image decompression. Moreover, in the method provided in the present application, the entire process can be implemented based on AI lossless compression of the AI ​​chip, including AI model and entropy coding, avoiding the transmission problem between system memory and AI chip memory, and improving coding efficiency.

[0236] For ease of understanding, the following describes the decoding method provided by this application in conjunction with specific application scenarios. Fig.13 , a flow chart of another decoding method provided by the present application is as follows.

[0237] First, latent variable code 1301 and residual code 1302 are obtained.

[0238] The latent variable code 1301 and the residual code 1302 may be read locally or received from the encoding end, and may be adjusted according to the actual application scenario. Figure 7 The latent variable encoding 708 and the residual encoding 709 mentioned in .

[0239] Subsequently, the latent variable encoding 1301 is input into the static entropy encoder 1303, and the latent variable 1304 is output.

[0240] Typically, the bits corresponding to each probability in entropy coding can be as shown in Table 1 above. After obtaining the bit stream encoded by the latent variable, the probability corresponding to each character can be determined according to the corresponding relationship, thereby outputting the latent variable, which can be understood as an important feature in the decompressed image.

[0241] The latent variable 1304 is then used as the input of the decoding model in the autoencoder model 1305, and the prediction residual 1306 is output.

[0242] Among them, the decoding model is the same as the above Figure 7 The prediction residual 1306 is similar to the above-mentioned prediction residual 707, and will not be described in detail here.

[0243] The residual coding 1302 and the predicted residual 1306 are then used as inputs to a semi-dynamic entropy encoder, and an image residual 1308 is output.

[0244] The image residual 1308 is similar to the aforementioned image residual 704 and will not be described again here.

[0245] Among them, the decoding process of semi-dynamic entropy coding can be understood as the inverse operation of the aforementioned semi-dynamic entropy coding, that is, given the prediction residual and the residual coding, the image residual is deduced in reverse. For example: find the state value of the current symbol: s = S' % 2 M , find the symbol x corresponding to s, and decompress x. It is necessary to satisfy CDF(x) ≤ s < CDF(x) + PMF(x). According to the decompressed symbol x, restore the state value of the previous step: S = S' / 2 M *PMF(x) + S' % 2 M –PMF(x).

[0246] After obtaining the image residual 1308, this image residual can be used as the input for the backpropagation of the autoregressive model 1309, thereby deducing the decompressed image 1310.

[0247] It can be understood that the autoregressive model 1309 is a trained model, which is the same as the aforementioned autoregressive model 702. It can be understood that given the image residual, the input image 701 is deduced in reverse.

[0248] Optionally, if the encoding end uses the pixel values of the predicted pixel points to predict the pixel values of the pixel points on the same line in parallel when outputting the prediction residual through the autoregressive model, then when performing backpropagation on the autoregressive model, the pixel values of the pixel points on the same line can be decoded, thereby achieving parallel decoding.

[0249] For example, given an m×n image and a hyperparameter h (0 ≤ h < n), if for any pixel point (i, j), all points (i′, j′) in the autoregressive model that predict (i, j) satisfy: h×i′ + j′ < h×i + j, then this image can be decompressed through n + (m - 1)×h parallel computations. The decompression order includes:

[0250] Decompress the points in the first row in sequence: (0,0), (0,1),…, (0,n - 1). When decompressing the point (0, j), if j - h ≥ 0, then decompress (1, j - h) simultaneously; if j - h×2 ≥ 0, then decompress (2, j - h×2) simultaneously, and so on;

[0251] Decompress the points in the second row in sequence: (1,n - h - 1),…, (1,n - 1). When decompressing the point (1, j), if j - h ≥ 0, then decompress (2, j - h) simultaneously; if j - h×2 ≥ 0, then decompress (3, j - h×2) simultaneously, and so on;

[0252] Decompress according to this rule until all points are decompressed.

[0253] Therefore, the method of encoding and decoding the same line in parallel provided by the present application can greatly improve the encoding and decoding efficiency and achieve more efficient image compression.

[0254] For ease of understanding, the effects achieved by this application are exemplified below using some specific application scenarios as examples.

[0255] First, it is necessary to construct a neural network model with autoregressive model and autoencoder model as the core. In this technical solution, the autoregressive model implements a lightweight design and contains only 12 parameters. For a three-channel image, only 4 parameters are needed for prediction for each channel. The autoencoder model uses a vector quantized autoencoder, which uses a vector codebook to reduce the space of latent variables and sets the codebook size to 256, that is, the value space of the latent variables in the autoencoder is limited to 256 integers. The encoder and decoder of the autoencoder both use four residual convolution blocks, and the number of channels of each layer of features is 32.

[0256] The model training and testing process are as follows:

[0257] Training: Train on the training set of a single data set to obtain the parameters of the autoregressive model, the autoencoder model, and the statistics of the latent variables for the compression of the latent variables.

[0258] Compression: Through the method provided by this application, all test images of a single dataset are stacked together in the batch dimension to form a four-dimensional tensor. This four-dimensional tensor is used as the input of the process at one time, and the residual encoding of all images and the encoding of the latent variables are output in parallel.

[0259] Decompression: The residual coding and latent variables of all images are used as input to the decompression process at one time through the method provided in this application, and the original images of all images are output in parallel.

[0260] Compared with some commonly used lossless compression, such as L3C (Practical Full Resolution Learned Lossless Image Compression), FLIF (free lossless image format based on maniac compression), WebP or PNG (Portable Network Graphics), the method provided in this application is called PILC (Practical Image Lossless Compression), see Table 3.

[0261]

[0262]

[0263] Table 3

[0264] As can be seen from Table 3, compared with the previous AI image lossless compression algorithm - L3C, this technical invention increases the throughput by 14 times while maintaining basically the same compression rate. At the same time, this technical invention is also superior to traditional methods such as PNG, WebP, and FLIF in terms of compression rate and throughput.

[0265] Therefore, the method provided in this application combines the autoregressive model with the autoencoder model, which greatly reduces the model size compared to using the autoencoder model alone. And the autoregressive model provided in this application can realize parallel encoding and parallel decompression, can realize efficient encoding and decoding, and realize efficient image compression and decompression. And the process of the method provided in this application can be run on the AI ​​chip, avoiding the information transmission between the system memory and the AI ​​chip memory, and further improving the encoding and decoding efficiency.

[0266] In addition, in the actual production environment, the sizes of images are basically different, and the resolution of the images is relatively high. In order to efficiently compress and decompress high-definition images of different sizes, this embodiment is designed as follows.

[0267] Model training: During the model training phase, high-definition large data sets such as OpenImage and ImageNet64 are used for model training to obtain the parameters of the autoregressive model and the autoencoder model.

[0268] compression:

[0269] First, the image is preprocessed to slice large high-definition images of different sizes into the same size (such as 32x32), and the size information of each image is stored separately for image restoration;

[0270] Stack all slices together in the batch dimension as input to the process;

[0271] Output the residual encoding of all images in parallel with the encoding of the latent variables;

[0272] The statistical information of the latent variables for each dataset (same dataset) / each image (different datasets) is recorded as another output of the process.

[0273] The achieved effects can be seen in Table 4.

[0274]

[0275] Table 4

[0276] Obviously, the method provided by this application can achieve higher throughput and realize efficient encoding and decoding.

[0277] More specifically, some commonly used compression methods are compared in more detail below.

[0278] Refer to Table 5. Under the premise that the maximum likelihood (an indicator for evaluating the prediction accuracy of the generative model, the smaller the better) is basically consistent with the previous fastest AI algorithm L3C, the reasoning speed is increased by 9.6 times.

[0279]

[0280] Table 5

[0281] Referring to Table 6, for the same autoregressive model, the decompression speed is increased by 7.9 times using the parallel solution provided by the present application compared with the non-parallel solution. The parallel solution has a limit on the receptive field, but the influence of this receptive field on the compression rate is limited.

[0282] Receptive Field Is it parallel? BPD Throughput (MB / s) 3 yes 5.77 382.5 3 no 5.77 48.5 4 no 5.77 47.5 7 no 5.74 44.0

[0283] Table 6

[0284] Referring to Table 7, compared with dynamic entropy coding (rANS), the semi-dynamic entropy coding (ANS-AI) proposed in this application has a 20-fold increase in encoding speed, a 100-fold increase in decoding speed, and a BPD loss of less than 0.55 and 0.17. Moreover, this semi-dynamic entropy coding can be run on an AI chip, and on a single V100, the peak speed can reach 1GB / s.

[0285]

[0286] Table 7

[0287] In addition, compared with dynamic entropy coding, the number of distribution types required for semi-dynamic entropy coding is reduced from 2048 to 8, the memory size required for preprocessing is reduced to 1 / 256 of the original, and the BPD loss is less than 0.03, which can reduce the computing resources required for entropy coding and improve coding efficiency.

[0288] The above describes the process of the image encoding method and the image decompression method provided by the present application. The following describes the device for executing the above method.

[0289] See also Fig.14 , a schematic diagram of the structure of an image encoding device provided by the present application, the image encoding device comprising:

[0290] An autoregressive module 1401 is used to take an input image as an input of an autoregressive model and output a first image and an autoregressive model;

[0291] A residual calculation module 1402 is used to obtain a residual between the first image and the input image to obtain a first residual image;

[0292] The autoencoder module 1403 is used to use the input image as an input of the autoencoder model, and output a latent variable and a first residual distribution, wherein the latent variable includes features extracted from the input image, and the first residual distribution includes a residual value output by the autoencoder model for representing each pixel in the input image and each corresponding pixel in the first residual image;

[0293] A residual coding module 1404 is used to encode the first residual image and the first residual distribution to obtain residual coding data;

[0294] The latent variable encoding module 1405 is used to encode the latent variable to obtain latent variable encoded data. The latent variable encoded data and the residual encoded data are used to obtain the input image after decompression.

[0295] In one possible implementation, the residual coding module 1404 is specifically used to use the first residual image and the first residual distribution as inputs of a semi-dynamic entropy encoder, and output residual coding data. The semi-dynamic entropy encoder is used to perform entropy coding using a first preset type of coding operation. The first preset type of coding operation includes addition, subtraction or bit operation, and the semi-dynamic entropy encoder does not include a second preset type of coding operation. The second preset type includes at least one of multiplication, division or remainder operation, that is, the semi-dynamic entropy encoder does not include time-consuming operations such as multiplication, division or remainder operation.

[0296] In a possible implementation, the semi-dynamic entropy encoder may be obtained by converting a dynamic upper encoder. Specifically, the operation of the dynamic entropy encoder may be approximated, such as replacing the operation of the dynamic entropy encoder with an approximate operation, reducing or removing operations such as multiplication, division, and remainder, and then performing a transformation process to transform the operation, thereby converting all operations that take more than a certain time (such as remaining remainder, multiplication, and division losses) into table access, and lightweight operations such as addition, subtraction, and bit operations to obtain the semi-dynamic entropy encoder provided by the present application. It can be understood that the semi-dynamic entropy encoder may be an entropy encoder obtained by replacing or converting some operations in the dynamic entropy encoder. When the semi-dynamic entropy encoder is used for entropy encoding, simple operations such as addition, subtraction, bit operations, and other efficient encoding operations can be used, thereby achieving efficient encoding.

[0297] In a possible implementation, the latent variable encoding module 1405 is specifically configured to use the latent variable as an input of a static entropy encoder to obtain latent variable encoded data.

[0298] In one possible implementation, the autoencoding model includes an encoding model and a decoding model, and the autoencoding module 1403 is specifically used to: use the input image as the input of the encoding model and output latent variables, and the encoding model is used to extract features from the input image; use the latent variables as the input of the decoding model to obtain a first residual distribution, and the decoding model is used to predict the residual value between the input image and the corresponding pixel distribution.

[0299] In a possible implementation, the autoregressive model is used to predict the values ​​of pixels on the same line using the pixel values ​​of predicted pixels.

[0300] See also Fig.15 , a schematic diagram of the structure of an image decompression device provided by the present application, the image decompression device comprises:

[0301] The transceiver module 1501 is used to obtain latent variable coded data and residual coded data, wherein the latent variable coded data includes data obtained by encoding the features extracted by the encoder from the input image, and the residual coded data includes data obtained by encoding the residual between the first image output by the autoregressive model and the input image;

[0302] A latent variable decoding module 1502 is used to decode the latent variable encoded data to obtain a latent variable, where the latent variable includes a feature extracted from the input image by the encoding end;

[0303] An autoencoder module 1503, configured to use the latent variable as an input of the autoencoder model and output a second residual distribution;

[0304] A residual decoding module 1504, configured to perform decoding in combination with the second residual distribution and the residual coding data to obtain a second residual image;

[0305] The autoregressive module 1505 is used to use the second residual image as the input of the back propagation of the autoregressive model and output the decompressed image.

[0306] In a possible implementation, the latent variable decoding module 1502 is specifically configured to use the latent variable encoded data as input of a static entropy encoder to output the latent variable.

[0307] In one possible implementation, the residual decoding module 1504 is specifically used to use the second residual distribution and the residual coding data as input of a semi-dynamic entropy encoder to output a second residual image. The semi-dynamic entropy encoder is used to perform entropy coding using a first preset type of coding operation. The first preset type of coding operation includes addition, subtraction or bit operation, and the semi-dynamic entropy encoder does not include a second preset type of coding operation. The second preset type includes at least one of multiplication, division or remainder operation, that is, the semi-dynamic entropy encoder does not include time-consuming operations such as multiplication, division or remainder operation.

[0308] In a possible implementation, the semi-dynamic entropy encoder may be obtained by converting a dynamic upper encoder. Specifically, the operation of the dynamic entropy encoder may be approximated, such as replacing the operation of the dynamic entropy encoder with an approximate operation, reducing or removing operations such as multiplication, division, and remainder, and then performing a transformation process to transform the operation, thereby converting all operations that take more than a certain time (such as remaining remainder, multiplication, and division losses) into table access, and lightweight operations such as addition, subtraction, and bit operations to obtain the semi-dynamic entropy encoder provided by the present application. It can be understood that the semi-dynamic entropy encoder may be an entropy encoder obtained by replacing or converting some operations in the dynamic entropy encoder. When the semi-dynamic entropy encoder is used for entropy encoding, simple operations such as addition, subtraction, bit operations, and other efficient encoding operations can be used, thereby achieving efficient encoding.

[0309] In a possible implementation, the autoregressive module 1505 is specifically configured to perform parallel decoding on pixel points on the same line in the second residual image through an autoregressive model to obtain a decompressed image.

[0310] See also Fig.16 , a structural schematic diagram of another image encoding device provided by the present application is described as follows.

[0311] The image coding device may include a processor 1601 and a memory 1602. The processor 1601 and the memory 1602 are interconnected via a line. The memory 1602 stores program instructions and data.

[0312] The memory 1602 stores the aforementioned Figure 6-Figure 11 The program instructions and data corresponding to the steps in .

[0313] Processor 1601 is used to execute the above Figure 6-Figure 11 The method steps are performed by the image encoding device shown in any one of the embodiments.

[0314] Optionally, the image encoding apparatus may further include a transceiver 1603 for receiving or sending data.

[0315] In an embodiment of the present application, a computer-readable storage medium is also provided, in which a program for generating a vehicle driving speed is stored. When the computer is running on a computer, the computer executes the above-mentioned Figure 6-Figure 11 The illustrated embodiments describe steps in a method.

[0316] Optionally, the aforementioned Fig.16 The image encoding device shown in is a chip.

[0317] The embodiment of the present application also provides an image coding device, which can also be called a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface. The program instructions are executed by the processing unit. The processing unit is used to execute the aforementioned Figure 6-Figure 11 The method steps are performed by the image encoding device shown in any one of the embodiments.

[0318] The embodiment of the present application also provides a digital processing chip. The digital processing chip integrates a circuit and one or more interfaces for implementing the above-mentioned processor 1601 or the function of the processor 1601. When the digital processing chip integrates a memory, the digital processing chip can complete the method steps of any one or more embodiments in the above-mentioned embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory through a communication interface. The digital processing chip implements the actions performed by the image encoding device in the above-mentioned embodiment according to the program code stored in the external memory.

[0319] The image encoding device provided in the embodiment of the present application may be a chip, and the chip includes: a processing unit and a communication unit, wherein the processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit. The processing unit may execute the computer execution instructions stored in the storage unit, so that the chip in the server executes the above Figure 6-Figure 11 The image encoding method described in the illustrated embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc., and the storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0320] See also Fig.17 , a structural schematic diagram of another image decompression device provided by the present application is described as follows.

[0321] The image decompression device may include a processor 1701 and a memory 1702. The processor 1701 and the memory 1702 are interconnected via a line. The memory 1702 stores program instructions and data.

[0322] The memory 1702 stores the aforementioned Figure 12-13 The program instructions and data corresponding to the steps in .

[0323] Processor 1701 is used to execute the above Figure 12-13 The method steps are performed by the image decompression device shown in any one of the embodiments.

[0324] Optionally, the image decompression device may further include a transceiver 1703 for receiving or sending data.

[0325] In an embodiment of the present application, a computer-readable storage medium is also provided, in which a program for generating a vehicle driving speed is stored. When the computer is running on a computer, the computer executes the above-mentioned Figure 12-13 The illustrated embodiments describe steps in a method.

[0326] Optionally, the aforementioned Fig.17 The image decompression device shown in is a chip.

[0327] The embodiment of the present application also provides an image decompression device, which can also be called a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface. The program instructions are executed by the processing unit. The processing unit is used to execute the aforementioned Figure 12-13 The method steps are performed by the image decompression device shown in any one of the embodiments.

[0328] The embodiment of the present application also provides a digital processing chip. The digital processing chip integrates a circuit and one or more interfaces for implementing the above-mentioned processor 1701 or the functions of the processor 1701. When the digital processing chip integrates a memory, the digital processing chip can complete the method steps of any one or more of the above-mentioned embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory through a communication interface. The digital processing chip implements the actions performed by the image decompression device in the above-mentioned embodiment according to the program code stored in the external memory.

[0329] The image decompression device provided in the embodiment of the present application may be a chip, and the chip includes: a processing unit and a communication unit, wherein the processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit. The processing unit may execute the computer execution instructions stored in the storage unit, so that the chip in the server executes the above Figure 6-Figure 11 The image decompression method described in the illustrated embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc., and the storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0330] The present application also provides a computer program product which, when executed on a computer, enables the computer to execute the above Figure 6-Figure 13 The illustrated embodiments describe an image decompression device or steps performed by the image decompression device in the method.

[0331] The present application also provides an image processing system, including an image encoding device and an image decompression device, wherein the image encoding device is used to perform the above Figure 6-Figure 11 Corresponding method steps, the image decompression device is used to perform the aforementioned Figure 12-13 The corresponding method steps.

[0332] Specifically, the aforementioned processing unit or processor may be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0333] For example, see Fig.18 , Fig.18 A schematic diagram of the structure of a chip provided in an embodiment of the present application, the chip can be expressed as a neural network processor NPU 180, NPU 180 is mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 1803, which is controlled by the controller 1804 to extract matrix data from the memory and perform multiplication operations.

[0334] In some implementations, the operation circuit 1803 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 1803 is a two-dimensional systolic array. The operation circuit 1803 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1803 is a general-purpose matrix processor.

[0335] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 1802 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1801 and performs matrix operations with matrix B. The partial results or final results of the obtained matrix are stored in the accumulator 1808.

[0336] The unified memory 1806 is used to store input data and output data. The weight data is directly transferred to the weight memory 1802 through the direct memory access controller (DMAC) 1805. The input data is also transferred to the unified memory 1806 through the DMAC.

[0337] The bus interface unit (BIU) 1810 is used for the interaction between the AXI bus and the DMAC and instruction fetch buffer (IFB) 1809 .

[0338] The bus interface unit 1810 (BIU) is used for the instruction fetch memory 1809 to obtain instructions from the external memory, and is also used for the storage unit access controller 1805 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0339] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1806 or to transfer weight data to the weight memory 1802 or to transfer input data to the input memory 1801.

[0340] The vector calculation unit 1807 includes multiple operation processing units, which further process the output of the operation circuit when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, upsampling of feature planes, etc.

[0341] In some implementations, the vector calculation unit 1807 can store the processed output vector to the unified memory 1806. For example, the vector calculation unit 1807 can apply a linear function and / or a nonlinear function to the output of the operation circuit 1803, such as linear interpolation of the feature plane extracted by the convolution layer, and then, for example, a vector of accumulated values ​​to generate an activation value. In some implementations, the vector calculation unit 1807 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1803, for example, for use in a subsequent layer in a neural network.

[0342] An instruction fetch buffer 1809 connected to the controller 1804 is used to store instructions used by the controller 1804;

[0343] Unified memory 1806, input memory 1801, weight memory 1802 and instruction fetch memory 1809 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0344] Among them, the operations of each layer in the recurrent neural network can be performed by the operation circuit 1803 or the vector calculation unit 1807.

[0345] The processor mentioned in any of the above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more processors for controlling the above Figure 6-Figure 13 The method is performed by a program of an integrated circuit.

[0346] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0347] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0348] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0349] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)), etc.

[0350] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0351] Finally, it should be noted that the above is only a specific implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. An image coding method, characterized in that: include: Taking the input image as the input of the autoregressive model and outputting the first image; Obtaining a residual between the first image and the input image to obtain a first residual image; Taking the input image as an input of an autoencoder model, outputting a latent variable and a first residual distribution, wherein the latent variable includes features extracted from the input image, and the first residual distribution includes a residual value output by the autoencoder model for representing each pixel in the input image and each corresponding pixel in the first residual image; Encoding the first residual image and the first residual distribution to obtain residual encoding data; The latent variable is encoded to obtain latent variable encoded data, and the latent variable encoded data and the residual encoded data are used to obtain the input image after decompression.

2. The method according to claim 1, characterized in that The encoding of the first residual image and the first residual distribution to obtain residual encoding data includes: The first residual image and the first residual distribution are used as inputs of a semi-dynamic entropy encoder, and the residual coding data is outputted. The semi-dynamic entropy encoder is used to perform entropy coding using a first preset type of coding operation, wherein the first preset type of coding operation includes addition, subtraction or bit operation, and the semi-dynamic entropy encoder does not include a second preset type of coding operation, wherein the second preset type includes at least one of multiplication, division or modulo operation.

3. The method according to claim 1, characterized in that The step of encoding the latent variable to obtain residual coded data includes: The latent variable is used as the input of a static entropy encoder to obtain the latent variable encoded data.

4. The method according to any one of claims 1 to 3, characterized in that The autoencoder model includes an encoding model and a decoding model, and the input image is used as an input of the autoencoder model, and the hidden variable and the first residual distribution are output, including: Taking the input image as input of the encoding model and outputting the latent variable, the encoding model is used to extract features from the input image; The latent variable is used as an input of the decoding model to obtain the first residual distribution, and the decoding model is used to predict the residual value between the input image and the corresponding pixel distribution.

5. The method according to any one of claims 1 to 3, characterized in that: The autoregressive model is used to predict the values ​​of pixels on the same line using the pixel values ​​of the predicted pixels.

6. An image decompression method, characterized in that: include: Obtaining latent variable encoded data and residual encoded data, wherein the latent variable encoded data is obtained by encoding features extracted from an input image by an encoding end, and the residual encoded data includes data obtained by encoding a residual between the input image and an image output by forward propagation of an autoregressive model; Decoding the latent variable encoded data to obtain a latent variable, wherein the latent variable includes features extracted from the input image; Using the latent variable as an input of the autoencoder model, and outputting a second residual distribution; Decoding the second residual distribution and the residual coded data to obtain a second residual image; The second residual image is used as the input of the back propagation of the autoregressive model, and the decompressed image is output.

7. The method according to claim 6, characterized in that The decoding of the latent variable encoded data to obtain the latent variable includes: The latent variable encoded data is used as input of a static entropy encoder to output the latent variable.

8. The method according to claim 6 or 7, characterized in that: The decoding in combination with the second residual distribution and the residual coded data to obtain a second residual image includes: The second residual distribution and the residual coding data are used as input of a semi-dynamic entropy encoder to output the second residual image. The semi-dynamic entropy encoder is used to perform entropy coding using a first preset type of coding operation, wherein the first preset type of coding operation includes addition, subtraction or bit operation, and the semi-dynamic entropy encoder does not include a second preset type of coding operation, wherein the second preset type includes at least one of multiplication, division or modulo operation.

9. The method according to claim 6 or 7, characterized in that: The step of using the second residual image as the input of the back propagation of the autoregressive model and outputting the decompressed image comprises: The pixel points on the same line in the second residual image are decoded in parallel through the autoregressive model to obtain the decompressed image.

10. An image encoding device, characterized in that: include: An autoregressive module, used to take the input image as an input of the autoregressive model and output a first image; A residual calculation module, used to obtain a residual between the first image and the input image to obtain a first residual image; An autoencoder module, configured to use the input image as an input of an autoencoder model, and output a latent variable and a first residual distribution, wherein the latent variable includes features extracted from the input image, and the first residual distribution includes a residual value output by the autoencoder model for representing each pixel in the input image and each corresponding pixel in the first residual image; A residual encoding module, used for encoding the first residual image and the first residual distribution to obtain residual encoding data; The latent variable encoding module is used to encode the latent variable to obtain latent variable encoded data, and the latent variable encoded data and the residual encoded data are used to obtain the input image after decompression.

11. The device according to claim 10, characterized in that The residual coding module is specifically used to use the first residual image and the first residual distribution as input of a semi-dynamic entropy encoder, and output the residual coding data. The semi-dynamic entropy encoder is used to perform entropy coding using a first preset type of coding operation, the first preset type of coding operation includes addition, subtraction or bit operation, and the semi-dynamic entropy encoder does not include a second preset type of coding operation, the second preset type includes at least one of multiplication, division or modulo operation.

12. The device according to claim 10, characterized in that The latent variable encoding module is specifically used to use the latent variable as the input of the static entropy encoder to obtain the latent variable encoded data.

13. The device according to any one of claims 10 to 12, characterized in that The auto-encoding model includes an encoding model and a decoding model, and the auto-encoding module is specifically used for: Taking the input image as input of the encoding model and outputting the latent variable, the encoding model is used to extract features from the input image; The latent variable is used as an input of the decoding model to obtain the first residual distribution, and the decoding model is used to predict the residual value between the input image and the corresponding pixel distribution.

14. The device according to any one of claims 10 to 12, characterized in that The autoregressive model is used to predict the values ​​of pixels on the same line using the pixel values ​​of the predicted pixels.

15. An image decompression device, characterized in that: include: A transceiver module, used to obtain latent variable coded data and residual coded data, wherein the latent variable coded data includes data obtained by encoding the features extracted by the encoder from the input image, and the residual coded data includes data obtained by encoding the residual between the first image output by the forward propagation of the autoregressive model and the input image; A latent variable decoding module, used for decoding the latent variable encoded data to obtain a latent variable, wherein the latent variable includes a feature extracted from the input image; An autoencoder module, used for taking the latent variable as an input of an autoencoder model and outputting a second residual distribution; A residual decoding module, used for decoding the second residual distribution and the residual coded data to obtain a second residual image; The autoregressive module is used to use the second residual image as the input of the back propagation of the autoregressive model and output the decompressed image.

16. The device according to claim 15, characterized in that The latent variable decoding module is specifically used to use the latent variable encoded data as the input of the static entropy encoder and output the latent variable.

17. The device according to claim 15 or 16, characterized in that The residual decoding module is specifically used to use the second residual distribution and the residual encoding data as input of a semi-dynamic entropy encoder to output the second residual image, wherein the semi-dynamic entropy encoder is used to perform entropy encoding using a first preset type of encoding operation, wherein the first preset type of encoding operation includes addition, subtraction or bit operation, and the semi-dynamic entropy encoder does not include a second preset type of encoding operation, wherein the second preset type includes at least one of multiplication, division or remainder operation.

18. The device according to claim 15 or 16, characterized in that The autoregressive module is specifically used to perform parallel decoding on pixel points on the same line in the second residual image through the autoregressive model to obtain the decompressed image.

19. An image encoding device, characterized in that: The method comprises a processor, wherein the processor is coupled to a memory, wherein the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the steps of the method according to any one of claims 1 to 5 are implemented.

20. An image decompression device, characterized in that: The method comprises a processor, wherein the processor is coupled to a memory, wherein the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the steps of the method according to any one of claims 6 to 9 are implemented.

21. An image processing system, characterized in that: The invention comprises an image encoding device and an image decompression device, wherein the image encoding device is used to implement the steps of the method described in any one of claims 1 to 5, and the image decompression device is used to implement the steps of the method described in any one of claims 6 to 9.

22. A computer-readable storage medium comprising a program, which, when executed by a processing unit, performs the steps of the method according to any one of claims 1 to 9.

23. A computer program product, characterized in that The computer program product comprises software code for executing the steps of the method according to any one of claims 1 to 9.

Images