Entropy estimation method and apparatus for image compression

Abstract

The present application provides an entropy estimation method and apparatus for image compression. The method comprises: acquiring a first feature matrix to be processed, the shape of the first feature matrix being [Cin, hin, win], wherein Cin represents the number of channels of the first feature matrix, hin represents the height of the first feature matrix, and win represents the width of the first feature matrix; and inputting the first feature matrix into an entropy estimation network to obtain a second feature matrix, the shape of the second feature matrix being [Cout, hout, wout], wherein Cout represents the number of channels of the second feature matrix, hout represents the height of the second feature matrix, and wout represents the width of the second feature matrix. The second feature matrix and the first feature matrix meet the following conditions: Cout=Cin, hout=s·hin, wout=s·win, wherein s is an integer greater than 1. The entropy estimation network comprises a first convolutional layer and pointwise operations, the first convolutional layer comprises two convolution processes, one of the two convolution processes is a two-dimensional grouped convolution, and inputs of the pointwise operations are respective outputs of the two convolution processes. The present application can improve the compression performance.

Description

Entropy estimation method and device for image compression Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an entropy estimation method and device for image compression. Background Art

[0002] As convolutional neural networks (CNNs) outperform traditional algorithms in computer vision tasks such as image recognition and object detection, a growing number of researchers are exploring deep learning-based image / video compression methods. Some researchers have designed end-to-end deep learning image / video compression networks, comprising modules such as an encoding network, an entropy estimation network, an entropy encoding network, an entropy decoding network, and a decoding network. The entropy estimation network is a fundamental component of image / video compression networks, primarily used to estimate the probability distribution (mean or variance) of the signal to be encoded / decoded. The accuracy of the entropy estimation network's probability distribution estimate directly affects the size of the bitstream, and thus the overall compression performance. Therefore, designing an efficient entropy estimation network is key to improving image / video compression performance.

[0003] Summary of the Invention

[0004] The embodiments of the present application provide an entropy estimation method and apparatus for image compression, which can improve compression performance.

[0005] In a first aspect, the present application provides an entropy estimation method for image compression, comprising: obtaining a first feature matrix to be processed, wherein the shape of the first feature matrix is ​​[C in ,h in ,w in ], where C in represents the number of channels of the first feature matrix, h in represents the height of the first feature matrix, w in Represents the width of the first feature matrix; the first feature matrix is ​​input into the entropy estimation network to obtain the second feature matrix, the shape of the second feature matrix is ​​[C out ,h out ,w out ], where C out represents the number of channels of the second feature matrix, h out represents the height of the second characteristic matrix, w out represents the width of the second characteristic matrix; the second characteristic matrix and the first characteristic matrix meet the following conditions: C out =C in , h out =s·h in , w out =s·win , s is an integer greater than 1; the entropy estimation network includes a first convolutional layer and a point-by-point operation, the first convolutional layer includes two convolution processes, one of the two convolution processes is a two-dimensional group convolution (also called group convolution process), and the input of the point-by-point operation is the output of each of the two convolution processes.

[0006] In a possible implementation, the second characteristic matrix and the first characteristic matrix satisfy the following conditions: out =C in , h out =4h in , w out =4w in ; The entropy estimation network includes the third convolutional layer, the first activation function, the first convolutional layer, the point-by-point operation, the second activation function and the 4x upsampling process in sequence.

[0007] In an embodiment of the present application, in an entropy estimation network, convolution operations and point-by-point operations are combined to implement a local attention mechanism, which uses the local information of the input feature matrix to modulate the response of each position in the feature matrix, thereby improving compression performance. In addition, the use of group convolution can expand the receptive field while reducing the amount of computation.

[0008] According to the above description, the entropy estimation network can obtain the estimated probability value of each feature element based on the image feature estimation, which is used for processing by the entropy coding network and the entropy decoding network. Based on this, the first feature matrix can be the feature matrix obtained by quantizing the original image after being processed by the coding network. For example, the first feature matrix can be represented as input[C in ,h in ,w in ], C in Indicates the number of channels of the first feature matrix, h in Represents the height of the first characteristic matrix, w in Represents the width of the first characteristic matrix, that is, [C in ,h in ,w in ] represents the shape of the first feature matrix.

[0009] The second characteristic matrix can be expressed as output[C out ,h out ,w out ], C out Indicates the number of channels of the second feature matrix, h out Represents the height of the second characteristic matrix, w out Represents the width of the second characteristic matrix, that is, [C out ,h out ,w out] represents the shape of the second characteristic matrix.

[0010] In the embodiment of the present application, the second characteristic matrix and the first characteristic matrix satisfy:

[0011] C out =C out

[0012] h out =s·h in

[0013] w out =s·w in

[0014] Wherein, s is an integer greater than 1, which is used to represent the stride of the sliding window. Optionally, s=4.

[0015] The entropy estimation network sequentially includes a third convolutional layer, a first activation function, a first convolutional layer, a point-by-point operation, a second activation function, and an s-fold upsampling process, wherein the first convolutional layer includes two convolution processes, and the input of the point-by-point operation is the output of each of the two convolution processes.

[0016] The input feature matrix of any of the above convolutional layers is convolved to obtain an output feature matrix with unchanged height and width. The output feature matrix can be considered as a local response of the input feature matrix. The convolution operation can be expressed as the following formula:

[0017] Conv(x)=∑w*x+b

[0018] Among them, w is the weight of the convolutional layer, and b is the bias parameter of the convolutional layer, both of which are unrestricted learnable parameters.

[0019] Any of the above activation operations is a point-by-point operation, that is, an independent mapping operation is performed on each element of the input feature matrix. The activation operation can be an identity operation (Identity), a rectified linear unit (Rectified Linear Unit, ReLU), a rectified linear unit (LeakyReLU), a parametric rectified linear unit (PReLU), Sigmoid, Tanh, etc., which can provide linear or nonlinear transformation characteristics. In the embodiment of the present application, the first activation function and the second activation function can use the same operation or different operations, and there is no specific limitation on this. It should be noted that the activation function can also use other deformation methods of ReLU, and the embodiment of the present application does not make any specific limitations on this.

[0020] These point-by-point operations can include point-by-point multiplication, addition, subtraction, or division. The input to these point-by-point operations is the output feature matrices of the two convolutional processes of the first convolutional layer. These two output feature matrices have the same size, and the point-by-point operations multiply, add, subtract, or divide the values ​​of the elements at corresponding positions. The combination of convolutional operations and point-by-point operations implements a local attention mechanism, leveraging the local information of the input feature matrix to modulate the response at each position in the feature matrix.

[0021] In the embodiment of the present application, the s-fold upsampling process can be implemented in multiple ways:

[0022] (1) The s-fold upsampling process includes the second convolutional layer and the s-fold pixel shuffling layer in sequence.

[0023] The eighth feature matrix is ​​input into the second convolutional layer and outputs the ninth feature matrix. The ninth feature matrix has the same size as the eighth feature matrix, but the number of channels of the ninth feature matrix is ​​n times the number of channels of the first feature matrix. The ninth feature matrix is ​​input into the pixel shuffling layer and outputs the second feature matrix. The height of the second feature matrix is ​​s times the height of the ninth feature matrix, the width of the second feature matrix is ​​s times the width of the ninth feature matrix, and the number of channels of the second feature matrix is ​​1 / n of the number of channels of the ninth feature matrix, and the same as the number of channels of the first feature matrix. Optionally, n=s 2 .

[0024] The pixel shuffle layer, also known as sub-pixel convolution, is expressed as PixelShuffle(s), where s>1. The pixel shuffle layer can rearrange the elements in the input feature matrix (whose shape is [C4,h4,w4]) to obtain the output feature matrix (whose shape is [C out ,h out ,w out ]), where h out =s·h4,w out =s·w4,C out =C4 / s 2 .

[0025] The pixel shuffling layer is used for upsampling in the last layer of the entropy estimation network, which only requires memory operations. Therefore, it is highly adaptable to different hardware (for example, it can be converted into convolution operations on AI accelerators as needed). It can even be combined with subsequent steps to merge memory access, thereby improving overall encoding / decoding efficiency.

[0026] (2) The s-fold upsampling process sequentially includes a second convolutional layer and an s-fold upsampling deconvolution layer.

[0027] The eighth feature matrix is ​​input into the second convolution layer and outputs the ninth feature matrix. The ninth feature matrix has the same size as the eighth feature matrix, but the number of channels of the ninth feature matrix is ​​n times the number of channels of the first feature matrix. The ninth feature matrix is ​​input into the deconvolution layer with s times upsampling and outputs the second feature matrix. The height of the second feature matrix is ​​s times the height of the ninth feature matrix, the width of the second feature matrix is ​​s times the width of the ninth feature matrix, and the number of channels of the second feature matrix is ​​1 / n of the number of channels of the ninth feature matrix, and the same as the number of channels of the first feature matrix. Optionally, n=s 2 .

[0028] (3) The s-fold upsampling process includes a deconvolution layer with s-fold upsampling.

[0029] The eighth feature matrix is ​​input into the deconvolution layer with s times upsampling and outputs the second feature matrix. The height of the second feature matrix is ​​s times the height of the eighth feature matrix, the width of the second feature matrix is ​​s times the width of the eighth feature matrix, and the number of channels of the second feature matrix is ​​1 / n of the number of channels of the eighth feature matrix, and the same as the number of channels of the first feature matrix. Optionally, n=s 2 .

[0030] (4) The s-fold upsampling process includes the second convolutional layer and two s / 2-fold pixel shuffling layers.

[0031] The eighth feature matrix is ​​input into the second convolutional layer and outputs the ninth feature matrix. The ninth feature matrix has the same size as the eighth feature matrix, but the number of channels of the ninth feature matrix is ​​n times the number of channels of the first feature matrix. The ninth feature matrix is ​​input into the first s / 2 times pixel shuffle layer and outputs the tenth feature matrix. The height of the tenth feature matrix is ​​s / 2 times the height of the ninth feature matrix, the width of the tenth feature matrix is ​​s / 2 times the width of the ninth feature matrix, and the number of channels of the tenth feature matrix is ​​n / 2 of the number of channels of the ninth feature matrix. The tenth feature matrix is ​​input into the second s / 2 times pixel shuffle layer and outputs the eleventh feature matrix. The height of the eleventh feature matrix is ​​s / 2 times the height of the tenth feature matrix, the width of the eleventh feature matrix is ​​s / 2 times the width of the tenth feature matrix, and the number of channels of the eleventh feature matrix is ​​n / 2 of the number of channels of the tenth feature matrix, which is the same as the number of channels of the first feature matrix. Optionally, n=s 2 .

[0032] It should be noted that in addition to the above-mentioned implementation methods, the s-fold upsampling processing can also be implemented in other ways. As long as the height of the second characteristic matrix can be s times the height of the eighth characteristic matrix, the width of the second characteristic matrix can be s times the width of the eighth characteristic matrix, and the second characteristic matrix has the same number of channels as the first characteristic matrix, it can be used as an implementation method of the s-fold upsampling processing. The embodiments of the present application do not make specific restrictions on this.

[0033] In one possible implementation, one of the two convolutions in the first convolutional layer can use a two-dimensional group convolutional layer (also called a group convolutional layer). The group convolution is represented by CONV(K ver *K hor ,C1,C2,G,t↓). The input feature matrix of this convolutional layer has a size of [C1,h1,w1], and the output feature matrix has a size of [C2,h2,w2], where h1 = t·h2, w1 = t·w2, and G represents the number of groups, which controls the connection between the input feature matrix and the output feature matrix. C1 and C2 are divisible by G, and t represents the step size of the sliding window. Without the step size parameter, no spatial resolution change is "performed." Using group convolution in the entropy estimation network can reduce the amount of computation while expanding the receptive field.

[0034] In an embodiment of the present application, the entropy estimation network may include a variance prediction network and a mean prediction network. The input feature matrices of the two networks (i.e., the first feature matrix may be the same) are different, and the output feature matrices are different. The output feature matrix of the variance prediction network can be called the variance matrix, and the output feature matrix of the mean prediction network can be called the mean matrix. The structures of the two networks are similar, except that, since the variance is a positive number, but the mean is not limited to this, any convolution layer in the variance prediction network uses a quantized convolution layer, and an absolute value operation is provided inside or outside the convolution layer to ensure the positive characteristic of the variance.

[0035] Quantized convolution can be expressed as qCONV(K ver *K hor ,C5,C6,G,q↓,d,p). The convolution process receives a feature matrix of shape [C5,h5,w5] and outputs a feature matrix of shape [C6,h6,w6], where h6=q·h5 and w6=q·w5, where q represents the convolution step size and the default value of step size is 1. The corresponding outputs have the same spatial resolution. d is a non-negative integer that defines the maximum value of the elements of the input feature matrix after clipping. p[c7] contains the scaling coefficient and translation coefficient of each channel of the output feature matrix.

[0036] Optionally, the input of the entropy estimation network is the first feature matrix, starting with a 1*1 convolution (the first convolution layer, whose convolution kernel size is 1*1), followed by an activation function a (the first activation function, for example, ReLU, LeakyReLU, etc.), followed by two convolutions (the second convolution layer), respectively 1*1 convolution and 3*3 group convolution (group convolution), and then the outputs of each of the two convolutions are operated point by point, followed by an activation function b (the second activation function, for example, ReLU, LeakyReLU, etc.), and then the number of channels is increased to 16 times the number of channels of the first feature matrix by using a 1*1 convolution, and finally the number of channels is reduced by 16 times by using a 4-fold sampling PixelShuffle (4), and the height and width are upsampled by 4 times respectively to obtain the final output, which is the second feature matrix.

[0037] In a second aspect, the present application provides an entropy estimation device for image compression, comprising: an acquisition module for acquiring a first feature matrix to be processed, wherein the shape of the first feature matrix is ​​[C in ,h in ,w in ], where C in represents the number of channels of the first feature matrix, h in represents the height of the first feature matrix, w in represents the width of the first feature matrix; a processing module, used to input the first feature matrix into the entropy estimation network to obtain a second feature matrix, the shape of the second feature matrix is ​​[C out ,h out ,w out ], where C out represents the number of channels of the second feature matrix, h out represents the height of the second characteristic matrix, w out represents the width of the second characteristic matrix; the second characteristic matrix and the first characteristic matrix meet the following conditions: C out =C in , h out =s·h in , w out =s·w in , s is an integer greater than 1; the entropy estimation network includes a first convolutional layer and a point-by-point operation, the first convolutional layer includes two convolution processes, one of the two convolution processes is a two-dimensional grouped convolution, and the input of the point-by-point operation is the output of each of the two convolution processes.

[0038] In one possible implementation, the input feature matrix and output feature matrix of the two-dimensional grouped convolution satisfy the following conditions: h1=t·h2, w1=t·w2, C1 and C2 are divisible by G; t is an integer greater than 1; C1 represents the number of channels of the input feature matrix, h1 represents the height of the input feature matrix, and w1 represents the width of the input feature matrix; C2 represents the number of channels of the output feature matrix, h2 represents the height of the output feature matrix, and w2 represents the width of the output feature matrix; G represents the number of groups, which is used to control the connection between the input feature matrix and the output feature matrix.

[0039] In a possible implementation, the entropy estimation network further includes an s-fold upsampling process.

[0040] In a possible implementation, the s-fold upsampling process includes a second convolution layer and a pixel shuffling layer in sequence, and the third feature matrix output by the second convolution layer and the first feature matrix meet the following conditions: C3 = nC in , h3=h in , w3=w in , where n is an integer greater than 1, C3 represents the number of channels of the third characteristic matrix, h3 represents the height of the third characteristic matrix, w3 represents the width of the third characteristic matrix, the pixel shuffling layer is used to upsample the height and width of the third characteristic matrix respectively, and the characteristic matrix output by the pixel shuffling layer is the second characteristic matrix.

[0041] In one possible implementation, the s-fold upsampling process sequentially includes a second convolution layer and an s-fold upsampling deconvolution layer, and the third feature matrix output by the second convolution layer and the first feature matrix satisfy the following condition: C3=nC in , h3=h in , w3=w in , where n is an integer greater than 1, C3 represents the number of channels of the third feature matrix, h3 represents the height of the third feature matrix, w3 represents the width of the third feature matrix, the deconvolution layer is used to upsample the height and width of the third feature matrix respectively, and the feature matrix output by the deconvolution layer is the second feature matrix.

[0042] In one possible implementation, the s-fold upsampling process includes an s-fold upsampling deconvolution layer, where the deconvolution layer is used to upsample the height and width of the input feature matrix respectively, and the feature matrix output by the deconvolution layer is the second feature matrix.

[0043] In one possible implementation, the entropy estimation network also includes a first activation function and a second activation function; the first activation function and the second activation function respectively include an identity operation Identity, a rectified linear unit ReLU processing, a leaky rectified linear unit LeakyReLU processing, or a parameter rectified linear unit PReLU processing.

[0044] In a possible implementation, the point-by-point operation includes point-by-point multiplication, point-by-point addition, point-by-point subtraction, or point-by-point division.

[0045] In a possible implementation, the entropy estimation network further includes a third convolutional layer, and a feature matrix input to the third convolutional layer is the first feature matrix.

[0046] In a third aspect, an embodiment of the present application provides an encoder comprising: one or more processors; and a non-transitory computer-readable storage medium coupled to the processors and storing a program executed by the processors, wherein the program, when executed by the processors, causes the decoder to perform any of the methods described in the first aspect. Optionally, after obtaining the second characteristic matrix, the encoder's processor is further configured to perform entropy encoding based on the second characteristic matrix to obtain a bitstream. The encoder also includes a communication interface for transmitting the bitstream.

[0047] In a fourth aspect, an embodiment of the present application provides a decoder comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the processor and storing a program executed by the processor, wherein the program, when executed by the processor, causes the decoder to perform any of the methods described in the first aspect. Optionally, after obtaining the second characteristic matrix, the processor of the decoder is further configured to perform entropy decoding based on the second characteristic matrix to obtain a reconstructed image or video. The decoder also includes a communication interface for receiving a video stream to be decoded.

[0048] In a fifth aspect, an embodiment of the present application provides a computer program product comprising a program code, which, when executed on a computer or a processor, is used to execute any of the methods described in the first aspect above.

[0049] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium comprising instructions, which, when executed on a computer, enables the computer to execute any one of the methods described in the first aspect above.

[0050] In the seventh aspect, an embodiment of the present application provides a chip comprising one or more interface circuits and one or more processors; the interface circuit is used to receive and / or send signals, wherein the signals include computer instructions stored in a memory; when the processor executes the computer instructions, it is used to implement any one of the methods described in the first aspect above.

[0051] In an eighth aspect, an embodiment of the present application provides a video code stream transmission system, comprising: an acquisition device for acquiring a video code stream, wherein the video code stream is encoded according to the encoder described in the third aspect above; and a transmission device for transmitting the video code stream.

[0052] In a ninth aspect, an embodiment of the present application provides a method for storing a video code stream, wherein the video code stream is encoded according to the encoder described in the third aspect above.

[0053] In the tenth aspect, an embodiment of the present application provides a storage device for a video code stream, comprising: a communication interface for obtaining a video code stream, wherein the video code stream is encoded according to the encoder described in the third aspect above; and a memory for storing the video code stream.

[0054] In an eleventh aspect, an embodiment of the present application provides a method for sending a video code stream, wherein the video code stream is encoded according to the encoder described in the third aspect above.

[0055] In the twelfth aspect, an embodiment of the present application provides a video code stream sending system, including: an acquisition device for obtaining a video code stream, which is encoded according to the encoder described in the third aspect above; and a communication interface for sending the video code stream to a destination device. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is an example diagram of a typical deep learning image compression framework;

[0057] Figure 2 is an example diagram of a typical deep learning video compression framework;

[0058] Figure 3 is an example diagram of an image / video storage / transmission application scenario;

[0059] FIG4 is an example diagram of an image / video storage application scenario;

[0060] Figure 5 is an example diagram of a Huawei Cloud application scenario;

[0061] FIG6 is a schematic diagram of a workflow of a streaming media system applicable to an embodiment of the present application;

[0062] FIG7 is a schematic diagram of the architecture of a streaming media system according to an embodiment of the present application;

[0063] FIG8 is a flow chart of a process 800 of an entropy estimation method for image compression according to an embodiment of the present application;

[0064] FIG9 is a schematic diagram of the structure of an entropy estimation network according to an embodiment of the present application;

[0065] FIG10 a is a schematic diagram of the structure of the variance prediction network for entropy estimation;

[0066] FIG10 b is a schematic diagram of the structure of the mean prediction network for entropy estimation;

[0067] Figure 11a is a schematic diagram of the structure of the variance prediction network for entropy estimation;

[0068] FIG11b is a schematic diagram of the structure of the mean prediction network for entropy estimation;

[0069] FIG12a is a schematic diagram of the structure of the variance prediction network for entropy estimation;

[0070] FIG12 b is a schematic diagram of the structure of the mean prediction network for entropy estimation;

[0071] FIG13a is a schematic diagram of the structure of the variance prediction network for entropy estimation;

[0072] FIG13 b is a schematic diagram of the structure of the mean prediction network for entropy estimation;

[0073] FIG14 is a schematic diagram of an entropy estimation device 1400 for image compression according to an embodiment of the present application;

[0074] FIG15 is a schematic structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0075] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0076] The terms "first," "second," and the like in the description, embodiments, claims, and drawings of this application are used solely for descriptive purposes and are not to be construed as indicating or implying relative importance or order. Furthermore, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions, such as, for example, inclusion of a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0077] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0078] Since the embodiments of the present application involve the application of neural networks, for ease of understanding, the following first explains the relevant nouns or terms used in the embodiments of the present application:

[0079] 1. Neural Networks

[0080] A neural network (NN) is a machine learning model. A neural network can be composed of neural units. A neural unit can refer to a network with x as its input. s The output of the operation unit can be shown as formula (1):

[0081] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s is x s The weight of the neural unit, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a nonlinear function such as ReLU. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.

[0082] 2. Multi-layer perception (MLP)

[0083] MLP is a simple deep neural network (DNN) (different layers are fully connected), also known as a multi-layer neural network, which can be understood as a neural network with many hidden layers. The "many" here does not have a special metric. According to the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since there are many DNN layers, the coefficient W and the offset vector The definition of these parameters in DNN is as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscript corresponds to the output of the third layer index 2 and the input of the second layer index 4. In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as It's important to note that the input layer has no W parameter. In deep neural networks, more hidden layers allow the network to better capture complex real-world situations. Theoretically, a model with more parameters has higher complexity and greater "capacity," meaning it can handle more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrices for all layers of a trained deep neural network (a weight matrix formed by the vectors W across many layers).

[0084] 3. Convolutional Neural Networks

[0085] A convolutional neural network (CNN) is a deep neural network with a convolutional structure and a deep learning architecture. A deep learning architecture involves multiple levels of learning at different levels of abstraction using machine learning algorithms. As a deep learning architecture, a CNN is a feed-forward artificial neural network in which individual neurons respond to input images. A CNN consists of a feature extractor consisting of convolutional and pooling layers. The feature extractor can be thought of as a filter, and the convolution process can be thought of as convolving an input image or feature map with a trainable filter.

[0086] A convolutional layer is a layer of neurons in a convolutional neural network that performs convolution on the input signal. A convolutional layer can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter that extracts specific information from the input image matrix. A convolution operator is essentially a weight matrix, which is usually predefined. During the convolution operation, the weight matrix is ​​typically applied horizontally to the input image, pixel by pixel (or two pixels by two pixels, depending on the stride), to extract specific features from the image. The size of the weight matrix should be proportional to the image size. It is important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during the convolution operation, the weight matrix extends across the entire depth of the input image. Therefore, convolution with a single weight matrix produces a convolved output with a single depth dimension. However, in most cases, multiple weight matrices of the same size (rows * columns) are applied instead. This is known as multiple homogeneous matrices. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image. This dimension can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features from the image. For example, one weight matrix can be used to extract edge information, another to extract specific colors, and yet another to blur unwanted noise. These multiple weight matrices have the same size (rows * columns), and the feature maps extracted by these weight matrices of the same size are also of the same size. These extracted feature maps of the same size are then combined to form the output of the convolution operation. In practical applications, the weight values ​​in these weight matrices require extensive training. The weight matrices formed from these trained weight values ​​can be used to extract information from the input image, enabling the convolutional neural network to make accurate predictions. When a convolutional neural network has multiple convolutional layers, the initial convolutional layers often extract more general features, which can also be called low-level features. As the depth of the convolutional neural network increases, the features extracted by subsequent convolutional layers become increasingly complex, such as high-level semantic features. Features with higher semantic meaning are more applicable to the problem being solved.

[0087] Because it's often necessary to reduce the number of trainable parameters, pooling layers are often periodically introduced after convolutional layers. This can be done in a single convolutional layer followed by a pooling layer, or in a multi-layered system followed by one or more pooling layers. In image processing, the sole purpose of a pooling layer is to reduce the spatial size of an image. Pooling layers can include average pooling and / or max pooling operators, which are used to downsample the input image to produce a smaller image. The average pooling operator calculates the average value of pixel values ​​within a specific range, producing the average pooling result. The max pooling operator takes the pixel with the largest value within a specific range as the max pooling result. Furthermore, just as the size of the weight matrix used in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The output image size after processing by a pooling layer can be smaller than the size of the image input to the pooling layer. Each pixel in the output image represents the average or maximum value of the corresponding subregion of the image input to the pooling layer.

[0088] After being processed by the convolution layer / pooling layer, the convolutional neural network is still not sufficient to output the required output information. Because as mentioned above, the convolution layer / pooling layer only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolutional neural network needs to use the neural network layer to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer may include multiple hidden layers, and the parameters contained in the multiple hidden layers can be pre-trained based on relevant training data of a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0089] Optionally, after the multiple hidden layers in the neural network layer, an output layer of the entire convolutional neural network is also included. The output layer has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network is completed, the backpropagation will begin to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network and the error between the result output by the convolutional neural network through the output layer and the ideal result.

[0090] 4. Recurrent Neural Networks

[0091] Recurrent neural networks (RNNs) are designed to process sequential data. In traditional neural network models, layers are fully connected, from the input layer to the hidden layer to the output layer, while nodes within each layer are disconnected. While these conventional neural networks have solved many difficult problems, they are still inadequate for many others. For example, to predict the next word in a sentence, you generally need to use the previous words, as the previous and next words in a sentence are not independent. RNNs are called recurrent neural networks because the current output of a sequence is dependent on the previous output. Specifically, the network memorizes previous information and applies it to the calculation of the current output. This means that nodes within the hidden layer are no longer disconnected but connected, and the input to a hidden layer includes not only the output of the input layer but also the output of the previous hidden layer. In theory, RNNs can process sequence data of any length. Training an RNN is similar to training a traditional CNN or DNN. This approach also uses the backpropagation algorithm, but with one key difference: if the RNN is expanded, its parameters, such as W, are shared; this is not the case with traditional neural networks, as in the example above. Furthermore, when using gradient descent, the output of each step depends not only on the state of the network at the current step but also on the state of the network at several previous steps. This learning algorithm is called backpropagation through time (BPTT).

[0092] Since we already have convolutional neural networks, why do we still need recurrent neural networks? The reason is simple. Convolutional neural networks assume that elements are independent of each other, and that inputs and outputs are also independent, such as cats and dogs. However, in the real world, many elements are interconnected, such as the changes in stock prices over time. Or, for example, someone says, "I love traveling, and my favorite place is Yunnan. I must visit it someday." Humans should know to fill in the blank with "Yunnan." This is because humans make inferences based on context, but how can machines do the same? That's where RNNs come in. RNNs aim to give machines the ability to remember, like humans do. Therefore, the output of an RNN depends on both the current input and historical memory.

[0093] 5. Loss Function

[0094] During the training of a deep neural network, because we want the output of the deep neural network to be as close as possible to the desired predicted value, we can compare the current network's predicted value with the desired target value and then update the weight vector of each layer of the neural network based on the difference between the two. (Of course, before the first update, there is usually an initialization process, which pre-configures the parameters for each layer in the deep neural network.) For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value. This adjustment is continued until the deep neural network can predict the desired target value or a value very close to the desired target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value." This is the loss function (or objective function), which is an important equation used to measure the difference between the predicted value and the target value. For example, the loss function output value (loss) indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss.

[0095] 6. Backpropagation algorithm

[0096] Convolutional neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during training, reducing the reconstruction error loss of the super-resolution model. Specifically, the forward propagation of the input signal to the output generates an error loss. This error loss information is then backpropagated to update the parameters of the initial super-resolution model, thereby converging the error loss. The BP algorithm is a backward propagation movement dominated by the error loss, aiming to obtain the optimal super-resolution model parameters, such as the weight matrix.

[0097] 7. Generative Adversarial Networks

[0098] Generative adversarial networks (GANs) are a type of deep learning model. They consist of at least two modules: a generative model and a discriminative model. These two modules learn through interaction to produce better outputs. Both the generative and discriminative models can be neural networks, specifically deep neural networks or convolutional neural networks. The basic principle of a GAN is as follows: For example, consider a GAN that generates images. Suppose there are two networks, G (Generator) and D (Discriminator). G is the image generator network, which receives random noise z and generates an image from it, denoted as G(z). D is the discriminator network, which determines whether an image is "real." Its input parameter is x, representing an image. Its output, D(x), represents the probability that x is real. A value of 1 indicates a 130% probability of authenticity, while a value of 0 indicates an impossibility of authenticity. During the training of this generative adversarial network, the goal of the generative network G is to generate realistic images as much as possible to deceive the discriminative network D, while the goal of the discriminative network D is to distinguish the images generated by G from real images as much as possible. This creates a dynamic "game" between G and D, which is the "game" in "generative adversarial network." Ultimately, under ideal conditions, G can generate images G(z) that are sufficiently realistic, while D has difficulty determining whether the images generated by G are real, i.e., D(G(z)) = 0.5. This results in an excellent generative model G that can be used to generate images.

[0099] FIG1 is an example diagram of a typical deep learning image compression framework. As shown in FIG1 , the image compression framework is an implementation of current artificial intelligence (AI) image compression technology.

[0100] On the encoding side, the original image is transformed from the image domain to the feature domain through the encoding network. The transformed image features are then processed by the quantization module and the entropy coding network and encoded into a bitstream for transmission or storage. On the decoding side, the bitstream is decoded into image features through the entropy decoding network. The image features are then transformed from the feature domain to the image domain through the decoding network to produce a reconstructed image. The entropy estimation network estimates the image features and derives an estimated probability value for each feature element for processing by the entropy coding and decoding networks.

[0101] In this embodiment, both the encoding network (Encoder) and the decoding network (Decoder) have nonlinear transformation units.

[0102] FIG2 is an example diagram of a typical deep learning video compression framework. As shown in FIG2 , the video compression framework includes a prediction module (predict model) and a residual compression module (residual compress).

[0103] The prediction module uses the reconstructed image of the previous frame to predict the current frame and generate a predicted image. The residual compression module compresses the residual between the original image and the predicted image of the current frame and decompresses it to obtain a reconstructed residual. The reconstructed residual and the predicted image are summed to obtain the reconstructed image of the current frame. Both the encoding and decoding subnetworks in the prediction and residual compression modules have nonlinear transformation units.

[0104] In this embodiment, both the prediction module (predict model) and the residual compression module (residual compress) have nonlinear transformation units.

[0105] Figure 3 is an example diagram of an image / video storage / transmission application scenario. As shown in Figure 3, the application scenario can be a business involving image / video acquisition, storage, or transmission in a terminal, Huawei Cloud, or video surveillance, such as terminal photo / video recording, photo album, Huawei Cloud photo album, video surveillance, etc.

[0106] Encoding: The camera captures images / videos. The AI ​​image / video encoding network extracts features from the images / video to obtain low-redundancy image features, and then compresses them to produce a bitstream / image file.

[0107] Decoding end: When reconstructed images / videos need to be output, the AI ​​image / video decoding network decompresses the bitstream / image file to obtain image features, and then performs inverse feature extraction on the image features to obtain the reconstructed image / video.

[0108] The storage / transmission module stores the compressed code stream / image files for different services (for example, terminal photography, video surveillance, cloud servers, etc.) or transmits them (for example, cloud services, live broadcast technology, etc.).

[0109] Figure 4 is an example diagram of an image / video storage application scenario. As shown in Figure 4, the application scenario can be a terminal album, Cyberverse (a new "digital reality" technology based on the integration of virtual and real released by Huawei), Huawei Cloud Album, video surveillance, live broadcast, etc.

[0110] Encoding: The camera captures digital images / videos. The AI ​​encoding unit transforms the images / videos into image features with lower redundancy. This unit typically includes a nonlinear transformation unit and exhibits nonlinear characteristics. The entropy estimation unit calculates the encoding probability of each data element in the image features. The entropy coding network losslessly encodes the image features based on the corresponding probabilities to generate a bitstream / image file, further reducing the amount of data transmitted during the image compression process. The storage module saves the bitstream / image file to the corresponding storage location on the terminal.

[0111] Decoding: The loading module loads the bitstream / image file from the corresponding storage location on the terminal and inputs it into the entropy decoding network. The entropy estimation unit is responsible for calculating the decoding probability of each data point in the image feature. Based on the corresponding probabilities of each data point, the entropy decoding network performs lossless decoding on the bitstream / image file to reconstruct the image features. The AI ​​decoding unit performs an inverse transformation on the image features output by entropy decoding, parsing them into a reconstructed image / video. Similar to the AI ​​encoding unit, it typically includes a nonlinear transformation unit and exhibits nonlinear characteristics.

[0112] Optionally, the AI ​​encoding unit, the AI ​​decoding unit and the entropy estimation unit may adopt a network processor (NPU), and the entropy encoding network, the entropy decoding network, the saving module and the loading module may adopt a central processing unit (CPU).

[0113] Figure 5 is an example diagram of Huawei Cloud application scenarios. As shown in Figure 5, the application scenarios can be terminal albums, Cyberverse, Huawei Cloud albums, video surveillance, live broadcasts, etc.

[0114] Encoding end: The original image is obtained locally, image (JPEG) encoded on it to obtain a compressed image, and then the compressed image is sent to the cloud. The cloud performs JPEG decoding on the compressed image to obtain an image, and then the AI ​​encoding unit transforms the image into image features with lower redundancy. It usually includes a nonlinear transformation unit and has nonlinear characteristics. The entropy estimation unit is responsible for calculating the encoding probability of each data in the image feature. The entropy coding network performs lossless encoding on the image features according to the probability corresponding to each data to obtain a code stream / image file, further reducing the amount of data transmitted during the image compression process. The saving module saves the code stream / image file to the corresponding storage location of the terminal.

[0115] Decoding: When a local device needs to obtain an image from the cloud, the cloud loading module loads the bitstream / image file from the corresponding storage location on the terminal and inputs it into the entropy decoding network. The entropy estimation unit is responsible for calculating the decoding probability of each data point in the image features. The entropy decoding network losslessly decodes the bitstream / image file based on the corresponding probabilities of each data point to obtain reconstructed image features. The AI ​​decoding unit inversely transforms the image features output by entropy decoding and parses them into a reconstructed image. Similar to the AI ​​encoding unit, it typically includes a nonlinear transformation unit and has nonlinear characteristics. The cloud then performs JPEG encoding on the image to obtain a compressed image, which is then sent to the local device. The local device performs JPEG decoding on the compressed image to obtain and display the image.

[0116] Optionally, the entropy estimation unit can adopt NPU, the JPEG encoding module, JPEG decoding module, entropy encoding network, entropy decoding network, saving module and loading module can adopt CPU, and the AI ​​encoding unit and AI decoding unit can adopt graphics processing unit (GPU).

[0117] FIG6 is a schematic diagram of a streaming media system workflow applicable to an embodiment of the present application. As shown in FIG6 , the streaming media system includes a content creation module that generates the required content data, such as video or audio. The streaming media system also includes a video encoding module that encodes the generated content through an encoder. The streaming media system also includes a video stream transmission module that transmits the encoded video in the form of a code stream. Optionally, the format of the video stream can be converted into a code stream format of a commonly used transmission protocol for an open Internet video service (over the top TV, OTT) device, for example, the protocol includes but is not limited to real-time streaming protocol (RTSP), hypertext transfer protocol (HTTP), HTTP Live streaming protocol (HLS), MPEG HTTP dynamic adaptive streaming (MPEG Dynamic Adaptive Streaming over HTTP, MPEG-DASH), real-time transport protocol (RTP), real-time messaging protocol (RTMP) or any combination thereof. Optionally, video stream storage can be performed to store the original format of the video stream and / or the converted multiple code stream formats for easy use. Furthermore, the streaming system also includes a video stream encapsulation module for encapsulating the video stream to generate an encapsulated video stream. The encapsulated video stream can be referred to as a video streaming package. Exemplarily, the video streaming package can be generated based on a transcoded video stream or a stored video stream. Furthermore, the streaming system also includes a content distribution network (CDN) for distributing the video streaming package to multiple OTT devices, such as mobile phones, computers, tablets, and home projectors.

[0118] It should be noted that the video encoding, video streaming transmission, video stream transcoding, video stream storage, video streaming package generation, and content distribution network can all be implemented on a cloud platform (or understood as a cloud server). In one possible implementation, the video encoding module can include the integration of the encoding network, quantization, entropy encoding network, and entropy estimation network in Figure 1; the video stream transcoding module can include the integration of the entropy estimation network, entropy decoding network, and decoding network in Figure 1; and the OTT device can include the display of the reconstructed image in Figure 1.

[0119] FIG7 is a schematic diagram of the architecture of a streaming media system according to an embodiment of the present application. As shown in FIG7 , the streaming media system architecture includes: a client device, a content distribution network, and a cloud server.

[0120] The user on the client device sends a play or playback request to the cloud platform (ie, the cloud server). Optionally, the content of the request sent may be the title of the movie or television program to be played.

[0121] The cloud platform makes a decision and responds to the client, sending the client the address of the requested content on the CDN. Optionally, the content sent to the client can be a URL (uniform resource locator). Specifically, the playback application service in the cloud platform checks user authorization and permissions, and then considers individual client characteristics and current network conditions to determine which specific files are required to process the playback request. It should be noted that the CDN regularly reports its operational status, learned routes, and available content (files) to the cache control service in the cloud platform.

[0122] The client then requests the CDN to play the content based on the address, and the CDN provides the content to the client, ultimately completing the client's request.

[0123] Based on the above application scenarios, an embodiment of the present application provides an entropy estimation method for image compression to improve compression performance.

[0124] FIG8 is a flow chart of process 800 of an entropy estimation method for image compression according to an embodiment of the present application. Process 800 may be performed by the encoding end or the decoding end in the above-described embodiments. Process 800 is described as a series of steps or operations. It should be understood that process 800 may be performed in various orders and / or concurrently, and is not limited to the execution order shown in FIG8. Process 800 includes the following steps:

[0125] Step 801: Obtain a first feature matrix to be processed.

[0126] According to the above description, the entropy estimation network can obtain the estimated probability value of each feature element based on the image feature estimation, which is used for processing by the entropy coding network and the entropy decoding network. Based on this, the first feature matrix can be the feature matrix obtained by quantizing the original image after being processed by the coding network. For example, the first feature matrix can be represented as input[C in ,h in ,w in ], C in Indicates the number of channels of the first feature matrix, h in Represents the height of the first characteristic matrix, w in Represents the width of the first characteristic matrix, that is, [C in ,h in ,win ] represents the shape of the first feature matrix.

[0127] Step 802: Input the first feature matrix into an entropy estimation network to obtain a second feature matrix.

[0128] The second characteristic matrix can be expressed as output[C out ,h out ,w out ], C out Indicates the number of channels of the second feature matrix, h out Represents the height of the second characteristic matrix, w out Represents the width of the second characteristic matrix, that is, [C out ,h out ,w out ] represents the shape of the second characteristic matrix.

[0129] In the embodiment of the present application, the second characteristic matrix and the first characteristic matrix satisfy:

[0130] C out =C in

[0131] h out =s·h in

[0132] w out =s·w in

[0133] Wherein, s is an integer greater than 1, which is used to represent the stride of the sliding window. Optionally, s=4.

[0134] The entropy estimation network sequentially includes a third convolutional layer, a first activation function, a first convolutional layer, a point-by-point operation, a second activation function, and an s-fold upsampling process, wherein the first convolutional layer includes two convolution processes, and the input of the point-by-point operation is the output of each of the two convolution processes.

[0135] The input feature matrix of any of the above convolutional layers is convolved to obtain an output feature matrix with unchanged height and width. The output feature matrix can be considered as a local response of the input feature matrix. The convolution operation can be expressed as the following formula:

[0136] Conv(x)=∑w*x+b

[0137] Among them, w is the weight of the convolutional layer, and b is the bias parameter of the convolutional layer, both of which are unrestricted learnable parameters.

[0138] Any of the above activation operations is a point-by-point operation, that is, an independent mapping operation is performed on each element of the input feature matrix. The activation operation can be an identity operation (Identity), a rectified linear unit (Rectified Linear Unit, ReLU), a rectified linear unit (LeakyReLU), a parametric rectified linear unit (PReLU), Sigmoid, Tanh, etc., which can provide linear or nonlinear transformation characteristics. In the embodiment of the present application, the first activation function and the second activation function can use the same operation or different operations, and there is no specific limitation on this. Among them,

[0139] Identity refers to taking the original value of the input feature value. It can be expressed as the following formula:

[0140] y=x

[0141] ReLU is a piecewise linear mapping method. For the input eigenvalue, the output is 0 if the eigenvalue is less than 0, and the output is constant if the eigenvalue is greater than or equal to 0. It can be expressed as follows:

[0142] LeakyReLU is a piecewise linear mapping method that scales input feature values ​​less than 0 with a pre-set weight based on ReLU, which is usually 0.01. It can be expressed as follows:

[0143] Here, a represents a preset value, which is usually set to 0.01.

[0144] The definition of PReLU is similar to LeakyReLU:

[0145] The difference from LeakyReLU is that a is a trainable parameter that can be obtained through training.

[0146] Sigmoid can be expressed as the following operation:

[0147] Tanh can be expressed as the following operation:

[0148] It should be noted that the activation function may also adopt other variations of ReLU, and the embodiments of the present application do not specifically limit this.

[0149] These point-by-point operations can include point-by-point multiplication, addition, subtraction, or division. The input to these point-by-point operations is the output feature matrices of the two convolutional processes of the first convolutional layer. These two output feature matrices have the same size, and the point-by-point operations multiply, add, subtract, or divide the values ​​of the elements at corresponding positions. The combination of convolutional operations and point-by-point operations implements a local attention mechanism, leveraging the local information of the input feature matrix to modulate the response at each position in the feature matrix.

[0150] FIG9 is a schematic diagram of the structure of the entropy estimation network according to an embodiment of the present application. As shown in FIG9 , the input of the entropy estimation network is the first characteristic matrix.

[0151] The first feature matrix is ​​input into the third convolutional layer, and the output of the third convolutional layer is the third feature matrix. The third feature matrix has the same height and width as the first feature matrix (also called the same size), but the number of channels is different.

[0152] The third feature matrix inputs the first activation function and outputs a fourth feature matrix. The fourth feature matrix has the same size and number of channels as the third feature matrix.

[0153] The fourth feature matrix is ​​input into the two convolutions (convolution 1 and convolution 2) of the first convolutional layer, and outputs the fifth feature matrix and the sixth feature matrix respectively. The fifth feature matrix and the sixth feature matrix have the same size and number of channels. They have the same size as the fourth feature matrix but different number of channels.

[0154] The fifth characteristic matrix and the sixth characteristic matrix are input point by point and the seventh characteristic matrix is ​​output. The seventh characteristic matrix has the same size and number of channels as the fifth / sixth characteristic matrices.

[0155] The seventh characteristic matrix inputs the second activation function and outputs an eighth characteristic matrix. The eighth characteristic matrix has the same size and number of channels as the seventh characteristic matrix.

[0156] The eighth characteristic matrix is ​​upsampled s times to output a second characteristic matrix, the height of the second characteristic matrix is ​​s times the height of the eighth characteristic matrix, the width of the second characteristic matrix is ​​s times the width of the eighth characteristic matrix, and the number of channels of the second characteristic matrix is ​​the same as that of the first characteristic matrix.

[0157] As an example, the entropy estimation network of the embodiment of the present application can use a hyper-scale decoder (HSD). All operations in HSD are integers, the accumulators involved in all calculations are 32-bit integers, and the model parameters are quantized to 8-bit integers. This ensures the cross-platform bit consistency of the neural network module. HSD uses customized quantized convolution. For each quantized convolution in the process, a set of clipping values ​​{d k} and the scaling parameter {p k}, 1≤k≤4. All clipping values ​​in quantized convolution are d k =2 15 -1. Where the scaling shift {p k}Acquired through training.

[0158] It is worth noting that the size of the weight in the quantization model does not exceed 2 15 The combination of -1, shift and clipping values ​​ensures that the registers for the quantized convolution fit within 32 bits.

[0159] The HSD structure is as follows. First, it receives an input three-dimensional matrix of size [C, H, W], which is converted into a matrix of [C1, H, W] through a 1*1 quantized convolution and an activation function. This matrix is ​​then converted into two output matrices A and B of [C2, H, W] through a 1*1 quantized convolution and a 3*3 quantized grouped convolution. Then, matrices A and B are converted into a matrix C [C2, H, W] through a point-by-point operation. This is then converted into a matrix of [16C, H, W] through an activation function and a 1*1 quantized convolution. The final output is formed through a PixelShuffle operation, with a size of [C, 4H, 4W].

[0160] In the embodiment of the present application, the s-fold upsampling process can be implemented in multiple ways:

[0161] (1) The s-fold upsampling process includes the second convolutional layer and the s-fold pixel shuffling layer in sequence.

[0162] The eighth feature matrix is ​​input into the second convolutional layer and outputs the ninth feature matrix. The ninth feature matrix has the same size as the eighth feature matrix, but the number of channels of the ninth feature matrix is ​​n times the number of channels of the first feature matrix. The ninth feature matrix is ​​input into the pixel shuffling layer and outputs the second feature matrix. The height of the second feature matrix is ​​s times the height of the ninth feature matrix, the width of the second feature matrix is ​​s times the width of the ninth feature matrix, and the number of channels of the second feature matrix is ​​1 / n of the number of channels of the ninth feature matrix, and the same as the number of channels of the first feature matrix. Optionally, n=s 2 .

[0163] The pixel shuffle layer, also known as sub-pixel convolution, is expressed as PixelShuffle(s), where s>1. The pixel shuffle layer can rearrange the elements in the input feature matrix (whose shape is [C4,h4,w4]) to obtain the output feature matrix (whose shape is [C out ,h out ,w out ]), where h out =s·h4,w out =s·w4,C out =C4 / s 2 .

[0164] The pixel shuffling layer can be expressed as:

[0165] output[c,i,j]=input[s 2 c+si′+j′,i / s+i′,j / s+j′]

[0166] Where c is from 0 to C out -1, i is from 0 to h out -1, j is from 0 to w out -1, i' is from 0 to s-1, j' is from 0 to s-1.

[0167] The pixel shuffling layer is used for upsampling in the last layer of the entropy estimation network, which only requires memory operations. Therefore, it is highly adaptable to different hardware (for example, it can be converted into convolution operations on AI accelerators as needed). It can even be combined with subsequent steps to merge memory access, thereby improving overall encoding / decoding efficiency.

[0168] (2) The s-fold upsampling process sequentially includes a second convolutional layer and an s-fold upsampling deconvolution layer.

[0169] The eighth feature matrix is ​​input into the second convolution layer and outputs the ninth feature matrix. The ninth feature matrix has the same size as the eighth feature matrix, but the number of channels of the ninth feature matrix is ​​n times the number of channels of the first feature matrix. The ninth feature matrix is ​​input into the deconvolution layer with s times upsampling and outputs the second feature matrix. The height of the second feature matrix is ​​s times the height of the ninth feature matrix, the width of the second feature matrix is ​​s times the width of the ninth feature matrix, and the number of channels of the second feature matrix is ​​1 / n of the number of channels of the ninth feature matrix, and the same as the number of channels of the first feature matrix. Optionally, n=s 2 .

[0170] (3) The s-fold upsampling process includes a deconvolution layer with s-fold upsampling.

[0171] The eighth feature matrix is ​​input into the deconvolution layer with s times upsampling and outputs the second feature matrix. The height of the second feature matrix is ​​s times the height of the eighth feature matrix, the width of the second feature matrix is ​​s times the width of the eighth feature matrix, and the number of channels of the second feature matrix is ​​1 / n of the number of channels of the eighth feature matrix, and the same as the number of channels of the first feature matrix. Optionally, n=s 2 .

[0172] (4) The s-fold upsampling process includes the second convolutional layer and two s / 2-fold pixel shuffling layers.

[0173] The eighth feature matrix is ​​input into the second convolutional layer and outputs the ninth feature matrix. The ninth feature matrix has the same size as the eighth feature matrix, but the number of channels of the ninth feature matrix is ​​n times the number of channels of the first feature matrix. The ninth feature matrix is ​​input into the first s / 2 times pixel shuffle layer and outputs the tenth feature matrix. The height of the tenth feature matrix is ​​s / 2 times the height of the ninth feature matrix, the width of the tenth feature matrix is ​​s / 2 times the width of the ninth feature matrix, and the number of channels of the tenth feature matrix is ​​n / 2 of the number of channels of the ninth feature matrix. The tenth feature matrix is ​​input into the second s / 2 times pixel shuffle layer and outputs the eleventh feature matrix. The height of the eleventh feature matrix is ​​s / 2 times the height of the tenth feature matrix, the width of the eleventh feature matrix is ​​s / 2 times the width of the tenth feature matrix, and the number of channels of the eleventh feature matrix is ​​n / 2 of the number of channels of the tenth feature matrix, which is the same as the number of channels of the first feature matrix. Optionally, n=s 2 .

[0174] It should be noted that in addition to the above-mentioned implementation methods, the s-fold upsampling processing can also be implemented in other ways. As long as the height of the second characteristic matrix can be s times the height of the eighth characteristic matrix, the width of the second characteristic matrix can be s times the width of the eighth characteristic matrix, and the second characteristic matrix has the same number of channels as the first characteristic matrix, it can be used as an implementation method of the s-fold upsampling processing. The embodiments of the present application do not make specific restrictions on this.

[0175] In one possible implementation, one of the two convolutions in the first convolutional layer can use a two-dimensional group convolutional layer (also called a group convolutional layer). The group convolution is represented by CONV(K ver *K hor , C1, C2, G, t↓). The input feature matrix of this convolutional layer has dimensions [C1, h1, w1], and the output feature matrix has dimensions [C2, h2, w2], where h1 = t·h2 and w1 = t·w2. G represents the number of groups, which controls the connections between the input and output feature matrices. C1 and C2 are divisible by G, and t represents the stride of the sliding window. Without the stride parameter, no spatial resolution change is performed.

[0176] Group convolution can be expressed as:

[0177] Among them, 0≤i<h2, 0≤j<w2;

[0178] ★ is the kernel size K ver ·K hor 2D cross-correlation operator;

[0179] Characteristic matrix weigth[C1 / G,C2,K ver ,Khor ] contains the learnable weights, and the feature matrix bias[C2] contains the learnable bias parameters, both of which are unrestricted learnable parameters.

[0180] Comparison of convolution and group convolution:

[0181] Convolution can be expressed as Group convolution can be expressed as It can be seen that using group convolution in the entropy estimation network can reduce the amount of calculation while expanding the receptive field.

[0182] In an embodiment of the present application, in an entropy estimation network, convolution operations and point-by-point operations are combined to implement a local attention mechanism, which uses the local information of the input feature matrix to modulate the response of each position in the feature matrix, thereby improving compression performance. In addition, the use of group convolution can reduce the amount of computation while expanding the receptive field.

[0183] In an embodiment of the present application, the entropy estimation network may include a variance prediction network and a mean prediction network, the input feature matrices of the two networks (i.e., the first feature matrix may be the same), and the output feature matrices are different. The output feature matrix of the variance prediction network can be called a variance matrix, and the output feature matrix of the mean prediction network can be called a mean matrix. The structures of the two networks are similar, the difference being that, since the variance is a positive number, but the mean is not limited to this, any convolution layer in the variance prediction network uses a quantized convolution layer, and an absolute value operation is provided inside or outside the convolution layer to ensure the positive characteristic of the variance. In addition, all operations in the quantized convolution layer are integers, and all accumulators involved in the calculation process are integers, and the model parameters are quantized to integers, which ensures the cross-platform bit consistency of the network module.

[0184] Quantized convolution can be expressed as qCONV(K ver *K hor ,C5,C6,G,q↓,d,p). The convolution process receives a feature matrix of shape [C5,h5,w5] and outputs a feature matrix of shape [C6,h6,w6], where h6=q·h5 and w6=q·w5, where q represents the convolution step size and the default value of step size is 1, and the corresponding outputs have the same spatial resolution. d is a non-negative integer that defines the maximum value of the elements of the input feature matrix after clipping. p[c7] contains the scaling coefficient and translation coefficient of each channel of the output feature matrix.

[0185] The operations in quantized convolution can be described as three steps:

[0186] temp[c5,i,j]=clip(-d,d-1,input[c5,i,j])

[0187] Where, i = 0,…,h5-1; j = 0,…,w5-1; c5 = 0,…,C5-1;

[0188] Where “★” is the kernel size K ver *K hor The two-dimensional cross-correlation operator (convolution operation) of .

[0189] out[c6,i,j]=(R[c6,i,j])>>p[c6]

[0190] Among them, i=0,…,h6-1; j=0,…,w6-1; c6=0,…,C6-1.

[0191] The shape is [c5,c6,K ver ,K hor ] is a learnable integer weight, and the feature matrix of shape [c6] is a learnable integer bias.

[0192] Optionally, the input of the entropy estimation network is the first feature matrix, starting with a 1*1 convolution (the first convolution layer, whose convolution kernel size is 1*1), followed by an activation function a (the first activation function, for example, ReLU, LeakyReLU, etc.), followed by two convolutions (the second convolution layer), respectively 1*1 convolution and 3*3 group convolution (group convolution), and then the outputs of each of the two convolutions are operated point by point, followed by an activation function b (the second activation function, for example, ReLU, LeakyReLU, etc.), and then the number of channels is increased to 16 times the number of channels of the first feature matrix by using a 1*1 convolution, and finally the number of channels is reduced by 16 times by using a 4-fold sampling PixelShuffle (4), and the height and width are upsampled by 4 times respectively to obtain the final output, which is the second feature matrix.

[0193] FIG10a is a schematic diagram of the structure of the variance prediction network of entropy estimation. As shown in FIG10a, the first feature matrix z[C,h in ,w in] Input qConv(1*1,C,C1) (1*1 convolution, C represents the number of input channels, C1 represents the number of output channels), followed by activation function a, followed by two convolutions, namely qConv(1*1,C1,C2) (1*1 convolution, C1 represents the number of input channels, C2 represents the number of output channels) and qConv(3*3,C1,C2,G=g) (3*3group convolution, C1 represents the number of input channels, C2 represents the number of output channels), and then these two convolutions The outputs of each matrix are subjected to point-by-point operation a, followed by activation function b, and then qConv(1*1,C2,16C) (1*1 convolution, C2 represents the number of input channels, 16C represents the number of output channels) is used to increase the number of channels to 16 times the number of channels of the first feature matrix. Finally, PixelShuffle(4) is used to reduce the number of channels by 16 times, and the height and width are upsampled by 4 times respectively. Then, the absolute value operation ABS() is performed in sequence to obtain the second feature matrix, that is, the variance matrix δ[C,4h in ,4w in ].

[0194] FIG10b is a schematic diagram of the structure of the mean prediction network of entropy estimation. As shown in FIG10b, the first characteristic matrix z[C,h in ,w in ] Input Conv(1*1,C,C3) (1*1 convolution, C represents the number of input channels, C3 represents the number of output channels), followed by activation function c, followed by two convolutions, namely Conv(1*1,C3,C4) (1*1 convolution, C3 represents the number of input channels, C4 represents the number of output channels) and Conv(3*3,C3,C4,G=g) (3*3group convolution, C3 represents the number of input channels, C4 represents the number of output channels), then perform point-by-point operation b on the outputs of these two convolutions, followed by activation function d, and then use Conv(1*1,C4,16C) (1*1 convolution, C4 represents the number of input channels, 16C represents the number of output channels) to increase the number of channels to 16 times the number of channels of the first feature matrix, and finally use PixelShuffle(4) to reduce the number of channels by 16 times, and upsample the height and width by 4 times respectively to obtain the second feature matrix, that is, the mean matrix p[C,4h in ,4w in ].

[0195] Optionally, there are multiple options for the activation function, number of channels, etc. in the embodiments shown in Figures 10a and 10b above. For example, activation function a-activation function d can be specified as any activation function, including but not limited to ReLU, LeakyReLU, Tanh, etc. For another example, the number of intermediate channels C1, C2, C3, and C4 in the entropy estimation network can also be adjusted at will. In addition, depending on the inference speed of the hardware, the 4x upsampling process of the last layer can also be replaced by other operations. Therefore, the embodiments of the present application do not specifically limit the aforementioned operations.

[0196] The technical solution of this application is described below through several specific embodiments.

[0197] Example 1

[0198] Figure 11a is a structural diagram of the variance prediction network for entropy estimation. As shown in Figure 11a, based on the embodiment shown in Figure 10a, the activation function a and the activation function b use ReLU, and the element-wise operation a uses element-wise multiplication (EMUL).

[0199] That is, the first characteristic matrix z[C,h in ,w in ] Input qConv(1*1,C,C1) (1*1 convolution, C represents the number of input channels, C1 represents the number of output channels), followed by ReLU, followed by two convolutions, namely qConv(1*1,C1,C2) (1*1 convolution, C1 represents the number of input channels, C2 represents the number of output channels) and qConv(3*3,C1,C2,G=g) (3*3 group convolution, C1 represents the number of input channels, C2 represents the number of output channels), then perform EMUL on the outputs of these two convolutions, followed by ReLU, and then use qConv(1*1,C2,16C) (1*1 convolution, C2 represents the number of input channels, 16C represents the number of output channels) to increase the number of channels to 16 times the number of channels of the first feature matrix, finally use PixelShuffle(4) to reduce the number of channels by 16 times, and upsample the height and width by 4 times respectively, and then obtain the second feature matrix through the absolute value operation ABS(), that is, the variance matrix δ[C,4h in ,4w in ].

[0200] Figure 11b is a structural diagram of the mean prediction network for entropy estimation. As shown in Figure 11b, based on the embodiment shown in Figure 10b, the activation function c and the activation function d use LeakyReLU, and the element-by-element operation b uses EMUL.

[0201] That is, the first characteristic matrix z[C,h in ,win ] Input Conv(1*1,C,C3) (1*1 convolution, C represents the number of input channels, C3 represents the number of output channels), followed by LeakyReLU, followed by two convolutions, namely Conv(1*1,C3,C4) (1*1 convolution, C3 represents the number of input channels, C4 represents the number of output channels) and Conv(3*3,C3,C4,G=g) (3*3group convolution, C3 represents the number of input channels, C4 represents the number of output channels), then perform EMUL on the outputs of these two convolutions, followed by LeakyReLU, and then use Conv(1*1,C4,16C) (1*1 convolution, C4 represents the number of input channels, 16C represents the number of output channels) to increase the number of channels to 16 times the number of channels of the first feature matrix, and finally use PixelShuffle(4) to reduce the number of channels by 16 times, and upsample the height and width by 4 times respectively to obtain the second feature matrix, that is, the mean matrix p[C,4h in ,4w in ].

[0202] Compared with the division operation, the present embodiment adopts the multiplication operation. On the one hand, it is not necessary to accommodate the restriction that the square root of the variance calculation must be a positive number. On the other hand, it is not necessary to accommodate the restriction that the denominator of the division cannot be 0.

[0203] Example 2

[0204] FIG12a is a schematic diagram of the structure of the variance prediction network for entropy estimation. As shown in FIG12a , based on the embodiment shown in FIG10a , the 4-fold upsampling process is replaced by the PixelShuffle (4) with a 4-fold upsampling deconvolution layer.

[0205] That is, the first characteristic matrix z[C,h in ,w in ] Input qConv(1*1,C,C1) (1*1 convolution, C represents the number of input channels, C1 represents the number of output channels), followed by activation function a, followed by two convolutions, namely qConv(1*1,C1,C2) (1*1 convolution, C1 represents the number of input channels, C2 represents the number of output channels) and qConv(3*3,C1,C2,G=g) (3*3group convolution, C1 represents the number of input channels, C2 represents the number of output channels), then perform point-by-point operation a on the outputs of these two convolutions, followed by activation function b, and then use qConv(1*1,C2,16C) (1*1 convolution, C2 represents the number of input channels, 16C represents the number of output channels) to increase the number of channels to 16 times the number of channels of the first feature matrix, and finally use qConv -1(4*4,16C,C,stride=4) reduces the number of channels by 16 times, and upsamples the height and width by 4 times respectively. Then, the absolute value operation ABS() is performed in sequence to obtain the second feature matrix, that is, the variance matrix δ[C,4h in ,4w in ].

[0206] FIG12b is a schematic diagram of the structure of the mean prediction network of entropy estimation. As shown in FIG12b , based on the embodiment shown in FIG10b , the 4-fold upsampling process is replaced by the PixelShuffle (4) with a 4-fold upsampling deconvolution layer.

[0207] That is, the first characteristic matrix z[C,h in ,w in ] Input Conv(1*1,C,C3) (1*1 convolution, C represents the number of input channels, C3 represents the number of output channels), followed by activation function c, followed by two convolutions, namely Conv(1*1,C3,C4) (1*1 convolution, C3 represents the number of input channels, C4 represents the number of output channels) and Conv(3*3,C3,C4,G=g) (3*3group convolution, C3 represents the number of input channels, C4 represents the number of output channels), then perform point-by-point operation b on the outputs of these two convolutions, followed by activation function d, and then use Conv(1*1,C4,16C) (1*1 convolution, C4 represents the number of input channels, 16C represents the number of output channels) to increase the number of channels to 16 times the number of channels of the first feature matrix, and finally use Conv -1 (4*4,16C,C,stride=4) reduces the number of channels by 16 times and upsamples the height and width by 4 times respectively to obtain the second feature matrix, that is, the mean matrix p[C,4h in ,4w in ].

[0208] In this embodiment, the deconvolution layer with 4x upsampling is placed at the end, which can meet the upsampling requirement and avoid the increase of computational complexity because there is no subsequent processing.

[0209] Example 3

[0210] FIG13a is a schematic diagram of the structure of the variance prediction network for entropy estimation. As shown in FIG13a , based on the embodiment shown in FIG10a , the 4-fold upsampling process is replaced by the last convolution layer and PixelShuffle (4) with a 4-fold upsampling deconvolution layer.

[0211] That is, the first characteristic matrix z[C,h in ,w in] Input qConv(1*1,C,C1) (1*1 convolution, C represents the number of input channels, C1 represents the number of output channels), followed by activation function a, followed by two convolutions, namely qConv(1*1,C1,C2) (1*1 convolution, C1 represents the number of input channels, C2 represents the number of output channels) and qConv(3*3,C1,C2,G=g) (3*3group convolution, C1 represents the number of input channels, C2 represents the number of output channels), then perform point-by-point operation a on the outputs of each of the two convolutions, followed by activation function b, and finally use qConv -1 (4*4, C2, C, stride = 4) adjusts the number of channels to the number of channels of the first feature matrix, and upsamples the height and width by 4 times respectively, and then performs the absolute value operation ABS() to obtain the second feature matrix, that is, the variance matrix δ[C,4h in ,4w in ].

[0212] FIG13b is a schematic diagram of the structure of the mean prediction network for entropy estimation. As shown in FIG13b , based on the embodiment shown in FIG10b , the 4-fold upsampling process is replaced by the last convolution layer and PixelShuffle (4) with a 4-fold upsampling deconvolution layer.

[0213] That is, the first characteristic matrix z[C,h in ,w in ] Input Conv(1*1,C,C3) (1*1 convolution, C represents the number of input channels, C3 represents the number of output channels), followed by activation function c, followed by two convolutions, namely Conv(1*1,C3,C4) (1*1 convolution, C3 represents the number of input channels, C4 represents the number of output channels) and Conv(3*3,C3,C4,G=g) (3*3group convolution, C3 represents the number of input channels, C4 represents the number of output channels), then perform point-by-point operation b on the outputs of each of the two convolutions, followed by activation function d, and finally use Conv -1 (4*4,C4,C,stride=4) adjusts the number of channels to the number of channels of the first feature matrix, and upsamples the height and width by 4 times to obtain the second feature matrix, that is, the mean matrix p[C,4hi n, 4w in ].

[0214] FIG14 is a schematic diagram of an entropy estimation device 1400 for image compression according to an embodiment of the present application. As shown in FIG14 , the device 1400 of this embodiment can be applied to an encoding end or a decoding end. The device 1400 may include: an acquisition module 1401 and a processing module 1402.

[0215] The acquisition module 1401 is used to acquire a first feature matrix to be processed. The shape of the first feature matrix is ​​[C in ,h in ,w in ], where C in represents the number of channels of the first feature matrix, h in represents the height of the first characteristic matrix, w in represents the width of the first feature matrix; processing module 1402, for inputting the first feature matrix into the entropy estimation network to obtain a second feature matrix, the shape of the second feature matrix is ​​[C out ,h out ,w out ], where C out represents the number of channels of the second feature matrix, h out represents the height of the second characteristic matrix, w out represents the width of the second characteristic matrix; the second characteristic matrix and the first characteristic matrix meet the following conditions: C out =C in , h out =s·h in , w out =s·w in , s is an integer greater than 1; the entropy estimation network includes a first convolutional layer and a point-by-point operation, the first convolutional layer includes two convolution processes, one of the two convolution processes is a two-dimensional grouped convolution, and the input of the point-by-point operation is the output of each of the two convolution processes.

[0216] In one possible implementation, the input feature matrix and output feature matrix of the two-dimensional grouped convolution satisfy the following conditions: h1=t·h2, w1=t·w2, C1 and C2 are divisible by G; t is an integer greater than 1; C1 represents the number of channels of the input feature matrix, h1 represents the height of the input feature matrix, and w1 represents the width of the input feature matrix; C2 represents the number of channels of the output feature matrix, h2 represents the height of the output feature matrix, and w2 represents the width of the output feature matrix; G represents the number of groups, which is used to control the connection between the input feature matrix and the output feature matrix.

[0217] In a possible implementation, the entropy estimation network further includes an s-fold upsampling process.

[0218] In a possible implementation, the s-fold upsampling process includes a second convolution layer and a pixel shuffling layer in sequence, and the third feature matrix output by the second convolution layer and the first feature matrix meet the following conditions: C3 = nC in , h3=h in , w3=w in, where n is an integer greater than 1, C3 represents the number of channels of the third characteristic matrix, h3 represents the height of the third characteristic matrix, w3 represents the width of the third characteristic matrix, the pixel shuffling layer is used to upsample the height and width of the third characteristic matrix respectively, and the characteristic matrix output by the pixel shuffling layer is the second characteristic matrix.

[0219] In one possible implementation, the s-fold upsampling process sequentially includes a second convolution layer and an s-fold upsampling deconvolution layer, and the third feature matrix output by the second convolution layer and the first feature matrix satisfy the following condition: C3=nC in , h3=h in , w3=w in , where n is an integer greater than 1, C3 represents the number of channels of the third feature matrix, h3 represents the height of the third feature matrix, w3 represents the width of the third feature matrix, the deconvolution layer is used to upsample the height and width of the third feature matrix respectively, and the feature matrix output by the deconvolution layer is the second feature matrix.

[0220] In one possible implementation, the s-fold upsampling process includes an s-fold upsampling deconvolution layer, where the deconvolution layer is used to upsample the height and width of the input feature matrix respectively, and the feature matrix output by the deconvolution layer is the second feature matrix.

[0221] In one possible implementation, the entropy estimation network also includes a first activation function and a second activation function; the first activation function and the second activation function respectively include an identity operation Identity, a rectified linear unit ReLU processing, a leaky rectified linear unit LeakyReLU processing, or a parameter rectified linear unit PReLU processing.

[0222] In a possible implementation, the point-by-point operation includes point-by-point multiplication, point-by-point addition, point-by-point subtraction, or point-by-point division.

[0223] In a possible implementation, the entropy estimation network further includes a third convolutional layer, and a feature matrix input to the third convolutional layer is the first feature matrix.

[0224] The device of this embodiment can be used to execute the technical solution of the method embodiment shown in Figure 8. Its implementation principle and technical effects are similar and will not be repeated here.

[0225] Figure 15 is a schematic diagram of the structure of an electronic device in an embodiment of the present application. As shown in Figure 15, the electronic device can be used as the encoding end (or encoder) or decoding end (or decoder) in the above embodiment. The electronic device includes at least one processor 151, at least one memory 152, at least one transceiver 153 (also referred to as a communication interface), at least one network interface 154 and one or more antennas 155. The processor 151, the memory 152, the transceiver 153 and the network interface 154 are connected, for example, via a bus. In an embodiment of the present application, the connection may include various interfaces, transmission lines or buses, etc., which are not limited to this. Antenna 155 is connected to transceiver 153. The network interface 154 is used to enable the electronic device to be connected to other communication devices through a communication link.

[0226] The processor 151 is primarily used to process data, control the entire electronic device, execute software programs, and process data from software programs, for example, to support the electronic device in performing the actions described in the above embodiments. The electronic device may include a baseband processor and a central processing unit (CPU). The baseband processor is primarily used to process communication protocols and communication data, while the CPU is primarily used to control the entire electronic device, execute software programs, and process data from software programs. The processor 151 in Figure 15 may integrate the functions of both a baseband processor and a CPU. Those skilled in the art will appreciate that the baseband processor and the CPU may also be independent processors interconnected via a bus or other technology. Those skilled in the art will appreciate that an electronic device may include multiple baseband processors to accommodate different network standards, multiple CPUs to enhance its processing capabilities, and various components of the electronic device may be connected via various buses. The baseband processor may also be referred to as a baseband processing circuit or a baseband processing chip. The CPU may also be referred to as a central processing circuit or a central processing chip. The functionality for processing communication protocols and communication data may be built into the processor or stored in memory as a software program, which is then executed by the processor to implement the baseband processing functionality.

[0227] The memory 152 is primarily used to store software programs and data. The memory 152 can exist independently and be connected to the processor 151. Alternatively, the memory 152 can be integrated with the processor 151, for example, within a single chip. The memory 152 can store program code for executing the technical solutions of the embodiments of the present application, and the execution is controlled by the processor 151. The various computer program codes executed can also be considered as drivers for the processor 151.

[0228] Figure 15 shows only one memory and one processor. In actual electronic devices, there may be multiple processors and multiple memories. The memory may also be referred to as a storage medium or a storage device. The memory may be a storage element on the same chip as the processor, i.e., an on-chip storage element, or an independent storage element, which is not limited in the embodiments of the present application.

[0229] The transceiver 153 can be used to support the reception or transmission of radio frequency signals between electronic devices, and the transceiver 153 can be connected to the antenna 155. The transceiver 153 includes a transmitter Tx and a receiver Rx. Specifically, one or more antennas 155 can receive radio frequency signals. The receiver Rx of the transceiver 153 is used to receive the radio frequency signal from the antenna, convert the radio frequency signal into a digital baseband signal or a digital intermediate frequency signal, and provide the digital baseband signal or digital intermediate frequency signal to the processor 151 so that the processor 1511 can further process the digital baseband signal or digital intermediate frequency signal, such as demodulation and decoding. In addition, the transmitter Tx in the transceiver 153 is also used to receive a modulated digital baseband signal or a digital intermediate frequency signal from the processor 1511, convert the modulated digital baseband signal or digital intermediate frequency signal into a radio frequency signal, and transmit the radio frequency signal through one or more antennas 155. Digital baseband signals and digital intermediate frequency signals can be collectively referred to as digital signals.

[0230] The transceiver 153 may also be referred to as a transceiver unit, a transceiver, a transceiver device, etc. Optionally, a device in the transceiver unit that implements a receiving function may be referred to as a receiving unit, and a device in the transceiver unit that implements a transmitting function may be referred to as a transmitting unit. That is, the transceiver unit includes a receiving unit and a transmitting unit. The receiving unit may also be referred to as a receiver, an input port, a receiving circuit, etc., and the transmitting unit may be referred to as a transmitter, a transmitter, or a transmitting circuit, etc.

[0231] During implementation, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware coding processor, or can be executed by a combination of hardware and software modules in the coding processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0232] The memory mentioned in the above embodiments may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0233] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of this application.

[0234] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0235] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0236] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0237] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0238] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to make a computer device (personal computer, server, or network equipment, etc.) perform all or part of the steps of the method described in each embodiment of the embodiment of the present application. Or the computer software product can be transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instruction can be transmitted from a website, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode to another website, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center that includes one or more available media integrations. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD), a read-only memory (ROM), a random access memory (RAM)), etc.

[0239] The above description is merely a specific implementation of the embodiments of the present application, but the scope of protection of the embodiments of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the embodiments of the present application should be included in the scope of protection of the embodiments of the present application. Therefore, the scope of protection of the embodiments of the present application should be based on the scope of protection of the claims.

Claims

1. A method for estimating entropy for image compression, characterized in that: include: Get the first feature matrix to be processed, the shape of the first feature matrix is ​​[C in ,h in ,w in ], where C in represents the number of channels of the first feature matrix, h in represents the height of the first feature matrix, w in represents the width of the first feature matrix; The first feature matrix is ​​input into the entropy estimation network to obtain a second feature matrix, the shape of the second feature matrix is ​​[C out ,h out ,w out ], where C out represents the number of channels of the second feature matrix, h out represents the height of the second characteristic matrix, w out represents the width of the second characteristic matrix; the second characteristic matrix and the first characteristic matrix satisfy the following conditions: C out =C in ,h out =s·h in , w out =s·w in , s is an integer greater than 1; the entropy estimation network includes a first convolutional layer and a point-by-point operation, the first convolutional layer includes two convolution processes, one of the two convolution processes is a two-dimensional grouped convolution, and the input of the point-by-point operation is the output of each of the two convolution processes.

2. The method according to claim 1, characterized in that The input feature matrix and output feature matrix of the two-dimensional grouped convolution satisfy the following conditions: h1=t·h2, w1=t·w2, C1 and C2 are divisible by G; wherein t is an integer greater than 1; C1 represents the number of channels of the input feature matrix, h1 represents the height of the input feature matrix, and w1 represents the width of the input feature matrix; C2 represents the number of channels of the output feature matrix, h2 represents the height of the output feature matrix, and w2 represents the width of the output feature matrix; G represents the number of groups, which is used to control the connection between the input feature matrix and the output feature matrix.

3. The method according to claim 1 or 2, characterized in that: The entropy estimation network also includes an s-fold upsampling process.

4. The method according to claim 3, characterized in that The s-fold upsampling process includes a second convolution layer and a pixel shuffling layer in sequence, and the third feature matrix output by the second convolution layer and the first feature matrix satisfy the following condition: C3 = nC in , h3=h in , w3=w in , wherein n is an integer greater than 1, C3 represents the number of channels of the third feature matrix, h3 represents the height of the third feature matrix, w3 represents the width of the third feature matrix, the pixel shuffling layer is used to upsample the height and width of the third feature matrix respectively, and the feature matrix output by the pixel shuffling layer is the second feature matrix.

5. The method according to claim 3, characterized in that: The s-fold upsampling process includes a second convolution layer and an s-fold upsampling deconvolution layer in sequence, and the third feature matrix output by the second convolution layer and the first feature matrix satisfy the following condition: C3=nC in , h3=h in , w3=w in , wherein n is an integer greater than 1, C3 represents the number of channels of the third feature matrix, h3 represents the height of the third feature matrix, w3 represents the width of the third feature matrix, the deconvolution layer is used to upsample the height and width of the third feature matrix respectively, and the feature matrix output by the deconvolution layer is the second feature matrix.

6. The method according to claim 3, characterized in that The s-fold upsampling process includes an s-fold upsampling deconvolution layer, the deconvolution layer is used to upsample the height and width of the input feature matrix respectively, and the feature matrix output by the deconvolution layer is the second feature matrix.

7. The method according to any one of claims 1 to 6, characterized in that The entropy estimation network also includes a first activation function and a second activation function; the first activation function and the second activation function respectively include an identity operation Identity, a rectified linear unit ReLU processing, a leaky rectified linear unit LeakyReLU processing, or a parameter rectified linear unit PReLU processing.

8. The method according to any one of claims 1 to 7, characterized in that The point-by-point operation includes point-by-point multiplication, point-by-point addition, point-by-point subtraction or point-by-point division.

9. The method according to any one of claims 1 to 8, characterized in that The entropy estimation network also includes a third convolutional layer, and the feature matrix input to the third convolutional layer is the first feature matrix.

10. The method according to any one of claims 1 to 9, characterized in that s＝4。 11. An entropy estimation device for image compression, characterized in that: include: An acquisition module is used to acquire a first feature matrix to be processed, wherein the shape of the first feature matrix is ​​[C in ,h in ,w in ], where C in surface represents the number of channels of the first feature matrix, h in represents the height of the first feature matrix, w in represents the width of the first feature matrix; A processing module is used to input the first feature matrix into an entropy estimation network to obtain a second feature matrix, wherein the shape of the second feature matrix is ​​[C out ,h out ,w out ], where C out represents the number of channels of the second feature matrix, h out represents the height of the second characteristic matrix, w out represents the width of the second characteristic matrix; the second characteristic matrix and the first characteristic matrix satisfy the following conditions: C out =C in ,h out =s·h in , w out =s·w in , s is an integer greater than 1; the entropy estimation network includes a first convolutional layer and a point-by-point operation, the first convolutional layer includes two convolution processes, one of the two convolution processes is a two-dimensional grouped convolution, and the input of the point-by-point operation is the output of each of the two convolution processes.

12. The device according to claim 11, characterized in that The input feature matrix and output feature matrix of the two-dimensional grouped convolution satisfy the following conditions: h1=t·h2, w1=t·w2, C1 and C2 are divisible by G; t is an integer greater than 1; C1 represents the number of channels of the input feature matrix, h1 represents the height of the input feature matrix, and w1 represents the width of the input feature matrix; C2 represents the number of channels of the output feature matrix, h2 represents the height of the output feature matrix, and w2 represents the width of the output feature matrix; G represents the number of groups, which is used to control the connection between the input feature matrix and the output feature matrix.

13. The device according to claim 11 or 12, characterized in that The entropy estimation network also includes an s-fold upsampling process.

14. The device according to claim 13, characterized in that The s-fold upsampling process includes a second convolution layer and a pixel shuffling layer in sequence, and the third feature matrix output by the second convolution layer and the first feature matrix satisfy the following condition: C3 = nC in , h3=h in , w3=w in , wherein n is an integer greater than 1, C3 represents the number of channels of the third feature matrix, h3 represents the height of the third feature matrix, w3 represents the width of the third feature matrix, the pixel shuffling layer is used to upsample the height and width of the third feature matrix respectively, and the feature matrix output by the pixel shuffling layer is the second feature matrix.

15. The device according to claim 13, characterized in that The s-fold upsampling process includes a second convolution layer and an s-fold upsampling deconvolution layer in sequence, and the third feature matrix output by the second convolution layer and the first feature matrix satisfy the following condition: C3=nC in , h3=h in , w3=w in , wherein n is an integer greater than 1, C3 represents the number of channels of the third feature matrix, h3 represents the height of the third feature matrix, w3 represents the width of the third feature matrix, the deconvolution layer is used to upsample the height and width of the third feature matrix respectively, and the feature matrix output by the deconvolution layer is the second feature matrix.

16. The device according to claim 13, characterized in that The s-fold upsampling process includes an s-fold upsampling deconvolution layer, the deconvolution layer is used to upsample the height and width of the input feature matrix respectively, and the feature matrix output by the deconvolution layer is the second feature matrix.

17. The device according to any one of claims 11 to 16, characterized in that The entropy estimation network also includes a first activation function and a second activation function; the first activation function and the second activation function respectively include an identity operation Identity, a rectified linear unit ReLU processing, a leaky rectified linear unit LeakyReLU processing, or a parameter rectified linear unit PReLU processing.

18. The device according to any one of claims 11 to 17, characterized in that The point-by-point operation includes point-by-point multiplication, point-by-point addition, point-by-point subtraction or point-by-point division.

19. The device according to any one of claims 11 to 18, characterized in that The entropy estimation network also includes a third convolutional layer, and the feature matrix input to the third convolutional layer is the first feature matrix.

20. The device according to any one of claims 11 to 19, characterized in that s＝4。 21. An encoder, characterized in that: include: one or more processors; A non-transitory computer-readable storage medium, coupled to the processor and storing a program executed by the processor, wherein the program, when executed by the processor, causes the decoder to perform the method of any one of claims 1-10.

22. A decoder, characterized in that: include: one or more processors; A non-transitory computer-readable storage medium, coupled to the processor and storing a program executed by the processor, wherein the program, when executed by the processor, causes the decoder to perform the method of any one of claims 1-10.

23. A computer-readable storage medium, characterized in that: The method comprises instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 10.

Images