Entropy estimation method and device for image compression

Abstract

The invention provides an entropy estimation method and device for image compression. The method comprises the steps that a to-be-processed first feature matrix is acquired, the shape of the first feature matrix is [Cin, hin, win], Gin represents the number of channels of the first feature matrix, hin represents the height of the first feature matrix, and win represents the width of the first feature matrix; the first feature matrix is input into an entropy estimation network to obtain a second feature matrix, the shape of the second feature matrix is [Cout, hout and wout], Cout represents the number of channels of the second feature matrix, hout represents the height of the second feature matrix, and wout represents the width of the second feature matrix; the second feature matrix and the first feature matrix meet the following conditions: Cout = Gin, hoout = s * hin, and wout = s * win, and s is an integer greater than 1; the entropy estimation network comprises a first convolutional layer and point-by-point operation, the first convolutional layer comprises two pieces of convolution processing, one of the two pieces of convolution processing is two-dimensional grouping convolution, and the input of the point-by-point operation is the respective output of the two pieces of convolution processing. The compression performance can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an entropy estimation method and device for image compression. Background Art

[0002] As the performance of convolutional neural networks (CNNs) in computer vision tasks such as image recognition and target detection far exceeds that of traditional algorithms, more and more researchers have begun to explore image / video compression methods based on deep learning. Some researchers have designed end-to-end deep learning image / video compression networks, including encoding networks, entropy estimation networks, entropy encoding networks, entropy decoding networks, decoding networks and other modules. The entropy estimation network is one of the basic components of the image / video compression network. It is mainly used to estimate the probability distribution (mean or variance) of the encoded / decoded signal. The accuracy of the entropy estimation network's estimation of the probability distribution directly affects the size of the bitstream, and thus affects the overall compression performance. Therefore, designing an efficient entropy estimation network is the key to improving image / video compression performance. Summary of the invention

[0003] The embodiments of the present application provide an entropy estimation method and device for image compression, which can improve compression performance.

[0004] In a first aspect, the present application provides an entropy estimation method for image compression, comprising: obtaining a first feature matrix to be processed, wherein the shape of the first feature matrix is ​​[C in ,h in , w in ], where C in represents the number of channels of the first feature matrix, h in represents the height of the first feature matrix, w in represents the width of the first feature matrix; the first feature matrix is ​​input into the entropy estimation network to obtain a second feature matrix, the shape of the second feature matrix is ​​[C out ,h out , w out ], where C out represents the number of channels of the second feature matrix, h out represents the height of the second characteristic matrix, w out represents the width of the second characteristic matrix; the second characteristic matrix and the first characteristic matrix satisfy the following conditions: C out =C in ,h out =s·h in , w out =s·w in, s is an integer greater than 1; the entropy estimation network includes a first convolutional layer and a point-by-point operation, the first convolutional layer includes two convolution processes, one of the two convolution processes is a two-dimensional group convolution (also called a group convolution process), and the input of the point-by-point operation is the output of each of the two convolution processes.

[0005] In a possible implementation, the second characteristic matrix and the first characteristic matrix satisfy the following conditions: out =C in ,h out =4h in , w ont =4w in ; The entropy estimation network sequentially includes the third convolution layer, the first activation function, the first convolution layer, the point-by-point operation, the second activation function and a 4-fold upsampling process.

[0006] In the embodiment of the present application, in the entropy estimation network, the convolution operation and the point-by-point operation are combined to realize the local attention mechanism, and the local information of the input feature matrix is ​​used to modulate the response of each position in the feature matrix to improve the compression performance; in addition, the use of group convolution can reduce the amount of calculation while expanding the receptive field.

[0007] According to the above description, the entropy estimation network can obtain the estimated probability value of each feature element according to the image feature estimation, so as to be used for the processing of the entropy coding network and the entropy decoding network. Based on this, the first feature matrix can be the feature matrix obtained by quantizing the original image after being processed by the coding network. For example, the first feature matrix can be represented as input[C in ,h in , w in ], C in Indicates the number of channels of the first feature matrix, h in represents the height of the first characteristic matrix, w in Represents the width of the first characteristic matrix, that is, [C in ,h in , w in ] represents the shape of the first feature matrix.

[0008] The second characteristic matrix can be expressed as output[C out ,h out , w out ], C out Indicates the number of channels of the second feature matrix, h out represents the height of the second characteristic matrix, w ont Represents the width of the second characteristic matrix, that is, [C out ,h out , w out] represents the shape of the second feature matrix.

[0009] In the embodiment of the present application, the second characteristic matrix and the first characteristic matrix satisfy:

[0010] C out =C in

[0011] h out =s·h in

[0012] w out =s·w in

[0013] Wherein, s is an integer greater than 1, and is used to represent the stride of the sliding window. Optionally, s=4.

[0014] The entropy estimation network sequentially includes a third convolution layer, a first activation function, a first convolution layer, a point-by-point operation, a second activation function, and an s-fold upsampling process, wherein the first convolution layer includes two convolution processes, and the input of the point-by-point operation is the output of each of the two convolution processes.

[0015] The input feature matrix of any of the above convolutional layers is converted into an output feature matrix with unchanged height and width through convolution operation. The output feature matrix can be considered as a local response of the input feature matrix. The convolution operation can be expressed as the following formula:

[0016] Conv(x)=∑w*x+b

[0017] Among them, w is the weight of the convolutional layer, and b is the bias parameter of the convolutional layer, both of which are unrestricted learnable parameters.

[0018] Any of the above activation operations is a point-by-point operation, that is, an independent mapping operation is performed on each element of the input feature matrix. The activation operation can be an identity operation (Identity), a rectified linear unit (Rectified Linear Unit, ReLU), a rectified linear unit (LeakyReLU), a parametric rectified linear unit (Parametric RectifiedLinearUnit, PReLU), Sigmoid, Tanh, etc., which can provide linear or nonlinear transformation characteristics. In the embodiment of the present application, the first activation function and the second activation function can use the same operation or different operations, which is not specifically limited. It should be noted that the activation function can also use other deformation methods of ReLU, which is not specifically limited in the embodiment of the present application.

[0019] The above-mentioned point-by-point operation may include point-by-point multiplication, point-by-point addition, point-by-point subtraction or point-by-point division. The input of the point-by-point operation is the output feature matrix of the two convolution processes of the first convolutional layer. The two output feature matrices have the same size, and the point-by-point operation multiplies, adds, subtracts or divides the element values ​​at corresponding positions. The combination of convolution operation and point-by-point operation realizes the local attention mechanism, which uses the local information of the input feature matrix to modulate the response of each position in the feature matrix.

[0020] In the embodiment of the present application, the s-fold upsampling process can be implemented in multiple ways:

[0021] (1) The s-fold upsampling process sequentially includes a second convolutional layer and an s-fold pixel shuffling layer.

[0022] The eighth feature matrix is ​​input into the second convolutional layer and outputs the ninth feature matrix. The ninth feature matrix has the same size as the eighth feature matrix, but the number of channels of the ninth feature matrix is ​​n times the number of channels of the first feature matrix. The ninth feature matrix is ​​input into the pixel shuffle layer and outputs the second feature matrix. The height of the second feature matrix is ​​s times the height of the ninth feature matrix, the width of the second feature matrix is ​​s times the width of the ninth feature matrix, and the number of channels of the second feature matrix is ​​1 / n of the number of channels of the ninth feature matrix, but the same as the number of channels of the first feature matrix. Optionally, n=s 2 .

[0023] The pixel shuffle layer, also called sub-pixel convolution, is represented by PixelShuffle(s), where s>1. The pixel shuffle layer can convert the input feature matrix (whose shape is [C 4 ,h 4 , w 4 ]) to obtain the output feature matrix (whose shape is [C out ,h out , w out ]), where h out =s·h 4 , w out =s·w 4 , C out =C 4 / s 2 .

[0024] In the last layer of the entropy estimation network, a pixel shuffling layer is used for upsampling, and only memory operations can be performed. Therefore, it is highly adaptable to different hardware (for example, it can be converted into a convolution operation on an AI accelerator as appropriate), and can even be combined with subsequent steps to merge memory access, thereby improving the overall encoding / decoding efficiency.

[0025] (2) The s-fold upsampling process sequentially includes a second convolutional layer and an s-fold upsampling deconvolution layer.

[0026] The eighth feature matrix is ​​input into the second convolution layer and outputs the ninth feature matrix. The ninth feature matrix has the same size as the eighth feature matrix, but the number of channels of the ninth feature matrix is ​​n times the number of channels of the first feature matrix. The ninth feature matrix is ​​input into the deconvolution layer with s times upsampling and outputs the second feature matrix. The height of the second feature matrix is ​​s times the height of the ninth feature matrix, the width of the second feature matrix is ​​s times the width of the ninth feature matrix, and the number of channels of the second feature matrix is ​​1 / n of the number of channels of the ninth feature matrix, but the same as the number of channels of the first feature matrix. Optionally, n=s 2 .

[0027] (3) The s-fold upsampling process includes a deconvolution layer with s-fold upsampling.

[0028] The eighth feature matrix is ​​input to the deconvolution layer with s times upsampling and outputs the second feature matrix, the height of the second feature matrix is ​​s times the height of the eighth feature matrix, the width of the second feature matrix is ​​s times the width of the eighth feature matrix, the number of channels of the second feature matrix is ​​1 / n of the number of channels of the eighth feature matrix, and the same as the number of channels of the first feature matrix. Optionally, n=s 2 .

[0029] (4) The s-fold upsampling process includes the second convolutional layer and two s / 2-fold pixel shuffling layers.

[0030] The eighth feature matrix is ​​input into the second convolutional layer to output the ninth feature matrix. The ninth feature matrix has the same size as the eighth feature matrix, but the number of channels of the ninth feature matrix is ​​n times the number of channels of the first feature matrix. The ninth feature matrix is ​​input into the first s / 2 times pixel shuffle layer to output the tenth feature matrix. The height of the tenth feature matrix is ​​s / 2 times the height of the ninth feature matrix, the width of the tenth feature matrix is ​​s / 2 times the width of the ninth feature matrix, and the number of channels of the tenth feature matrix is ​​n / 2 of the number of channels of the ninth feature matrix. The tenth feature matrix is ​​input into the second s / 2 times pixel shuffle layer to output the eleventh feature matrix. The height of the eleventh feature matrix is ​​s / 2 times the height of the tenth feature matrix, the width of the eleventh feature matrix is ​​s / 2 times the width of the tenth feature matrix, and the number of channels of the eleventh feature matrix is ​​n / 2 of the number of channels of the tenth feature matrix, which is the same as the number of channels of the first feature matrix. Optionally, n=s 2 .

[0031] It should be noted that in addition to the above-mentioned implementation methods, the s-fold upsampling processing can also be implemented in other ways, as long as the height of the second characteristic matrix can be s times the height of the eighth characteristic matrix, the width of the second characteristic matrix can be s times the width of the eighth characteristic matrix, and the second characteristic matrix has the same number of channels as the first characteristic matrix, all of which can be used as implementation methods of the s-fold upsampling processing, and the embodiments of the present application do not make specific limitations on this.

[0032] In one possible implementation, one of the two convolutions in the first convolutional layer can use a two-dimensional group convolutional layer (also called a group convolutional layer). The group convolution is represented by CONV(K ver *K hor , C 1 , C 2 , G, t↓). The size of the input feature matrix of the convolutional layer is [C 1 ,h 1 , w 1 ], the size of the output feature matrix is ​​[C 2 ,h 2 , w 2 ], where h 1 =t·h 2 , w 1 =t·w 2 , G represents the number of groups, which controls the connection between the input feature matrix and the output feature matrix. 1 and C 2 It is divisible by G, and t represents the step size of the sliding window. In the absence of a step size parameter, no spatial resolution change is "performed". Using group convolution in the entropy estimation network can reduce the amount of calculation while expanding the receptive field.

[0033] In an embodiment of the present application, the entropy estimation network may include a variance prediction network and a mean prediction network. The input feature matrices of the two networks (i.e., the first feature matrix may be the same) are different, and the output feature matrices are different. The output feature matrix of the variance prediction network may be called a variance matrix, and the output feature matrix of the mean prediction network may be called a mean matrix. The structures of the two networks are similar, except that, since the variance is a positive number, but the mean is not limited to this, any convolution layer in the variance prediction network uses a quantized convolution layer, and an absolute value operation is provided inside or outside the convolution layer to ensure the positive characteristic of the variance.

[0034] Quantized convolution can be expressed as qCONV(K ver *K hor， C 5 , C 6 , G, q↓, d, p). The convolution process receives the shape [C 5 ,h 5 , w 5 ] feature matrix, and the output shape is [C 6 ,h 6 , w 6 ], where h 6 =q·h 5 , w 6 =q·w 5, q represents the convolution step size, the default value of the step size is 1, and the corresponding outputs have the same spatial resolution; d is a non-negative integer, which defines the maximum value of the elements of the input feature matrix after clipping; p[c 7 ] contains the scaling and translation coefficients for each channel of the output feature matrix.

[0035] Optionally, the input of the entropy estimation network is the first feature matrix, starting with a 1*1 convolution (the first convolution layer, whose convolution kernel size is 1*1), followed by an activation function a (the first activation function, for example, ReLU, LeakyReLU, etc.), followed by two convolutions (the second convolution layer), which are 1*1 convolution and 3*3 group convolution (group convolution), and then the outputs of the two convolutions are point-by-point operated, followed by an activation function b (the second activation function, for example, ReLU, LeakyReLU, etc.), and then a 1*1 convolution is used to increase the number of channels to 16 times the number of channels of the first feature matrix, and finally a 4-fold sampling PixelShuffle (4) is used to reduce the number of channels by 16 times, and the height and width are upsampled by 4 times respectively to obtain the final output, which is the second feature matrix.

[0036] In a second aspect, the present application provides an entropy estimation device for image compression, comprising: an acquisition module, configured to acquire a first feature matrix to be processed, wherein the shape of the first feature matrix is ​​[C in ,h in , w in ], where C in represents the number of channels of the first feature matrix, h in represents the height of the first feature matrix, w in represents the width of the first feature matrix; a processing module, used to input the first feature matrix into an entropy estimation network to obtain a second feature matrix, the shape of the second feature matrix is ​​[C out ,h out , w out ], where C out represents the number of channels of the second feature matrix, h out represents the height of the second characteristic matrix, w out represents the width of the second characteristic matrix; the second characteristic matrix and the first characteristic matrix satisfy the following conditions: C out =C in ,h out =s·h in , w out =s·w in, s is an integer greater than 1; the entropy estimation network includes a first convolutional layer and a point-by-point operation, the first convolutional layer includes two convolution processes, one of the two convolution processes is a two-dimensional grouped convolution, and the input of the point-by-point operation is the output of each of the two convolution processes.

[0037] In a possible implementation, the input feature matrix and the output feature matrix of the two-dimensional group convolution satisfy the following conditions: 1 =t·h 2 , w 1 =t·w 2 , C 1 and C 2 Divisible by G; t is an integer greater than 1; C 1 represents the number of channels of the input feature matrix, h 1 represents the height of the input feature matrix, w 1 represents the width of the input feature matrix; C 2 represents the number of channels of the output feature matrix, h 2 represents the height of the output feature matrix, w 2 represents the width of the output feature matrix; G represents the number of groups, which is used to control the connection between the input feature matrix and the output feature matrix.

[0038] In a possible implementation, the entropy estimation network further includes an s-fold upsampling process.

[0039] In a possible implementation, the s-fold upsampling process includes a second convolution layer and a pixel shuffling layer in sequence, and the third feature matrix output by the second convolution layer and the first feature matrix satisfy the following conditions: C 3 =nC in ,h 3 =h in , w 3 =w in , where n is an integer greater than 1, C 3 represents the number of channels of the third characteristic matrix, h 3 represents the height of the third characteristic matrix, w 3 represents the width of the third feature matrix, the pixel shuffling layer is used to upsample the height and width of the third feature matrix respectively, and the feature matrix output by the pixel shuffling layer is the second feature matrix.

[0040] In a possible implementation, the s-fold upsampling process sequentially includes a second convolution layer and an s-fold upsampling deconvolution layer, and the third feature matrix output by the second convolution layer and the first feature matrix satisfy the following conditions: C 3 =nC in ,h 3=h in , w 3 =w in , where n is an integer greater than 1, C 3 represents the number of channels of the third characteristic matrix, h 3 represents the height of the third characteristic matrix, w 3 represents the width of the third feature matrix, the deconvolution layer is used to upsample the height and width of the third feature matrix respectively, and the feature matrix output by the deconvolution layer is the second feature matrix.

[0041] In a possible implementation, the s-fold upsampling process includes an s-fold upsampling deconvolution layer, the deconvolution layer is used to upsample the height and width of the input feature matrix respectively, and the feature matrix output by the deconvolution layer is the second feature matrix.

[0042] In a possible implementation, the entropy estimation network also includes a first activation function and a second activation function; the first activation function and the second activation function respectively include an identity operation Identity, a rectified linear unit ReLU processing, a leaky rectified linear unit LeakyReLU processing, or a parameter rectified linear unit PReLU processing.

[0043] In a possible implementation, the point-by-point operation includes point-by-point multiplication, point-by-point addition, point-by-point subtraction, or point-by-point division.

[0044] In a possible implementation, the entropy estimation network further includes a third convolutional layer, and a feature matrix input to the third convolutional layer is the first feature matrix.

[0045] In a third aspect, an embodiment of the present application provides an encoder, comprising: one or more processors; a non-transitory computer-readable storage medium, coupled to the processor and storing a program executed by the processor, wherein the program, when executed by the processor, causes the decoder to perform any of the methods described in the first aspect above. Optionally, after obtaining the second characteristic matrix, the processor of the encoder is further configured to perform entropy encoding based on the second characteristic matrix to obtain a code stream. The encoder also includes a communication interface for sending a code stream.

[0046] In a fourth aspect, an embodiment of the present application provides a decoder, comprising: one or more processors; a non-transitory computer-readable storage medium, coupled to the processor and storing a program executed by the processor, wherein the program, when executed by the processor, causes the decoder to execute any of the methods described in the first aspect above. Optionally, after obtaining the second characteristic matrix, the processor of the decoder is further used to perform entropy decoding based on the second characteristic matrix to obtain a reconstructed image or video. The decoder also includes a communication interface for receiving a video stream to be decoded.

[0047] In a fifth aspect, an embodiment of the present application provides a computer program product comprising a program code, which, when executed on a computer or a processor, is used to execute any of the methods described in the first aspect above.

[0048] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enables the computer to execute any of the methods described in the first aspect above.

[0049] In the seventh aspect, an embodiment of the present application provides a chip comprising one or more interface circuits and one or more processors; the interface circuit is used to receive and / or send signals, and the signals include computer instructions stored in a memory; when the processor executes the computer instructions, it is used to implement any one of the methods described in the first aspect above.

[0050] In an eighth aspect, an embodiment of the present application provides a video code stream transmission system, comprising: an acquisition device for acquiring a video code stream, wherein the video code stream is encoded according to the encoder described in the third aspect above; and a transmission device for transmitting the video code stream.

[0051] In a ninth aspect, an embodiment of the present application provides a method for storing a video code stream, wherein the video code stream is encoded according to the encoder described in the third aspect above.

[0052] In the tenth aspect, an embodiment of the present application provides a storage device for a video code stream, comprising: a communication interface for obtaining a video code stream, wherein the video code stream is encoded according to the encoder described in the third aspect above; and a memory for storing the video code stream.

[0053] In an eleventh aspect, an embodiment of the present application provides a method for sending a video code stream, wherein the video code stream is encoded according to the encoder described in the third aspect above.

[0054] In the twelfth aspect, an embodiment of the present application provides a system for sending a video code stream, comprising: an acquisition device for acquiring a video code stream, which is encoded according to the encoder described in the third aspect above; and a communication interface for sending the video code stream to a destination device. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is an example diagram of a typical deep learning image compression framework;

[0056] Figure 2 This is an example diagram of a typical deep learning video compression framework;

[0057] Figure 3 This is an example diagram of an image / video storage / transmission application scenario;

[0058] Figure 4 Example diagram for image / video storage application scenarios;

[0059] Figure 5 This is an example diagram of Huawei Cloud application scenarios;

[0060] Figure 6 A schematic diagram of a streaming media system workflow applicable to an embodiment of the present application;

[0061] Figure 7 A schematic diagram of the architecture of a streaming media system according to an embodiment of the present application;

[0062] Figure 8 It is a flowchart of a process 800 of an entropy estimation method for image compression according to an embodiment of the present application;

[0063] Fig. 9 A schematic diagram of the structure of an entropy estimation network according to an embodiment of the present application;

[0064] Fig.10a This is a schematic diagram of the structure of the variance prediction network for entropy estimation;

[0065] Fig.10b This is a schematic diagram of the structure of the mean prediction network for entropy estimation;

[0066] Fig.11a This is a schematic diagram of the structure of the variance prediction network for entropy estimation;

[0067] Fig.11b This is a schematic diagram of the structure of the mean prediction network for entropy estimation;

[0068] Fig.12a This is a schematic diagram of the structure of the variance prediction network for entropy estimation;

[0069] Figure 12b This is a schematic diagram of the structure of the mean prediction network for entropy estimation;

[0070] Fig.13a This is a schematic diagram of the structure of the variance prediction network for entropy estimation;

[0071] Fig.13b This is a schematic diagram of the structure of the mean prediction network for entropy estimation;

[0072] Fig.14 Schematic diagram of an entropy estimation device 1400 for image compression according to an embodiment of the present application;

[0073] Fig.15 A schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0075] The terms "first", "second", etc. in the specification embodiments, claims, and drawings of the present application are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, including a series of steps or units. The method, system, product, or device is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0076] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0077] Since the embodiments of the present application involve the application of a neural network, for ease of understanding, the following first explains the relevant nouns or terms used in the embodiments of the present application:

[0078] 1. Neural Networks

[0079] A neural network (NN) is a machine learning model. A neural network can be composed of neural units. A neural unit can refer to a s The output of the operation unit can be shown in formula (1):

[0080]

[0081] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s For x s , b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a nonlinear function such as ReLU. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.

[0082] 2. Multi-layer perception (MLP)

[0083] MLP is a simple deep neural network (DNN) (different layers are fully connected), also known as a multi-layer neural network, which can be understood as a neural network with many hidden layers. There is no special metric for "many" here. From the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The definition of these parameters in DNN is as follows: Take coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W).

[0084] 3. Convolutional Neural Networks

[0085] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. It is a deep learning architecture. A deep learning architecture refers to multiple levels of learning at different levels of abstraction through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network in which each neuron can respond to the image input into it. A convolutional neural network contains a feature extractor consisting of a convolutional layer and a pooling layer. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve an input image or convolution feature plane (feature map).

[0086] A convolutional layer is a layer of neurons in a convolutional neural network that performs convolution on the input signal. A convolutional layer can include many convolution operators, also known as kernels. In image processing, the convolution operator is essentially a weight matrix, which is usually predefined. During the convolution operation on the image, the weight matrix is ​​usually processed horizontally on the input image one pixel by one pixel (or two pixels by two pixels... depending on the value of the stride) to complete the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. During the convolution operation, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, instead of using a single weight matrix, multiple weight matrices of the same size (rows*columns) are applied, that is, multiple isotype matrices. The output of each weight matrix is ​​stacked to form the depth dimension of the convolution image, where the dimension can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in the image, for example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unnecessary noise points in the image. The multiple weight matrices have the same size (rows * columns), and the size of the feature maps extracted by the multiple weight matrices of the same size is also the same. The extracted multiple feature maps of the same size are then merged to form the output of the convolution operation. The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, so that the convolutional neural network can make correct predictions. When the convolutional neural network has multiple convolutional layers, the initial convolutional layer often extracts more general features, which can also be called low-level features; as the depth of the convolutional neural network deepens, the features extracted by the later convolutional layers become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0087] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer. It can be a convolution layer followed by a pooling layer, or multiple convolution layers followed by one or more pooling layers. In the image processing process, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer can include an average pooling operator and / or a maximum pooling operator to sample the input image to obtain an image of smaller size. The average pooling operator can calculate the pixel values ​​in the image within a specific range to produce an average value as the result of average pooling. The maximum pooling operator can take the pixel with the largest value in the range as the result of maximum pooling within a specific range. In addition, just as the size of the weight matrix used in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer can be smaller than the size of the image input to the pooling layer. Each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0088] After being processed by the convolution layer / pooling layer, the convolution neural network is not sufficient to output the required output information. Because as mentioned above, the convolution layer / pooling layer will only extract features and reduce the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network needs to use the neural network layer to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer may include multiple hidden layers, and the parameters contained in the multiple hidden layers can be pre-trained according to the relevant training data of the specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0089] Optionally, after the multiple hidden layers in the neural network layer, an output layer of the entire convolutional neural network is also included. The output layer has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network is completed, the back propagation will begin to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network and the error between the result output by the convolutional neural network through the output layer and the ideal result.

[0090] 4. Recurrent Neural Networks

[0091] Recurrent neural networks (RNN) are used to process sequence data. In the traditional neural network model, from the input layer to the hidden layer and then to the output layer, the layers are fully connected, and the nodes within each layer are disconnected. Although this ordinary neural network has solved many difficult problems, it is still powerless for many problems. For example, if you want to predict the next word in a sentence, you generally need to use the previous word, because the previous and next words in a sentence are not independent. The reason why RNN is called a recurrent neural network is that the current output of a sequence is also related to the previous output. The specific manifestation is that the network will remember the previous information and apply it to the calculation of the current output, that is, the nodes between the hidden layers are no longer disconnected but connected, and the input of the hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous moment. In theory, RNN can process sequence data of any length. The training of RNN is the same as the training of traditional CNN or DNN. The same error back propagation algorithm is used, but there is a difference: that is, if the RNN is expanded, the parameters, such as W, are shared; this is not the case with the traditional neural network mentioned above. And when using the gradient descent algorithm, the output of each step depends not only on the network at the current step, but also on the state of the network at the previous steps. This learning algorithm is called the Back Propagation Through Time (BPTT).

[0092] Since we already have convolutional neural networks, why do we need recurrent neural networks? The reason is simple. In convolutional neural networks, there is a premise that the elements are independent of each other, and the input and output are also independent, such as cats and dogs. But in the real world, many elements are interconnected, such as the change of stocks over time, or someone said: I like traveling, and my favorite place is Yunnan. I must go there if I have the chance in the future. Fill in the blank here, humans should all know to fill in "Yunnan". Because humans will make inferences based on the content of the context, but how can machines do this? RNN came into being. RNN is designed to give machines the ability to remember like humans. Therefore, the output of RNN needs to rely on the current input information and historical memory information.

[0093] 5. Loss Function

[0094] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0095] 6. Back Propagation Algorithm

[0096] Convolutional neural networks can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the error loss information is back-propagated to update the parameters in the initial super-resolution model, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weight matrix.

[0097] 7. Generative Adversarial Networks

[0098] Generative adversarial networks (GAN) are a type of deep learning model. The model includes at least two modules: one is the generative model and the other is the discriminative model. These two modules learn from each other through competition to produce better output. Both the generative model and the discriminative model can be neural networks, specifically deep neural networks or convolutional neural networks. The basic principle of GAN is as follows: Taking the GAN that generates images as an example, assume that there are two networks, G (Generator) and D (Discriminator), where G is a network that generates images. It receives a random noise z and generates images through this noise, denoted as G(z); D is a discriminative network that is used to determine whether a picture is "real". Its input parameter is x, which represents a picture. The output D(x) represents the probability that x is a real picture. If it is 1, it means that 130% is a real picture, and if it is 0, it means that it cannot be a real picture. In the process of training the generative adversarial network, the goal of the generative network G is to generate realistic images as much as possible to deceive the discriminative network D, while the goal of the discriminative network D is to distinguish the images generated by G from the real images as much as possible. In this way, G and D constitute a dynamic "game" process, which is also the "adversary" in the "generative adversarial network". The final result of the game is that, under ideal conditions, G can generate images G(z) that are "real" enough, while D has difficulty determining whether the images generated by G are real, that is, D(G(z)) = 0.5. In this way, an excellent generative model G is obtained, which can be used to generate images.

[0099] Figure 1 This is an example diagram of a typical deep learning image compression framework, such as Figure 1 As shown, the image compression framework is an implementation of current artificial intelligence (AI) image compression technology.

[0100] At the encoding end, the original image is transformed from the image domain to the feature domain through the processing of the encoding network. The transformed image features are encoded into a code stream to be transmitted or stored through the processing of the quantization module and the entropy coding network. At the decoding end, the code stream is decoded into image features through the processing of the entropy decoding network. The image features are transformed from the feature domain to the image domain through the processing of the decoding network to obtain a reconstructed image. The entropy estimation network estimates the estimated probability value of each feature element based on the image features for processing by the entropy coding network and the entropy decoding network.

[0101] In this embodiment, both the encoding network (Encoder) and the decoding network (Decoder) have nonlinear transformation units.

[0102] Figure 2 This is an example diagram of a typical deep learning video compression framework, such as Figure 2 As shown, the video compression framework includes a prediction module (predict model) and a residual compression module (residual compress).

[0103] The prediction module uses the reconstructed image of the previous frame to predict the current frame to obtain the predicted image. The residual compression module compresses the residual between the original image and the predicted image of the current frame on the one hand, and decompresses it on the other hand to obtain the reconstructed residual. The reconstructed residual and the predicted image are summed to obtain the reconstructed image of the current frame. The encoding subnetwork and decoding subnetwork in the prediction module and the residual compression module both have nonlinear transformation units.

[0104] In this embodiment, both the prediction module (predict model) and the residual compression module (residual compress) have nonlinear transformation units.

[0105] Figure 3 This is an example diagram of an image / video storage / transmission application scenario, such as Figure 3 As shown, the application scenario can be a business involving image / video acquisition, storage or transmission in a terminal, Huawei Cloud, or video surveillance, such as terminal photo / video recording, photo album, Huawei Cloud photo album, video surveillance, etc.

[0106] Encoding end: The camera collects images / videos. The AI ​​image / video encoding network extracts features from the image / video to obtain image features with low redundancy, and then compresses the image features to obtain a bitstream / image file.

[0107] Decoding end: When the reconstructed image / video needs to be output, the AI ​​image / video decoding network decompresses the bitstream / image file to obtain image features, and then performs inverse feature extraction on the image features to obtain the reconstructed image / video.

[0108] The storage / transmission module stores the compressed code stream / image files for different services (for example, terminal photography, video surveillance, cloud servers, etc.) or transmits (for example, cloud services, live broadcast technology, etc.).

[0109] Figure 4 Example diagram for image / video storage application scenarios, such as Figure 4As shown, the application scenarios can be terminal albums, Cyberverse (a new "digital reality" technology based on the integration of virtual and real released by Huawei), Huawei Cloud Albums, video surveillance, live broadcasts, etc.

[0110] Encoding end: Camera collects digital images / videos. The AI ​​encoding unit transforms the image / video into image features with lower redundancy, which usually includes a nonlinear transformation unit and has nonlinear characteristics. The entropy estimation unit is responsible for calculating the encoding probability of each data in the image feature. The entropy coding network performs lossless encoding of the image features according to the probability corresponding to each data to obtain the code stream / image file, further reducing the amount of data transmission during the image compression process. The saving module saves the code stream / image file to the corresponding storage location of the terminal.

[0111] Decoding end: The loading module loads the code stream / image file from the corresponding storage location of the terminal and inputs it into the entropy decoding network. The entropy estimation unit is responsible for calculating the decoding probability of each data in the image feature. The entropy decoding network performs lossless decoding on the code stream / image file according to the probability corresponding to each data to obtain the reconstructed image features. The AI ​​decoding unit performs a reverse transformation on the image features output by entropy decoding and parses it into a reconstructed image / video. Corresponding to the AI ​​encoding unit, it usually includes a nonlinear transformation unit and has nonlinear characteristics.

[0112] Optionally, the AI ​​encoding unit, the AI ​​decoding unit and the entropy estimation unit may adopt a network processor (Network Processing Unit, NPU), and the entropy encoding network, the entropy decoding network, the saving module and the loading module may adopt a central processing unit (Central Processing Unit, CPU).

[0113] Figure 5 This is an example diagram of Huawei Cloud application scenarios, such as Figure 5 As shown, the application scenario can be terminal album, Cyberverse, Huawei Cloud album, video surveillance, live broadcast, etc.

[0114] Encoding end: The original image is obtained locally, and image (JPEG) encoding is performed on it to obtain a compressed image, and then the compressed image is sent to the cloud. The cloud performs JPEG decoding on the compressed image to obtain an image, and then the AI ​​encoding unit transforms the image into image features with lower redundancy, which usually includes a nonlinear transformation unit and has nonlinear characteristics. The entropy estimation unit is responsible for calculating the encoding probability of each data in the image feature. The entropy coding network performs lossless encoding on the image features according to the probability corresponding to each data to obtain a code stream / image file, further reducing the amount of data transmission during the image compression process. The saving module saves the code stream / image file to the corresponding storage location of the terminal.

[0115] Decoding end: When the local needs to obtain an image from the cloud, the cloud loading module loads the code stream / image file from the corresponding storage location of the terminal and inputs it into the entropy decoding network. The entropy estimation unit is responsible for calculating the decoding probability of each data in the image feature. The entropy decoding network performs lossless decoding on the code stream / image file according to the probability corresponding to each data to obtain the reconstructed image features. The AI ​​decoding unit performs a reverse transformation on the image features output by the entropy decoding and parses it into a reconstructed image. Corresponding to the AI ​​encoding unit, it usually includes a nonlinear transformation unit and has nonlinear characteristics. The cloud then performs JPEG encoding on the image to obtain a compressed image and sends the compressed image to the local. The local JPEG decoding of the compressed image obtains the image and displays it.

[0116] Optionally, the entropy estimation unit may adopt an NPU, the JPEG encoding module, the JPEG decoding module, the entropy encoding network, the entropy decoding network, the saving module and the loading module may adopt a CPU, and the AI ​​encoding unit and the AI ​​decoding unit may adopt a graphics processing unit (GPU).

[0117] Figure 6 A schematic diagram of a streaming media system workflow applicable to the embodiment of the present application is shown in FIG. Figure 6As shown, the streaming media system includes a content creation module to generate the required content data, such as video or audio. The streaming media system also includes a video encoding module, which encodes the generated content through an encoder. The streaming media system also includes a video stream transmission module, which transmits the encoded video in the form of a code stream. Optionally, the format of the video stream can be converted into a code stream format of a common transmission protocol of an open Internet video service (over the top TV, OTT) device, for example, the protocol includes but is not limited to real-time streaming protocol (real time streaming protocol, RTSP), hypertext transfer protocol (hyper text transfer protocol, HTTP), HTTP live streaming protocol (HTTP Live streaming protocol, HLS), MPEGHTTP dynamic adaptive streaming media (MPEG Dynamic Adaptive Streaming over HTTP, MPEG-DASH), real-time transport protocol (Real-time Transport protocol, RTP), real-time messaging protocol (Real Time Messaging Protocol, RTMP) or any combination thereof. Optionally, video stream storage can be performed, and the original format of the video stream and / or the converted multiple code stream formats are stored for easy use. Furthermore, the streaming media system also includes a video stream encapsulation module for encapsulating the video stream to generate an encapsulated video stream, which can be called a video streaming media package. Exemplarily, the video streaming media package can be generated according to the transcoded video stream or the stored video stream. Furthermore, the streaming media system also includes a content distribution network (CDN), which is used to distribute the video streaming media package to multiple OTT devices, such as mobile phones, computers, tablets, home projectors, etc.

[0118] It should be noted that the video encoding, video stream transmission, video stream transcoding, video stream storage, video streaming package generation and content distribution network can all be implemented on a cloud platform (or understood as a cloud server). In a possible implementation, the video encoding module may include Figure 1 The video stream transcoding module may include: Figure 1 The OTT device may include an entropy estimation network, an entropy decoding network, and an integration of a decoding network; Figure 1 Display of the reconstructed image in .

[0119] Figure 7 Schematic diagram of the architecture of a streaming media system according to an embodiment of the present application. Figure 7 As shown, the streaming media system architecture includes: client devices, content distribution networks, and cloud servers.

[0120] The user on the client device sends a play or playback request to the cloud platform (ie, the cloud server). Optionally, the content of the request sent may be the title of the movie or television program being played.

[0121] The cloud platform makes a decision, replies to the client, and sends the address of the content requested by the client on the CDN to the client. Optionally, the content sent to the client can be a URL link (uniform resource locator). Specifically, the playback application service in the cloud platform checks user authorization and permission, and then considers the characteristics of each client and the current network conditions to determine which specific files are needed to process the playback request. It should be noted that the CDN regularly reports the operating status, learned routes, and available content (files) to the cache control service in the cloud platform.

[0122] The client then requests the CDN to play the content based on the address, and the CDN provides the content to the client, ultimately completing the client's request.

[0123] Based on the above application scenarios, an embodiment of the present application provides an entropy estimation method for image compression to improve compression performance.

[0124] Figure 8 800 is a flowchart of a process 800 of an entropy estimation method for image compression according to an embodiment of the present application. The process 800 may be performed by an encoding end or a decoding end in the above-mentioned embodiment. The process 800 is described as a series of steps or operations. It should be understood that the process 800 may be performed in various orders and / or may occur simultaneously, not limited to Figure 8 The execution order shown. Process 800 includes the following steps:

[0125] Step 801: Obtain a first feature matrix to be processed.

[0126] According to the above description, the entropy estimation network can obtain the estimated probability value of each feature element according to the image feature estimation, so as to be used for the processing of the entropy coding network and the entropy decoding network. Based on this, the first feature matrix can be the feature matrix obtained by quantizing the original image after being processed by the coding network. For example, the first feature matrix can be represented as input[C in ,h in , w in ], C in Indicates the number of channels of the first feature matrix, h in represents the height of the first characteristic matrix, w in Represents the width of the first characteristic matrix, that is, [C in ,h in , win ] represents the shape of the first feature matrix.

[0127] Step 802: Input the first feature matrix into an entropy estimation network to obtain a second feature matrix.

[0128] The second characteristic matrix can be expressed as output[C out ,h out , w out ], C out Indicates the number of channels of the second feature matrix, h out represents the height of the second characteristic matrix, w out Represents the width of the second characteristic matrix, that is, [C out ,h out , w out ] represents the shape of the second feature matrix.

[0129] In the embodiment of the present application, the second characteristic matrix and the first characteristic matrix satisfy:

[0130] C out =C in

[0131] h out =s·h in

[0132] w out =s·w in

[0133] Wherein, s is an integer greater than 1, and is used to represent the stride of the sliding window. Optionally, s=4.

[0134] The entropy estimation network sequentially includes a third convolution layer, a first activation function, a first convolution layer, a point-by-point operation, a second activation function, and an s-fold upsampling process, wherein the first convolution layer includes two convolution processes, and the input of the point-by-point operation is the output of each of the two convolution processes.

[0135] The input feature matrix of any of the above convolutional layers is converted into an output feature matrix with unchanged height and width through convolution operation. The output feature matrix can be considered as a local response of the input feature matrix. The convolution operation can be expressed as the following formula:

[0136] Conv(x)=∑w*x+b

[0137] Among them, w is the weight of the convolutional layer, and b is the bias parameter of the convolutional layer, both of which are unrestricted learnable parameters.

[0138] Any of the above activation operations is a point-by-point operation, that is, an independent mapping operation is performed on each element of the input feature matrix. The activation operation can be an identity operation (Identity), a rectified linear unit (Rectified Linear Unit, ReLU), a rectified linear unit (LeakyReLU), a parametric rectified linear unit (PReLU), Sigmoid, Tanh, etc., which can provide linear or nonlinear transformation characteristics. In the embodiment of the present application, the first activation function and the second activation function can use the same operation or different operations, and there is no specific limitation on this. Among them,

[0139] Identity refers to taking the original value of the input feature value. It can be expressed by the following formula:

[0140] y=x

[0141] ReLU is a piecewise linear mapping method. For the input eigenvalue, the eigenvalue less than 0 is output as 0, and the eigenvalue greater than or equal to 0 is output as constant. It can be expressed by the following formula:

[0142]

[0143] LeakyReLU is a piecewise linear mapping method. Based on ReLU, it scales the input feature values ​​less than 0 with a preset weight, which is usually 0.01. It can be expressed as follows:

[0144]

[0145] Here, a represents a preset value, which is usually set to 0.01.

[0146] The definition of PReLU is similar to LeakyReLU:

[0147]

[0148] The difference from LeakyReLU is that a is a trainable parameter that can be obtained through training.

[0149] Sigmoid can be expressed as the following operation:

[0150]

[0151] Tanh can be expressed as the following operation:

[0152]

[0153] It should be noted that the activation function may also adopt other deformation modes of ReLU, and the embodiments of the present application do not specifically limit this.

[0154] The above-mentioned point-by-point operation may include point-by-point multiplication, point-by-point addition, point-by-point subtraction or point-by-point division. The input of the point-by-point operation is the output feature matrix of the two convolution processes of the first convolutional layer. The two output feature matrices have the same size, and the point-by-point operation multiplies, adds, subtracts or divides the element values ​​at corresponding positions. The combination of convolution operation and point-by-point operation realizes the local attention mechanism, which uses the local information of the input feature matrix to modulate the response of each position in the feature matrix.

[0155] Fig. 9 Schematic diagram of the structure of the entropy estimation network of the embodiment of the present application, such as Fig. 9 As shown, the input of the entropy estimation network is the first feature matrix.

[0156] The first feature matrix is ​​input into the third convolutional layer, and the output of the third convolutional layer is the third feature matrix. The third feature matrix has the same height and width as the first feature matrix (also called the same size), but the number of channels of the two is different.

[0157] The third feature matrix inputs the first activation function and outputs a fourth feature matrix, and the fourth feature matrix has the same size and number of channels as the third feature matrix.

[0158] The fourth feature matrix is ​​respectively input into the two convolutions (convolution 1 and convolution 2) of the first convolutional layer, and outputs the fifth feature matrix and the sixth feature matrix respectively. The fifth feature matrix and the sixth feature matrix have the same size and number of channels, and both have the same size as the fourth feature matrix but different number of channels.

[0159] The fifth characteristic matrix and the sixth characteristic matrix are input point by point and the seventh characteristic matrix is ​​output. The seventh characteristic matrix has the same size and number of channels as the fifth / sixth characteristic matrices.

[0160] The seventh characteristic matrix inputs the second activation function and outputs an eighth characteristic matrix, and the eighth characteristic matrix has the same size and number of channels as the seventh characteristic matrix.

[0161] The eighth characteristic matrix is ​​upsampled s times to output a second characteristic matrix, the height of the second characteristic matrix is ​​s times the height of the eighth characteristic matrix, the width of the second characteristic matrix is ​​s times the width of the eighth characteristic matrix, and the number of channels of the second characteristic matrix is ​​the same as that of the first characteristic matrix.

[0162] As an example, the entropy estimation network of the embodiment of the present application can use a hyper-scale decoder (HSD). All operations in HSD are integers, and all accumulators involved in the calculation process are 32-bit integers. The model parameters are quantized to 8-bit integers. This ensures the cross-platform bit consistency of the neural network module. HSD uses customized quantized convolution. For each quantized convolution in the process, a set of clipping values ​​{d k} and the scaling parameter {p k}, 1≤k≤4. All clipping values ​​in quantized convolution are d k =2 15 -1. Where the scaling shift {p k}Acquired through training.

[0163] It is worth noting that the size of the weight in the quantized model does not exceed 2 15 The combination of -1, shift and clip values ​​ensures that the registers for the quantized convolution fit within 32 bits.

[0164] The HSD structure is as follows. First, it receives an input three-dimensional matrix of size [C, H, W], which is converted into a matrix of [C1, H, W] through a 1*1 quantized convolution and an activation function. The matrix is ​​also converted into two output matrices A and B of [C2, H, W] through 1*1 quantized convolution and 3*3 quantized grouped convolution. Then, matrices A and B are converted into matrices C [C2, H, W] through point-by-point operations, and then converted into a matrix of [16C, H, W] through activation functions and 1*1 quantized convolutions, and then the final output is formed through the PixelShuffle operation, with a size of [C, 4H, 4W].

[0165] In the embodiment of the present application, the s-fold upsampling process can be implemented in multiple ways:

[0166] (1) The s-fold upsampling process sequentially includes a second convolutional layer and an s-fold pixel shuffling layer.

[0167] The eighth feature matrix is ​​input into the second convolutional layer and outputs the ninth feature matrix. The ninth feature matrix has the same size as the eighth feature matrix, but the number of channels of the ninth feature matrix is ​​n times the number of channels of the first feature matrix. The ninth feature matrix is ​​input into the pixel shuffle layer and outputs the second feature matrix. The height of the second feature matrix is ​​s times the height of the ninth feature matrix, the width of the second feature matrix is ​​s times the width of the ninth feature matrix, and the number of channels of the second feature matrix is ​​1 / n of the number of channels of the ninth feature matrix, but the same as the number of channels of the first feature matrix. Optionally, n=s 2 .

[0168] The pixel shuffle layer, also called sub-pixel convolution, is represented by PixelShuffle(s), where s>1. The pixel shuffle layer can convert the input feature matrix (whose shape is [C 4 ,h 4 , w 4 ]) to obtain the output feature matrix (whose shape is [C out ,h out , w out ]), where h out =s·h 4 , w out =s·w 4 , C out =C 4 / s 2 .

[0169] The pixel shuffling layer can be expressed as:

[0170] output[c,i,j]=input[s 2 c+si′+j′, i / s+i′, j / s+j′]

[0171] Where c is from 0 to C out -1, i is from 0 to h out -1, j is from 0 to w out -1, i' is from 0 to s-1, j' is from 0 to s-1.

[0172] In the last layer of the entropy estimation network, a pixel shuffling layer is used for upsampling, and only memory operations can be performed. Therefore, it is highly adaptable to different hardware (for example, it can be converted into a convolution operation on an AI accelerator as appropriate), and can even be combined with subsequent steps to merge memory access, thereby improving the overall encoding / decoding efficiency.

[0173] (2) The s-fold upsampling process sequentially includes a second convolutional layer and an s-fold upsampling deconvolution layer.

[0174] The eighth feature matrix is ​​input into the second convolution layer and outputs the ninth feature matrix. The ninth feature matrix has the same size as the eighth feature matrix, but the number of channels of the ninth feature matrix is ​​n times the number of channels of the first feature matrix. The ninth feature matrix is ​​input into the deconvolution layer with s times upsampling and outputs the second feature matrix. The height of the second feature matrix is ​​s times the height of the ninth feature matrix, the width of the second feature matrix is ​​s times the width of the ninth feature matrix, and the number of channels of the second feature matrix is ​​1 / n of the number of channels of the ninth feature matrix, but the same as the number of channels of the first feature matrix. Optionally, n=s 2 .

[0175] (3) The s-fold upsampling process includes a deconvolution layer with s-fold upsampling.

[0176] The eighth feature matrix is ​​input to the deconvolution layer with s times upsampling and outputs the second feature matrix, the height of the second feature matrix is ​​s times the height of the eighth feature matrix, the width of the second feature matrix is ​​s times the width of the eighth feature matrix, the number of channels of the second feature matrix is ​​1 / n of the number of channels of the eighth feature matrix, and the same as the number of channels of the first feature matrix. Optionally, n=s 2 .

[0177] (4) The s-fold upsampling process includes the second convolutional layer and two s / 2-fold pixel shuffling layers.

[0178] The eighth feature matrix is ​​input into the second convolutional layer to output the ninth feature matrix. The ninth feature matrix has the same size as the eighth feature matrix, but the number of channels of the ninth feature matrix is ​​n times the number of channels of the first feature matrix. The ninth feature matrix is ​​input into the first s / 2 times pixel shuffle layer to output the tenth feature matrix. The height of the tenth feature matrix is ​​s / 2 times the height of the ninth feature matrix, the width of the tenth feature matrix is ​​s / 2 times the width of the ninth feature matrix, and the number of channels of the tenth feature matrix is ​​n / 2 of the number of channels of the ninth feature matrix. The tenth feature matrix is ​​input into the second s / 2 times pixel shuffle layer to output the eleventh feature matrix. The height of the eleventh feature matrix is ​​s / 2 times the height of the tenth feature matrix, the width of the eleventh feature matrix is ​​s / 2 times the width of the tenth feature matrix, and the number of channels of the eleventh feature matrix is ​​n / 2 of the number of channels of the tenth feature matrix, which is the same as the number of channels of the first feature matrix. Optionally, n=s 2 .

[0179] It should be noted that in addition to the above-mentioned implementation methods, the s-fold upsampling processing can also be implemented in other ways, as long as the height of the second characteristic matrix can be s times the height of the eighth characteristic matrix, the width of the second characteristic matrix can be s times the width of the eighth characteristic matrix, and the second characteristic matrix has the same number of channels as the first characteristic matrix, all of which can be used as implementation methods of the s-fold upsampling processing, and the embodiments of the present application do not make specific limitations on this.

[0180] In one possible implementation, one of the two convolutions in the first convolutional layer can use a two-dimensional group convolutional layer (also called a group convolutional layer). The group convolution is represented by CONV(K ver *K hor , C 1 , C 2 , G, t↓). The size of the input feature matrix of the convolutional layer is [C 1 ,h 1 , w 1 ], the size of the output feature matrix is ​​[C 2 ,h 2 , w 2 ], where h 1 =t·h2 , w 1 =t·w 2 , G represents the number of groups, which controls the connection between the input feature matrix and the output feature matrix. 1 and C 2 is divisible by G, and t represents the step size of the sliding window. In the absence of a step size parameter, no spatial resolution change is "performed".

[0181] The group convolution can be expressed as:

[0182]

[0183] Where 0≤i<h 2 , 0≤j<w 2 ;

[0184] ★ is the kernel size K ver *K hor 2D cross-correlation operator;

[0185]

[0186] The characteristic matrix weight[C 1 / G,C 2 , K ver , K hor ] contains learnable weights, feature matrix bias[C 2 ] contains a learnable bias parameter, both of which are unrestricted learnable parameters.

[0187] Comparison of convolution and group convolution:

[0188] Convolution can be expressed as The group convolution can be expressed as It can be seen that using group convolution in the entropy estimation network can reduce the amount of calculation while expanding the receptive field.

[0189] In the embodiment of the present application, in the entropy estimation network, the convolution operation and the point-by-point operation are combined to realize the local attention mechanism, and the local information of the input feature matrix is ​​used to modulate the response of each position in the feature matrix to improve the compression performance; in addition, the use of group convolution can reduce the amount of calculation while expanding the receptive field.

[0190] In an embodiment of the present application, the entropy estimation network may include a variance prediction network and a mean prediction network, and the input feature matrices of the two networks (i.e., the first feature matrix may be the same) are different, and the output feature matrices are different. The output feature matrix of the variance prediction network can be called a variance matrix, and the output feature matrix of the mean prediction network can be called a mean matrix. The structures of the two networks are similar, except that, since the variance is a positive number, but the mean is not limited to this, any convolutional layer in the variance prediction network uses a quantized convolutional layer, and an absolute value operation is provided inside or outside the convolutional layer to ensure the positive characteristic of the variance. In addition, all operations in the quantized convolutional layer are integers, and the accumulators involved in all calculation processes are integers, and the model parameters are quantized to integers, which ensures the cross-platform bit consistency of the network module.

[0191] Quantized convolution can be expressed as qCONV(K ver *K hor , C 5 , C 6 , G, q↓, d, p). The convolution process receives the shape [C 5 ,h 5 , w 5 ] feature matrix, and the output shape is [C 6 ,h 6 , w 6 ], where h 6 =q·h 5 , w 6 =q·w 5 , q represents the convolution step size, the default value of the step size is 1, and the corresponding outputs have the same spatial resolution; d is a non-negative integer, which defines the maximum value of the elements of the input feature matrix after clipping; p[c 7 ] contains the scaling and translation coefficients for each channel of the output feature matrix.

[0192] The operations in quantized convolution can be described as three steps:

[0193] temp[c 5 ,i,j]=clip(-d,d-1,input[c 5 , i, j])

[0194] Where i = 0, ..., h 5 -1; j = 0, ..., w 5 -1;c 5 =0, ..., C 5 -1;

[0195]

[0196] Where "★" is the kernel size K ver*K hor The two-dimensional cross-correlation operator (convolution operation) of .

[0197] out[c 6 , i, j] = (R[c 6 , i, j])>>p[c 6 ]

[0198] Where i = 0, ..., h 6 -1; j = 0, ..., w 6 -1;c 6 =0, ..., C 6 -1.

[0199] The shape is [c 5 , c 6 , K ver , K hor ] is a learnable integer weight matrix with shape [c 6 ] is a learnable integer bias.

[0200] Optionally, the input of the entropy estimation network is the first feature matrix, starting with a 1*1 convolution (the first convolution layer, whose convolution kernel size is 1*1), followed by an activation function a (the first activation function, for example, ReLU, LeakyReLU, etc.), followed by two convolutions (the second convolution layer), which are 1*1 convolution and 3*3 group convolution (group convolution), and then the outputs of the two convolutions are point-by-point operated, followed by an activation function b (the second activation function, for example, ReLU, LeakyReLU, etc.), and then a 1*1 convolution is used to increase the number of channels to 16 times the number of channels of the first feature matrix, and finally a 4-fold sampling PixelShuffle (4) is used to reduce the number of channels by 16 times, and the height and width are upsampled by 4 times respectively to obtain the final output, which is the second feature matrix.

[0201] Fig.10a This is a schematic diagram of the structure of the variance prediction network for entropy estimation, such as Fig.10a As shown, the first characteristic matrix z[C, h in , w in] Input qConv(1*1, C, C1) (1*1 convolution, C represents the number of input channels, C1 represents the number of output channels), followed by activation function a, followed by two convolutions, namely qConv(1*1, C1, C2) (1*1 convolution, C1 represents the number of input channels, C2 represents the number of output channels) and qConv(3*3, C1, C2, G=g) (3*3 group convolution, C1 represents the number of input channels, C2 represents the number of output channels), and then these two convolutions The outputs of each are subjected to point-by-point operation a, followed by activation function b, and then qConv(1*1, C2, 16C) (1*1 convolution, C2 represents the number of input channels, 16C represents the number of output channels) is used to increase the number of channels to 16 times the number of channels of the first feature matrix. Finally, PixelShuffle(4) is used to reduce the number of channels by 16 times, and the height and width are upsampled by 4 times respectively. Then, the absolute value operation ABS() is performed in sequence to obtain the second feature matrix, that is, the variance matrix δ[C, 4h in , 4w in ].

[0202] Fig.10b This is a schematic diagram of the structure of the mean prediction network for entropy estimation, such as Fig.10b As shown, the first characteristic matrix z[C, h in , w in ] Input Conv(1*1, C, C3) (1*1 convolution, C represents the number of input channels, C3 represents the number of output channels), followed by activation function c, followed by two convolutions, namely Conv(1*1, C3, C4) (1*1 convolution, C3 represents the number of input channels, C4 represents the number of output channels) and Conv(3*3, C3, C4, G=g) (3*3 group convolution, C3 represents the number of input channels, C4 represents the number of output channels), then perform point-by-point operation b on the outputs of these two convolutions, followed by activation function d, and then use Conv(1*1, C4, 16C) (1*1 convolution, C4 represents the number of input channels, 16C represents the number of output channels) to increase the number of channels to 16 times the number of channels of the first feature matrix, and finally use PixelShuffle(4) to reduce the number of channels by 16 times, and upsample the height and width by 4 times respectively to obtain the second feature matrix, that is, the mean matrix p[C, 4h in , 4w in ].

[0203] Optional, above Fig.10a and Fig.10bThere are multiple options for the activation function, number of channels, etc. in the illustrated embodiment. For example, activation function a-activation function d can be specified as any activation function, including but not limited to ReLU, LeakyReLU, Tanh, etc. For another example, the number of intermediate channels C1, C2, C3, and C4 of the entropy estimation network can also be adjusted at will. In addition, depending on the inference speed on the hardware, the 4x upsampling process of the last layer can also be replaced by other operations. Therefore, the embodiments of the present application do not specifically limit the aforementioned operations.

[0204] The technical solution of the present application is described below through several specific embodiments.

[0205] Embodiment 1

[0206] Fig.11a This is a schematic diagram of the structure of the variance prediction network for entropy estimation, such as Fig.11a As shown, in Fig.10a Based on the illustrated embodiment, activation function a and activation function b adopt ReLU, and element-wise operation a adopts element-wise multiplication (EMUL).

[0207] That is, the first characteristic matrix z[C, h in , w in ] Input qConv(1*1, C, C1) (1*1 convolution, C represents the number of input channels, C1 represents the number of output channels), followed by ReLU, followed by two convolutions, namely qConv(1*1, C1, C2) (1*1 convolution, C1 represents the number of input channels, C2 represents the number of output channels) and qConv(3*3, C1, C2, G=g) (3*3 group convolution, C1 represents the number of input channels, C2 represents the number of output channels), then perform EMUL on the outputs of these two convolutions, followed by ReLU, and then use qConv(1*1, C2, 16C) (1*1 convolution, C2 represents the number of input channels, 16C represents the number of output channels) to increase the number of channels to 16 times the number of channels of the first feature matrix, and finally use PixelShuffle(4) to reduce the number of channels by 16 times, and upsample the height and width by 4 times respectively, and then obtain the second feature matrix through absolute value operations ABS(), that is, the variance matrix δ[C, 4h in , 4w in ].

[0208] Fig.11b This is a schematic diagram of the structure of the mean prediction network for entropy estimation, as shown in Fig.11b As shown, in Fig.10b Based on the illustrated embodiment, activation function c and activation function d adopt LeakyReLU, and element-by-element operation b adopts EMUL.

[0209] That is, the first characteristic matrix z[C, h in , w in ] Input Conv(1*1, C, C3) (1*1 convolution, C represents the number of input channels, C3 represents the number of output channels), followed by LeakyReLU, followed by two convolutions, namely Conv(1*1, C3, C4) (1*1 convolution, C3 represents the number of input channels, C4 represents the number of output channels) and Conv(3*3, C3, C4, G=g) (3*3 group convolution, C3 represents the number of input channels, C4 represents the number of output channels), then perform EMUL on the outputs of these two convolutions, followed by LeakyReLU, and then use Conv(1*1, C4, 16C) (1*1 convolution, C4 represents the number of input channels, 16C represents the number of output channels) to increase the number of channels to 16 times the number of channels of the first feature matrix, and finally use PixelShuffle(4) to reduce the number of channels by 16 times, and upsample the height and width by 4 times respectively to obtain the second feature matrix, that is, the mean matrix p[C, 4h in , 4w in ].

[0210] Compared with the division operation, the present embodiment adopts the multiplication operation. On the one hand, it is not necessary to accommodate the restriction that the square root of the variance calculation must be a positive number. On the other hand, it is not necessary to accommodate the restriction that the denominator cannot be 0 in the division operation.

[0211] Embodiment 2

[0212] Fig.12a This is a schematic diagram of the structure of the variance prediction network for entropy estimation, such as Fig.12a As shown, in Fig.10a Based on the illustrated embodiment, the 4x upsampling process is replaced by a 4x upsampling deconvolution layer by PixelShuffle(4).

[0213] That is, the first characteristic matrix z[C, h in , w in] Input qConv(1*1, C, C1) (1*1 convolution, C represents the number of input channels, C1 represents the number of output channels), followed by activation function a, followed by two convolutions, namely qConv(1*1, C1, C2) (1*1 convolution, C1 represents the number of input channels, C2 represents the number of output channels) and qConv(3*3, C1, C2, G=g) (3*3 group convolution, C1 represents the number of input channels, C2 represents the number of output channels), then perform point-by-point operation a on the outputs of these two convolutions, followed by activation function b, and then use qConv(1*1, C2, 16C) (1*1 convolution, C2 represents the number of input channels, 16C represents the number of output channels) to increase the number of channels to 16 times the number of channels of the first feature matrix, and finally use qConv -1 (4*4, 16C, C, stride = 4) The number of channels is reduced by 16 times, and the height and width are upsampled by 4 times respectively. Then, the absolute value operation ABS() is performed in sequence to obtain the second feature matrix, that is, the variance matrix δ[C, 4h in , 4w in ].

[0214] Figure 12b This is a schematic diagram of the structure of the mean prediction network for entropy estimation, such as Figure 12b As shown, in Fig.10b Based on the illustrated embodiment, the 4x upsampling process is replaced by a 4x upsampling deconvolution layer by PixelShuffle(4).

[0215] That is, the first characteristic matrix z[C, h in , w in ] Input Conv(1*1, C, C3) (1*1 convolution, C represents the number of input channels, C3 represents the number of output channels), followed by activation function c, followed by two convolutions, namely Conv(1*1, C3, C4) (1*1 convolution, C3 represents the number of input channels, C4 represents the number of output channels) and Conv(3*3, C3, C4, G=g) (3*3group convolution, C3 represents the number of input channels, C4 represents the number of output channels), then perform point-by-point operation b on the outputs of these two convolutions, followed by activation function d, and then use Conv(1*1, C4, 16C) (1*1 convolution, C4 represents the number of input channels, 16C represents the number of output channels) to increase the number of channels to 16 times the number of channels of the first feature matrix, and finally use Conv -1 (4*4, 16C, C, stride = 4) The number of channels is reduced by 16 times, and the height and width are upsampled by 4 times respectively to obtain the second feature matrix, that is, the mean matrix p[C, 4h in , 4w in ].

[0216] In this embodiment, the deconvolution layer with 4 times upsampling is placed at the end, which can not only meet the upsampling requirement, but also avoid the increase of calculation amount because there is no subsequent processing.

[0217] Embodiment 3

[0218] Fig.13a This is a schematic diagram of the structure of the variance prediction network for entropy estimation, such as Fig.13a As shown, in Fig.10a Based on the illustrated embodiment, the 4x upsampling process is performed by the last convolution layer and PixelShuffle(4) is replaced by a 4x upsampling deconvolution layer.

[0219] That is, the first characteristic matrix z[C, h in , w in ] Input qConv(1*1, C, C1) (1*1 convolution, C represents the number of input channels, C1 represents the number of output channels), followed by activation function a, followed by two convolutions, namely qConv(1*1, C1, C2) (1*1 convolution, C1 represents the number of input channels, C2 represents the number of output channels) and qConv(3*3, C1, C2, G=g) (3*3 group convolution, C1 represents the number of input channels, C2 represents the number of output channels), then perform point-by-point operation a on the outputs of each of the two convolutions, followed by activation function b, and finally use qConv -1 (4*4, C2, C, stride = 4) The number of channels is adjusted to the number of channels of the first feature matrix, and the height and width are upsampled by 4 times respectively, and then the absolute value operation ABS() is performed in sequence to obtain the second feature matrix, that is, the variance matrix δ[C, 4h in , 4w in ].

[0220] Fig.13b This is a schematic diagram of the structure of the mean prediction network for entropy estimation, as shown in Fig.13b As shown, in Fig.10b Based on the illustrated embodiment, the 4x upsampling process is performed by the last convolution layer and PixelShuffle(4) is replaced by a 4x upsampling deconvolution layer.

[0221] That is, the first characteristic matrix z[C, h in , w in] Input Conv(1*1, C, C3) (1*1 convolution, C represents the number of input channels, C3 represents the number of output channels), followed by activation function c, followed by two convolutions, namely Conv(1*1, C3, C4) (1*1 convolution, C3 represents the number of input channels, C4 represents the number of output channels) and Conv(3*3, C3, C4, G=g) (3*3 group convolution, C3 represents the number of input channels, C4 represents the number of output channels), then perform point-by-point operations b on the outputs of the two convolutions, followed by activation function d, and finally use Conv -1 (4*4, C4, C, stride=4) The number of channels is adjusted to the number of channels of the first feature matrix, and the height and width are upsampled by 4 times respectively to obtain the second feature matrix, that is, the mean matrix p[C, 4h in , 4w in ].

[0222] Fig.14 Schematic diagram of an entropy estimation device 1400 for image compression according to an embodiment of the present application, as shown in FIG. Fig.14 As shown, the device 1400 of this embodiment can be applied to an encoding end or a decoding end. The device 1400 may include: an acquisition module 1401 and a processing module 1402.

[0223] The acquisition module 1401 is used to acquire a first feature matrix to be processed. The shape of the first feature matrix is ​​[C in ,h in , w in ], where C in represents the number of channels of the first feature matrix, h in represents the height of the first feature matrix, w in represents the width of the first feature matrix; processing module 1402, used to input the first feature matrix into the entropy estimation network to obtain a second feature matrix, the shape of the second feature matrix is ​​[C out ,h out , w out ], where C out represents the number of channels of the second feature matrix, h out represents the height of the second characteristic matrix, w out represents the width of the second characteristic matrix; the second characteristic matrix and the first characteristic matrix satisfy the following conditions: C out =C in ,h out =s·h in , w out =s·w in, s is an integer greater than 1; the entropy estimation network includes a first convolutional layer and a point-by-point operation, the first convolutional layer includes two convolution processes, one of the two convolution processes is a two-dimensional grouped convolution, and the input of the point-by-point operation is the output of each of the two convolution processes.

[0224] In a possible implementation, the input feature matrix and the output feature matrix of the two-dimensional group convolution satisfy the following conditions: 1 =t·h 2 , w 1 =t·w 2 , C 1 and C 2 Divisible by G; t is an integer greater than 1; C 1 represents the number of channels of the input feature matrix, h 1 represents the height of the input feature matrix, w 1 represents the width of the input feature matrix; C 2 represents the number of channels of the output feature matrix, h 2 represents the height of the output feature matrix, w 2 represents the width of the output feature matrix; G represents the number of groups, which is used to control the connection between the input feature matrix and the output feature matrix.

[0225] In a possible implementation, the entropy estimation network further includes an s-fold upsampling process.

[0226] In a possible implementation, the s-fold upsampling process includes a second convolution layer and a pixel shuffling layer in sequence, and the third feature matrix output by the second convolution layer and the first feature matrix satisfy the following conditions: C 3 =nC in ,h 3 =h in , w 3 =w in , where n is an integer greater than 1, C 3 represents the number of channels of the third characteristic matrix, h 3 represents the height of the third characteristic matrix, w 3 represents the width of the third feature matrix, the pixel shuffling layer is used to upsample the height and width of the third feature matrix respectively, and the feature matrix output by the pixel shuffling layer is the second feature matrix.

[0227] In a possible implementation, the s-fold upsampling process sequentially includes a second convolution layer and an s-fold upsampling deconvolution layer, and the third feature matrix output by the second convolution layer and the first feature matrix satisfy the following conditions: C 3 =nC in ,h 3=h in , w 3 =w in , where n is an integer greater than 1, C 3 represents the number of channels of the third characteristic matrix, h 3 represents the height of the third characteristic matrix, w 3 represents the width of the third feature matrix, the deconvolution layer is used to upsample the height and width of the third feature matrix respectively, and the feature matrix output by the deconvolution layer is the second feature matrix.

[0228] In a possible implementation, the s-fold upsampling process includes an s-fold upsampling deconvolution layer, the deconvolution layer is used to upsample the height and width of the input feature matrix respectively, and the feature matrix output by the deconvolution layer is the second feature matrix.

[0229] In a possible implementation, the entropy estimation network also includes a first activation function and a second activation function; the first activation function and the second activation function respectively include an identity operation Identity, a rectified linear unit ReLU processing, a leaky rectified linear unit LeakyReLU processing, or a parameter rectified linear unit PReLU processing.

[0230] In a possible implementation, the point-by-point operation includes point-by-point multiplication, point-by-point addition, point-by-point subtraction, or point-by-point division.

[0231] In a possible implementation, the entropy estimation network further includes a third convolutional layer, and a feature matrix input to the third convolutional layer is the first feature matrix.

[0232] The device of this embodiment can be used to perform Figure 8 The technical solution of the method embodiment shown has similar implementation principles and technical effects, which will not be repeated here.

[0233] Fig.15 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application, such as Fig.15 As shown, the electronic device can be used as the encoding end (or encoder) or decoding end (or decoder) in the above embodiments. The electronic device includes at least one processor 151, at least one memory 152, at least one transceiver 153 (also referred to as a communication interface), at least one network interface 154 and one or more antennas 155. The processor 151, the memory 152, the transceiver 153 and the network interface 154 are connected, for example, via a bus. In an embodiment of the present application, the connection may include various interfaces, transmission lines or buses, etc., which are not limited. The antenna 155 is connected to the transceiver 153. The network interface 154 is used to connect the electronic device to other communication devices through a communication link.

[0234] The processor 151 is mainly used to process data, control the entire electronic device, execute software programs, and process the data of the software programs, for example, to support the electronic device to perform the actions described in the above embodiments. The electronic device may include a baseband processor and a central processing unit, the baseband processor is mainly used to process the communication protocol and communication data, and the central processing unit is mainly used to control the entire electronic device, execute software programs, and process the data of the software programs. Fig.15 The processor 151 in the embodiment can integrate the functions of the baseband processor and the central processing unit. It can be understood by those skilled in the art that the baseband processor and the central processing unit can also be independent processors, which are interconnected through technologies such as buses. It can be understood by those skilled in the art that an electronic device can include multiple baseband processors to adapt to different network formats, an electronic device can include multiple central processing units to enhance its processing capabilities, and various components of the electronic device can be connected through various buses. The baseband processor can also be described as a baseband processing circuit or a baseband processing chip. The central processing unit can also be described as a central processing circuit or a central processing chip. The function of processing the communication protocol and the communication data can be built into the processor, or it can be stored in the memory in the form of a software program, and the processor executes the software program to realize the baseband processing function.

[0235] The memory 152 is mainly used to store software programs and data. The memory 152 can be independent and connected to the processor 151. Optionally, the memory 152 can be integrated with the processor 151, for example, integrated into a chip. Among them, the memory 152 can store program codes for executing the technical solutions of the embodiments of the present application, and the execution is controlled by the processor 151. The various types of computer program codes executed can also be regarded as drivers of the processor 151.

[0236] Fig.15 Only one memory and one processor are shown. In an actual electronic device, there may be multiple processors and multiple memories. The memory may also be referred to as a storage medium or a storage device, etc. The memory may be a storage element on the same chip as the processor, i.e., an on-chip storage element, or an independent storage element, which is not limited in the embodiments of the present application.

[0237] The transceiver 153 can be used to support the reception or transmission of radio frequency signals between electronic devices, and the transceiver 153 can be connected to the antenna 155. The transceiver 153 includes a transmitter Tx and a receiver Rx. Specifically, one or more antennas 155 can receive radio frequency signals, and the receiver Rx of the transceiver 153 is used to receive the radio frequency signal from the antenna, and convert the radio frequency signal into a digital baseband signal or a digital intermediate frequency signal, and provide the digital baseband signal or the digital intermediate frequency signal to the processor 151, so that the processor 1511 further processes the digital baseband signal or the digital intermediate frequency signal, such as demodulation processing and decoding processing. In addition, the transmitter Tx in the transceiver 153 is also used to receive a modulated digital baseband signal or a digital intermediate frequency signal from the processor 1511, and convert the modulated digital baseband signal or the digital intermediate frequency signal into a radio frequency signal, and send the radio frequency signal through one or more antennas 155. Digital baseband signals and digital intermediate frequency signals can be collectively referred to as digital signals.

[0238] The transceiver 153 may also be referred to as a transceiver unit, a transceiver, a transceiver device, etc. Optionally, a device in the transceiver unit for implementing a receiving function may be regarded as a receiving unit, and a device in the transceiver unit for implementing a sending function may be regarded as a sending unit, that is, the transceiver unit includes a receiving unit and a sending unit, the receiving unit may also be referred to as a receiver, an input port, a receiving circuit, etc., and the sending unit may be referred to as a transmitter, a transmitter, or a transmitting circuit, etc.

[0239] In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The processor can be a general-purpose processor, a digital signal processor (digital signal processor, DSP), an application-specific integrated circuit (application-specific integrated circuit, ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware coding processor to be executed, or the hardware and software modules in the coding processor are combined to be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0240] The memory mentioned in the above embodiments may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0241] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0242] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0243] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0244] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0245] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0246] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to make a computer device (personal computer, server, or network device, etc.) perform all or part of the steps of the method described in each embodiment of the present application. Or the computer software product can be transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instruction can be transmitted from a website site, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD), a read-only memory (ROM), a random access memory (RAM)), etc.

[0247] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiment of the present application, which should be included in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.

Claims

1. A method for estimating entropy for image compression, characterized in that: include: Get the first feature matrix to be processed, the shape of the first feature matrix is ​​[C in ,h in ,w in ], where C in represents the number of channels of the first feature matrix, h in represents the height of the first feature matrix, w in represents the width of the first feature matrix; The first feature matrix is ​​input into the entropy estimation network to obtain a second feature matrix, the shape of the second feature matrix is ​​[C out ,h out ,w out ], where C out represents the number of channels of the second feature matrix, h out represents the height of the second characteristic matrix, w out represents the width of the second characteristic matrix; the second characteristic matrix and the first characteristic matrix satisfy the following conditions: C out =C in ,h out =s·h in , w out =s·w in , s is an integer greater than 1; the entropy estimation network includes a first convolutional layer and a point-by-point operation, the first convolutional layer includes two convolution processes, one of the two convolution processes is a two-dimensional grouped convolution, and the input of the point-by-point operation is the output of each of the two convolution processes.

2. The method according to claim 1, characterized in that The input feature matrix and output feature matrix of the two-dimensional grouped convolution satisfy the following conditions: h1=t·h2, w1=t·w2, C1 and C2 are divisible by G; wherein t is an integer greater than 1; C1 represents the number of channels of the input feature matrix, h1 represents the height of the input feature matrix, and w1 represents the width of the input feature matrix; C2 represents the number of channels of the output feature matrix, h2 represents the height of the output feature matrix, and w2 represents the width of the output feature matrix; G represents the number of groups, which is used to control the connection between the input feature matrix and the output feature matrix.

3. The method according to claim 1 or 2, characterized in that: The entropy estimation network also includes an s-fold upsampling process.

4. The method according to claim 3, characterized in that The s-fold upsampling process includes a second convolution layer and a pixel shuffling layer in sequence, and the third feature matrix output by the second convolution layer and the first feature matrix satisfy the following condition: C3 = nC in , h3=h in , w3=w in , wherein n is an integer greater than 1, C3 represents the number of channels of the third feature matrix, h3 represents the height of the third feature matrix, w3 represents the width of the third feature matrix, the pixel shuffling layer is used to upsample the height and width of the third feature matrix respectively, and the feature matrix output by the pixel shuffling layer is the second feature matrix.

5. The method according to claim 3, characterized in that: The s-fold upsampling process includes a second convolution layer and an s-fold upsampling deconvolution layer in sequence, and the third feature matrix output by the second convolution layer and the first feature matrix satisfy the following condition: C3=nC in , h3=h in , w3=w in , wherein n is an integer greater than 1, C3 represents the number of channels of the third feature matrix, h3 represents the height of the third feature matrix, w3 represents the width of the third feature matrix, the deconvolution layer is used to upsample the height and width of the third feature matrix respectively, and the feature matrix output by the deconvolution layer is the second feature matrix.

6. The method according to claim 3, characterized in that The s-fold upsampling process includes an s-fold upsampling deconvolution layer, the deconvolution layer is used to upsample the height and width of the input feature matrix respectively, and the feature matrix output by the deconvolution layer is the second feature matrix.

7. The method according to any one of claims 1 to 6, characterized in that The entropy estimation network also includes a first activation function and a second activation function; the first activation function and the second activation function respectively include an identity operation Identity, a rectified linear unit ReLU processing, a leaky rectified linear unit LeakyReLU processing, or a parameter rectified linear unit PReLU processing.

8. The method according to any one of claims 1 to 7, characterized in that The point-by-point operation includes point-by-point multiplication, point-by-point addition, point-by-point subtraction or point-by-point division.

9. The method according to any one of claims 1 to 8, characterized in that The entropy estimation network also includes a third convolutional layer, and the feature matrix input to the third convolutional layer is the first feature matrix.

10. The method according to any one of claims 1 to 9, characterized in that s＝4。 11. An entropy estimation device for image compression, characterized in that: include: An acquisition module is used to acquire a first feature matrix to be processed, wherein the shape of the first feature matrix is ​​[C in ,h in ,w in ], where C in represents the number of channels of the first feature matrix, h in represents the height of the first feature matrix, w in represents the width of the first feature matrix; A processing module is used to input the first feature matrix into an entropy estimation network to obtain a second feature matrix, wherein the shape of the second feature matrix is ​​[C out ,h out ,w out ], where C out represents the number of channels of the second feature matrix, h out represents the height of the second characteristic matrix, w out represents the width of the second characteristic matrix; the second characteristic matrix and the first characteristic matrix satisfy the following conditions: C out =C in ,h out =s·h in , w out =s·w in , s is an integer greater than 1; the entropy estimation network includes a first convolutional layer and a point-by-point operation, the first convolutional layer includes two convolution processes, one of the two convolution processes is a two-dimensional grouped convolution, and the input of the point-by-point operation is the output of each of the two convolution processes.

12. The device according to claim 11, characterized in that The input feature matrix and output feature matrix of the two-dimensional grouped convolution satisfy the following conditions: h1=t·h2, w1=t·w2, C1 and C2 are divisible by G; t is an integer greater than 1; C1 represents the number of channels of the input feature matrix, h1 represents the height of the input feature matrix, and w1 represents the width of the input feature matrix; C2 represents the number of channels of the output feature matrix, h2 represents the height of the output feature matrix, and w2 represents the width of the output feature matrix; G represents the number of groups, which is used to control the connection between the input feature matrix and the output feature matrix.

13. The device according to claim 11 or 12, characterized in that The entropy estimation network also includes an s-fold upsampling process.

14. The device according to claim 13, characterized in that The s-fold upsampling process includes a second convolution layer and a pixel shuffling layer in sequence, and the third feature matrix output by the second convolution layer and the first feature matrix satisfy the following condition: C3 = nC in , h3=h in , w3=w in , wherein n is an integer greater than 1, C3 represents the number of channels of the third feature matrix, h3 represents the height of the third feature matrix, w3 represents the width of the third feature matrix, the pixel shuffling layer is used to upsample the height and width of the third feature matrix respectively, and the feature matrix output by the pixel shuffling layer is the second feature matrix.

15. The device according to claim 13, characterized in that The s-fold upsampling process includes a second convolution layer and an s-fold upsampling deconvolution layer in sequence, and the third feature matrix output by the second convolution layer and the first feature matrix satisfy the following condition: C3=nC in , h3=h in , w3=w in , wherein n is an integer greater than 1, C3 represents the number of channels of the third feature matrix, h3 represents the height of the third feature matrix, w3 represents the width of the third feature matrix, the deconvolution layer is used to upsample the height and width of the third feature matrix respectively, and the feature matrix output by the deconvolution layer is the second feature matrix.

16. The device according to claim 13, characterized in that The s-fold upsampling process includes an s-fold upsampling deconvolution layer, the deconvolution layer is used to upsample the height and width of the input feature matrix respectively, and the feature matrix output by the deconvolution layer is the second feature matrix.

17. The device according to any one of claims 11 to 16, characterized in that The entropy estimation network also includes a first activation function and a second activation function; the first activation function and the second activation function respectively include an identity operation Identity, a rectified linear unit ReLU processing, a leaky rectified linear unit LeakyReLU processing, or a parameter rectified linear unit PReLU processing.

18. The device according to any one of claims 11 to 17, characterized in that The point-by-point operation includes point-by-point multiplication, point-by-point addition, point-by-point subtraction or point-by-point division.

19. The device according to any one of claims 11 to 18, characterized in that The entropy estimation network also includes a third convolutional layer, and the feature matrix input to the third convolutional layer is the first feature matrix.

20. The device according to any one of claims 11 to 19, characterized in that s＝4。 21. An encoder, characterized in that: include: one or more processors; A non-transitory computer-readable storage medium, coupled to the processor and storing a program executed by the processor, wherein the program, when executed by the processor, causes the decoder to perform the method of any one of claims 1-10.

22. A decoder, characterized in that: include: one or more processors; A non-transitory computer-readable storage medium, coupled to the processor and storing a program executed by the processor, wherein the program, when executed by the processor, causes the decoder to perform the method of any one of claims 1-10.

23. A computer-readable storage medium, characterized in that: The method comprises instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 10.

Images