Image compression method, device and storage medium based on full integer stream model

Abstract

This paper discloses an image compression method based on a full integer stream model, comprising: training the full integer stream model using a training image dataset; calculating the target image input data using the trained full integer stream model to obtain a representation vector of the target image and a probability distribution of the representation vector; performing entropy coding based on the target image representation vector and its probability distribution to obtain a compressed coded bit string; wherein the full integer stream model includes multiple serially connected integer stream layers, each of which models the input vector and output vector of the integer stream layer as integer vectors. This image compression method can improve the speed of lossless image compression.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and in particular to an image compression method, device, and storage medium based on a full integer stream model. Background Art

[0002] Lossless image compression algorithms based on deep generative models typically only achieve a compression bandwidth of 1MB / s, far below the 100MB / s bandwidth of traditional lossless image compressors (which don't use neural networks). The inference speed of deep generative models is the primary bottleneck in overall compression speed.

[0003] In related technologies, the transformation of deep generative models usually uses floating-point numbers for calculations, which has a slow calculation speed. Summary of the Invention

[0004] The present invention provides an image compression method based on a full integer stream model, including:

[0005] Train the full integer stream model using the training image dataset;

[0006] Calculating the input target image data using a trained full integer stream model to obtain a representation vector of the target image and a probability distribution of the representation vector; performing entropy coding on the representation vector of the target image and its probability distribution to obtain a compressed coding bit string;

[0007] The full integer stream model includes multiple serially connected integer stream layers, and any integer stream layer models the input vector and output vector of the integer stream layer as integer vectors.

[0008] An embodiment of the present application provides an image compression device based on a full integer stream model, comprising: a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the above-mentioned image compression method based on the full integer stream model are implemented.

[0009] The embodiment of the present application provides an image compression device based on a full integer stream model, comprising:

[0010] A model training module configured to train a full integer stream model using a training image dataset;

[0011] A feature extraction module is configured to calculate the input target image data using a trained full integer stream model to obtain a representation vector of the target image and a probability distribution of the representation vector;

[0012] An encoding module configured to perform entropy encoding according to the representation vector of the target image and its probability distribution to obtain a compressed encoding bit string;

[0013] The full integer stream model includes multiple serially connected integer stream layers, and any integer stream layer models the input vector and output vector of the integer stream layer as integer vectors.

[0014] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-mentioned image compression method based on the full integer stream model.

[0015] The image compression method based on the full integer stream model provided in the embodiment of the present application uses a training picture data set to train the full integer stream model; the data of the input target picture is calculated by the trained full integer stream model to obtain the representation vector of the target picture and the probability distribution of the representation vector; entropy coding is performed according to the representation vector of the target picture and its probability distribution to obtain a compressed coding bit string; wherein, the full integer stream model includes multiple integer stream layers connected in series, and any integer stream layer models the input vector and output vector of the integer stream layer as integer vectors. The above method uses quantization and pruning technology to accelerate the flow model inference speed, replaces floating-point operations with integer operations, and reduces the number of model convolution kernels, thereby obtaining a model with less computational complexity, which can improve the speed of lossless compression of images.

[0016] Still other aspects will become apparent upon reading and understanding the accompanying drawings and detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to provide an understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0018] Figure 1 This is a flowchart of an image compression method based on a full integer stream model according to an embodiment of the present application;

[0019] Figure 2 This is a schematic diagram of the input and output of a full-integer stream model according to an embodiment of the present application;

[0020] Figure 3-1 This is a schematic diagram of a first integer stream layer of a full integer stream model according to an embodiment of the present application;

[0021] Figure 3-2 This is a schematic diagram of a second integer stream layer of a full integer stream model according to an embodiment of the present application;

[0022] Figure 4 A schematic diagram of a channel permutation transformation layer rearranging the channel order according to an embodiment of the present application;

[0023] Figure 5 This is a schematic structural diagram of a coupling conversion layer according to an embodiment of the present application;

[0024] Figure 6 A schematic structural diagram of a convolutional layer included in a first convolutional neural network of a coupled transformation layer according to an embodiment of the present application;

[0025] Figure 7 A schematic diagram of a structure of a divided layer according to an embodiment of the present application;

[0026] Figure 8 This is a structural diagram of an image compression device based on a full integer stream model according to an embodiment of the present application;

[0027] Figure 9 This is a structural diagram of another image compression device based on a full integer stream model according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] This application describes multiple embodiments, but this description is exemplary rather than restrictive, and it will be apparent to those skilled in the art that there may be more embodiments and implementations within the scope of the embodiments described herein. Although many possible feature combinations are shown in the drawings and discussed in the detailed description, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with any other feature or element in any other embodiment, or may replace any other feature or element in any other embodiment.

[0029] This application includes and contemplates combinations of features known to those of ordinary skill in the art. The embodiments and features disclosed in this application may also be combined with any conventional features to form a unique inventive solution defined by the appended claims. Any feature of any embodiment may also be combined with features from other inventive solutions to form another unique inventive solution defined by the appended claims. Therefore, it should be understood that any feature shown and / or discussed in this application may be implemented individually or in any appropriate combination. Therefore, the embodiments are not subject to other limitations except for the limitations made according to the appended claims and their equivalents. In addition, various modifications and changes may be made within the scope of protection of the appended claims.

[0030] In addition, when describing representative embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of the steps described herein, the method or process should not be limited to the steps in the specific order described. As will be understood by those skilled in the art, other orders of steps are also possible. Therefore, the specific order of the steps set forth in the specification should not be interpreted as a limitation on the appended claims. In addition, the claims for the method and / or process should not be limited to performing their steps in the order written, and those skilled in the art can readily understand that these orders can be changed and still remain within the spirit and scope of the embodiments of the present application.

[0031] like Figure 1 As shown, the embodiment of the present application provides an image compression method based on a full integer stream model, including:

[0032] Step S10, training a full integer stream model using a training image dataset;

[0033] Step S20, calculating the input target image data using the trained full integer stream model to obtain a representation vector of the target image and a probability distribution of the representation vector;

[0034] Step S30, performing entropy coding according to the representation vector of the target image and its probability distribution to obtain a compressed coded bit string;

[0035] The full integer stream model includes multiple serially connected integer stream layers, and any integer stream layer models the input vector and output vector of the integer stream layer as integer vectors.

[0036] The image compression method based on the full integer stream model provided in the embodiment of the present application uses a training picture data set to train the full integer stream model; the data of the input target picture is calculated by the trained full integer stream model to obtain the representation vector of the target picture and the probability distribution of the representation vector; entropy coding is performed according to the representation vector of the target picture and its probability distribution to obtain a compressed coding bit string; wherein, the full integer stream model includes multiple integer stream layers connected in series, and any integer stream layer models the input vector and output vector of the integer stream layer as integer vectors. The above method uses quantization and pruning technology to accelerate the flow model inference speed, replaces floating-point operations with integer operations, and reduces the number of model convolution kernels, thereby obtaining a model with less computational complexity, which can improve the speed of lossless compression of images.

[0037] In some exemplary embodiments, Figure 2As shown, the full integer stream model includes M integer stream layers connected in series; wherein the first M-1 integer stream layers include M-1 first integer stream layers, and the last integer stream layer includes a second integer stream layer; M is an integer greater than 1;

[0038] The first integer stream layer outputs a portion of the representation vector y i1 As part of the representation vector z output by the full integer flow model i , the first integer stream layer output representation vector y i1 The probability distribution p(y i1 ) as part of the probability distribution p(z) representing the vector z output by the full integer flow model i ), the other part y of the representation vector output by the first integer stream layer i i2 Transmit to the next level integer stream layer; 1≤i≤M-1;

[0039] The second integer stream layer outputs the representation vector y M As part of the representation vector z output by the full integer flow model M , the second integer stream layer outputs the representation vector y M The probability distribution p(y M ) as part of the probability distribution p(z) representing the vector z output by the full integer flow model M )

[0040] In some exemplary embodiments, Figure 3-1 As shown, the first integer stream layer includes a size compression transformation layer, N1 groups of cascaded channel permutation coupling transformation layers and a partitioning layer connected in series; the channel permutation coupling transformation layer includes a channel permutation transformation layer and a coupling transformation layer connected in series; N1 is an integer;

[0041] like Figure 3-2 As shown, the second integer stream layer includes a size compression transform layer, N2 groups of cascaded channel permutation coupling transform layers and probability distribution layers connected in series; the channel permutation coupling transform layer includes a channel permutation transform layer and a coupling transform layer connected in series; N2 is an integer;

[0042] The input vector and output vector of the first integer stream layer and the second integer stream layer include four dimensions: the first dimension is the number of input pictures, the second dimension is the number of channels, the third dimension is the length of the picture, and the fourth dimension is the width of the picture.

[0043] The first integer stream layer obtains partial output vectors by setting a partitioning layer, thereby realizing feature extraction at different resolutions.

[0044] In some exemplary embodiments, N1 and N2 may be equal or unequal.

[0045] In some exemplary embodiments, the size compression transformation layer is used to expand the second dimension of the input vector and compress the third and fourth dimensions of the input vector.

[0046] In some exemplary embodiments, the size compression transformation layer is used to expand the second dimension of the input vector to n1 times of the original, reduce the third dimension of the input vector to 1 / n2 of the original, and reduce the fourth dimension of the input vector to 1 / n3 of the original; n1=n2*n3.

[0047] In some exemplary embodiments, Figure 4 As shown, the channel permutation transformation layer is used to shuffle and rearrange different channels of the input vector according to a preset order.

[0048] In some exemplary embodiments, different channel permutation transform layers in any integer stream layer have different preset orders. By changing the channel order of the input vector, the channels entering the coupled transform layer for convolution operations can be divided differently.

[0049] In some exemplary embodiments, Figure 5 As shown, the j-th coupled transformation layer of any integer stream layer includes a first convolutional neural network and a rounding layer; when the integer stream layer is the first integer stream layer, 1≤j≤N1, and when the integer stream layer is the second integer stream layer, 1≤j≤N2;

[0050] The first convolutional neural network takes the input vector x j The first part is the input vector x j1 The number of channels occupied is a1, and the second part input vector x j2 The number of channels occupied is a2; a1+a2=a0; a1>a2; a0 is the input vector x j The total number of channels;

[0051] The first part is the input vector x j1 Directly as the output vector y of the coupling transformation layer j The first part of the output vector y j1 ; The first part of the input vector x j1 After the operation of the first convolutional neural network and the rounding layer, the second part of the input vector x j2 The sum of the additions is the output vector y of the coupled transformation layer j The second part of the output vector y j2 , output vector y j Output vector y for the first part j1 and the second part output vector y j2 splicing in the channel dimension;

[0052] Among them, the rounding layer is used to convert the numerical value of the input element into an integer by rounding, and the coupled transformation layer provides an expressive reversible transformation from integer to integer.

[0053] The purpose of convolution operation is to provide rich transformation modes to extract image features and provide powerful expression capabilities for the coupled transformation layer.

[0054] In some exemplary embodiments, the first convolutional neural network includes at least one convolutional layer; Figure 6 As shown, the convolution layer includes: a parameter quantizer, an input quantizer, a convolution operation unit and a gate controller;

[0055] The parameter quantizer is used to quantize the floating point precision convolution kernel parameter vector w of the convolution layer to generate the quantized approximate convolution kernel parameter vector

[0056] The input quantizer is used to quantize the floating point precision input vector x of the convolution layer to generate an approximate input vector

[0057] The convolution operation unit is used to quantize the approximate convolution kernel parameter vector and approximate input vector Perform convolution operation to obtain the convolution result vector;

[0058] The gate controller is used to zero or retain the elements of the convolution result vector in the channel dimension according to the gate parameters.

[0059] In some exemplary embodiments, the floating point precision convolution kernel parameter vector w of the convolution layer is quantized to generate the quantized approximate convolution kernel parameter vector The floating point precision input vector x of the convolutional layer is quantized to generate an approximate input vector Approximate convolution kernel parameter vector and approximate input vector Performing a convolution operation is equivalent to approximating the convolution kernel parameter integer vector with approximate input integer vector Perform convolution operation and then multiply element-wise by real number s w *s x , which is expressed as follows:

[0060]

[0061] in, is the convolution operation, * is the multiplication operation, s w is the quantization step size of the parameter quantizer, s x is the quantization step size of the input quantizer.

[0062] In the above formula, the convolution operation between the original floating-point vectors w and x is converted into an integer vector The convolution operation between and is then element-wise multiplied by a common real number s. w *s x When deployed on specific hardware, integer matrix operations are faster than floating-point matrix operations, so convolutional layers with quantizers are faster than convolutional layers without quantizers.

[0063] In some exemplary embodiments, the gate controller is configured to set the elements of the convolution result vector to zero or retain them in the channel dimension according to the gate parameters in the following manner:

[0064] After the convolution result vector is input into the gate controller, the tth element g of the gate parameter vector of the gate controller t Acting on the tth channel of the convolution result vector, if g t If g is less than the preset first threshold, all convolution result vector elements of the channel are set to 0. t If the value is greater than or equal to the preset first threshold, all the convolution result vector elements of the channel remain unchanged; where 0≤g t ≤1.

[0065] In some exemplary embodiments, the first threshold is 0.5.

[0066] In some exemplary embodiments, Figure 7 As shown, the partitioning layer of the i-th integer stream layer is used to permute the vector y output by the channel coupling transformation layer in the channel dimension. i Divide into the first vector y i1 and the second vector y i2 ; The first vector y i1 As part of the representation vector z output by the full integer flow model i , the first vector y i1 Input the second convolutional neural network to calculate and obtain the representation vector z of the i-th integer stream layer i The probability distribution parameters of the characterization vector z are calculated according to the probability distribution parameters i The probability distribution p(z i ) and output the probability distribution p(z i ), the second vector y i2 It is directly output to the next level integer stream layer and serves as the input vector of the next level integer stream layer; wherein, 1≤i≤M-1.

[0067] In some exemplary embodiments, the probability distribution layer of the second integer stream layer is used to permute the vector y output by the channel coupling transformation layer iAs part of the representation vector z output by the full integer flow model M , according to the preset characterization vector z M The probability distribution parameter calculation characterization vector z M The probability distribution p(z M ) and output the probability distribution p(z M ).

[0068] In some exemplary embodiments, training a full integer stream model using a training image dataset includes:

[0069] The full integer stream model is trained using the training image dataset in the following manner to obtain the convolution kernel parameters, quantizer parameters, and gate controller parameters of the full integer stream model:

[0070] Initializing the quantization step size of the parameter quantizer of the full integer stream model, the quantization step size of the input quantizer, and the gate parameter vector of the gate controller and setting them to be non-updatable;

[0071] Setting a first objective function for training, wherein the first objective function is -logp(z); 1≤i≤M, M is the total number of integer stream layers contained in the full integer stream model, z i is the representation vector output by the i-th integer stream layer;

[0072] The optimizer is used for training, the initial learning rate is set to b1, the learning rate decays by c1% at the end of each round of training, and the training ends after d1 rounds of training.

[0073] In some exemplary embodiments, b1, c1, and d1 can be set according to actual needs.

[0074] In some exemplary embodiments, training a full integer stream model using a training image dataset includes:

[0075] The full integer stream model is trained using the training image dataset in the following manner to obtain the convolution kernel parameters, quantizer parameters, and gate controller parameters of the full integer stream model:

[0076] Setting the quantization step size of the parameter quantizer and the quantization step size of the input quantizer of the full integer stream model to be non-updatable, and setting the gate parameter vector of the gate controller to be updatable;

[0077] A second threshold is set for the sum k of the number of elements of the gate parameter vector of the full integer stream model that is greater than the first threshold. k i is the number of elements in the gate parameter vector of the i-th integer stream layer that are greater than the first threshold, λ i is the adjustment factor corresponding to the i-th integer flow layer; M is the total number of integer flow layers contained in the full integer flow model;

[0078] Set the second objective function for training. The second objective function is 1≤i≤M, M is the total number of integer stream layers contained in the full integer stream model, z i is the representation vector output by the i-th integer stream layer;

[0079] The optimizer is used for training, the initial learning rate is set to b2, and the learning rate is decayed by c2% at the end of each round of training. Training is stopped when the sum of the number of elements k of the gate parameter vector of the full integer stream model is greater than the first threshold and is lower than the second threshold.

[0080] In some exemplary embodiments, the second threshold value can be set according to actual needs. b2 and c2 can also be set according to actual needs.

[0081] In some exemplary embodiments, training a full integer stream model using a training image dataset includes:

[0082] The full integer stream model is trained using the training image dataset in the following way to obtain the convolution kernel parameters, quantizer parameters, and gate controller parameters of the full integer stream model:

[0083] Setting the gate parameter vector of the gate controller of the full integer stream model to be non-updatable, and setting the quantization step size of the parameter quantizer and / or the quantization step size of the input quantizer to be updateable;

[0084] Setting a third objective function for training, wherein the third objective function is -logp(z); 1≤i≤M, M is the total number of integer stream layers contained in the full integer stream model, z i is the representation vector output by the i-th integer stream layer;

[0085] updating the quantization step size of the parameter quantizer and / or the quantization step size of the input quantizer using a gradient optimization algorithm for the quantization step size s based on the third objective function;

[0086] The optimizer is used for training, the initial learning rate is set to b3, the learning rate decays by c3% at the end of each round of training, and the training ends after d3 rounds of training.

[0087] In some exemplary embodiments, b3, c3, and d3 can be set according to actual needs.

[0088] In some exemplary embodiments, the gradient of the third objective function with respect to the quantization step size s is calculated using the following chain rule formula:

[0089]

[0090]

[0091] in, is the indicator function, Act on a vector element by element, and judge whether each element satisfies the inequality condition in the brackets. If it does, it will get 1, and if it does not, it will get 0. r is the vector before quantization. is a quantized vector, ⊙ represents the element-wise product of two vectors; [-Q N ,Q P ] is the quantization range of the quantization step s.

[0092] In some exemplary embodiments, the optimizer may be an adamax optimizer, or an adam optimizer, or an sgd optimizer, etc.

[0093] During training, Methods 1, 2, and 3 can be used individually or in combination. For example, Methods 1 and 2 can be used together, or Methods 1 and 3 can be used together, or Methods 1, 2, and 3 can be used together. When all three methods are used together, Method 1 can be used first, and the order of executing Methods 2 and 3 is not limited.

[0094] In some exemplary embodiments, the probability distribution p(z) of the representation vector z output by the full integer flow model is a discrete logistic distribution;

[0095] 1≤i≤M, M is the total number of integer stream layers contained in the full integer stream model, z i is the representation vector output by the i-th integer stream layer;

[0096]

[0097] 1≤j≤D i , D i is the representation vector z i The number of elements, σ(.) is the sigmoid activation function; z i,j is the representation vector z i The jth element of μ i,j and s i,j is the representation vector z i The probability distribution parameters corresponding to the j-th element of .

[0098] The full integer stream model outputs a representation vector of the image and the probability distribution of the representation vector. The encoder uses an entropy coding algorithm for compression. The entropy coding algorithm encodes the input into a 0 / 1 bit stream based on the probability distribution of the input vector, preserving the information intact and enabling reverse recovery. Common entropy coding algorithms include Huffman coding and arithmetic coding. As an implementation of an embodiment of the present application, an asymmetric numerical system algorithm can be used to entropy encode the representation vector dimension by dimension to obtain an encoded bit string. The asymmetric numerical system algorithm is a type of arithmetic coding with the advantages of high speed and high coding efficiency. Because the entropy coding process is lossless, the representation vector can be fully recovered from the encoded bit string. The decoder uses the inverse process of the entropy coding algorithm to decompress the encoded bit string to obtain the representation vector. Since each transformation layer of the full integer stream model is reversible, the full integer stream model has an inverse transformation. Using the representation vector as the input of the inverse transformation, the original image data can be recovered by working backward from the last layer of the full integer stream model to the first layer.

[0099] The following uses a specific example to explain the dimensional changes of the input vector and output vector of the full integer stream model during the training phase.

[0100] Prepare an image dataset for training. The image dataset can be the publicly available ImageNet32 training set. The full integer stream model includes three integer stream layers: the first, second, and third integer stream layers. Both the first and second integer stream layers include a partitioning layer, while the third integer stream layer does not include a partitioning layer but includes a probability distribution layer. All integer stream layers include a size reduction transform layer and at least one channel permutation coupling transform layer. The channel permutation coupling transform layer includes a channel permutation transform layer and a coupling transform layer connected in series.

[0101] The image data used for training serves as the input vector of the full integer stream model, and includes four dimensions: [d1, d2, d3, d4], where d1 is the first dimension, indicating the number of input images, d2 is the second dimension, indicating the number of channels, d3 is the third dimension, indicating the length of the image, and d4 is the fourth dimension, indicating the width of the image.

[0102] Assuming that the dimension of the input vector of the full integer stream model is [d1,3,32,32], the size compression transformation layer expands the second dimension of the input vector to 4 times its original size, reduces the third dimension of the input vector to 1 / 2 of its original size, and reduces the fourth dimension of the input vector to 1 / 2 of its original size.

[0103] The image data vector is input to the first integer stream layer. After the size compression transformation layer, the vector dimension becomes [d1, 12, 16, 16]. After the partitioning layer in the first integer stream layer, it is divided into two equal vectors: the first vector and the second vector. The first vector is output to the output vector of the full integer stream model. The first vector has a dimension of [d1, 6, 16, 16], and the second vector is output to the second integer stream layer. The dimension of the second vector is [d1, 6, 16, 16].

[0104] The input vector [d1, 6, 16, 16] in the second integer stream layer is reduced to [d1, 24, 8, 8] after the size reduction transformation layer. After the partitioning layer in the second integer stream layer, it is divided into two equal vectors: the third vector and the fourth vector. The third vector is output to the output vector of the full integer stream model, and the dimension of the third vector is [d1, 12, 8, 8]. The fourth vector is output to the third integer stream layer, and the dimension of the fourth vector is [d1, 12, 8, 8].

[0105] The input vector [d1, 12, 8, 8] in the third integer stream layer is reduced to [d1, 48, 4, 4] after the size reduction transformation layer. The vector dimensions remain unchanged and are all output to the output vector of the full integer stream model.

[0106] The output vector (representation vector) of the full integer flow model is the concatenation of the output vectors of the three integer flow layers.

[0107] like Figure 8 As shown, the embodiment of the present disclosure provides an image compression device based on a full integer stream model, comprising:

[0108] A model training module 10 is configured to train a full integer stream model using a training image dataset;

[0109] The feature extraction module 20 is configured to calculate the input target image data using a trained full integer stream model to obtain a representation vector of the target image and a probability distribution of the representation vector;

[0110] The encoding module 30 is configured to perform entropy encoding according to the representation vector of the target image and its probability distribution to obtain a compressed encoding bit string;

[0111] The full integer stream model includes multiple serially connected integer stream layers, and any integer stream layer models the input vector and output vector of the integer stream layer as integer vectors.

[0112] In some exemplary embodiments, the full integer flow model includes M integer flow layers connected in series; wherein the first M-1 integer flow layers include M-1 first integer flow layers, and the last integer flow layer includes one second integer flow layer; M is an integer greater than 1;

[0113] The first integer stream layer outputs a portion of the representation vector y i1 As part of the representation vector z output by the full integer flow model i , the first integer stream layer output representation vector y i1 The probability distribution p(y i1 ) as part of the probability distribution p(z) representing the vector z output by the full integer flow model i ), the other part y of the representation vector output by the first integer stream layer i i2 Transmit to the next level integer stream layer; 1≤i≤M-1;

[0114] The second integer stream layer outputs the representation vector y M As part of the representation vector z output by the full integer flow model M , the second integer stream layer outputs the representation vector y M The probability distribution p(y M ) as part of the probability distribution p(z) representing the vector z output by the full integer flow model M ).

[0115] In some exemplary embodiments, the first integer stream layer includes a size compression transform layer, N1 groups of cascaded channel permutation coupling transform layers, and a partitioning layer connected in series; the channel permutation coupling transform layer includes a channel permutation transform layer and a coupling transform layer connected in series;

[0116] The second integer stream layer includes a size compression transform layer and N2 groups of cascaded channel permutation coupling transform layers connected in series; the channel permutation coupling transform layer includes a channel permutation transform layer and a coupling transform layer connected in series;

[0117] The input vector and output vector of the first integer stream layer and the second integer stream layer include four dimensions: the first dimension is the number of input pictures, the second dimension is the number of channels, the third dimension is the length of the picture, and the fourth dimension is the width of the picture.

[0118] In some exemplary embodiments, the size compression transformation layer is used to expand the second dimension of the input vector to n1 times of the original, reduce the third dimension of the input vector to 1 / n2 of the original, and reduce the fourth dimension of the input vector to 1 / n3 of the original; n1=n2*n3.

[0119] In some exemplary embodiments, the channel permutation transformation layer is used to shuffle and rearrange different channels of the input vector according to a preset order; different channel permutation transformation layers correspond to different preset orders.

[0120] In some exemplary embodiments, the j-th coupled transform layer of any integer stream layer includes a first convolutional neural network and a rounding layer; when the integer stream layer is a first integer stream layer, 1≤j≤N1; when the integer stream layer is a second integer stream layer, 1≤j≤N2;

[0121] The first convolutional neural network takes the input vector x j The first part is the input vector x j1 The number of channels occupied is a1, and the second part input vector x j2 The number of channels occupied is a2; a1+a2=a0; a1>a2; a0 is the input vector x j The total number of channels;

[0122] The first part is the input vector x j1 Directly as the output vector y of the coupling transformation layer j The first part of the output vector y j1 ; The first part of the input vector x j1 After the operation of the first convolutional neural network and the rounding layer, the second part of the input vector x j2 The sum of the additions is the output vector y of the coupled transformation layer j The second part of the output vector y j2 , output vector y j Output vector y for the first part j1 and the second part output vector y j2 splicing in the channel dimension;

[0123] Among them, the rounding layer is used to convert the numerical value of the input element into an integer by rounding, and the coupled transformation layer provides an expressive reversible transformation from integer to integer.

[0124] In some exemplary embodiments, the first convolutional neural network includes at least one convolutional layer;

[0125] The convolution layer includes: a parameter quantizer, an input quantizer, a convolution operation unit and a gate controller;

[0126] The parameter quantizer is used to quantize the floating point precision convolution kernel parameter vector w of the convolution layer to generate the quantized approximate convolution kernel parameter vector

[0127] The input quantizer is used to quantize the floating point precision input vector x of the convolution layer to generate an approximate input vector

[0128] The convolution operation unit is used to quantize the approximate convolution kernel parameter vector and approximate input vector Perform convolution operation to obtain the convolution result vector;

[0129] The gate controller is used to zero or retain the elements of the convolution result vector in the channel dimension according to the gate parameters.

[0130] In some exemplary embodiments, the gate controller is configured to set the elements of the convolution result vector to zero or retain them in the channel dimension according to the gate parameters in the following manner:

[0131] After the convolution result vector is input into the gate controller, the tth element g of the gate parameter vector of the gate controller t Acting on the tth channel of the convolution result vector, if g t If g is less than the preset first threshold, all convolution result vector elements of the channel are set to 0. t If the value is greater than or equal to the preset first threshold, all the convolution result vector elements of the channel remain unchanged; where 0≤g t ≤1.

[0132] In some exemplary embodiments, the splitting layer of the i-th integer stream layer is used to permute the vector y output by the channel coupling transformation layer in the channel dimension. i Divide into the first vector y i1 and the second vector y i2 ; The first vector y i1 As part of the representation vector z output by the full integer flow model i , the first vector y i1 Input the second convolutional neural network to calculate and obtain the representation vector z of the i-th integer stream layer i The probability distribution parameters of the characterization vector z are calculated according to the probability distribution parameters i The probability distribution p(z i ) and output the probability distribution p(z i ), the second vector y i2 Directly output to the next level integer stream layer and serve as the input vector of the next level integer stream layer; wherein 1≤i≤M-1;

[0133] The probability distribution layer of the second integer stream layer is used to permute the vector y output by the channel coupling transformation layer i As part of the representation vector z output by the full integer flow model M , according to the preset characterization vector z M The probability distribution parameter calculation characterization vector z M The probability distribution p(z M ) and output the probability distribution p(z M ).

[0134] In some exemplary embodiments, the model training module is configured to train a full integer stream model using a training image dataset in the following manner to obtain convolution kernel parameters, quantizer parameters, and gate controller parameters of the full integer stream model:

[0135] Initializing the quantization step size of the parameter quantizer of the full integer stream model, the quantization step size of the input quantizer, and the gate parameter vector of the gate controller and setting them to be non-updatable;

[0136] Setting a first objective function for training, wherein the first objective function is -logp(z); 1≤i≤M, M is the total number of integer stream layers contained in the full integer stream model, z i is the representation vector output by the i-th integer stream layer;

[0137] The optimizer is used for training, the initial learning rate is set to b1, the learning rate decays by c1% at the end of each round of training, and the training ends after d1 rounds of training.

[0138] In some exemplary embodiments, the model training module is further configured to train the full integer stream model using the training image dataset in the following manner to obtain convolution kernel parameters, quantizer parameters, and gate controller parameters of the full integer stream model:

[0139] Setting the quantization step size of the parameter quantizer and the quantization step size of the input quantizer of the full integer stream model to be non-updatable, and setting the gate parameter vector of the gate controller to be updatable;

[0140] A second threshold is set for the sum k of the number of elements of the gate parameter vector of the full integer stream model that is greater than the first threshold. k i is the number of elements in the gate parameter vector of the i-th integer stream layer that are greater than the first threshold, λ i is the adjustment factor corresponding to the i-th integer flow layer; M is the total number of integer flow layers contained in the full integer flow model;

[0141] Set the second objective function for training. The second objective function is 1≤i≤M, M is the total number of integer stream layers contained in the full integer stream model, z i is the representation vector output by the i-th integer stream layer;

[0142] The optimizer is used for training, the initial learning rate is set to b2, and the learning rate is decayed by c2% at the end of each round of training. Training is stopped when the sum of the number of elements k of the gate parameter vector of the full integer stream model is greater than the first threshold and is lower than the second threshold.

[0143] In some exemplary embodiments, the model training module is further configured to train the full integer stream model using the training image dataset in the following manner to obtain convolution kernel parameters, quantizer parameters, and gate controller parameters of the full integer stream model:

[0144] Setting the gate parameter vector of the gate controller of the full integer stream model to be non-updatable, and setting the quantization step size of the parameter quantizer and / or the quantization step size of the input quantizer to be updateable;

[0145] Setting a third objective function for training, wherein the third objective function is -logp(z); 1≤i≤M, M is the total number of integer stream layers contained in the full integer stream model, z i is the representation vector output by the i-th integer stream layer;

[0146] updating the quantization step size of the parameter quantizer and / or the quantization step size of the input quantizer using a gradient optimization algorithm for the quantization step size s based on the third objective function;

[0147] The optimizer is used for training, the initial learning rate is set to b3, the learning rate decays by c3% at the end of each round of training, and the training ends after d3 rounds of training.

[0148] In some exemplary embodiments, the gradient of the third objective function with respect to the quantization step size s is calculated using the following chain rule formula:

[0149]

[0150]

[0151] in, is the indicator function, Act on a vector element by element, and judge whether each element satisfies the inequality condition in the brackets. If it does, it will get 1, and if it does not, it will get 0. r is the vector before quantization. is a quantized vector, ⊙ represents the element-wise product of two vectors; [-Q N ,Q P ] is the quantization range of the quantization step s.

[0152] In some exemplary embodiments, the probability distribution p(z) of the representation vector z output by the full integer flow model is a discrete logistic distribution;

[0153] 1≤i≤M, M is the total number of integer stream layers contained in the full integer stream model, z i is the representation vector output by the i-th integer stream layer;

[0154]

[0155] 1≤j≤D i , D i is the representation vector z i The number of elements, σ(.) is the sigmoid activation function; z i,j is the representation vector z i The jth element of μ i,j and s i,j is the representation vector z i The probability distribution parameters corresponding to the j-th element of .

[0156] like Figure 9 As shown, an embodiment of the present disclosure provides an image compression device based on a full integer stream model, including: a memory and a processor, the memory storing a computer program, and the computer program implementing the steps of the above-mentioned image compression method based on the full integer stream model when executed by the processor.

[0157] An embodiment of the present disclosure provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-mentioned image compression method based on the full integer stream model.

[0158] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable, and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media generally embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

Claims

1. An image compression method based on a full integer stream model, comprising: Train the full integer stream model using the training image dataset; Calculating the input target image data using the trained full integer stream model to obtain a representation vector of the target image and a probability distribution of the representation vector; Performing entropy coding on the target image according to its representation vector and probability distribution to obtain a compressed coding bit string; In which, the full integer stream model includes M integer stream layers connected in series, and any integer stream layer models the input vector and output vector of the integer stream layer as integer vectors; the first M-1 integer stream layers include M-1 first integer stream layers, and the last integer stream layer includes a second integer stream layer; M is an integer greater than 1; the first integer stream layer includes a size compression transformation layer, N1 groups of cascaded channel permutation coupling transformation layers and a partitioning layer connected in series; the channel permutation coupling transformation layer includes a channel permutation transformation layer and a coupling transformation layer connected in series; the second integer stream layer includes a size compression transformation layer, N2 groups of cascaded channel permutation coupling transformation layers and a probability distribution layer connected in series; the channel permutation coupling transformation layer includes a channel permutation transformation layer and a coupling transformation layer connected in series; N1 and N2 are integers.

2. The method according to claim 1, wherein: The first integer stream layer outputs a portion of the representation vector y i1 As part of the representation vector z output by the full integer flow model i , the first integer stream layer output representation vector y i1 The probability distribution p(y i1 ) as part of the probability distribution p(z) representing the vector z output by the full integer flow model i ), the other part y of the representation vector output by the first integer stream layer i i2 Transmit to the next level integer stream layer; 1≤i≤M-1; The second integer stream layer outputs the representation vector y M As part of the representation vector z output by the full integer flow model M , the second integer stream layer outputs the representation vector y M The probability distribution p(y M ) as part of the probability distribution p(z) representing the vector z output by the full integer flow model M ).

3. The method according to claim 2, wherein: The input vector and output vector of the first integer stream layer and the second integer stream layer include four dimensions: the first dimension is the number of input pictures, the second dimension is the number of channels, the third dimension is the length of the picture, and the fourth dimension is the width of the picture.

4. The method according to claim 3, wherein: The size compression transformation layer is used to expand the second dimension of the input vector to n1 times of the original, reduce the third dimension of the input vector to 1 / n2 of the original, and reduce the fourth dimension of the input vector to 1 / n3 of the original; n1=n2*n3.

5. The method according to claim 3, wherein: The channel permutation transformation layer is used to shuffle and rearrange different channels of the input vector according to a preset order; different channel permutation transformation layers correspond to different preset orders.

6. The method according to claim 3, wherein: The j-th coupled transformation layer of any integer stream layer includes a first convolutional neural network and a rounding layer; when the integer stream layer is the first integer stream layer, 1≤j≤N1; when the integer stream layer is the second integer stream layer, 1≤j≤N2; The first convolutional neural network takes the input vector x j The first part is the input vector x j1 The number of channels occupied is a1, and the second part input vector x j2 The number of channels occupied is a2; a1+a2=a0; a1>a2; a0 is the input vector x j The total number of channels; The first part is the input vector x j1 Directly as the output vector y of the coupling transformation layer j The first part of the output vector y j1 ; The first part of the input vector x j1 After the operation of the first convolutional neural network and the rounding layer, the second part of the input vector x j2 The sum of the additions is the output vector y of the coupled transformation layer j The second part of the output vector y j2 , output vector y j Output vector y for the first part j1 and the second part output vector y j2 splicing in the channel dimension; Among them, the rounding layer is used to convert the numerical value of the input element into an integer by rounding, and the coupled transformation layer provides an expressive reversible transformation from integer to integer.

7. The method according to claim 6, wherein: The first convolutional neural network includes at least one convolutional layer; The convolution layer includes: a parameter quantizer, an input quantizer, a convolution operation unit and a gate controller; The parameter quantizer is used to quantize the floating point precision convolution kernel parameter vector w of the convolution layer to generate the quantized approximate convolution kernel parameter vector The input quantizer is used to quantize the floating-point precision input vector x of the convolutional layer to generate an approximate input vector x%; The convolution operation unit is used to quantize the approximate convolution kernel parameter vector and approximate input vector Perform convolution operation to obtain the convolution result vector; The gate controller is used to zero or retain the elements of the convolution result vector in the channel dimension according to the gate parameters.

8. The method according to claim 7, wherein: The gate controller is used to zero or retain the elements of the convolution result vector in the channel dimension according to the gate parameters in the following way: After the convolution result vector is input into the gate controller, the tth element g of the gate parameter vector of the gate controller t Acting on the tth channel of the convolution result vector, if g t If g is less than the preset first threshold, all convolution result vector elements of the channel are set to 0. t If the value is greater than or equal to the preset first threshold, all the convolution result vector elements of the channel remain unchanged; where 0≤g t ≤1.

9. The method according to claim 3, wherein: The partitioning layer of the i-th integer stream layer is used to permute the vector y output by the channel coupling transformation layer in the channel dimension. i Divide into the first vector y i1 and the second vector y i2 ; The first vector y i1 As part of the representation vector z output by the full integer flow model i , the first vector y i1 Input the second convolutional neural network to calculate and obtain the representation vector z of the i-th integer stream layer i The probability distribution parameters of the characterization vector z are calculated according to the probability distribution parameters i The probability distribution p(z i ) and output the probability distribution p(z i ), the second vector y i2 Directly output to the next level integer stream layer and serve as the input vector of the next level integer stream layer; wherein 1≤i≤M-1; The probability distribution layer of the second integer stream layer is used to permute the vector y output by the channel coupling transformation layer i As part of the representation vector z output by the full integer flow model M , according to the preset characterization vector z M The probability distribution parameter calculation characterization vector z M The probability distribution p(z M ) and output the probability distribution p(z M ).

10. The method according to claim 9, wherein: Use the training image dataset to train the full integer stream model, including: The full integer stream model is trained using the training image dataset in the following manner to obtain the convolution kernel parameters, quantizer parameters, and gate controller parameters of the full integer stream model: Initializing the quantization step size of the parameter quantizer of the full integer stream model, the quantization step size of the input quantizer, and the gate parameter vector of the gate controller and setting them to be non-updatable; Setting a first objective function for training, wherein the first objective function is -logp(z); 1≤i≤M, M is the total number of integer stream layers contained in the full integer stream model, z i is the representation vector output by the i-th integer stream layer; The optimizer is used for training, the initial learning rate is set to b1, the learning rate decays by c1% at the end of each round of training, and the training ends after d1 rounds of training.

11. The method according to claim 10, wherein: Using the training image dataset to train the full integer stream model, it also includes: The full integer stream model is trained using the training image dataset in the following manner to obtain the convolution kernel parameters, quantizer parameters, and gate controller parameters of the full integer stream model: Setting the quantization step size of the parameter quantizer and the quantization step size of the input quantizer of the full integer stream model to be non-updatable, and setting the gate parameter vector of the gate controller to be updatable; A second threshold is set for the sum k of the number of elements of the gate parameter vector of the full integer stream model that is greater than the first threshold. k i is the number of elements in the gate parameter vector of the i-th integer stream layer that are greater than the first threshold, λ i is the adjustment factor corresponding to the i-th integer flow layer; M is the total number of integer flow layers contained in the full integer flow model; Set the second objective function for training. The second objective function is 1≤i≤M, M is the total number of integer stream layers contained in the full integer stream model, z i is the representation vector output by the i-th integer stream layer; The optimizer is used for training, the initial learning rate is set to b2, and the learning rate is decayed by c2% at the end of each round of training. Training is stopped when the sum of the number of elements k of the gate parameter vector of the full integer stream model is greater than the first threshold and is lower than the second threshold.

12. The method according to claim 10 or 11, wherein: Using the training image dataset to train the full integer stream model, it also includes: The full integer stream model is trained using the training image dataset in the following way to obtain the convolution kernel parameters, quantizer parameters, and gate controller parameters of the full integer stream model: Setting the gate parameter vector of the gate controller of the full integer stream model to be non-updatable, and setting the quantization step size of the parameter quantizer and / or the quantization step size of the input quantizer to be updateable; Setting a third objective function for training, wherein the third objective function is -logp(z); 1≤i≤M, M is the total number of integer stream layers contained in the full integer stream model, z i is the representation vector output by the i-th integer stream layer; updating the quantization step size of the parameter quantizer and / or the quantization step size of the input quantizer using a gradient optimization algorithm for the quantization step size s based on the third objective function; The optimizer is used for training, the initial learning rate is set to b3, the learning rate decays by c3% at the end of each round of training, and the training ends after d3 rounds of training.

13. The method according to claim 12, wherein: The gradient of the third objective function with respect to the quantization step size s is calculated using the following chain rule formula: in, is the indicator function, Act on a vector element by element, and judge whether each element satisfies the inequality condition in the brackets. If it does, it will get 1, and if it does not, it will get 0. r is the vector before quantization. is a quantized vector, ⊙ represents the element-wise product of two vectors; [-Q N ,Q P ] is the quantization range of the quantization step s.

14. The method according to any one of claims 1 to 11, wherein: The probability distribution p(z) of the representation vector z output by the full integer flow model is a discrete logistic distribution; 1≤i≤M, M is the total number of integer stream layers contained in the full integer stream model, z i is the representation vector output by the i-th integer stream layer; 1≤j≤D i , D i is the representation vector z i The number of elements, σ(.) is the sigmoid activation function; z i,j is the representation vector z i The jth element of μ i,j and s i,j is the representation vector z i The probability distribution parameter corresponding to the jth element of i and s i is the representation vector z i The parameter vector of the probability distribution of .

15. An image compression device based on a full integer stream model, comprising: A model training module configured to train a full integer stream model using a training image dataset; A feature extraction module is configured to calculate the input target image data using a trained full integer stream model to obtain a representation vector of the target image and a probability distribution of the representation vector; An encoding module configured to perform entropy encoding according to the representation vector of the target image and its probability distribution to obtain a compressed encoding bit string; In which, the full integer stream model includes M integer stream layers connected in series, and any integer stream layer models the input vector and output vector of the integer stream layer as integer vectors; the first M-1 integer stream layers include M-1 first integer stream layers, and the last integer stream layer includes a second integer stream layer; M is an integer greater than 1; the first integer stream layer includes a size compression transformation layer, N1 groups of cascaded channel permutation coupling transformation layers and a partitioning layer connected in series; the channel permutation coupling transformation layer includes a channel permutation transformation layer and a coupling transformation layer connected in series; the second integer stream layer includes a size compression transformation layer, N2 groups of cascaded channel permutation coupling transformation layers and a probability distribution layer connected in series; the channel permutation coupling transformation layer includes a channel permutation transformation layer and a coupling transformation layer connected in series; N1 and N2 are integers.

16. An image compression device based on a full integer stream model, comprising: A memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the image compression method based on the full integer stream model described in any one of claims 1 to 14 are implemented.

17. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the steps of the image compression method based on the full integer stream model described in any one of claims 1 to 14 are implemented.

Images