An end-to-end image compression method based on contextual clustering transform
By using context clustering transformation and hyper-priori network in the end-to-end image compression method, the problem of insufficient interaction between local similar features of images in the CNN transform network is solved, and the image compression quality and rate distortion performance are improved.
Patent Information
- Application Number
- CN202311569161.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-21
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2043-11-21
AI Technical Summary
In the existing end-to-end image compression method, the CNN transform network has insufficient interaction of similar features in local images, resulting in unsatisfactory image compression effect.
Using a context-based clustering transformation method, by converting the image into feature points containing coordinates, grouping according to similarity, and performing multiple aggregation and redistribution, a latent representation is generated, and the latent representations are captured through a hyper-prior network. Redundant, generate prior features for entropy encoding.
It improves the interaction of local similar features of images, improves image compression quality and rate distortion performance, and provides a new end-to-end image compression design idea.
Smart Images

Figure CN117456017B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image compression and deep learning, and in particular to an end-to-end image compression method based on context clustering transformation. Background Art
[0002] With the rapid development of information technology and big data, multimedia data has shown a sharp growth trend in many fields, making image compression technology increasingly important. Traditional image compression methods are mainly based on image / video compression standards such as JPEG, JPEG2000, HEVC and VVC.
[0003] With the rapid development of deep learning in the field of image processing, researchers have proposed an end-to-end image compression method based on deep learning. Compared with traditional image compression methods, end-to-end image compression methods have the ability to adaptively balance image compression quality and transmission bitrate, and are more suitable for diverse image data, such as 360-degree panoramas, screen content, and light fields. Therefore, the research on end-to-end image compression methods has become one of the focuses of scholars in the field of image communication.
[0004] At present, end-to-end image compression methods mainly include four modules: transform network (or main codec), quantizer, super prior network and entropy coding. Most of the transform networks are designed based on the basic framework of convolutional neural network (CNN). However, existing research work mainly focuses on how to optimize CNN transform network and entropy coding, but ignores the problem of insufficient interaction of local similar information of images in CNN transform network itself, resulting in unsatisfactory image compression effect. Therefore, how to improve the end-to-end image compression effect is a technical problem that needs to be solved urgently. Summary of the invention
[0005] In view of the above-mentioned deficiencies in the prior art, the technical problem to be solved by the present invention is: how to provide an end-to-end image compression method based on context clustering transformation, which fully considers the correlation between adjacent pixels of the image, improves the problem of insufficient interaction of local similar features of the image in the current end-to-end image compression method based on CNN transformation, thereby ensuring the quality of end-to-end image compression and improving the rate-distortion performance of image compression, and providing a new idea for the design of end-to-end image compression methods.
[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:
[0007] An end-to-end image compression method based on context clustering transformation, comprising:
[0008] S1: Get the image to be compressed;
[0009] S2: Input the image to be compressed into the trained image compression model and output the reconstructed image;
[0010] The training process of the image compression model is as follows:
[0011] S201: Using the image x as a training sample as the input of the image compression model;
[0012] S202: Through the main encoder g a Encode image x: First, convert image x into feature points with coordinates, then divide the feature points into multiple clusters according to similarity, and finally learn image features by multiple aggregation and redistribution of feature points in each cluster to generate potential representation y;
[0013] S203: quantize the potential representation y through the quantizer Q to generate a discrete potential representation
[0014] S204: Capture the redundancy between potential representations y through the super prior network to generate prior features ψ;
[0015] S205: Taking the prior feature ψ as the potential representation to be encoded Prior information of
[0016] S206: Potential representation for encoding via entropy encoder Perform entropy encoding and entropy decoding to generate entropy value feature map
[0017] S207: Through the main decoder g s Entropy value feature map Perform decoding and reconstruction to generate a reconstructed image At the same time, the decoded channel group in the entropy feature map y and spatial datum features As a potential representation to be encoded Prior information of
[0018] S208: Calculate model loss and optimize parameters of the image compression model;
[0019] S209: repeating steps S201 to S208 until the image compression model converges;
[0020] S3: Using the reconstructed image output by the image compression model as the image decoding and reconstruction result of the image to be compressed.
[0021] Preferably, the main encoder g a It includes a coding embedding layer and a plurality of context clustering modules connected in sequence, wherein a feature point reduction module is arranged between two adjacent context clustering modules;
[0022] The encoding embedding layer is used to convert the input image into a feature map t1 containing coordinate points;
[0023] The context clustering module is used to perform multiple aggregation and redistribution operations on similar feature points in the feature map t1 to extract image features and obtain the feature map t2;
[0024] The feature point reduction module is used to downsample the feature map and increase the number of channels.
[0025] Preferably, the processing steps of the context clustering module are as follows:
[0026] S2021: Divide all feature points in feature map t1 into several clusters according to similarity, and record the pairwise cosine similarity matrix s between each feature point and the central feature i ;
[0027] S2022: Map feature points to value space. For feature points in the same cluster, map them to the value space according to their relationship with the central feature v. c Aggregate the similarity of to generate the aggregate feature d;
[0028] The formula is described as:
[0029]
[0030] Where: s i Represents the i-th feature point v in the cluster in the value space i and the central feature v c cosine similarity matrix; α and β represent learnable parameters; sig represents the sigmoid function; m represents a feature point v in a cluster i The number of; C represents the normalization factor;
[0031] S2023: Adaptively assign the aggregated feature d to each feature point p in the feature map t1 according to similarity i , and then restore each feature point p by matching the feature dimensions through the fully connected layer i , generate feature point p i ′;
[0032] The formula is described as:
[0033] p i '=p i +FC(sig(αs i +β)·d);
[0034] Where: s i is the i-th feature point v i and the central feature v cThe cosine similarity matrix of ; α and β represent learnable parameters; sig is the sigmoid function; FC represents the fully connected layer;
[0035] S2024: Perform channel interaction on the restored feature map through a multi-layer perceptron module to generate a feature map t2.
[0036] Preferably, for feature map t1, c centers are uniformly selected in the linear projection space of several feature points, and the features of their k neighboring feature points are averaged as the central features; then, by calculating the pairwise cosine similarity matrix between each feature point and the central feature, each feature point is assigned to the most similar center, thereby obtaining c clusters.
[0037] Preferably, the multilayer perceptron module includes two fully connected layers and an activation function connected between the two fully connected layers.
[0038] Preferably, the processing steps of the super prior network are as follows:
[0039] S2041: Encode the latent representation y through a super prior encoder to generate a latent representation z;
[0040] S2042: quantize the potential representation z through the quantizer Q to generate a discrete potential representation
[0041] S2043: Latent Representation via Super Prior Decoder Perform decoding processing to generate prior features ψ.
[0042] Preferably, the super prior encoder includes three convolutional layers connected in sequence, and a ReLu activation function is set between adjacent convolutional layers; wherein the convolution kernel size of the first convolutional layer is 3×3, and the step size is 1; the convolution kernel size of the second and third convolutional layers is 5×5, and the step size is 2;
[0043] The super-prior decoder includes two deconvolution layers and one convolution layer connected in sequence; ReLu activation functions are set between adjacent deconvolution layers and between deconvolution layers and convolution layers; the convolution kernel size of the first and second deconvolution layers is 5×5 and the step size is 2; the convolution kernel size of the third convolution layer is 3×3 and the step size is 1.
[0044] Preferably, the entropy model adopts an entropy model based on SCCTX;
[0045] The prior features ψ generated by the super prior network and the decoded channel groups in the entropy feature map y and spatial datum features As a potential representation to be encoded The entropy model is then used to encode each potential representation to be encoded based on the prior information. Gaussian modeling is performed to obtain a Gaussian probability model (μ, σ); the entropy encoder realizes the potential representation to be encoded based on the corresponding Gaussian probability model (μ, σ) Entropy encoding and decoding;
[0046] The formula is:
[0047]
[0048]
[0049]
[0050] Where: Represents the value based on ψ and Generated parameters; g cc (k) Represents the channel context model; Indicates the decoded channel group ψ represents the prior feature; represents the parameters generated in the spatial dimension; g sc (k) Represents a checkerboard parallel context model; represents the decoded spatial reference feature; φ k represents the entropy parameter for each potential representation Generate the mean μ and variance σ of the corresponding Gaussian probability model; g ep (k) Represents a network that generates entropy parameters.
[0051] Preferably, g cc (k) and g ep (k) Each of them includes three convolutional layers connected in sequence, and a ReLu activation function is set between adjacent convolutional layers; cc (k) The convolution kernel size of the first and second convolution layers is 5×5, and the convolution kernel size of the third convolution layer is 3×3; g ep (k) The convolution kernel sizes of the three convolutional layers are all 1×1.
[0052] Preferably, the model loss is calculated by the following formula:
[0053]
[0054] Where: L represents the model loss; R represents the potential representation and The bit rate generated; D represents the MSE mean square error between the reconstructed image and the input image; p xRepresents the probability distribution of the input image; and For potential and The probability distribution of the input image x and the reconstructed image The difference between λ and bit rate is λ.
[0055] Compared with the prior art, the end-to-end image compression method based on context clustering transformation in the present invention has the following beneficial effects:
[0056] The main encoder of the compressed image model in the present invention first converts the image into feature points with coordinates during encoding, then divides the feature points into multiple clusters according to similarity, and finally learns the image features by aggregating and redistributing the feature points in each cluster multiple times, which fully considers the correlation between adjacent pixels of the image, can more effectively eliminate the redundant information between image data, and improves the problem of insufficient interaction of local similar features of the image in the existing end-to-end image compression method based on CNN transformation, thereby improving the performance of the image compression model and ensuring the quality of end-to-end image compression. At the same time, the present invention captures the redundancy between potential representations through a super-prior network to generate prior features, and then uses the prior features, the decoded channel group and the spatial reference features in the entropy feature map as the prior information of the potential representation to be encoded, and improves the prediction probability of the entropy model for the potential representation to be encoded through these prior information, thereby further improving the quality of end-to-end image compression. In summary, the method of the present invention fully considers the correlation between adjacent pixels of the image, improves the problem of insufficient interaction of local similar features of the image in the end-to-end image compression method based on CNN transformation, thereby ensuring the quality of end-to-end image compression and improving the rate-distortion performance of image compression, providing a new idea for the design of end-to-end image compression methods. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:
[0058] Figure 1 This is the overall framework diagram of the image compression model;
[0059] Figure 2 This is the schematic diagram of the context clustering module;
[0060] Figure 3 (a) is the structural diagram of SCCTX. Figure 3 (b) is g cc (k) , g ep (k) Schematic diagram of the structure;
[0061] Figure 4 The rate-distortion performance evaluation diagram on the Kodak (a) and CLIC (b) datasets;
[0062] Figure 5 Visualization of reconstructed images for the Kodak dataset (kodim07.png, kodim23.png);
[0063] Figure 6 Visualization of the maximum entropy channel;
[0064] Figure 7 This is the rate-distortion performance evaluation diagram of CCT and CNT on the Kodak dataset. DETAILED DESCRIPTION
[0065] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.
[0066] It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. In the description of the present invention, it should be noted that the orientation or position relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, or the orientation or position relationship in which the invention product is usually placed when used, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance. In addition, the terms "horizontal", "vertical", etc. do not mean that the components are required to be absolutely horizontal or suspended, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted. In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0067] The following is a further detailed description through specific implementation methods:
[0068] Example:
[0069] This embodiment discloses an end-to-end image compression method based on context clustering transformation.
[0070] An end-to-end image compression method based on context clustering transformation, comprising:
[0071] S1: Get the image to be compressed;
[0072] S2: Input the image to be compressed into the trained image compression model (Context Cluster Transform, CCT) and output the reconstructed image;
[0073] Combination Figure 1 As shown in Figure 1, the training process of the image compression model is as follows:
[0074] S201: Using the image x as a training sample as the input of the image compression model;
[0075] In this embodiment, 30,000 images are randomly selected from the existing OpenImages dataset as a training sample set. In the dataset preprocessing stage, the images are cropped into samples with a size of 256×256.
[0076] S202: Through the main encoder g a Encode image x: First, convert image x into feature points with coordinates, then divide the feature points into multiple clusters according to similarity, and finally learn image features by multiple aggregation and redistribution of feature points in each cluster to generate potential representation y;
[0077] The formula is described as: y = g a (x);
[0078] S203: quantize the potential representation y through the quantizer Q to generate a discrete potential representation
[0079] The formula is described as:
[0080] S204: Capture the redundancy between potential representations y through the super prior network to generate prior features ψ;
[0081] S205: Taking the prior feature ψ as the potential representation to be encoded Prior information of
[0082] S206: Potential representation for encoding via entropy encoder Perform entropy encoding and entropy decoding to generate entropy value feature map
[0083] In this embodiment, the entropy model completes the potential representation of each to-be-encoded (without entropy encoding and decoding) ) The probability modeling is based on the output of the corresponding Gaussian probability model (μ, σ). At this time, the entropy encoder receives the potential representation to be encoded. The probability of , and then complete the entropy encoding and decoding (AE-AD).
[0084] S207: Through the main decoder g s Entropy value feature map Perform decoding and reconstruction to generate a reconstructed image (i.e. the compressed image); at the same time, the decoded channel group in the entropy feature map y and spatial datum features As a potential representation to be encoded Prior information of
[0085] The formula is described as:
[0086] S208: Calculate model loss and optimize parameters of the image compression model;
[0087] S209: repeating steps S201 to S208 until the image compression model converges;
[0088] S3: Using the reconstructed image output by the image compression model as the image decoding and reconstruction (compression) result of the image to be compressed.
[0089] The main encoder of the compressed image model in the present invention first converts the image into feature points with coordinates during encoding, then divides the feature points into multiple clusters according to similarity, and finally learns the image features by aggregating and redistributing the feature points in each cluster multiple times, which fully considers the correlation between adjacent pixels of the image, can more effectively eliminate the redundant information between image data, and improves the problem of insufficient interaction of local similar features of the image in the existing end-to-end image compression method based on CNN transformation, thereby improving the performance of the image compression model and ensuring the quality of end-to-end image compression. At the same time, the present invention captures the redundancy between potential representations through a super-prior network to generate prior features, and then uses the prior features, the decoded channel group and the spatial reference features in the entropy feature map as the prior information of the potential representation to be encoded, and improves the prediction probability of the entropy model for the potential representation to be encoded through these prior information, thereby further improving the quality of end-to-end image compression. In summary, the method of the present invention fully considers the correlation between adjacent pixels of the image, improves the problem of insufficient interaction of local similar features of the image in the end-to-end image compression method based on CNN transformation, thereby ensuring the quality of end-to-end image compression and improving the rate-distortion performance of image compression, providing a new idea for the design of end-to-end image compression methods.
[0090] In the specific implementation process, inspired by the idea of context clustering, the present invention considers the image to be compressed as a number of points with coordinates. By continuously aggregating and redistributing the feature points in the same cluster, the image features can be extracted to obtain a more compact image feature expression. Therefore, the present invention designs a transformation network based on context clustering (g a ,g s ) module to solve the problem of insufficient interaction of local similar features of images in the current CNN transformation network.
[0091] Main encoder g aIt includes a layer of encoding embedding (Encode Embedding) and several (four) context clustering modules (Context Cluster Module, CCM) connected in sequence, where a feature point reduction module (Points Reducer Module, PRM) is set between two adjacent context clustering modules. It extracts image features by performing multiple aggregation and redistribution operations on similar feature points in the input image x to fully utilize the correlation between adjacent pixels of the image. Main decoder g s It includes a feature point augmentation module (Points Increase Module, PIM), several (four) context clustering modules and decoding embedding (Decode Embedding), which are used to gradually restore feature points to decode and generate more accurate reconstructed images.
[0092] The encoding embedding layer is used to convert the input image into a feature map t1 containing coordinate points. The encoding embedding layer also contains a feature point reduction module. The context clustering module is used to perform multiple aggregation and redistribution operations on similar feature points in the feature map t1 to extract image features, achieve a more compact feature expression, and obtain the feature map t2. The feature point reduction module is used to downsample the feature map and increase the number of channels: the function of downsampling is to gradually reduce the size of the feature map and increase the receptive field of the model, so that the model has the ability to learn non-local information, which is conducive to CCM capturing richer feature information in the global dimension. At the same time, downsampling can effectively reduce the computational complexity of the model, reduce the points required for aggregation and redistribution of CCM, and reduce the training and inference time of the model. In addition, PRM gradually increases the number of channels, which can make the model have stronger feature expression capabilities and help the model capture more complex feature information.
[0093] Combination Figure 2 As shown in Figure 1, the context clustering module consists of a context cluster block (CCB) and a multi-layer perceptron block (MLPB). The processing steps of the context clustering module are as follows:
[0094] S2021: Divide all feature points in feature map t1 into several clusters according to similarity, and record the pairwise cosine similarity matrix s between each feature point and the central feature i ;
[0095] In this embodiment, for the feature map t1, c centers are uniformly selected in the linear projection space of several feature points, and the features of their k neighboring feature points are averaged as the central features; then, by calculating the pairwise cosine similarity matrix between each feature point and the central feature, each feature point is assigned to the most similar center, thereby obtaining c clusters.
[0096] S2022: Map feature points to value space. For feature points in the same cluster, map them to the value space according to their relationship with the central feature v. c Aggregate the similarity of to generate the aggregate feature d;
[0097] The formula is described as:
[0098]
[0099] Where: s i Represents the i-th feature point v in the cluster in the value space i and the central feature v c The cosine similarity matrix (in the value space, the central feature is defined as v c , the other feature points in the cluster are defined as v i ), α and β represent learnable parameters, which are s i The scaling and movement factors; sig represents the sigmoid function; m represents a feature point v in a cluster i The number of; C represents the normalization factor;
[0100] S2023: Adaptively assign the aggregated feature d to each feature point p in the feature map t1 according to similarity i This allocation method can realize the interaction between feature points, and each feature point can share the features of all points in the cluster. Then, the dimension of the feature is matched through the fully connected (FC) layer to restore each feature point p i , generate feature point p i ′;
[0101] The formula is described as:
[0102] p i '=p i +FC(sig(αs i +β)·d);
[0103] Where: s i is the i-th feature point v i and the central feature v c The cosine similarity matrix of ; α and β represent learnable parameters, which are s i The scaling and shifting factors; sig is the sigmoid function; FC represents the fully connected layer;
[0104] S2024: Perform channel interaction on the restored feature map through the multi-layer perceptron module to obtain a compact feature expression and generate a feature map t2. In this embodiment, the multi-layer perceptron module includes two fully connected layers and an activation function connected between the two fully connected layers.
[0105] In the present invention, during encoding, the image is first converted into feature points with coordinates, and then the feature points are divided into multiple clusters according to similarity. Finally, the image features are learned by performing multiple aggregation and redistribution of the feature points in each cluster. The correlation between adjacent pixels of the image is fully considered, and the redundant information between image data can be more effectively eliminated. The problem of insufficient interaction of local similar features of the image in the existing end-to-end image compression method based on CNN transformation is improved, thereby improving the performance of the image compression model and ensuring the quality of end-to-end image compression.
[0106] Combination Figure 1 As shown, the processing steps of the super prior network are as follows:
[0107] S2041: Through the super prior encoder h a Encode the potential representation y to generate the potential representation z;
[0108] The formula is described as: z = h a (y);
[0109] S2042: quantize the potential representation z through the quantizer Q to generate a discrete potential representation
[0110] The formula is described as:
[0111] S2043: Through the super prior decoder h s Potential Perform decoding processing to generate prior features ψ.
[0112] The formula is described as:
[0113] Specifically, the super-a priori encoder includes three convolutional layers connected in sequence, and a ReLu activation function is set between adjacent convolutional layers; the convolution kernel size of the first convolutional layer is 3×3, and the step size is 1; the convolution kernel size of the second and third convolutional layers is 5×5, and the step size is 2; the super-a priori decoder includes two deconvolutional layers and one convolutional layer connected in sequence; ReLu activation functions are set between adjacent deconvolutional layers and between deconvolutional layers and convolutional layers; the convolution kernel size of the first and second deconvolutional layers is 5×5, and the step size is 2; the convolution kernel size of the third convolutional layer is 3×3, and the step size is 1.
[0114] The present invention captures the redundancy between potential representations through a hyper-prior network to generate prior features, and then uses the prior features and the benchmark features of the channel group and space decoded by the main decoder as prior information of the entropy model. The prior information is used to improve the prediction probability of the entropy model for the potential representation to be encoded, thereby further improving the quality of end-to-end image compression.
[0115] In the specific implementation process, the entropy model uses a Gaussian probability model to model each potential representation for bit rate estimation and entropy coding. In order to better balance the network rate distortion performance and speed, in the entropy model part, in order to better balance the network rate distortion performance and speed, the entropy model of the present invention adopts an entropy model based on non-uniform grouping space-channel joint context (Space-Channel Context, SCCTX), which converts the potential representation It is divided into five channel groups with 16, 16, 32, 64, and 192 channels. The (μ, σ) Gaussian probability model is used to represent each potential Perform rate estimation and entropy coding.
[0116] Combination Figure 3 As shown in (a), the prior features ψ generated by the super prior network and the decoded channel group in the entropy feature map y and spatial datum features As a potential representation to be encoded The entropy model is then used to encode each potential representation to be encoded based on the prior information. Gaussian modeling is performed to obtain a Gaussian probability model (μ, σ); the entropy encoder realizes the potential representation to be encoded based on the corresponding Gaussian probability model (μ, σ) Entropy encoding and decoding;
[0117] The formula is:
[0118]
[0119]
[0120]
[0121] Where: Represents the value based on ψ and Generated parameters; g cc (k) Represents the channel context model; Indicates the decoded channel group ψ represents the prior feature; represents the parameters generated in the spatial dimension; g sc (k) Representation checkerboard parallel context model (checkerboard parallel context model is a latent representation to be encoded The context model is divided into anchor and non-anchor in a checkerboard manner. When encoding, only one encoding is required, while when decoding, two decodings are required. The anchor uses the prior information ψ and the decoded channel group For encoding and decoding, non-anchor can not only use the prior information ψ information and the decoded channel group The decoded anchor can also be encoded as contextual prior information); represents the decoded spatial reference feature; φ k represents the entropy parameter for each potential representation Generate the mean μ and variance σ of the corresponding Gaussian probability model; g ep (k) Represents a network that generates entropy parameters.
[0122] Combined with the above formula, as shown in formula 1, there is no decoded channel group at the beginning. At this time, the entropy parameter φ is obtained by combining the prior feature ψ with formulas 2 and 3 k , that is, the probability distribution of half of the features of the first channel group is obtained, and then the entropy encoder is used to decode half of the features of channel group 1, which is called the baseline feature Then, this benchmark feature is used as spatial context information and the spatial context model is used to sc (k) Modeling is shown in Formula 2. Then the distribution of the other half of the features is obtained by Formula 3, that is, the other half of the features are decoded by using the entropy encoder, and the decoding of the first channel group is completed.
[0123] Next, the second channel group is decoded, where the first channel group can be used as channel context information to input the channel context model. Similarly, the baseline features of the second channel group are decoded, and then the channel context model, spatial context model, and super prior information are simultaneously applied to model the other half of the features, and the spatial features of the other half of the second channel group are decoded, and so on.
[0124] Combination Figure 3 As shown in (b), g cc (k) and g ep (k) Each of them includes three convolutional layers connected in sequence, and a ReLu activation function is set between adjacent convolutional layers; cc (k) The convolution kernel size of the first and second convolution layers is 5×5, and the convolution kernel size of the third convolution layer is 3×3; g ep (k) The convolution kernel sizes of the three convolutional layers are all 1×1.
[0125] The present invention performs entropy encoding and entropy decoding on a potential representation through entropy coding to generate an entropy value feature map, which is then decoded by a main decoder to obtain a reconstructed image. By performing entropy encoding and entropy decoding on the potential representation, the uncertainty and redundancy of image feature data can be reduced, thereby achieving efficient compression of the image.
[0126] During the specific implementation, the model loss is calculated by the following formula:
[0127]
[0128] Where: L represents the model loss; R represents the potential representation and The resulting bit rate (or bit rate); D represents the MSE mean square error between the reconstructed image and the input image; p x Represents the probability distribution of the input image; and For potential and The probability distribution of the input image x and the reconstructed image λ represents the balance factor between the bit rate and distortion of the reconstructed image. Generally speaking, the larger the λ value, the higher the quality of the reconstructed image, but the larger the bit rate occupied.
[0129] In the loss function of the present invention, D is made as small as possible, that is, the original input image and the reconstructed image are ensured to be as close as possible, and R is also made as small as possible to ensure that the transmitted code stream is small, thereby further improving the performance of the image compression model and ensuring the quality of end-to-end image compression.
[0130] It should be noted that the model with the best performance evaluation is selected in the verification stage of the image compression model for testing the performance of the CCT method. The Peak Signal-to-Noise Ratio (PSNR) indicator is used to evaluate the quality of the reconstructed image, the Bit-rate is used to measure the bit rate required for entropy coding, and the BD-rate is used to evaluate the bit rate savings of each method compared with the benchmark algorithm at the same quality.
[0131] In order to better illustrate the advantages of the technical solution of the present invention, this embodiment discloses the following experiment.
[0132] 1. Training
[0133] In this experiment, 30,000 images were randomly selected from the OpenImages dataset as the training set. In the dataset preprocessing stage, the images were cropped into samples of size 256×256. The batch size of each training iteration was set to 16, and the Adam optimizer was used to train the model. All models were trained on a GPU model of 4090, starting with 1×10-4 The training was performed for 400 rounds with a learning rate of 1×10 -5 The learning rate is 0.01 and trained for 100 epochs.
[0134] 2. Model setup
[0135] In this experiment, the mean square error (MSE) is used as the image quality evaluation parameter of the optimization model. The λ value in the loss function L = R + λ·D is set to {0.0018, 0.0035, 0.0067, 0.0130, 0.025, 0.0483}. The potential representations of the main codec and the super prior part of all models are set to 320 and 192 channels respectively. During model training, each [y-μ] is encoded into a bit stream, and each symbol is restored using [y-μ] + μ. In the main encoder, the four PRM modules are set to 32, 64, 196, and 320 channels respectively, and are set to 2 times spatial downsampling. The feature extraction modules of the four stages are stacked by 2, 2, 6, and 2 CCMs respectively.
[0136] 3. Testing
[0137] This experiment selected two professional datasets in the field of image compression to complete the rate-distortion performance test, including the Kodak dataset with 24 images at a resolution of 768×512. The CLIC dataset contains 41 high-resolution images. The Peak Signal-to-Noise Ratio (PSNR) is used to evaluate the quality of the reconstructed image, and the Bit-rate measures the bit rate required for entropy coding. The BD-rate evaluates the bit rate savings of each method compared with the baseline algorithm at the same quality.
[0138] 3. Rate-distortion performance evaluation
[0139] Figure 4 This is a rate-distortion performance evaluation chart of the method of the present invention on Kodak and CLIC datasets and the traditional encoding method BPG, end-to-end image compression based on deep learning Balle2018, Minnen2018, Cheng2020, Hu2021, Qian2022. Table 1 is based on BPG, and the BD-rate (%) saved by each method on the Kodak and CLIC datasets. Among them, in the encoding parameters of BPG, the color sampling mode is set to 444, the encoding mode is set to x265, and the bit depth is set to 8 to test the two datasets. Figure 4 All comparative literature data on the rate-distortion curves shown are from the corresponding official websites.
[0140] Figure 4The experimental results in Table 1 show that compared with the comparative method, the proposed method achieves the best rate-distortion performance on both the Kodak and CLIC datasets, and performs better on the CLIC dataset. Figure 4 It can be seen that the rate-distortion performance curve of CCT is at the top of all methods, which indicates that the method of the present invention requires the least number of bits when the quality of the reconstructed image is the same. As can be seen from Table 1, when BPG encoding is used as the benchmark, the BD-rate of CCT saves 19.16% and 25.31% respectively, which is higher than the comparison method on both data sets. Among them, compared with the end-to-end image compression method Cheng2020 based on the CNN transform network, the bitstream saved by CCT on the CLIC dataset is more significant. CLIC has a higher spatial resolution than Kodak, and therefore exhibits higher local correlation, making it easier to use image neighborhood correlation to obtain better compression quality. It can be seen that the strategy of using a contextual clustering transform network to learn image neighborhood correlation proposed in the present invention makes fuller use of the similarity characteristics between local features in high-resolution images, and can more effectively improve the rate-distortion performance of compressed images in the CLIC dataset. The method of the present invention is more suitable for the compression of high-resolution images.
[0141] Table 1 BD-rate saved by each method on Kodak and CLIC datasets based on BPG
[0142]
[0143]
[0144] 4. Subjective visual evaluation
[0145] Figure 5The reconstruction images of kodim07.png and kodim23.png by CCT and traditional compression methods BPG and VVC (VTM 6.0) intra-frame coding are shown. The corresponding images are reconstructed by decoding at an approximate bit rate, and the corresponding bit rate (Bit-rate) and peak signal-to-noise ratio (PSNR) quantitative index values are marked. From the overall perspective of the two groups of reconstructed images, CCT is clearer than other encoding methods, and the effect is closer to the original image. Its Bit-rate and PSNR are both optimal, reaching 0.1525bpp and 32.3665dB in the kodim07 image; and 0.0895bpp and 33.6107dB in the kodim23 image. From the wall and petals in the partial enlarged image, it can be found that CCT retains more detailed information than the reconstruction images of other methods, especially the angularity of the wall is stronger. From the black spots and red feathers of the parrot in the partial enlarged image of Kodim23, it can be found that CCT reconstructs the black spots of the parrot more prominently and clearly than other methods, and the feathers have a stronger sense of layering. It can be seen that the subjective visual quality of the reconstructed image of the method of the present invention is better, which further demonstrates the effectiveness of the proposed method in the field of end-to-end image compression.
[0146] 5. Ablation experiment
[0147] In order to explore the advantages and disadvantages of the CCT proposed in the present invention and the current end-to-end image compression method based on CNN transform in terms of learning features, the present invention replaces the main codec of CCT with a CNN transform network composed of convolutional layers and GDN layers, which is called CNT (Convolutional Neural Network Transform, CNT). Figure 6 The potential representation of the method and CNTs are shown using kodim20.png to visualize the present invention, respectively. The maximum entropy channel information of the first 10 channels. The experimental results show that the feature map of the proposed method presents a very clear fuselage outline, and its maximum entropy channel reflects relatively complete information, such as Figure 6 (b) The characteristic graph of the CNT method shows incomplete fuselage information and reflects less information, such as Figure 6 As shown in (c), it can be seen that compared with the CNT method, the preceding channel group of the method of the present invention provides more complete prior information, which helps to improve the accuracy of the bit rate estimation during the entropy coding of the following channel group, thereby reducing the bit rate occupied by the image.
[0148] In order to further verify the effectiveness of the method of the present invention, CCT and CNT were trained in the same way and tested on the Kodak and CLIC datasets. Figure 7The test results on the Kodak dataset are presented. The experimental results show that the proposed method is superior to CNT in rate-distortion performance, especially in the high bit rate range. Specifically, on the Kodak dataset, CCT saves 2.75% of BD-rate compared to CNT. Similarly, the test results on the CLIC dataset show that the method of the present invention saves 4.20% of BD-rate compared to CNT. The above experimental results show that compared to CNT composed of convolutional layers and GDN layers, the method of the present invention can further improve the rate-distortion performance of compressed images by enhancing the interaction of local similar features in the image.
[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the technical solution. Those skilled in the art should understand that those modifications or equivalent substitutions of the technical solution of the present invention that do not depart from the purpose and scope of the technical solution should be included in the scope of the claims of the present invention.
Claims
1. An end-to-end image compression method based on context clustering transform, characterized in that: include: S1: Get the image to be compressed; S2: Input the image to be compressed into the trained image compression model and output the reconstructed image; The training process of the image compression model is as follows: S201: Using the image x as a training sample as the input of the image compression model; S202: Through the main encoder g a Encode image x: First, convert image x into feature points with coordinates, then divide the feature points into multiple clusters according to similarity, and finally learn image features by multiple aggregation and redistribution of feature points in each cluster to generate potential representation y; In step S202, the main encoder g a It includes a coding embedding layer and a plurality of context clustering modules connected in sequence, wherein a feature point reduction module is arranged between two adjacent context clustering modules; The encoding embedding layer is used to convert the input image into a feature map t1 containing coordinate points; The context clustering module is used to perform multiple aggregation and redistribution operations on similar feature points in the feature map t1 to extract image features and obtain the feature map t2; The feature point reduction module is used to downsample the feature map and increase the number of channels; The processing steps of the context clustering module are as follows: S2021: Divide all feature points in feature map t1 into several clusters according to similarity, and record the pairwise cosine similarity matrix s between each feature point and the central feature i ; S2022: Map feature points to value space. For feature points in the same cluster, map them to the value space according to their relationship with the central feature v. c Aggregate the similarity of to generate the aggregate feature d; The formula is described as: Where: s i Represents the i-th feature point v in the cluster in the value space i and the central feature v c cosine similarity matrix; α and β represent learnable parameters; sig represents the sigmoid function; m represents a feature point v in a cluster i The number of; C represents the normalization factor; S2023: Adaptively assign the aggregated feature d to each feature point p in the feature map t1 according to similarity i , and then restore each feature point p by matching the feature dimensions through the fully connected layer i , generate feature point p i ′; The formula is described as: p′ i (p i +FC(sig(αs). i +β)·d) Where: s i is the i-th feature point v i and the central feature v c The cosine similarity matrix of ; α and β represent learnable parameters; sig is the sigmoid function; FC represents the fully connected layer; S2024: Perform channel interaction on the restored feature map through a multi-layer perceptron module to generate a feature map t2; S203: quantize the potential representation y through the quantizer Q to generate a discrete potential representation S204: Capture the redundancy between potential representations y through the super prior network to generate prior features Ψ; S205: Taking the prior feature Ψ as the potential representation to be encoded Prior information of S206: Potential representation for encoding via entropy encoder Perform entropy encoding and entropy decoding to generate entropy value feature map S207: Through the main decoder g s Entropy value feature map Perform decoding and reconstruction to generate a reconstructed image At the same time, the entropy feature map The decoded channel group in and spatial datum features As a potential representation to be encoded Prior information of S208: Calculate model loss and optimize parameters of the image compression model; S209: repeating steps S201 to S208 until the image compression model converges; S3: Using the reconstructed image output by the image compression model as the image decoding and reconstruction result of the image to be compressed.
2. The end-to-end image compression method based on context clustering transform according to claim 1, characterized in that: In step S2021, for feature map t1, c centers are uniformly selected in the linear projection space of several feature points, and the features of k neighboring feature points are averaged as the central feature; Then, by calculating the pairwise cosine similarity matrix between each feature point and the center feature, each feature point is assigned to the most similar center, thus obtaining c clusters.
3. The end-to-end image compression method based on context clustering transform according to claim 1, characterized in that: In step S2024, the multilayer perceptron module includes two fully connected layers and an activation function connected between the two fully connected layers.
4. The end-to-end image compression method based on context clustering transform according to claim 1, characterized in that: In step S204, the processing steps of the super prior network are as follows: S2041: Encode the latent representation y through a super prior encoder to generate a latent representation z; S2042: quantize the potential representation z through the quantizer Q to generate a discrete potential representation S2043: Latent Representation via Super Prior Decoder Perform decoding processing to generate prior features ψ.
5. The end-to-end image compression method based on context clustering transform according to claim 4, characterized in that: The super prior encoder consists of three convolutional layers connected in sequence, with ReLu activation functions set between adjacent convolutional layers; the convolution kernel size of the first convolutional layer is 3×3, with a step size of 1; the convolution kernel size of the second and third convolutional layers is 5×5, with a step size of 2; The super-prior decoder includes two deconvolution layers and one convolution layer connected in sequence; ReLu activation functions are set between adjacent deconvolution layers and between deconvolution layers and convolution layers; the convolution kernel size of the first and second deconvolution layers is 5×5 and the step size is 2; the convolution kernel size of the third convolution layer is 3×3 and the step size is 1.
6. The end-to-end image compression method based on context clustering transform according to claim 1, characterized in that: In step S205, the entropy model adopts the entropy model based on SCCTX; The prior feature ψ and entropy feature map generated by the super prior network The decoded channel group in and spatial datum features As a potential representation to be encoded The entropy model is then used to encode each potential representation to be encoded based on the prior information. Gaussian modeling is performed to obtain a Gaussian probability model (μ, σ); the entropy encoder realizes the potential representation to be encoded based on the corresponding Gaussian probability model (μ, σ) Entropy encoding and entropy decoding; The formula is: Where: Represents the value based on ψ and Generated parameters; g cc (k) Represents the channel context model; Indicates the decoded channel group ψ represents the prior feature; represents the parameters generated in the spatial dimension; g sc (k) Represents a checkerboard parallel context model; represents the decoded spatial reference feature; φ k represents the entropy parameter for each potential representation Generate the mean μ and variance σ of the corresponding Gaussian probability model; g ep (k) Represents a network that generates entropy parameters.
7. The end-to-end image compression method based on context clustering transform according to claim 6, characterized in that: g cc (k) and g ep (k) Each of them includes three convolutional layers connected in sequence, and a ReLu activation function is set between adjacent convolutional layers; cc (k) The convolution kernel size of the first and second convolution layers is 5×5, and the convolution kernel size of the third convolution layer is 3×3; g ep (k) The convolution kernel sizes of the three convolutional layers are all 1×1.
8. The end-to-end image compression method based on context clustering transform according to claim 4, characterized in that: In step S208, the model loss is calculated by the following formula: Where: L represents the model loss; R represents the potential representation and The bit rate generated; D represents the MSE mean square error between the reconstructed image and the input image; p x Represents the probability distribution of the input image; and For potential and The probability distribution of the input image x and the reconstructed image The difference between λ and bit rate is λ.
