Image compression error detection method and error-resilient image compression method and system

Abstract

The present invention provides an image compression error detection method, comprising: acquiring a training image data set; using an image compression model to extract a first latent feature representation of a training image, the first latent feature representation being a multi-channel latent feature representation for image compression; processing the first latent feature representation to obtain a stability measurement area for detecting an image compression error; and extracting a first latent feature representation of a test image, and comparing the first latent feature representation of the test image with the stability measurement area to obtain an image compression error detection result. On the basis of the image compression error detection method, an error-resilient image compression method and system are correspondingly provided. According to the present invention, the quality error and damage caused by neural network-based image compression can be efficiently detected, stable continuous image compression is efficiently realized, and the methods are applied to an actual image communication scenarios.

Description

Image compression error detection method and error-resistant image compression method and system Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image compression error detection method, and an error-resistant image compression method and system based on the image compression error detection method. Background Art

[0002] In recent years, the rate-distortion performance of learnable image compression methods has far surpassed that of traditional image compression methods, garnering widespread attention. Unlike traditional image compression methods, learnable image compression models utilize neural networks to implement a nonlinear mapping between the input image and its latent representation (latent variable), and jointly optimize the quantization and entropy coding of the latent representation, as well as the nonlinear mapping. However, theoretical analysis of learnable image compression methods is currently insufficient, thus failing to ensure the credibility of learnable image compression models and limiting their practical application. Variational inference is currently a mainstream approach to explaining learnable image compression methods. It approximates the conditional posterior of the latent representation by parameterizing the variational density, while end-to-end optimization of the encoder and decoder is achieved through variational autoencoders. In addition to variational inference, optimal quantization and principal component analysis can also explain the superiority of learnable image compression models. Ballé et al., in their paper "Nonlinear transform coding," published in the 2021 IEEE Journal of Selected Topics in Signal Processing, summarized current learnable image compression models from the perspective of nonlinear transform coding and demonstrated that nonlinear transforms can achieve nonlinear optimal quantization based on uniform quantization to achieve superior rate-distortion performance. Bhadane et al., in their paper "Principal bit analysis: Autoencoding with schur-concave loss," published at the 2021 International Conference on Machine Learning (ICML), demonstrated that principal component decomposition can minimize rate-distortion performance loss for linear autoencoders with independent scalar quantization. However, because learnable image compression models employ nonlinear transforms, principal component analysis can lead to performance degradation in practice. Duan et al., in their paper "Opening the black box of learned image coders," published in 2022, visualized the different channels of latent feature representations and proposed hypotheses, but these were not fully verified.

[0003] While the aforementioned methods provide theoretical explanations, effective and reasonable tools for understanding and interpreting learnable image compression models are still lacking. Therefore, theoretical explanations of learnable image compression models are difficult to connect with practical applications. In real-world image communication scenarios, continuous image compression is very common. Because different users continuously receive and send the same image, the image is continuously compressed. However, when the same learnable image compression model is used to repeatedly compress and decompress the same image, image stability cannot be guaranteed, and the reconstructed image may be corrupted. However, existing methods, such as "Successive learned image compression: comprehensive analysis of instability" published by Kim et al. in the journal Neurocomputing in 2022, do not achieve efficient corruption detection, lack corresponding measurement metrics, and require retraining the learnable image compression model. Kim et al.'s paper proposed using a convolutional neural network to predict corrupted images based on the original image and the first compressed image, demonstrating that instabilities are predictable. However, the corrupted images used for training and testing were labeled using manually set thresholds, which may have overlooked some corrupted images.

[0004] Summary of the Invention

[0005] In view of the deficiencies in the prior art, the present invention provides an image compression error detection method and an error-resistant image compression method and system, which can efficiently realize stable and continuous image compression without the need for training.

[0006] A first aspect of the present invention provides an image compression error detection method, comprising:

[0007] Obtain a training image dataset;

[0008] extracting a first latent feature representation of the training image using an image compression model, where the first latent feature representation is a multi-channel latent feature representation for image compression;

[0009] processing the first latent feature representation to obtain a stability metric region for detecting image compression errors;

[0010] A multi-channel latent feature of a test image, that is, a first latent feature representation of the test image, is extracted, and the multi-channel latent feature representation is compared with the stability measurement region to obtain an image compression error detection result.

[0011] Optionally, the stability measure is the allowable region of each channel of the first latent feature representation, and the allowable region is the numerical range of the multi-channel latent feature representation of any natural image obtained through the image compression model; or, the stability measure is the in-range region of each channel of the first latent feature representation, and the in-range region is the numerical range of the multi-channel latent feature representation of the training image obtained through the image compression model.

[0012] Optionally, the allowable region is obtained by the following method:

[0013] Repeat the following steps for each channel of the first latent feature representation:

[0014] Randomly generate an RGB three-channel image of a set size and input it into the encoder of the image compression model, where each pixel value in the three-channel image is sampled from a normal distribution and normalized to a value in the range of 0 to 1;

[0015] With the goal of minimizing the value of the latent feature representation element, the input three-channel image is updated by gradient descent, and the element value of any channel in the traversal process of the encoder output latent feature representation is minimized as the lower bound of the allowed region;

[0016] With the goal of maximizing the value of the latent feature representation element, the input three-channel image is updated by gradient ascent, and the element value of any channel in the process of traversing the encoder output latent feature representation is maximized as the upper bound of the allowed region;

[0017] The interval range formed by the upper limit of the allowable area and the lower limit of the allowable area is the allowable area.

[0018] Optionally, the area within the range is obtained by the following method:

[0019] Slide to obtain image blocks of a set size from all images in the training image dataset, with a sliding step size of 1;

[0020] Passing all the acquired image blocks through the encoder of the image compression model, recording the element value of each channel of the latent feature representation output by the encoder, and obtaining each multi-channel latent feature representation;

[0021] For any c-th channel:

[0022] Obtain the statistical maximum value of the element values ​​of all image blocks in the c-th channel;

[0023] Obtain the statistical minimum value of the element values ​​of all image blocks in the c-th channel;

[0024] The closed interval defined by the statistical maximum value and the statistical minimum value is the area within the range of the c-th channel.

[0025] Optionally, the comparing the multi-channel latent feature representation with the stability metric region to obtain an image compression error detection result includes:

[0026] If the multi-channel latent feature is represented within the stability measurement region, the image compression error detection result is a stable image;

[0027] If the multi-channel latent feature representation exceeds the stability measurement region, the image compression error detection result is that image damage exists, and the damaged channel is output.

[0028] A second aspect of the present invention provides an error-resistant image compression method, comprising:

[0029] An image compression error detection method is used to perform image compression error detection to obtain a damaged channel;

[0030] Calculating a stability constraint region of the first latent feature representation, wherein the stability constraint region is a region where a stable reconstructed image is obtained after the image compression model is applied;

[0031] Adjusting the value range of the damaged channel according to the stability constraint region to obtain a second latent feature representation, and encoding the second latent feature representation to obtain a binary code stream;

[0032] The binary code stream is decoded to obtain a second latent feature representation, and the second latent feature representation is passed through a decoder of the image compression model to obtain a reconstructed image.

[0033] Optionally, calculating the stability constraint region of the first latent feature representation includes:

[0034] For any c-th channel, calculate the upper bound of the area within the range and the lower bound The difference between c :

[0035] Get the maximum value η of the channel importance metric of all C channels max :

[0036] For any c-th channel, according to the channel importance metric η c and the maximum value of the channel importance metric η max Calculate the stability constraint region ratio r c :

[0037] According to the ratio r of the stability constraint region c and the upper bound of the region within and the lower bound Calculate the upper bound a of the stability constraint region c and the lower bound b c :

[0038] Optionally, adjusting a value range of the damaged channel according to the stability constraint region to obtain a second latent feature representation, and encoding the second latent feature representation to obtain a binary code stream includes:

[0039] Detect and adjust all C channels of the first latent feature representation, for any c-th channel, where c is a positive integer not greater than C:

[0040] The image compression error detection method outputs the range region and stability constraint region of the cth channel, where the range region is a real number closed interval Stability constraint region [b c ,a c ] is the area within the range The closed subinterval of ;

[0041] Determine the value y of any element in the cth channel c Are they all within the stated area? If not, it is considered as a damaged channel;

[0042] If the cth channel is a damaged channel, compress the element value of the cth channel into the range of the stability constraint region, and for any element y c , perform the following operations:

[0043] y c =min(max(y c ,a c ),b c )

[0044] Where max(m,n) means taking the larger value of m and n, and min(m,n) means taking the smaller value of m and n;

[0045] If the cth channel is a damaged channel, whether to adjust the cth channel is determined based on the rate-distortion loss of the compressed cth channel compared to the original cth channel;

[0046] If it is determined to adjust the cth channel, the compressed cth channel will replace the original cth channel, otherwise the original cth channel will remain unchanged;

[0047] The adjusted multi-channel latent feature representation is output as the second latent feature representation and encoded to obtain a binary code stream.

[0048] Optionally, if the c-th channel is a damaged channel, determining whether to adjust the c-th channel according to a rate-distortion loss of the compressed c-th channel compared to an original c-th channel includes:

[0049] Encode the original c-th channel and the compressed c-th channel respectively, and record their bit rates as and Get the bitrate difference

[0050] Calculate the element-wise mean square error (MSE) between the compressed c-th channel and the original c-th channel as the distortion difference Δ c (D);

[0051] Calculate and sort all C channels from high to low according to the bit rate difference and distortion difference, and keep the first K channels unchanged, where K is a preset constant;

[0052] Determine whether the cth channel is adjusted and output the determination result.

[0053] A third aspect of the present invention provides an error-resistant image compression system, comprising:

[0054] An image compression error detection module, which uses an image compression error detection method to perform image compression error detection and obtain a damaged channel;

[0055] A stability constraint region calculation module, which calculates the stability constraint region of the first latent feature representation, wherein the stability constraint region is a region where a stable reconstructed image is obtained after the image compression model is applied;

[0056] a binary code stream encoding module, which adjusts the value range of the damaged channel according to the stability constraint region to obtain a second latent feature representation, and encodes the second latent feature representation to obtain a binary code stream;

[0057] A decoding module is configured to decode the binary code stream to obtain a second latent feature representation, and to pass the second latent feature representation through a decoder of the image compression model to obtain a reconstructed image.

[0058] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0059] The image compression error detection method in the embodiment of the present invention uses an image compression model to extract the first latent feature representation of the training image, and obtains a stability measurement area for detecting image compression errors by processing the first latent feature representation. This method can accurately realize image compression error detection and can be applied to error resistance of continuous image compression.

[0060] The image compression error detection method in the embodiment of the present invention uses an allowable area or an area within a range as a stability measurement area, and can directly output statistical values ​​based on the encoder output features of the image compression model (the maximum and minimum values ​​obtained by the encoder traversing the entire training set); except for the encoder and decoder of the image compression model, no additional computing resources such as neural networks are required.

[0061] The error-resistant image compression method and system in the embodiments of the present invention can significantly improve the stability of the neural network-based image coding model, avoid errors in continuous compression, and ensure image quality.

[0062] The error-resistant image compression method and system in the embodiments of the present invention can be directly used in a pre-trained image coding model without the need for re-training, which is impossible with existing methods.

[0063] Based on the aforementioned error-resistant image compression method and system, the embodiments of the present invention can achieve stable and continuous compression on natural images of varying resolutions acquired in various scenarios. The present invention has significant practical application value, particularly in the application of neural network-based image compression methods in actual image communications. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0065] FIG1 is a flow chart of an image compression error detection method according to an embodiment of the present invention.

[0066] FIG2 is a flow chart of obtaining the allowable region of each channel of the latent feature representation output by the encoding end channel by channel in one embodiment of the present invention.

[0067] FIG3 is a flow chart of obtaining the region within the range of each channel of the latent feature representation output by the encoder in one embodiment of the present invention.

[0068] FIG. 4 shows a method for comparing and obtaining image compression error detection results according to an embodiment of the present invention.

[0069] FIG5 is a flow chart of an error-resistant image compression method according to an embodiment of the present invention.

[0070] FIG6 is a flowchart of calculating the stability constraint region of the first latent feature representation according to an embodiment of the present invention.

[0071] FIG7 is a flow chart of adjusting damaged channel values ​​according to stability constraint regions in one embodiment of the present invention.

[0072] FIG8 is a flowchart of determining whether a latent feature representation channel needs to be adjusted in one embodiment of the present invention.

[0073] FIG9 is a diagram showing the effect of implementing the error-resistant image compression method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0074] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several variations and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0075] FIG1 is a flow chart of an image compression error detection method according to an embodiment of the present invention.

[0076] 1 , the image compression error detection method in this embodiment includes the following steps:

[0077] S100, obtaining a training image dataset;

[0078] In this step, the training image can be any natural image, and a large number of training images constitute a training image dataset. In this embodiment, the vimeo90k dataset can be used as the training image dataset.

[0079] S200, extracting a first latent feature representation of the training image using an image compression model, where the first latent feature representation is a multi-channel latent feature representation used for image compression;

[0080] In this step, the image compression model can be any image compression model in the prior art, for example, an image coding model based on a multi-layer neural network. The multi-layer neural network can be obtained by alternating convolutional layers and nonlinear activation layers, or by alternating self-attention layers and nonlinear activation layers. Of course, image coding models with other network structures are also possible, and the present invention is applicable to image coding models of different types of networks.

[0081] As a preferred embodiment, the following image compression model can be used in this embodiment: its encoder consists of four convolutional layers, the number of channels of the four convolutional layers are 128, 128, 128, and 192 respectively, and the first three convolutional layers are followed by a nonlinear unit GDN (generalized divisive normalization), the convolution kernel size of each convolutional layer is 5×5, and the convolution step size is 2. Therefore, when the input image size is H×W×3 (three-channel image), the size of the first latent feature representation obtained is

[0082] S300: Process the first latent feature representation to obtain a stability measurement region for detecting image compression errors.

[0083] In this step, the stability measurement area can be an allowable area or an area within a range, specifically:

[0084] The stability measurement region is the allowable region of each channel of the first latent feature representation, and the allowable region is the numerical range of the multi-channel latent feature representation obtained by the image compression model for any natural image.

[0085] The stability measurement region is the range of each channel of the first latent feature representation, and the range is the numerical range of the multi-channel latent feature representation obtained by the image compression model for the training image.

[0086] When applied to image compression, the allowed region or the region within the range can be used as a measure to measure the stability of the multi-channel latent feature representation of the test image obtained by the image compression model.

[0087] S400: extracting a first latent feature representation of a test image, comparing the first latent feature representation of the test image with a stability measurement region, and obtaining an image compression error detection result.

[0088] In this step, the test image is a compressed image. Multi-channel latent features, i.e., first latent feature representations, are extracted from the test image and compared with the aforementioned allowable region or within-range region to detect image damage. The multi-channel latent features of the test image are extracted using an image compression model that has the same or corresponding structure to the image compression model in S200.

[0089] The image compression error detection method of the above-described embodiment of the present invention can efficiently detect quality errors and damage caused by neural network-based image compression, effectively achieving stable and continuous image compression, and is suitable for practical image communication scenarios. Furthermore, it does not involve neural networks other than the image compression model and is universal, applicable to various pre-trained networks.

[0090] FIG2 is a flow chart of obtaining the allowable region of each channel of the first latent feature representation channel by channel in one embodiment of the present invention. Referring to FIG2 , the process of obtaining the allowable region in a preferred embodiment of the present invention is illustrated. In this preselected embodiment, obtaining the allowable region may specifically include the following steps:

[0091] Repeat the following steps for each channel of the first latent feature representation:

[0092] S310, for any channel of the first latent feature representation, randomly generate an RGB three-channel image of a set size, input the image into the encoder of the image compression model, and normalize the three-channel image so that the values ​​are in the range of 0 to 1;

[0093] In this step, a three-channel RGB image of a set size is randomly generated, where the image size can be set according to actual needs. In this embodiment, a three-channel input image of a size of 16×16 is randomly generated. Of course, in other embodiments, other sizes can also be used.

[0094] In this step, during the random generation of the three-channel image, each pixel value of the image is sampled from a normal distribution and can be normalized by a Sigmoid function so that the value is within the range of 0 to 1. Of course, in other embodiments, other normalization methods can also be used.

[0095] S320, with the goal of minimizing the value of the latent feature representation element, updating the input three-channel image by gradient descent, and minimizing the element value of any channel in the process of traversing the encoder output latent feature representation as the lower bound of the allowed region;

[0096] S330, with the goal of maximizing the value of the latent feature representation element, updating the input three-channel image by gradient ascent, and maximizing the element value of any channel in the process of traversing the encoder output latent feature representation as the upper bound of the allowed region;

[0097] S340: The interval range formed by the upper limit of the allowed area and the lower limit of the allowed area is the allowed area.

[0098] In this step, the allowable region describes the possible numerical range of the first latent feature representation obtained by the image compression model for any image. Therefore, the test image corresponding to the multi-channel latent feature that falls outside this numerical region can be considered unstable.

[0099] The method for calculating the allowable region in the above embodiment of the present invention does not require data set support and can estimate the stability region of multi-channel latent features under the premise of only having an image compression model;

[0100] S350, processing the next channel until all 192 channels are processed.

[0101] This embodiment uses the tolerance region obtained in the above steps to detect image compression errors. If the first latent feature representation of the test image exceeds this tolerance region, it indicates that the input image differs significantly from the training image, indicating the possibility of image corruption. Therefore, the relationship between the first latent feature representation of the input image obtained through the image compression model and the tolerance region can be used to determine whether the input image is corrupted.

[0102] In the embodiment of FIG. 2 , the specific image size is not limited to the above description, and may also be other sizes, which may be determined by the network structure of the adopted image compression model.

[0103] FIG3 is a flowchart of obtaining the first latent feature representation of the in-range region of each channel in one embodiment of the present invention. Referring to FIG3 , the process of obtaining the in-range region in a preferred embodiment is illustrated. In this preferred embodiment, obtaining the in-range region may specifically include the following steps:

[0104] S360, sliding to obtain image blocks of a set size from all images in the training image dataset, with a sliding step size of 1;

[0105] In this embodiment, obtaining an image block of a set size may be obtaining an image block of 16×16. Of course, in other embodiments, other sizes may also be obtained.

[0106] S370, passing all acquired image blocks through an encoder of an image compression model, and recording the element value of each channel of the first latent feature representation;

[0107] S380, for any c-th channel (1≤c≤192): obtain the statistical maximum value of the element values ​​of all image blocks in the c-th channel; obtain the statistical minimum value of the element values ​​of all image blocks in the c-th channel;

[0108] S390: The closed interval defined by the statistical maximum value and the statistical minimum value is the area within the range of the c-th channel.

[0109] In this step, the region within the range describes the possible value range of the first latent feature obtained by the image compression model for the training image. Multi-channel latent features falling outside this value range have corresponding test images with different distributions from the training images and are considered to be images that are different from natural images, that is, have unstable image content.

[0110] The method for calculating the area within the range in the above embodiment of the present invention uses a training data set and traverses various possibilities of image input, so it is more accurate than the allowable area.

[0111] This embodiment uses the region within the range obtained in the above steps to detect image compression errors. If the multi-channel latent features of the test image exceed this range, it indicates that the input image differs significantly from the training image, indicating the possibility of image corruption. Therefore, the relationship between the multi-channel latent feature representation of the input image obtained through the image compression model and the region within this range can be used to determine whether the input image is corrupted.

[0112] FIG4 shows a method for comparing and obtaining image compression error detection results in one embodiment of the present invention, wherein the image compression error detection results are obtained by comparing the first latent feature representation of the test image with the stability metric region. Specifically, in this preferred embodiment, the method may include:

[0113] S410, extracting a first latent feature representation of the test image, and comparing the first latent feature representation with a stability measurement region; the stability measurement region is either an allowable region or a region within a range;

[0114] The first latent feature representation (multi-channel latent feature) of the test image is extracted using an image compression model. The image compression model used is the same as the image compression model used in "Extracting the first latent feature representation of the training image using the image compression model" in S200. The first latent feature representation of the image is a tensor.

[0115] S420, if the first latent feature of the test image is represented within the stability measurement region, the image compression error detection result is a stable image;

[0116] S430: If the first latent feature representation of the test image exceeds the stability measurement region, the image compression error detection result is that image damage exists, and a damaged channel is output.

[0117] The image compression error detection method in the above-mentioned embodiment of the present invention has extremely strong practical application value, especially when applying a neural network-based image compression method in actual image communication. In actual image communication scenarios, images are transmitted between different users, which inevitably requires repeated compression of the image, that is, repeated encoding and decoding processes. In such actual application scenarios, current neural network-based image coding models produce severe reconstruction distortion as the number of repetitions increases. To address this problem, the present invention further provides an error-resistant image compression method and system. Based on the damaged image detected by the image compression error detection method in the above-mentioned embodiment, it is further compressed using the following image compression method. The error-resistant image compression method and system can significantly improve the image reconstruction quality in such actual scenarios, while significantly reducing the bit rate compared to currently widely used image compression standards such as JPEG and JPEG2000.

[0118] Specifically, the technical features of the error-resistant image compression method and system are described in detail below with reference to Figures 5-8.

[0119] FIG5 is a flow chart of an error-resistant image compression method according to an embodiment of the present invention.

[0120] 5 , this embodiment provides an error-resistant image compression method, which specifically includes the following steps:

[0121] S500, performing error detection on the image compression to obtain an image damage channel; wherein the error detection on the image compression can be performed using the image compression error detection method described in S100-S400;

[0122] S600, calculating the stability constraint region of the first latent feature representation, where the stability constraint region is the region where a stable reconstructed image is obtained after the image compression model is used; the first latent feature representation is the same as the image compression model in S200 in the image compression error detection method.

[0123] S700, adjusting the value range of the image damage channel obtained in S500 according to the stability constraint region to obtain a second latent feature representation, and encoding the second latent feature representation to obtain a binary code stream;

[0124] S800: Decode the binary code stream to obtain a second latent feature representation, and pass the second latent feature representation through a decoder of the image compression model to obtain a reconstructed image.

[0125] The error-resistant image compression method provided in this embodiment uses stability constraint regions to adjust detected damaged channels. This significantly improves the stability of the neural network-based image coding model, preventing errors during continuous compression and ensuring image quality. As described in the above steps, this image compression method can be directly applied to pre-trained image coding models without the need for retraining.

[0126] FIG6 is a flowchart of calculating the stability constraint region of the first latent feature representation according to an embodiment of the present invention.

[0127] 6 , in this preferred embodiment, calculating the stability constraint region of the first latent feature representation may specifically include the following steps:

[0128] S610, for any c-th channel, calculate the upper bound of the area within the range and the lower bound The difference between c :

[0129] S620, obtaining the maximum value η of the channel importance metric of all 192 channels represented by the first latent feature max :

[0130] S630, for any c-th channel, according to the channel importance metric η c and the maximum value of the channel importance metric η max Calculate the stability constraint region ratio r c :

[0131] S640, according to the ratio r of the stability constraint region c and the upper bound of the region within and the lower bound Calculate the upper bound a of the stability constraint region cand the lower bound b c :

[0132] In this step, the stability constraint region is a contraction of the range. The first latent feature representation that falls within this numerical region is numerically closer to the average value of the training image and is therefore considered stable. By constraining the unstable first latent feature representation to this region, the stability of the image reconstructed by the image compression model can be improved.

[0133] The method for calculating the stability constraint region in the above embodiment of the present invention only requires simple numerical calculations and does not require additional neural networks. Compared with existing methods, it has a small amount of calculation and is easy to deploy.

[0134] FIG7 is a method for adjusting damaged channel values ​​according to stability constraint regions in one embodiment of the present invention.

[0135] 7 , in this preferred embodiment, in order to better achieve error robustness in image compression, in S700, adjusting the value range of the image damage channel obtained according to the stability constraint region may specifically include the following steps:

[0136] S710, obtain the stability measurement area of ​​the c-th channel according to the image compression error detection method, and determine whether any element value y in the c-th channel c Are they all within the stability measurement area? If not, they are considered damaged channels.

[0137] In this step, the stability measurement area can be within the above range Or the stability constraint region mentioned above [b c ,a c ], the specific calculation method can refer to the description in the above embodiment.

[0138] S720: If the cth channel is a damaged channel, compress the element value of the cth channel into the range of the stability constraint region, and for any element y c , perform the following operations:

[0139] y c =min(max(y c ,a c ),b c )

[0140] Where max(m,n) means taking the larger value of m and n, and min(m,n) means taking the smaller value of m and n;

[0141] S730 , if the c th channel is a damaged channel, determine whether to adjust the c th channel according to the rate-distortion loss of the compressed c th channel compared to the original c th channel;

[0142] S740, if it is determined to adjust the cth channel, the compressed cth channel replaces the original cth channel; otherwise, the original cth channel remains unchanged;

[0143] S750: Output the adjusted multi-channel latent feature representation as the second latent feature representation, and further encode it to obtain a binary code stream.

[0144] FIG8 is a flowchart of determining whether a latent feature representation channel needs to be adjusted according to an embodiment of the present invention, wherein the method of determining whether to adjust the latent feature representation channel is based on the rate-distortion loss after compression.

[0145] 8 , in the above S730 , whether to adjust the c-th channel can be determined based on the rate-distortion loss after compression. Specifically, in a preferred embodiment, the following steps can be performed:

[0146] S731, encode the original c-th channel and the compressed c-th channel respectively, and record their bit rates as and Get the bitrate difference

[0147] S732, calculate the element-by-element mean square error (MSE) between the compressed c-th channel and the original c-th channel as the distortion difference Δ c (D);

[0148] S733, calculating and sorting all 192 channels from large to small according to the bit rate difference and the distortion difference, and keeping the first K channels unchanged, where K is a preset constant;

[0149] In this step, K / 2 channels are screened from large to small based on the bit rate difference and the distortion difference, and a total of K channels are obtained.

[0150] In this step, K can be selected according to the needs of actual application, for example, K can be 20.

[0151] S734: Determine whether the channel belongs to the K channels and output the determination result. If it belongs to the K channels, no adjustment is required; otherwise, adjustment is required.

[0152] By adopting the above method to determine whether the latent feature representation channel needs to be adjusted, during implementation, it is possible to effectively prevent the degradation of the image compression model's expressive power caused by unreasonable channel adjustment (i.e., stability constraint), and ensure the reconstruction quality in continuous compression.

[0153] Based on the same technical concept, in another embodiment of the present invention, an error-resistant image compression system is provided, comprising:

[0154] An image compression error detection module, which performs image compression error detection using the image compression error detection method shown in FIG1 to obtain a damaged channel;

[0155] A stability constraint region calculation module, which calculates the stability constraint region of the first latent feature representation. The stability constraint region is the region where a stable reconstructed image is obtained after the image compression model is passed;

[0156] A binary code stream encoding module adjusts the value range of the damaged channel according to the stability constraint region to obtain a second latent feature representation, and encodes the second latent feature representation to obtain a binary code stream;

[0157] The decoding module decodes the binary code stream to obtain the second latent feature representation, and passes the second latent feature representation through the decoder of the image compression model to obtain the reconstructed image.

[0158] The specific implementation methods of the above modules in this embodiment can refer to the implementation process of each step in the error-resistant image compression method, and will not be repeated here.

[0159] The error-resistant image compression system in this embodiment can be used to implement an error-resistant image compression method, can efficiently detect quality errors and damage caused by neural network-based image compression, and efficiently achieve stable continuous image compression, which is suitable for actual image communication scenarios.

[0160] Application examples:

[0161] The error-resistant image compression method in the above embodiment of the present invention is used to alleviate image damage caused by continuous image compression without training.

[0162] The following scenario describes continuous image compression: An original image, pic1, is compressed by user 1 using an end-to-end image compression model and then sent. User 2 receives the original image using the same image compression model to recover pic2. User 2 then compresses the recovered image pic2 using the same model and sends it to the next user. User 3 receives the original image and repeats the same process as user 2 before sending it to the next user. This process continues, and the same image undergoes multiple compression and reconstruction cycles using the same end-to-end image compression model as it is sent and received by different users. This scenario is called "continuous image compression."

[0163] In this application scenario:

[0164] First, the method in the above embodiment is used to obtain the in-range area, and whether pic1 is damaged is determined by judging whether the multi-channel latent feature representation f1 obtained by the image compression model of pic1 is in the "in-range area". If damaged, a stable representation f'1 is obtained by adjusting f1 to the "stability constraint area".

[0165] The adjusted image is then restored using the above image compression model to obtain pic2. Afterwards, pic2 is used as the input image to determine whether the image is damaged again, and the obtained multi-channel latent feature representation is adjusted to the "stability constraint region".

[0166] Repeat the above steps to achieve continuous image compression.

[0167] Implementation effect:

[0168] In the above embodiment, the error-resistant image compression method provided by the present invention is used, based on eight hyper-prior network floating-point pre-trained models of different qualities provided by compressAI. Quality 1 results in the highest distortion of the reconstructed image compared to the original image, while quality 8 results in the lowest distortion. A 16*16 input image is obtained using the Vimeo90k dataset, and the encoded segment output latent features representing the "in-range region" of each channel are obtained. Continuous compression damage detection is also performed using the Tecnick dataset.

[0169] Table 1

[0170] In the above example, the channel damage detection method provided by the present invention was used to detect the number of damaged images in the Tecnick dataset after 50 consecutive compression cycles. The method was then compared with the detection method proposed in Kim et al. Images with a PSNR degradation exceeding 0.2dB after 20 to 50 consecutive compression cycles, as well as images with visually irregular patterns after 50 consecutive compression cycles, were marked as damaged and used as reference images. Table 1 shows the detection results. Compared to Kim et al.'s paper, the method provided by the present invention detected more damaged images.

[0171] In the above embodiment, each image of the Tecnick dataset is compressed continuously for 50 times, and the error-resistant image compression method provided by the present invention is used to adjust the latent feature representation output by the encoder after each compression to detect the damaged channel. Figure 9 shows the results of continuously compressing the 30th and 61st images in the Tecnick dataset based on a pre-trained super-prior model with a quality level of 4. As shown in Figure 9, using the error-resistant image compression method provided by the present invention, the latent feature representation of the encoder is always maintained in the "within-range area" during the continuous compression process, and at the same time, the PSNR attenuation is alleviated. In addition, using the error-resistant image compression method provided by the present invention, the visual quality of the reconstructed image after continuous compression is significantly improved.

[0172] Therefore, the present invention can achieve stable and continuous compression on natural images of different resolutions collected in different scenarios, and has extremely strong practical application value, especially in the application of neural network-based image compression methods in actual image communications.

[0173] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware.

[0174] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0175] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0176] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0177] While preferred embodiments of the present invention have been described, it should be understood that the present invention is not limited to the specific embodiments described above. Those skilled in the art, once informed of the basic inventive concepts, may make additional changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as covering the preferred embodiments and all variations and modifications that fall within the scope of the present invention. The preferred technical features of the present invention described above may be used in any combination as long as they do not conflict with each other.

Claims

1. A method for detecting image compression errors, characterized in that: include: Get a training image dataset; extracting a first latent feature representation of the training image using an image compression model, wherein the first latent feature representation is a multi-channel latent feature representation for image compression; Processing the first latent feature representation to obtain a stability metric region for detecting image compression errors; A first latent feature representation of a test image is extracted, and the first latent feature representation of the test image is compared with the stability metric region to obtain an image compression error detection result.

2. The image compression error detection method according to claim 1, characterized in that: The stability measurement region is the allowable region of each channel of the first latent feature representation, and the allowable region is the value range of the multi-channel latent feature representation obtained by any natural image through the image compression model; or, The stability measurement region is a region within the range of each channel of the first latent feature representation, and the region within the range is a numerical range of a multi-channel latent feature representation of the training image obtained through the image compression model.

3. The image compression error detection method according to claim 2, characterized in that: The allowable region is obtained by the following method: Repeat the following steps for each channel of the first latent feature representation: Randomly generate an RGB three-channel image of a set size and input it into the encoder of the image compression model, wherein each pixel value in the three-channel image is sampled from a normal distribution and normalized so that the value is in the range of 0 to 1; With the goal of minimizing the value of the latent feature representation element, the input three-channel image is updated by gradient descent, and the element value of any channel in the traversal process of the encoder output latent feature representation is minimized as the lower bound of the allowable region; With the goal of maximizing the value of the latent feature representation element, the input three-channel image is updated by gradient ascent, and the element value of any channel in the process of traversing the encoder output latent feature representation is maximized as the upper bound of the allowable region; The interval range formed by the upper limit of the allowable area and the lower limit of the allowable area is the allowable area.

4. The image compression error detection method according to claim 1, characterized in that: The area within the range is obtained by the following method: Slide to obtain image blocks of a set size from all images in the training image dataset, with a sliding step of 1; Passing all the acquired image blocks through the encoder of the image compression model, recording the element value of each channel of the latent feature representation output by the encoder, and obtaining each multi-channel latent feature representation; For any c-th channel: Obtain the statistical maximum value of the element values ​​of all image blocks in the c-th channel; Obtain the statistical minimum value of the element values ​​of all image blocks in the c-th channel; The closed interval defined by the above statistical maximum value and statistical minimum value is the area within the range of the c-th channel.

5. The image compression error detection method according to claim 1, characterized in that: The step of comparing the first latent feature representation of the test image with the stability metric region to obtain an image compression error detection result includes: If the multi-channel latent feature is represented within the stability measurement region, the image compression error detection result is a stable image; If the multi-channel latent feature representation exceeds the stability metric region, the image compression error detection result is that image damage exists, and the damaged channel is output.

6. An error-resistant image compression method, characterized in that: include: Using the image compression error detection method according to any one of claims 1 to 5 to perform image compression error detection to obtain a damaged channel; Calculating a stability constraint region represented by the first latent feature, wherein the stability constraint region is a region where a stable reconstructed image is obtained after the image compression model is passed; adjusting the value range of the damaged channel according to the stability constraint region to obtain a second latent feature representation, and encoding the second latent feature representation to obtain a binary code stream; The binary code stream is decoded to obtain a second latent feature representation, and the second latent feature representation is passed through a decoder of the image compression model to obtain a reconstructed image.

7. The error-resistant image compression method according to claim 6, characterized in that: The calculating of the stability constraint region of the first latent feature representation includes: For any c-th channel, calculate the upper bound of the area within the range and lower bound The difference between c : Get the maximum value η of the channel importance metric of all C channels max : For any c-th channel, according to the channel importance metric η c and the maximum value of the channel importance metric η max Calculate the stability constraint region ratio r c : According to the ratio r of the stability constraint region c and the upper bound of the region within and lower bound Calculate the upper bound a of the stability constraint region c and the lower bound b c :

8. The error-resistant image compression method according to claim 7, characterized in that: The method further comprises: adjusting a value range of the damaged channel according to the stability constraint region to obtain a second latent feature representation, and encoding the second latent feature representation to obtain a binary code stream, including: Detect and adjust all C channels represented by the first latent feature, for any c-th channel, where c is a positive integer not greater than C: The image compression error detection method outputs the in-range region and the stability constraint region of the cth channel, where the in-range region is a real number closed interval Stability constraint region [b c ,a c ] is the area within the range The closed subinterval of ; Determine the value y of any element in the cth channel c Are they all within the stated area? If not, it is considered as a damaged channel; If the cth channel is a damaged channel, compress the element value of the cth channel into the range of the stability constraint region. c , perform the operation: y c =min(max(y c ,has c ),b c ) Where max(m,n) means taking the larger value of m and n, and min(m,n) means taking the smaller value of m and n; If the cth channel is a damaged channel, whether to adjust the cth channel is determined according to the rate distortion loss of the compressed cth channel compared to the original cth channel; If it is determined to adjust the cth channel, the compressed cth channel will replace the original cth channel, otherwise the original cth channel will remain unchanged; The adjusted multi-channel latent feature representation is output as the second latent feature representation, and encoded to obtain a binary code stream.

9. The error-resistant image compression method according to claim 8, characterized in that: If the c-th channel is a damaged channel, determining whether to adjust the c-th channel according to the rate distortion loss of the compressed c-th channel compared with the original c-th channel includes: Encode the original c-th channel and the compressed c-th channel respectively, and record their bit rates as and Get the bit rate difference Calculate the element-by-element mean square error (MSE) between the compressed c-th channel and the original c-th channel as the distortion difference Δ c (D); Calculate and sort all C channels from high to low according to the bit rate difference and distortion difference, and keep the first K channels unchanged, where K is a preset constant; Determine whether the cth channel is adjusted and output the determination result.

10. An error-resistant image compression system, characterized in that: include: An image compression error detection module, which uses the image compression error detection method according to any one of claims 1 to 5 to perform image compression error detection to obtain a damaged channel; A stability constraint region calculation module, which calculates the stability constraint region represented by the first latent feature, wherein the stability constraint region is a region where a stable reconstructed image is obtained after the image compression model is used; A binary code stream encoding module, which adjusts the value range of the damaged channel according to the stability constraint region to obtain a second latent feature representation, and encodes the second latent feature representation to obtain a binary code stream; A decoding module is used to decode the binary code stream to obtain a second latent feature representation, and the second latent feature representation is passed through a decoder of the image compression model to obtain a reconstructed image.

Images