A multi-rate deep image compression system and method for mixed context

Abstract

The present invention discloses a multi-rate deep image compression system based on residual enhancement mask in a mixed context, which includes a deep image encoding and decoding module, a latent vector importance analysis module, and a latent space encoding module. The latent space encoding module uses a residual method to divide the latent space vector into a base layer and an enhancement layer, thereby eliminating the redundancy of the latent space and improving the compression efficiency. The base layer contains the global features of the original image, ensuring the overall perceptual effect of the reconstructed image, and contains overall semantic information. In addition, when performing image compression encoding, the present invention performs a single encoding to obtain a latent vector, and achieves code rate and context adaptation through latent space encoding, so that the compressed transmission of the image has higher real-time performance.

Description

Technical Field

[0001] The present invention relates to the field of image compression technology, and in particular to a multi-rate deep image compression system and method based on residual enhancement mask in a mixed context. Background Art

[0002] Image compression technology enables the dissemination of massive amounts of image information over the internet, saving significant resources. The Deep Image Compression (DIC) algorithm employs deep neural networks for image compression, sharing a similar framework to traditional image compression algorithms. While traditional image compression algorithms optimize each module individually during the encoding and decoding process, Deep Image Compression utilizes an end-to-end learning approach for global optimization of the compression algorithm, achieving overall optimality. Furthermore, while traditional image compression algorithms utilize manually designed compressed representations, Deep Image Compression learns compressed representations based on large amounts of data for specific application scenarios. Consequently, Deep Image Compression offers superior performance in specific application scenarios.

[0003] Contexts define the different uses of images by recipients. Images transmitted on the current internet are often used for multiple purposes. For example, a real-time road traffic map can be used by recipients for various applications, such as road congestion assessment, traffic violation detection, and record keeping. Image compression algorithms in mixed contexts can improve the utility of reconstructed images while simultaneously meeting the needs of multiple applications. In practical applications with limited communication resources, mixed-context image compression methods should feature multi-rate compression to adapt to varying communication resources. Since the dimensions of the compressed vectors in a deep image compression model are fixed after training, training a series of compression models to achieve multi-rate compression is cumbersome and impractical. Currently, algorithms that implement multi-rate deep image compression using a single deep image compression model fall into two main categories. The first category is encoder-based multi-rate deep compression algorithms. These algorithms generate compressed representations at different bitrates by feeding different conditions into the encoder. The decoder then generates reconstructed images of corresponding quality based on the different compressed representations. The second type of method is multi-rate deep compression based on latent space coding. In this method, the encoder generates a unified latent space compressed representation, generates bitstreams at different bitrates through latent space coding, and the decoder generates reconstructed images of corresponding quality based on the different bitstreams. In mixed contexts, encoder-based multi-rate compression algorithms need to regenerate the compressed representation when the bitrate and context change, which incurs additional overhead and latency. Currently, image compression methods based on latent space coding are all targeted at a single context and cannot meet the multi-rate image compression requirements in mixed contexts. Summary of the Invention

[0004] In response to the problems existing in the prior art, the purpose of the present invention is to provide a multi-rate deep image compression system and method based on residual enhancement mask in a mixed context, which can quickly adapt to image compression of different contexts and different bit rates, and the reconstructed image has high classification accuracy and high-quality visual effects under a general classifier.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] A multi-rate deep image compression system based on residual enhancement mask in a mixed context, which includes a deep image encoding and decoding module, a latent vector importance analysis module and a latent space encoding module.

[0007] The deep image encoding and decoding module includes a depth encoder, a quantizer, an entropy encoder, an entropy decoder, a potential vector hierarchical merging unit and a depth decoder. The depth encoder and the depth decoder are composed of a deep convolutional neural network that has been trained and converged. The parameters of the deep convolutional neural network are trained in an end-to-end manner.

[0008] The deep encoder is used to input the original image x and extract the features of the original image x to generate a latent vector;

[0009] The quantizer is used to input the to-be-sent vector output by the latent space coding module and perform scalar quantization processing on each element of the to-be-sent vector;

[0010] The entropy encoder is used to input a quantized vector to be sent, and perform entropy coding on the quantized vector to be sent to generate a code stream for sending;

[0011] The entropy decoder is configured to receive a code stream sent by the entropy encoder, perform entropy decoding on the code stream, and reconstruct a base layer vector and an enhancement layer vector;

[0012] The latent vector hierarchical merging unit is configured to input the reconstructed base layer vector and enhancement layer vector and merge them to reconstruct the latent vector;

[0013] The depth decoder is used to input the reconstructed latent vector and perform depth image compression decoding on the reconstructed latent vector to obtain a reconstructed image of the original image;

[0014] The latent vector importance analysis module includes a semantic importance analysis unit, a reconstruction importance analysis unit and an importance map fusion unit;

[0015] The semantic importance analysis unit is used to input the original image x, analyze the importance of each pixel in the original image to the image classification result, and map it to each plane position of the latent vector to generate a semantic importance map of the latent vector;

[0016] A reconstruction importance analysis unit is used to input the latent vector output by the depth encoder, analyze the importance of each plane position of the latent vector to image reconstruction, and generate a reconstruction importance map of the latent vector;

[0017] The importance map fusion unit is used to input the semantic importance map of the latent vector and the reconstruction importance map, and fuse the two to generate a latent vector importance map in a mixed context;

[0018] The latent space encoding module includes a latent vector hierarchical unit, a latent space mask generation unit and an encoding unit;

[0019] The latent vector hierarchical unit is configured to input a latent vector and decompose the latent vector into a base layer vector and an enhancement layer vector using a residual method. The base layer vector represents the entire original image in a latent space, and the enhancement layer vector represents the details in the original image in a latent space. The bit rates of the base layer vector and the enhancement layer vector are then determined based on a range of a target bit rate.

[0020] The latent space mask generation unit is used to input the target code rate r of the channel feedback and the potential vector importance map I, calculate the maximum number of elements of the potential vector allowed to be transmitted under the current channel resources, and then calculate the potential vector importance map Figure 2 The threshold of the value and the importance map Figure 2 The value is converted into a latent space mask M;

[0021] The encoding unit is used to input a latent space mask, a base layer vector, and an enhancement layer vector, select elements to be sent to the decoding end from the base layer vector and the enhancement layer vector according to the latent space mask, and generate a vector to be sent, which includes the latent space mask, all base layer vectors, and elements corresponding to positions where the mask is 1 in the enhancement layer.

[0022] The deep encoder and deep decoder are iteratively trained in an end-to-end manner. In one iterative training process, the discriminator D is first trained, and then the generator G composed of the deep encoder and deep decoder is trained. When training the generator D, the parameters of the discriminator G remain unchanged, and when training the discriminator G, the parameters of the generator D remain unchanged. The process of iterative training of the deep encoder and deep decoder network model is as follows:

[0023] Step S1: Sample an input image x from the training set and generate a latent vector z through a deep encoder. The latent vector layer unit decomposes the vector z into a base layer vector z based on the residual technology. b and the enhancement layer vector z e ;

[0024] Step S2: Randomly sample a single-channel mask vector with the same dimension as the potential vector plane, and the mask vector is the same as the vector ze Multiply each channel plane to get vector z e1 , random mask vector, base layer vector z b and vector z e1 Splicing vectors , for the vector Noise is added to simulate the quantization process; the deep decoder inputs the noisy vector The reconstructed images y are obtained respectively;

[0025] Step S3: Train the GAN-based discriminator D and calculate each parameter pair of the discriminator The Adam optimizer is used to optimize and update the parameters of the discriminator D.

[0026] Step S4: Train the generator G consisting of a deep encoder and a deep decoder. The loss function of the generator is as shown in formula (1):

[0027] (1)

[0028] in, is a vector The bit rate, is the GAN loss term, To rebuild the lost projects, is the semantic feature matching loss term, 、 is the weight corresponding to each loss item;

[0029]

[0030] Among them, x and y are the original image and the reconstructed image, (H, W, 3) are the dimensions of x and y, is the semantic feature extraction function, are the dimensions of the semantic feature vector respectively;

[0031] Solve the gradient of each parameter of the deep encoder and deep decoder with respect to the loss term, and use the Adam optimizer to perform parameter optimization updates.

[0032] The semantic importance analysis unit classifies the input image using a deep convolutional network classifier, and uses a gradient-based class activation mapping method to calculate the importance of each pixel to the classification result. Then, based on the spatial invariance of the convolutional neural network, the pixel importance is mapped to the latent space to obtain a latent vector semantic importance map.

[0033] The reconstruction importance analysis unit calculates the variance of all elements at each plane position of the potential vector. The larger the variance, the greater the entropy of the position, and the more important all elements at the position are to the reconstruction. The variance value of each position is normalized to obtain a reconstruction importance map.

[0034] A multi-rate deep image compression method based on residual enhancement mask in a mixed context is implemented using the deep image compression system described above, and specifically comprises the following steps:

[0035] Step 1: The original image x is passed through a deep encoder to generate a latent vector z, which is then input into the reconstruction importance analysis unit and the latent vector hierarchical unit.

[0036] Step 2: The latent vector hierarchical unit in the latent space encoding module decomposes the latent vector z into the base layer vector z based on the residual technology b and the enhancement layer vector z e ;

[0037] Step 3: The original image x and the latent vector are input into the latent vector importance analysis module to generate a latent vector importance map I;

[0038] Step 4: The latent space encoding module calculates the latent vector importance map I and the target bit rate according to the latent vector importance analysis module output. r Generate the latent space mask vector M and generate the vector to be sent ;

[0039] Step 5: The quantizer quantizes the vector to be transmitted to obtain a vector ;

[0040] Step 6: The entropy encoder uses an adaptive arithmetic coding method to encode the vector Perform entropy coding to generate a bit stream, and send the bit stream to the entropy decoder through the channel;

[0041] Step 7: The entropy decoder applies an adaptive arithmetic decoding algorithm to the received bitstream and recovers the vector M, the base layer vector, and the enhancement layer vector.

[0042] Step 8: The latent vector hierarchical merging unit inputs the base layer vector and the enhancement layer vector, and reconstructs them to generate a reconstructed latent vector ;

[0043] Step 9: The deep decoder inputs the reconstructed latent vector and the vector M to generate the reconstructed image y.

[0044] The step 3 is specifically as follows:

[0045] Step 3-1: Input the image x into the semantic importance analysis unit to perform semantic importance analysis on the latent vector to obtain a latent space latent vector semantic importance map, denoted as vector e. Adaptive binarization algorithm is used to binarize vector e into vector B.

[0046] Step 3-2, the reconstruction importance analysis unit calculates the variance vector v of the latent vector z composed of the variance of the C element values ​​at each plane position, and uses v as the reconstruction importance map;

[0047] Step 3-3, the importance map fusion unit generates a latent vector importance map I according to the following formula;

[0048] (4).

[0049] The step 4 is specifically as follows:

[0050] Step 4-1: The latent space mask generation unit prioritizes the bit rate to the base layer, , are the base layer vector and enhancement layer vector to be sent respectively, and the code rate of the base layer vector is R( ), the code rate allocated to the enhancement layer is :

[0051] (5)

[0052] Among them, R( ) indicates bit rate calculation;

[0053] Step 4-2: When allocating bitrate to the enhancement layer, the minimum allocation granularity is C elements of one plane position of the latent vector. The maximum number of plane positions S allowed to be transmitted is determined based on the current target bitrate r. The Sth largest element in the importance map I is used as the threshold for binarizing vector I. Vector I is binarized into a mask vector M.

[0054] Step 4-3: The latent space coding unit extracts the elements of the enhancement layer according to formula (7).

[0055] (7)

[0056] in, for The element to be sent, z e Elements, is an element of M; the elements extracted from the enhancement layer, the vector M, and the base layer vector are combined into a vector to be sent.

[0057] After adopting the above technical solution, the present invention has the following beneficial effects:

[0058] 1. The present invention uses a residual method to divide the latent space vector into a base layer and an enhancement layer, eliminating the redundancy of the latent space and improving the compression efficiency;

[0059] 2. The base layer in the compression method of the present invention contains the global features of the original image, ensuring the overall perceptual effect of the reconstructed image and containing the overall semantic information;

[0060] 3. The present invention uses a GAN generation model to ensure that the distribution of the reconstructed image is similar to that of the original image, further improving the overall perception of the reconstructed image;

[0061] 4. When performing image compression encoding, the proposed method uses a deep encoder to perform a single encoding operation to obtain a latent vector. This latent space encoding achieves bitrate and context adaptation, enabling more real-time image compression and transmission. Furthermore, this bitrate and context adaptation approach is more suitable for multicast and broadcast scenarios, saving significant computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a schematic diagram of the overall framework of a progressive depth image compression method based on residual enhancement mask in a mixed context according to the present invention;

[0063] Figure 2 This is a schematic diagram of the deep encoding and decoding network model training process in the present invention;

[0064] Figure 3 This is the implementation case deep encoding and decoding network model diagram;

[0065] The present invention will be described in further detail below with reference to the accompanying drawings. DETAILED DESCRIPTION

[0066] The present invention discloses a multi-rate deep image compression system based on residual enhancement mask in a mixed context, which includes a deep image encoding and decoding module, a latent vector importance analysis module and a latent space encoding module.

[0067] Among them, the deep image encoding and decoding module includes a deep encoder, a quantizer, an entropy encoder, an entropy decoder, a latent vector hierarchical merging unit and a deep decoder. The deep encoder and deep decoder are composed of a deep convolutional neural network that has been trained and converged. The parameters of the deep convolutional neural network are trained in an end-to-end manner.

[0068] The depth encoder is used to input the original image x and extract the features of the original image x to generate a latent vector. Specifically, in this embodiment, the dimension of the original image x is ( H, W, 3 ), the depth encoder downsamples the original image x rows by 8 times, and the dimension is ( ). In order to adapt to different channel conditions and different contexts, the latent vector generated by the deep encoder needs to be fed into the latent space encoding module for further encoding.

[0069] The quantizer takes the transmitted vector output by the latent space encoding module as input and performs scalar quantization on each element of the transmitted vector to reduce the entropy of the transmitted vector for transmission over the communication channel. The quantizer is a scalar quantizer, and in this embodiment, the quantization level is 5, with the quantization center being {−2, −1, 0, 1, 2}.

[0070] The entropy encoder takes a quantized vector to be transmitted and performs entropy encoding on it to generate a bitstream for transmission. This entropy encoder is an adaptive arithmetic encoder. The encoding is performed to remove redundancy from the vector to be transmitted, improving coding efficiency. This process is lossless to the information in the vector to be transmitted.

[0071] The entropy decoder is an adaptive arithmetic decoder, which is used to receive the code stream sent by the entropy encoder, perform entropy decoding on the code stream and reconstruct the base layer vector and the enhancement layer vector.

[0072] The latent vector hierarchical merging unit is used to input the base layer vector and the enhancement layer vector and merge them to reconstruct the latent vector.

[0073] The depth decoder is configured to input the reconstructed latent vector and perform deep image compression decoding on the reconstructed latent vector to obtain a reconstructed image of the original image. Specifically, in this embodiment, the depth decoder performs an 8x upsampling on the reconstructed latent vector to generate a reconstructed image of the original image.

[0074] The latent vector importance analysis module includes a semantic importance analysis unit, a reconstruction importance analysis unit and an importance map fusion unit.

[0075] The semantic importance analysis unit is configured to input an original image x, analyze the importance of each pixel in the original image to the image classification result, and map the importance to each plane position of the latent vector, thereby generating a semantic importance map of the latent vector. Specifically, the semantic importance analysis unit of this embodiment uses VGG16 to classify the input image and employs the gradient-based class activation mapping (CAM) method, Grad-CAM++, to calculate the importance of each pixel to the classification result. Based on the spatial invariance of the convolutional neural network, the pixel importance is mapped to the latent space to obtain a semantic importance map of the latent vector.

[0076] The reconstruction importance analysis unit is configured to input the latent vector output by the depth encoder, analyze the importance of each plane position of the latent vector to image reconstruction, and generate a reconstruction importance map of the latent vector. Specifically, in this embodiment, the reconstruction importance analysis unit calculates the variance of all elements at each plane position of the latent vector. A larger variance indicates a greater entropy at that position, and all elements at that position are more important to reconstruction. The variance value at each position is normalized to obtain the reconstruction importance map.

[0077] The importance map fusion unit is used to input the semantic importance map of the latent vector and the reconstruction importance map, and fuse the two to generate the latent vector importance map in the mixed context.

[0078] The latent space encoding module includes a latent vector hierarchical unit, a latent space mask generation unit and an encoding unit.

[0079] Among them, the latent vector hierarchical unit is used to input the latent vector and use the residual method to decompose the latent vector into a base layer vector and an enhancement layer vector. The base layer vector is the representation of the original image as a whole in the latent space, and the enhancement layer vector is the representation of the details in the original image in the latent space. Then, the bit rates of the base layer vector and the enhancement layer vector are determined according to the range of change of the target bit rate.

[0080] The latent space mask generation unit is used to input the target code rate r and the potential vector importance map I of the channel feedback, calculate the maximum number of elements of the potential vector allowed to be transmitted under the current channel resources, and then calculate the potential vector importance map Figure 2 The threshold of the value and the importance map Figure 2 The value is converted into the latent space mask M.

[0081] The encoding unit is used to input the latent space mask, the base layer vector, and the enhancement layer vector. Based on the latent space mask, it selects elements from the base layer vector and the enhancement layer vector to be sent to the decoder, generating a vector to be sent. The vector to be sent includes the latent space mask, all base layer vectors, and elements corresponding to positions in the enhancement layer where the mask is 1. In other words, the enhancement layer vector elements corresponding to positions where the mask is 1 need to be sent to the decoder, while positions where the mask is 0 are not sent. The extracted enhancement layer vector elements to be sent, all elements of the base layer vector, and the mask are concatenated and sent to the quantizer.

[0082] In this embodiment, the depth encoder and the depth decoder are iteratively trained in an end-to-end manner. The depth encoder and the depth decoder are iteratively trained in an end-to-end manner. In one iterative training process, the discriminator D is first trained, and then the generator G composed of the depth encoder and the depth decoder is trained. When training the generator D, the parameters of the discriminator G remain unchanged, and when training the discriminator G, the parameters of the generator D remain unchanged. The ImageNet dataset is used as the training set, the size of the input image is adjusted to 256×256, and the Adam optimizer is used with a learning rate of 2e-4 and a batch size of 1. Figure 2 As shown in Figure 2, the iterative training process of the deep encoder and deep decoder network models is as follows:

[0083] Step S1: Sample an input image x from the training set and generate a latent vector z through a deep encoder. The latent vector layer unit decomposes the vector z into a base layer vector z based on the residual technology. b and the enhancement layer vector z e ;

[0084] Step S2: Randomly sample a single-channel mask vector with the same dimension as the potential vector plane, and the mask vector is the same as the vector z e Multiply each channel plane to get vector z e1 , random mask vector, base layer vector z b and vector z e1 Splicing vectors , for the vector Noise is added to simulate the quantization process; the deep decoder inputs the noisy vector The reconstructed images y are obtained respectively;

[0085] Step S3: train the GAN-based discriminator D. When training D, select different types of GAN loss items according to actual needs, such as LSGAN, WGAN, infoGAN, etc., and calculate each parameter pair of the discriminator. The Adam optimizer is used to optimize and update the parameters of the discriminator D.

[0086] Step S4: Train the generator G consisting of a deep encoder and a deep decoder. The loss function of the generator is as shown in formula (1):

[0087] (1)

[0088] in, is a vector The bit rate, is the GAN loss term, which has different calculation methods depending on the different GANs used. To rebuild the lost projects, is the semantic feature matching loss term, 、 is the weight corresponding to each loss item;

[0089]

[0090] Among them, x and y are the original image and the reconstructed image, (H, W, 3) are the dimensions of x and y, is the semantic feature extraction function, are the dimensions of the semantic feature vector respectively;

[0091] Solve the gradient of each parameter of the deep encoder and deep decoder with respect to the loss term, and use the Adam optimizer to perform parameter optimization updates.

[0092] Continue to refer to Figure 1 Combined with Figure 3 As shown, based on the above system, the present invention also discloses a multi-rate deep image compression method based on residual enhancement mask in a mixed context, which includes the following steps:

[0093] Step 1: The original image x is passed through a deep encoder to generate a latent vector z, which is then input into the reconstruction importance analysis unit and the latent vector hierarchical unit.

[0094] Step 2: The latent vector hierarchical unit in the latent space encoding module decomposes the latent vector z into the base layer vector z based on the residual technology b and the enhancement layer vector z e ;

[0095] Step 3: Original image x and enhanced layer vector z e Input the latent vector importance analysis module to generate a latent vector importance map I;

[0096] Step 3-1: Input the image x into the semantic importance analysis unit to perform semantic importance analysis on the latent vector to obtain a latent space latent vector semantic importance map, denoted as vector e. Adaptive binarization algorithm is used to binarize vector e into vector B.

[0097] Step 3-2, the reconstruction importance analysis unit calculates the variance vector v of the latent vector z composed of the variance of the C element values ​​at each plane position, and uses v as the reconstruction importance map;

[0098] Step 3-3: The importance map fusion unit generates a latent vector importance map I according to the following formula:

[0099] (4)

[0100] Step 4: The latent space encoding module calculates the latent vector importance map I and the target bit rate according to the latent vector importance analysis module output.r Generate the latent space mask vector M and generate the vector to be sent ;

[0101] Step 4-1: The latent space mask generation unit prioritizes the bit rate to the base layer, , are the base layer vector and enhancement layer vector to be sent respectively, and the code rate of the base layer vector is R( ), the code rate allocated to the enhancement layer is :

[0102] (5)

[0103] Among them, R( ) indicates bit rate calculation;

[0104] Step 4-2: When allocating bitrate to the enhancement layer, the minimum allocation granularity is C elements of one plane position of the latent vector. The maximum number of plane positions S allowed to be transmitted is determined based on the current target bitrate r. The Sth largest element in the importance map I is used as the threshold for binarizing vector I. Vector I is binarized into a mask vector M.

[0105] S From formula 6),

[0106] (6),

[0107] Where P is the number of elements in vector M, S is the number of 1 positions in vector M;

[0108] Step 4-3: The latent space coding unit extracts the elements of the enhancement layer according to formula (7).

[0109] (7)

[0110] in, for The element to be sent, z e Elements, is an element of M; the elements extracted from the enhancement layer, the vector M, and the base layer vector are combined into a vector to be sent;

[0111] Step 5: The quantizer quantizes the vector to be transmitted to obtain a vector ;

[0112] Step 6: The entropy encoder uses an adaptive arithmetic coding method to encode the vector Perform entropy coding to generate a bit stream, and send the bit stream to the entropy decoder through the channel;

[0113] Step 7: The entropy decoder applies an adaptive arithmetic decoding algorithm to the received bitstream and recovers the vector M, the base layer vector, and the enhancement layer vector.

[0114] Step 8: The latent vector hierarchical merging unit inputs the base layer vector and the enhancement layer vector to reconstruct the latent vector and concatenates it with the vector M to generate the vector ;

[0115] Step 9. Depth Decoder Input Vector Generate the reconstructed image y.

[0116] In summary, the present invention has the following beneficial effects:

[0117] 1. The present invention uses a residual method to divide the latent space vector into a base layer and an enhancement layer, eliminating the redundancy of the latent space and improving the compression efficiency;

[0118] 2. The base layer in the compression method of the present invention contains the global features of the original image, ensuring the overall perceptual effect of the reconstructed image and containing the overall semantic information;

[0119] 3. The present invention uses a GAN generation model to ensure that the distribution of the reconstructed image is similar to that of the original image, further improving the overall perception of the reconstructed image;

[0120] 4. When performing image compression encoding, the proposed method uses a deep encoder to perform a single encoding operation to obtain a latent vector. This latent space encoding achieves bitrate and context adaptation, enabling more real-time image compression and transmission. Furthermore, this bitrate and context adaptation approach is more suitable for multicast and broadcast scenarios, saving significant computing resources.

[0121] The above description is merely an embodiment of the present invention and does not limit the technical scope of the present invention. Therefore, any minor modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A multi-rate deep image compression system based on residual enhancement mask in a mixed context, characterized by: Including deep image encoding and decoding module, latent vector importance analysis module and latent space encoding module, The deep image encoding and decoding module includes a deep encoder, a quantizer, a latent vector hierarchical merging unit and a deep decoder. The deep encoder and the deep decoder are composed of a deep convolutional neural network that has been trained and converged. The parameters of the deep convolutional neural network are trained in an end-to-end manner. The deep encoder is used to input the original image x and extract the features of the original image x to generate a latent vector; The quantizer is used to input the to-be-sent vector output by the latent space coding module and perform scalar quantization processing on each element of the to-be-sent vector; The latent vector hierarchical merging unit is configured to input the reconstructed base layer vector and enhancement layer vector and merge them to reconstruct the latent vector; The depth decoder is used to input the reconstructed latent vector and perform depth image compression decoding on the reconstructed latent vector to obtain a reconstructed image of the original image; The latent vector importance analysis module includes a semantic importance analysis unit, a reconstruction importance analysis unit and an importance map fusion unit; The semantic importance analysis unit is used to input the original image x, analyze the importance of each pixel in the original image to the image classification result, and map it to each plane position of the latent vector to generate a semantic importance map of the latent vector; A reconstruction importance analysis unit is used to input the latent vector output by the depth encoder, analyze the importance of each plane position of the latent vector to image reconstruction, and generate a reconstruction importance map of the latent vector; The importance map fusion unit is used to input the semantic importance map of the latent vector and the reconstruction importance map, and fuse the two to generate a latent vector importance map in a mixed context; The latent space encoding module includes a latent vector hierarchical unit, a latent space mask generation unit and an encoding unit; The latent vector hierarchical unit is configured to input a latent vector and decompose the latent vector into a base layer vector and an enhancement layer vector using a residual method. The base layer vector represents the entire original image in a latent space, and the enhancement layer vector represents the details in the original image in a latent space. The bit rates of the base layer vector and the enhancement layer vector are then determined based on a range of a target bit rate. The latent space mask generation unit is configured to input a target bit rate r and a latent vector importance map I fed back by the channel, calculate the maximum number of elements of the latent vector allowed to be transmitted under the current channel resources, calculate a threshold for binarizing the latent vector importance map, and binarize the importance map into a latent space mask M; The encoding unit is used to input a latent space mask, a base layer vector, and an enhancement layer vector, select elements to be sent to the decoding end from the base layer vector and the enhancement layer vector according to the latent space mask, and generate a vector to be sent, which includes the latent space mask, all base layer vectors, and elements corresponding to positions where the mask is 1 in the enhancement layer.

2. The multi-rate deep image compression system based on residual enhancement mask in a mixed context according to claim 1, characterized in that: The deep encoder and deep decoder are iteratively trained in an end-to-end manner. In one iterative training process, the discriminator D is first trained, and then the generator G composed of the deep encoder and deep decoder is trained. When training the generator D, the parameters of the discriminator G remain unchanged, and when training the discriminator G, the parameters of the generator D remain unchanged. The process of iterative training of the deep encoder and deep decoder network model is as follows: Step S1: Sample an input image x from the training set and generate a latent vector z through a deep encoder. The latent vector layer unit decomposes the vector z into a base layer vector z based on the residual technology. b and the enhancement layer vector z e ; Step S2: Randomly sample a single-channel mask vector with the same dimension as the potential vector plane, and the mask vector is the same as the vector z e Multiply each channel plane to get vector z e1 , random mask vector, base layer vector z b and vector z e1 Splicing vectors , for the vector Noise is added to simulate the quantization process; the deep decoder inputs the noisy vector The reconstructed images y are obtained respectively; Step S3: Train the GAN-based discriminator D and calculate each parameter pair of the discriminator The Adam optimizer is used to optimize and update the parameters of the discriminator D. Step S4: Train the generator G consisting of a deep encoder and a deep decoder. The loss function of the generator is as shown in formula (1): （1） in, is a vector The bit rate, is the GAN loss term, To rebuild the lost projects, is the semantic feature matching loss term, 、 is the weight corresponding to each loss item; Among them, x and y are the original image and the reconstructed image, (H, W, 3) are the dimensions of x and y, is the semantic feature extraction function, are the dimensions of the semantic feature vector respectively; Solve the gradient of each parameter of the deep encoder and deep decoder with respect to the loss term, and use the Adam optimizer to perform parameter optimization updates.

3. The multi-rate deep image compression system based on residual enhancement mask in a mixed context according to claim 1, characterized in that: The semantic importance analysis unit classifies the input image using a deep convolutional network classifier, and uses a gradient-based class activation mapping method to calculate the importance of each pixel to the classification result. Then, based on the spatial invariance of the convolutional neural network, the pixel importance is mapped to the latent space to obtain a latent vector semantic importance map.

4. The multi-rate deep image compression system based on residual enhancement mask in a mixed context according to claim 1, characterized in that: The reconstruction importance analysis unit calculates the variance of all elements at each plane position of the potential vector. The larger the variance, the greater the entropy of the position, and the more important all elements at the position are to the reconstruction. The variance value of each position is normalized to obtain a reconstruction importance map.

5. A multi-rate deep image compression method based on residual enhancement mask in a mixed context, characterized by: The method is implemented using the deep image compression system according to any one of claims 1 to 4, and specifically comprises the following steps: Step 1: The original image x is passed through a deep encoder to generate a latent vector z, which is then input into the reconstruction importance analysis unit and the latent vector hierarchical unit. Step 2: The latent vector hierarchical unit in the latent space encoding module decomposes the latent vector z into the base layer vector z based on the residual technology b and the enhancement layer vector z e ; Step 3: The original image x and the latent vector are input into the latent vector importance analysis module to generate a latent vector importance map I; Step 4: The latent space encoding module calculates the latent vector importance map I and the target bit rate according to the latent vector importance analysis module output. r Generate the latent space mask vector M and generate the vector to be sent ; Step 5: The quantizer quantizes the vector to be transmitted to obtain a vector ; Step 6: The entropy encoder uses an adaptive arithmetic coding method to encode the vector Perform entropy coding to generate a bit stream, and send the bit stream to the entropy decoder through the channel; Step 7: The entropy decoder applies an adaptive arithmetic decoding algorithm to the received bitstream and recovers the vector M, the base layer vector, and the enhancement layer vector. Step 8: The latent vector hierarchical merging unit inputs the base layer vector and the enhancement layer vector, and reconstructs them to generate a reconstructed latent vector ; Step 9: The deep decoder inputs the reconstructed latent vector and the vector M to generate the reconstructed image y.

6. The multi-rate deep image compression method based on residual enhancement mask in a mixed context according to claim 5, characterized in that: The step 3 is specifically as follows: Step 3-1: Input the image x into the semantic importance analysis unit to perform semantic importance analysis on the latent vector to obtain a latent space latent vector semantic importance map, denoted as vector e. Adaptive binarization algorithm is used to binarize vector e into vector B. Step 3-2, the reconstruction importance analysis unit calculates the variance vector v of the latent vector z composed of the variance of the C element values ​​at each plane position, and uses v as the reconstruction importance map; Step 3-3, the importance map fusion unit generates a latent vector importance map I according to the following formula; （4）。 7. The multi-rate deep image compression method based on residual enhancement mask in a mixed context according to claim 5, characterized in that: The step 4 is specifically as follows: Step 4-1: The latent space mask generation unit prioritizes the bit rate to the base layer, , are the base layer vector and enhancement layer vector to be sent respectively, and the code rate of the base layer vector is R( ), the code rate allocated to the enhancement layer is : (5) Among them, R( ) indicates bit rate calculation; Step 4-2: When allocating bitrate to the enhancement layer, the minimum allocation granularity is C elements of one plane position of the latent vector. The maximum number of plane positions S allowed to be transmitted is determined based on the current target bitrate r. The Sth largest element in the importance map I is used as the threshold for binarizing vector I. Vector I is binarized into a mask vector M. Step 4-3: The latent space coding unit extracts the elements of the enhancement layer according to formula (7). (7) in, for The element to be sent, z e Elements, is an element of M; the extracted enhancement layer element, vector M, and base layer vector are combined into a vector to be sent.

Images