Task-driven adversarial learning image compression method and system

Through the task-driven adversarial learning image compression method, combined with deep separation convolutional network and entropy codec technology, the contradiction between visual quality and analysis task performance in traditional image compression methods is solved, efficient image compression and storage is achieved, and improved Image transmission and storage efficiency.

CN119052497AActive Publication Date: 2024-11-29THREE GORGES JINSHAJIANG CHUANYUN HYDROPOWER DEV CO LTD

Patent Information

Application Number
CN202411029615.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-29
Estimated Expiration
2044-07-30

Smart Images

  • Figure CN119052497A_ABST
    Figure CN119052497A_ABST
Patent Text Reader

Abstract

The invention discloses a task-driven adversarial learning image compression method and a task-driven adversarial learning image compression system, which relate to the field of image compression and are used for improving the visual task performance of a machine and keeping the visual quality of human eyes at the same time. According to the invention, the image is coded through the trained image compression system so as to realize compression; and performing super coding and decoding and entropy coding and decoding on the compressed image to obtain priori information, performing entropy coding and decoding based on the priori information to generate a second potential representation, and further decoding to obtain a reconstructed image to realize decompression. And utilizing the trained multi-branch discriminator network to carry out true and false discrimination on the reconstructed image, and carrying out category identification on the original image. According to the method, the size of the image file can be reduced, meanwhile, high image quality is kept, key details are reserved, the transmission efficiency is improved, the storage cost is saved, and the downstream image classification precision is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image compression, and in particular to a task-driven adversarial learning image compression method and system. Background Art

[0002] In order to efficiently store and transmit the increasingly large image data with the development of Internet technology, image compression technology has emerged. Traditional image compression methods are based on specific mathematical models and compress image data through transformation, quantization and entropy coding. Such methods have a trade-off between compression efficiency and image quality, and high compression rate and high image quality often cannot coexist. In addition, the performance of such methods usually depends on their preset assumptions about the image content, which means that their compression effect may not be ideal for images of different types and characteristics.

[0003] In many practical applications, compressed images are not only for human viewing, but are often used for various image analysis tasks. For example, in the fields of security monitoring, medical analysis, and autonomous driving, images are further used for target detection, semantic segmentation, and environmental perception in autonomous driving. However, traditional image compression methods mainly focus on the visual quality of reconstructed images. The design and optimization of the methods often do not take into account the final use scenario of the image, resulting in performance degradation of compressed images in subsequent image analysis tasks.

[0004] The image compression method based on deep learning has brought new breakthroughs in the field of image compression. It uses deep learning networks to capture the essential structure of data and uses it to efficiently represent data, thus achieving effective representation of images extracted through automatic learning. However, among the image compression methods based on deep learning, the existing Compress then Analyze (CTA) mode also has the problem of preferring to retain machine vision information during the compression process and removing human vision information as redundant information, resulting in poor visual image quality. Summary of the invention

[0005] The purpose of the present invention is to provide a task-driven adversarial learning image compression method and system to improve the performance of machine vision tasks while maintaining the visual quality of the human eye.

[0006] The technical solution adopted by the present invention is as follows:

[0007] A task-driven adversarial learning image compression method, comprising:

[0008] Construct image compression network and multi-branch discriminator network respectively;

[0009] The image compression network processes the input image by:

[0010] S1, preprocessing the original image;

[0011] S2, encoding the preprocessed original image to generate a first latent representation;

[0012] S3, performing super encoding and decoding and entropy encoding and decoding on the first potential representation to obtain prior information;

[0013] S4, quantizing the first latent representation, and performing entropy encoding and decoding based on the prior information to generate a second latent representation;

[0014] S5. Decode the second latent representation to obtain a reconstructed image;

[0015] The multi-branch discriminator network processes the input image including:

[0016] S6, inputting the original image and the reconstructed image into a multi-branch discriminator network to obtain category information and discriminant information;

[0017] S7, training the image compression network and the multi-branch discriminator network in stages;

[0018] S8. Use the trained image compression network to process the image to be compressed according to steps S1-S2, and use the first latent representation of the encoded output as the compressed image; or use the trained image compression network to process the compressed image according to steps S3-S5, and use the reconstructed image of the decoded output as the decompressed image.

[0019] Furthermore, the step S2 comprises:

[0020] S21, constructing a deep separable convolutional network, wherein the deep separable convolutional network includes multiple convolutional layers;

[0021] S22, add LeakyReLU activation function and normalization layer to each convolution layer;

[0022] S23, adding a residual block to each convolutional layer, wherein the residual block adds the input of the convolutional layer to the output of the convolutional layer;

[0023] S24, inputting the preprocessed original image into the depth-separable convolutional network for downsampling;

[0024] S25. The feature map output by the depthwise separable convolutional network is used as the first potential representation.

[0025] Furthermore, the step S3 includes:

[0026] S31, inputting the first potential representation into a super encoder for super encoding to obtain a first super encoding feature map; the super encoder includes a convolution layer, and ReLU is used as an activation function between adjacent convolution layers;

[0027] S32, quantizing the first super-encoded feature map, and then performing entropy coding and entropy decoding to generate a second super-encoded feature map;

[0028] S33: Input the second super-encoded feature map into a super-decoder symmetrical to the super-encoder to obtain prior information.

[0029] Further, the step S4 includes:

[0030] S41, quantizing the first potential representation;

[0031] S42, performing entropy coding on the quantized first potential representation through a probability model generated by prior information;

[0032] S43. Perform entropy decoding on the entropy-encoded first latent representation through a probability model generated by prior information to generate a second latent representation.

[0033] Furthermore, the step S5 comprises:

[0034] Depth-wise separable convolution and pixel reorganization are performed on the second latent representation to obtain a reconstructed image.

[0035] Further, the step S6 includes:

[0036] S61, inputting the original image and the reconstructed image into a feature extraction module to obtain feature representation; the feature extraction module adopts a multi-layer convolutional neural network, followed by a batch normalization layer and a LeakyReLU activation function;

[0037] S62, inputting the feature representation into an image classification module to generate category information; the image classification module uses a fully connected layer and a nonlinear activation function to classify the input feature representation;

[0038] S63, inputting the feature representation into an authenticity discrimination module to generate discrimination information; the authenticity discrimination module uses a fully connected layer and a binary mapping function to perform authenticity discrimination on the input feature representation.

[0039] Furthermore, in step S7, the staged training is as follows:

[0040] The image compression network is first trained independently, and after the training converges, the multi-branch discriminator network is introduced to perform adversarial joint training.

[0041] Furthermore, the step S7 includes:

[0042] S71, constructing a training sample image set and dividing it into a training set and a validation set, wherein the training set is used in a network training process;

[0043] S72, independently training the image compression network using a rate-distortion loss function, and adjusting network parameters during training using a back-propagation algorithm and a gradient descent optimizer;

[0044] S73, iteratively training the image compression network until convergence, or iterating a predetermined number of rounds;

[0045] S74, training the multi-branch discriminator network using an adversarial discriminant loss function, and adjusting network parameters during training using a back propagation algorithm and a gradient descent optimizer;

[0046] S75, training the image compression network using a rate-distortion loss function, and adjusting network parameters during training using a back-propagation algorithm and a gradient descent optimizer;

[0047] S76, repeating alternately step S74 and step S75 until the image compression network converges, or iterating a predetermined number of rounds;

[0048] S77, using the verification set to verify the performance of the trained image compression network;

[0049] S78. Save the network parameters of the trained image compression model.

[0050] Furthermore, in step S8, compressing the image to be compressed includes:

[0051] S81, obtaining original data of the image to be compressed;

[0052] S82, inputting the original data of the image to be compressed into a trained image compression network, and obtaining a first potential representation using an encoder;

[0053] Alternatively, decompression of a compressed image involves:

[0054] S83, inputting the compressed image into a trained image compression network, and obtaining prior information through super encoding and decoding and entropy encoding and decoding;

[0055] S84, quantizing the compressed image, performing entropy encoding and decoding based on prior information, and obtaining a corresponding second potential representation;

[0056] S85. Decode the obtained second latent representation to generate a reconstructed image.

[0057] A task-driven adversarial learning image compression system, which is configured to execute the above method.

[0058] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0059] 1. Ability to reduce image file size while maintaining high image quality. By optimizing compression algorithms and sampling strategies, this technical solution can efficiently compress image data, reduce storage space usage, and provide faster speeds and lower bandwidth requirements for image transmission and storage.

[0060] 2. Preserve key details: Despite compression, the present invention can still preserve key details and important features of the image. By precisely controlling the parameters of the compression algorithm and optimizing the image encoding process, the scheme can minimize information loss and maintain important details in the compressed image, so that the image remains visually clear, natural and recognizable.

[0061] 3. Improve transmission efficiency: Since the size of compressed image files is reduced, the present invention can improve the efficiency of image transmission. In network transmission or storage, smaller file sizes can reduce transmission time and resource consumption, speed up image loading, and provide faster response time. This is of great significance for online image display, video communication, cloud storage and other scenarios.

[0062] 4. Save storage costs: By adopting the image compression technology based on adversarial learning, the present invention can significantly reduce storage costs. Smaller image file size means that more image data can be stored in the same storage space, thereby saving the demand and cost of storage devices, especially in large-scale image storage and processing applications.

[0063] 5. Improve downstream image classification accuracy: By adopting the image compression technology based on adversarial learning, the images compressed by the present invention achieve higher accuracy in image classification tasks. This means that the images compressed by this model have obvious advantages over other compression schemes when applied to image analysis tasks.

[0064] 6. The present invention retains low-level human visual information and high-level semantic information through adversarial learning, allowing humans to intuitively evaluate the reconstructed image while being more adaptable to specific downstream tasks. The present invention maintains the human visual quality of the image while ensuring the performance of machine vision tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The present invention will now be described by way of example with reference to the accompanying drawings, in which:

[0066] Figure 1 A flowchart of a task-driven adversarial learning image compression method according to an embodiment of the present invention;

[0067] Figure 2A schematic diagram of the principle of the task-driven adversarial learning image compression method according to an embodiment of the present invention;

[0068] Figure 3 A flowchart of encoding an original image using a task-driven adversarial learning image compression method according to an embodiment of the present invention;

[0069] Figure 4 A flowchart of the task-driven adversarial learning image compression method for obtaining prior information of the original image according to an embodiment of the present invention;

[0070] Figure 5 A flowchart of entropy encoding and decoding of an original image by a task-driven adversarial learning image compression method according to an embodiment of the present invention;

[0071] Figure 6 A flowchart of a multi-branch discriminator network for acquiring category information and discriminant information in a task-driven adversarial learning image compression method according to an embodiment of the present invention;

[0072] Figure 7 A flowchart of an embodiment of the present invention for optimizing an image compression network and a multi-branch discriminator network using a loss function through a training set;

[0073] Figure 8 The present invention is a flowchart of encoding and decoding an image based on a trained network to achieve image compression and decompression. DETAILED DESCRIPTION

[0074] All features disclosed in this specification, or steps in all methods or processes disclosed, except mutually exclusive features and / or steps, can be combined in any manner.

[0075] Any feature disclosed in this specification (including any additional claims and abstract), unless otherwise stated, may be replaced by other equivalent or alternative features having similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

[0076] refer to Figure 1 , Figure 2 , a task-driven adversarial learning image compression method provided by an embodiment of the present invention comprises the following steps:

[0077] Build an image compression network and configure it as follows:

[0078] Step S100: pre-processing the original image, including rotation, scaling, cropping, etc.

[0079] First, we select an image to be processed from the image dataset. Random cropping is to select a random position on the image and crop a part of the image from that position according to the given cropping size. This randomness makes the position and size of each crop different, which increases the diversity of the data and helps improve the robustness and generalization ability of the model.

[0080] During the cropping process, we can set the size of the crop, such as specifying a square or rectangular area to crop, and specifying the width and height of the crop. The resulting image area after cropping will be a subset of the input image, retaining important visual features and information. Next, for subsequent calculations and analysis, we convert the cropped image into a tensor format. Tensors are an abstract concept of multidimensional arrays and are widely used in deep learning. By converting images into tensors, we can take advantage of the parallel processing capabilities of computers to efficiently perform various image processing and analysis tasks.

[0081] Finally, after cropping and conversion, we get preprocessed image data represented in the form of tensors. These processed image data can be used to train deep learning models and perform image compression tasks.

[0082] Step S200: The preprocessed original image x i Input into the encoder to encode the image. The encoder consists of a convolutional layer, a generalized separable nonlinear (GDN) layer, and a residual block to extract the features of the original image and generate the first potential representation z i ;

[0083] The following takes an image of size H×W as an example to explain in detail how the lightweight image compression model compresses the original image into a feature map. Figure 3 As shown, step S200 includes:

[0084] Step S210: construct a depth-separable convolutional network structure including multiple convolutional layers. Send the original image to the convolutional network, and obtain a downsampled feature map through convolution.

[0085] Specifically, first, we take an image of size H×W as an example and use it as the original input image. After random cropping and tensor conversion in step S100, the image becomes a tensor of shape (H, W, 3), where 3 represents the RGB channels of the image. Next, we construct a depthwise separable convolutional network structure. Depthwise separable convolution is a convolution operation that decomposes the standard convolution of the convolution layer into two steps: depthwise grouped convolution and pointwise convolution.

[0086] In deep grouped convolution, we convolve the input tensor through a set of convolution kernels. The convolution kernels here can be thought of as a set of filters used to extract different features from the image. Through the convolution operation, we can slide the convolution kernel in the image and perform a weighted combination of the pixels at each position with the convolution kernel to obtain the output feature map. The purpose of deep grouped convolution is to group the number of channels of the input feature map to reduce the amount of computation and the number of parameters.

[0087] Next, the feature map after the deep group convolution will be sent to the point-by-point convolution layer. Point-by-point convolution is a convolution operation with a convolution kernel of only one pixel, which can be regarded as a linear combination of channels of the feature map. Through point-by-point convolution, we can effectively integrate features and transform dimensions to obtain the downsampled feature map.

[0088] Step S220: Add a LeakyReLU activation function and a normalization layer to each convolutional layer. The asymmetry of LeakyReLU helps capture asymmetric features in the image and better preserve image detail information. The normalization layer is used to accelerate network convergence and improve the robustness of the model.

[0089] In step S210, the downsampled feature map is obtained. In order to increase the expressive power of the network and introduce nonlinearity, an activation function is added after each convolution layer. In this embodiment, LeakyReLU (Leaky RectifiedLinear Unit) is used as the activation function. LeakyReLU is a variant of the rectified linear unit, which introduces a small slope in the negative region to avoid the problem of neuron death. Specifically, for the input x, the LeakyReLU function is defined as: f(x) = max(ax,x), where a is a parameter less than 1, usually set to 0.1 or 0.01. The LeakyReLU function retains the positive input and introduces a certain negative slope in the negative region, thereby increasing the nonlinear ability of the network. After adding the activation function, a normalization layer can also be added between adjacent convolutional layers to accelerate the convergence of the network and improve the robustness of the model. In this embodiment, GDN is used as the normalization layer.

[0090] GDN is a normalization technique that normalizes each channel of a feature map by dividing it by an estimate of the local pixel mean and variance. Specifically, for each channel in a feature map, GDN first calculates the local pixel mean and variance estimate for the channel, and then divides each pixel value of the channel by the corresponding mean and variance. The normalization operation can make each channel of the feature map have a relatively consistent scale and dynamic range, which is helpful for network training and optimization. At the same time, GDN is also adaptable and can make adaptive adjustments based on the statistical characteristics of the input data, thereby improving the robustness of the model to different data distributions.

[0091] By adding the LeakyReLU activation function and the GDN normalization layer, the nonlinear expression ability of the depthwise separable convolutional network is further enhanced in step S220, and the convergence speed and robustness of the model are improved. These operations enable the network to better extract the features of the input image and provide more accurate and reliable feature representation for subsequent image compression and processing tasks.

[0092] Step S230: adding a residual block structure to each convolutional layer.

[0093] The residual block used in this embodiment mainly consists of two parts: a convolutional layer and a jump connection. The jump connection adds the input of the residual block to the output of the convolutional layer, and by enhancing the transmission and learning of features, the network can better retain the information of the image during the compression process.

[0094] Step S240: Through forward propagation, the preprocessed original image is input into the deep separable convolutional network for downsampling. The original image undergoes multiple operations of the convolutional layer to gradually extract higher-level features. The output of the deep separable convolutional network is a feature map, in which each channel corresponds to a specific feature.

[0095] In step S220, a deep separable convolutional network with LeakyReLU activation function, GDN normalization layer and residual block has been constructed. Here, the original image is input into the network for forward propagation operation to obtain feature maps. First, the original image is fed as input into the first convolutional layer of the network. This convolutional layer has been defined in step S210, and it contains deep grouped convolution and point-by-point convolution operations. Through this convolutional layer, the original image is convolved to obtain a set of downsampled feature maps.

[0096] Next, this set of feature maps will be passed as input to the next convolutional layer of the network, which is similar to the first convolutional layer and also contains deep grouped convolution and point-by-point convolution operations. Through the operation of this convolutional layer, the feature map is further convolved to extract higher-level features. This process will be repeated many times, each time passing through a new convolutional layer, taking the feature map of the previous layer as input, and performing deep grouped convolution and point-by-point convolution operations. Each convolution operation will further extract features of the image, thereby gradually forming a more abstract and high-level feature representation.

[0097] Finally, after the original image passes through all the convolutional layers, a set of feature maps are obtained as output. The size of these feature maps is usually smaller than the original image, and each channel corresponds to a specific feature. These features can capture different structures, textures, or semantic information in the image.

[0098] Step S250: The feature map output by the deep separable convolutional network is used as the first potential representation. The feature map is passed to the subsequent image compression process as an encoded representation. The feature map has a lower spatial resolution and a higher channel dimension, and contains important feature information of the original image. This encoded representation can be stored and transmitted more efficiently, providing a basis for subsequent image reconstruction.

[0099] In step S240, we obtain a set of feature maps as the output of the depthwise separable convolutional network through the forward propagation operation. These feature maps have lower spatial resolution and higher channel dimension, and each channel corresponds to a specific feature. Next, in step S240, we pass these feature maps as encoded representations to the subsequent image compression process. The encoded representation has the following advantages over the original image:

[0100] 1) Lower spatial resolution: Due to multiple convolution operations, the spatial resolution of the feature map is usually smaller than that of the original image. This means that the feature map can use fewer pixels to represent the structure and texture information in the original image, thereby achieving spatial compression.

[0101] 2) Higher channel dimension: The channel dimension of the feature map represents the presence and strength of different features. The number of channels is usually much larger than the number of color channels of the original image. By increasing the channel dimension, the feature map can better capture the semantic information and abstract features in the image.

[0102] Step S300: Super-encode and entropy-encode the first latent representation. The encoded latent representation is input into the super-prior branch module, and the first latent representation is further compressed by the super-codec and entropy model to obtain the prior information σ i .

[0103] Specifically, Figure 4 As shown, step S300 is as follows:

[0104] Step S310: Transform the encoded latent representation z i Input super encoder, the super encoder includes convolutional layers, ReLU is used as the activation function between convolutional layers, and the output is the first super encoding feature map y i ;

[0105] Step S320: The first super-encoding feature map y i Quantize, and then convert the quantized y i The second super-encoded feature map is generated after encoding and decoding using the entropy model

[0106] Step S330: The second super-coding feature map Input super decoder, super decoder is symmetrical with super encoder, including a series of deconvolution layers and ReLU activation function. The first potential representation z is generated i Prior information σ i .

[0107] Step S400: The first potential representation z i Quantize and use prior information σ i Auxiliary, the quantized first potential representation z i Generate the second latent representation after encoding and decoding using the entropy model

[0108] Specifically, Figure 5 As shown, step S400 is as follows:

[0109] Step S410: The first potential representation z i Quantization is performed. Quantization is the process of converting continuous feature values ​​into discrete representations to reduce the representation space and storage requirements of the data. The quantization operation can be achieved by mapping the values ​​of the feature map to a set of predefined discrete levels.

[0110] The quantification process can be divided into the following steps:

[0111] 1) Select the quantization level: First, you need to determine the quantization level or resolution. The choice of quantization level determines the degree of discretization of the feature map. A higher quantization level can provide a more refined representation, but will generate more data, and a lower quantization level can reduce the amount of data, but will lose some detail information. When selecting the quantization level, you need to consider the storage and transmission limitations, as well as the requirements for the quality of the reconstructed image.

[0112] 2) Establish a quantization table: According to the selected quantization level, establish a quantization table or mapping function. The quantization table maps continuous feature values ​​to discrete quantization levels.

[0113] 3) Apply quantization: Map the value of each pixel in the feature map to the corresponding level in the quantization table. This can be done by comparing the pixel value in the latent representation with the quantization table and finding the closest quantization level. Discretization is achieved by replacing each pixel value in the latent representation with the corresponding quantization level.

[0114] The quantization operation can reduce the representation space and storage requirements of the latent representation, because discrete quantization levels can be represented by fewer bits. However, quantization also introduces information loss because continuous feature values ​​are mapped to discrete levels. Therefore, it is necessary to balance the relationship between compression efficiency and reconstructed image quality when selecting the quantization level. iQuantization is performed to map continuous eigenvalues ​​into discrete representations, thereby achieving data compression and storage optimization.

[0115] Step S420: The quantized first potential representation z is quantized by a probability model generated by prior information. i Entropy coding is performed. Entropy coding is a lossless data compression technique used to further reduce the size of the encoded data. By utilizing the statistical characteristics of the data and the prior information σ i The generated probabilistic model represents common patterns and symbols as shorter sequences of encoded bytes.

[0116] The entropy coding process can be divided into the following steps:

[0117] 1) Establish a probability model: First, a probability model needs to be established to describe the probability of occurrence of different symbols in the feature map. The probability model can be constructed based on the statistical distribution of the potential representation, usually using frequency statistics or probability estimation methods. The probability model of this method is based on the prior information σ i To construct.

[0118] 2) Symbol mapping: According to the probability model, the symbols in the potential representation are mapped to the encoding dictionary of entropy coding. Each symbol in the encoding dictionary corresponds to a short encoding sequence, common symbols correspond to shorter codes, and uncommon symbols correspond to longer codes. By using the probability information in the probability model, symbols with higher frequency can use shorter codes, thereby achieving higher compression efficiency.

[0119] 3) Code generation: Traverse the quantized potential representation and replace each symbol with the corresponding short code. The code generation process can be achieved by looking up the code dictionary or using a probability model for dynamic coding. According to the probability model, symbols with higher frequency are represented by shorter codes, thereby reducing the size of the encoded data.

[0120] 4) Output coded data: Output the generated entropy coded data as the final compressed coded sequence. Entropy coding usually generates coded data of shorter length, which can further reduce the representation space and storage requirements of the potential representation.

[0121] Step S430: The decoding end uses the probability model generated by the prior information to decode the first potential representation z i Entropy decoding is performed. Entropy decoding is the process of decoding the entropy-encoded data back to the original data. i The generated probabilistic model and decoding algorithm convert the entropy coded data back to the quantized latent representation.

[0122] In the decoding process, the probability model is needed to restore the state information of the encoding process. The encoded bytes are read one by one from the entropy encoded data.

[0123] According to the probability model and decoding algorithm, the read encoded bytes are converted back to the quantized potential representation. After decoding is completed, the decoded second potential representation is obtained. As the final decoding result, this feature map should be exactly the same as the quantized potential representation before encoding, achieving lossless data decoding.

[0124] Step S500: The second potential representation The decoder is symmetrical with the encoder and includes a series of deconvolution layers and inverse GDN layers. Restore to the original image x i Reconstructed images of the same size

[0125] Specifically, step S500 is as follows:

[0126] Using depthwise separable convolution technique and pixel reorganization, the second latent representation Restore to the original image. Pixel reorganization is to rearrange and combine the pixels in the potential representation to restore the spatial structure and pixel values ​​of the original image. The specific application of pixel reorganization and depth-separable convolution technology is as follows:

[0127] 1) Pixel reorganization: In the feature map, pixels are usually rearranged and combined to restore the spatial structure of the original image. This can be achieved by remapping the pixels of the latent representation back to the pixel positions of the original image. Through pixel reorganization, the pixels of the latent representation can be restored to their positions in the original image, thus restoring the spatial structure of the original image.

[0128] 2) Depthwise separable convolution: Depthwise separable convolution is a convolution operation that can effectively reduce the number of parameters and improve computational efficiency. Applying depthwise separable convolution on the latent representation can further extract features and enhance image details. This can restore the pixel values ​​and features of the original image while maintaining high efficiency.

[0129] Through the application of depthwise separable convolution technology and pixel reorganization, the latent representation can be restored to the original image. Pixel reorganization restores the spatial structure of the image, while depthwise separable convolution helps to restore the pixel values ​​and features of the image. In this way, the final image will be closer to the original image with higher quality and accuracy.

[0130] Build a multi-branch discriminator network and configure it as follows:

[0131] Step S600: convert the original image xi and reconstructed image Input the multi-branch discriminator network to obtain category information and discrimination information.

[0132] In detail, Figure 6 As shown, the step S600 specifically includes:

[0133] Step S610: convert the original image x i and reconstructed image The input feature extraction module adopts a multi-layer convolutional neural network structure and introduces a batch normalization layer and a LeakyReLU activation function to obtain a multi-scale and multi-level feature representation of the input image.

[0134] The input of the feature extraction module includes the original image and the image reconstructed by the graph compression network. Through multi-layer convolution and downsampling operations, the original 3×H×W dimensional image is mapped to a 512×H×W dimensional feature space.

[0135] Step S620: input the extracted feature representation into an image classification module to generate category information.

[0136] The image classification module is designed to evaluate the impact of the reconstructed image on the downstream classification task at the feature level. This module uses the feature representation output by the feature extraction module as input, and uses a fully connected layer to integrate the features and map nonlinear functions to achieve classification of image content. During the training process, the classification loss generated by the image classification module will be fed back to the image compression network to guide it to retain semantic information related to task decisions during the compression process and maintain the recognizability of the image content.

[0137] Step S630: input the extracted features into the authenticity determination module to generate determination information.

[0138] The authenticity discrimination module focuses on the closeness of the reconstructed image to the original image at the pixel level, making up for the shortcomings of the image classification module in terms of perceptual quality. Based on a fully connected layer network, this module maps the feature representation output by the feature extraction module to a binary classification label of 0 or 1 through a binary mapping function to determine whether the input is an original image or a reconstructed image. The discrimination loss generated by the authenticity discrimination module prompts the image compression network to generate a more realistic and better quality reconstructed image, thereby ensuring good visual quality of the reconstruction result.

[0139] Through the task-driven multi-branch discriminator network, the image compression network can learn to generate reconstructed images that can be correctly recognized and have high visual quality during training.

[0140] Step S700: Train the image compression network and the multi-branch discriminator network in stages. First, use the rate-distortion loss function in deep learning image compression to independently train the image compression network. When the performance of the image compression network reaches convergence, introduce the multi-branch discriminator network to start adversarial joint training. First, load the previously trained weights of the image compression network, and then use the loss function to alternately train the image compression network and the multi-branch discriminator network.

[0141] Preferably, if Figure 7 As shown, the step S700 includes:

[0142] Step S710: prepare a training sample image set and divide it into a training set and a validation set. The training sample image set includes a series of diverse and representative images.

[0143] Preparing a training sample image set is an important step in training the network. By preparing diverse and representative image samples and reasonably dividing the training set and validation set, we can ensure that the network has good generalization ability and performance evaluation accuracy. In this way, the trained network can be better applied to the processing and compression of unknown data in subsequent tasks.

[0144] Step S720: independently train the image compression network using the rate-distortion loss function in deep learning image compression. Adjust the network parameters to minimize the loss function through the back propagation algorithm and the gradient descent optimizer.

[0145] Preferably, a rate-distortion loss function is defined: The rate-distortion loss function is an indicator that comprehensively considers compression quality and compression bit rate. It measures the distortion between the compressed image and the original image, combined with the compression bit rate. Initialize the image compression network parameters: Before starting training, the parameters of the image compression network need to be initialized. Randomly initialized or pre-trained parameters can be used as a starting point. The training sample image is input into the image compression network and forward propagated to generate a compressed image. Then, the rate-distortion loss function is calculated to measure the difference between the compressed image and the original image. Through the back-propagation algorithm, the gradient of the loss function with respect to the network parameters is calculated. Then, a gradient descent optimizer (such as stochastic gradient descent) is used to update the network parameters so that the loss function gradually decreases. Repeat the training process of forward propagation, loss calculation, back-propagation, and parameter update.

[0146] By using the rate-distortion loss function as the objective function to train the image compression network, the network parameters can be optimized so that it can minimize the distortion between the reconstructed image and the original image while maintaining a high compression bit rate. In this way, the trained network can provide better quality and efficiency in subsequent image compression tasks.

[0147] Step S730: Repeat step S720 until the network converges or reaches a predetermined number of training rounds.

[0148] Step S740: Train the multi-branch discriminator network using the adversarial discriminant loss function. Adjust the network parameters to minimize the loss function through the back propagation algorithm and the gradient descent optimizer.

[0149] In this embodiment, the multi-branch discriminant network needs to perform two tasks: one is to distinguish whether the input image is an original image or generated by the image compression network G, that is, the task of generating discriminant information; the other is to accurately identify the category of the input image when it is an original image, that is, the task of generating category information; the information of the two tasks is back-propagated to the image compression network for network parameter optimization. In order to optimize the visual quality of the reconstructed image and the performance of the discriminant network, the proposed adversarial discriminant loss consists of two parts: the discriminant loss of the discriminant network for the original image and the reconstructed image, and the classification loss of the discriminant network for the original image.

[0150] Step S750: train the image compression network using the rate-distortion loss function. Adjust the network parameters to minimize the loss function through the back-propagation algorithm and the gradient descent optimizer.

[0151] Step S760: Repeat alternate steps S740 and S750 until the two networks converge or a predetermined number of training rounds is reached.

[0152] Step S770: Evaluate the performance of the two trained networks on the validation set. Using a set of independent image samples, calculate the image classification accuracy, peak signal-to-noise ratio (PSNR), and multi-scale structural similarity (MS-SSIM) of the compressed images to measure the compression effect and image classification performance of the network.

[0153] Calculate the evaluation indicators of all images on the validation set, and perform statistics and analysis. The mean, standard deviation, and distribution can be calculated to understand the overall performance of the network on the validation set. It is important to evaluate the performance of the trained network on the validation set, which can help evaluate the generalization ability and compression effect of the network. Based on the evaluation results, the network design or training strategy can be further adjusted to improve the performance of the network and image quality.

[0154] Step S780: Save the trained network model, that is, the network parameters of the two networks. Save the network parameters (such as weights and configurations) in a file for subsequent image compression tasks. This can avoid the need to retrain the network every time an image is compressed, thereby improving the efficiency and speed of compression.

[0155] Specifically, step S780 includes:

[0156] 1) Determine the save path: Choose a suitable path to save the network model file. This path should be accessible to ensure that the model can be loaded correctly in subsequent tasks.

[0157] 2) Save network parameters: Save the network's weights and parameters to a file. Generally, modern deep learning frameworks provide functions and tools for saving and loading models. You can use the corresponding functions to save the network's parameters to a specified file.

[0158] 3) Save training configuration: In addition to the network weights and parameters, you can also save the training configuration information, including training parameters, optimizer status, etc. This information can help ensure that when loading the model, it can be restored to the state at the end of training.

[0159] After saving the trained network model, you can directly load the model and use it to compress and reconstruct images when you need to perform image compression tasks. This can save time and computing resources and ensure that a consistent network model is used in subsequent tasks.

[0160] The trained image compression network is used to process the image to be compressed in steps S100-S200, and the first latent representation of the encoding output is used as the compressed image; or, the trained image compression network is used to process the compressed image in steps S300-S500, and the reconstructed image of the decoding output is used as the decompressed image.

[0161] Step S800: When an image to be compressed is received, the image is encoded based on the trained image compression network to achieve image compression. The compressed image can be sent to the target device or stored locally for subsequent use. When a compressed image to be decompressed is received, the image is decoded based on the trained image compression network to achieve image decompression.

[0162] Preferably, if Figure 8 As shown, the step S800 specifically includes:

[0163] Compression method:

[0164] Step S810: Obtain the original data of the image to be compressed. This data may include pixel values, color channel information, etc. of the image. The goal is to encode this data effectively to reduce the size of the image while maintaining the image quality as much as possible.

[0165] Step S820: Input the acquired raw data into the trained image compression network and compress it using the encoder to obtain the first latent representation. Through the encoder part of the network, the raw data is converted into the first latent representation, which contains the important features of the image.

[0166] Decompression method:

[0167] Step S830: Input the compressed image into the trained image compression network, and obtain prior information through super encoding and decoding and entropy encoding and decoding.

[0168] Step S840: quantize the compressed image, use the prior information to assist, and use the entropy model to perform entropy encoding and decoding to obtain a corresponding second potential representation.

[0169] Step S850: Based on the pixel reconstruction technology, the second latent representation is decoded to generate a reconstructed image. This process is the inverse process of encoding, and aims to restore a reconstructed image as close as possible to the original image from the latent representation.

[0170] Step S860: Deliver the reconstructed image. The decoded reconstructed image is sent to the target device or stored locally. The compressed image can be transmitted to the remote device via the network or stored in the local storage medium for subsequent use.

[0171] Example 2

[0172] This embodiment proposes a task-driven adversarial learning image compression system, which is configured to execute the task-driven adversarial learning image compression method in Embodiment 1.

[0173] Specifically, the system includes a processor and a memory, the memory stores a computer program, and running the computer program can execute the method of embodiment 1, and the processor is connected to the memory to run the computer program therein to execute the method of embodiment 1. In addition, the training sample image set is also stored in the memory.

[0174] The present invention is not limited to the above-mentioned specific embodiments, but extends to any new features or any new combination disclosed in this specification, as well as any new method or process steps or any new combination disclosed.

Claims

1. A task-driven adversarial learning image compression method, characterized in that: include: Construct image compression network and multi-branch discriminator network respectively; The image compression network processes the input image by: S1, preprocessing the original image; S2, encoding the preprocessed original image to generate a first latent representation; S3, performing super encoding and decoding and entropy encoding and decoding on the first potential representation to obtain prior information; S4, quantizing the first latent representation, and performing entropy encoding and decoding based on the prior information to generate a second latent representation; S5. Decode the second latent representation to obtain a reconstructed image; The multi-branch discriminator network processes the input image including: S6, inputting the original image and the reconstructed image into a multi-branch discriminator network to obtain category information and discriminant information; S7, training the image compression network and the multi-branch discriminator network in stages; S8. Use the trained image compression network to process the image to be compressed according to steps S1-S2, and use the first latent representation of the encoded output as the compressed image; or use the trained image compression network to process the compressed image according to steps S3-S5, and use the reconstructed image of the decoded output as the decompressed image.

2. The task-driven adversarial learning image compression method according to claim 1, characterized in that: The step S2 comprises: S21, constructing a deep separable convolutional network, wherein the deep separable convolutional network includes multiple convolutional layers; S22, add LeakyReLU activation function and normalization layer to each convolution layer; S23, adding a residual block to each convolutional layer, wherein the residual block adds the input of the convolutional layer to the output of the convolutional layer; S24, inputting the preprocessed original image into the depth-separable convolutional network for downsampling; S25. The feature map output by the depthwise separable convolutional network is used as the first potential representation.

3. The task-driven adversarial learning image compression method according to claim 1, characterized in that: The step S3 comprises: S31, inputting the first potential representation into a super encoder for super encoding to obtain a first super encoding feature map; the super encoder includes a convolution layer, and ReLU is used as an activation function between adjacent convolution layers; S32, quantizing the first super-encoded feature map, and then performing entropy coding and entropy decoding to generate a second super-encoded feature map; S33: Input the second super-encoded feature map into a super-decoder symmetrical to the super-encoder to obtain prior information.

4. The task-driven adversarial learning image compression method according to claim 1, characterized in that: The step S4 comprises: S41, quantizing the first potential representation; S42, performing entropy coding on the quantized first potential representation through a probability model generated by prior information; S43. Perform entropy decoding on the entropy-encoded first latent representation through a probability model generated by prior information to generate a second latent representation.

5. The task-driven adversarial learning image compression method according to claim 2, characterized in that: The step S5 comprises: Depth-wise separable convolution and pixel reorganization are performed on the second latent representation to obtain a reconstructed image.

6. The task-driven adversarial learning image compression method of claim 1, characterized in that: The step S6 comprises: S61, inputting the original image and the reconstructed image into a feature extraction module to obtain feature representation; the feature extraction module adopts a multi-layer convolutional neural network, followed by a batch normalization layer and a LeakyReLU activation function; S62, inputting the feature representation into an image classification module to generate category information; the image classification module uses a fully connected layer and a nonlinear activation function to classify the input feature representation; S63, inputting the feature representation into an authenticity discrimination module to generate discrimination information; the authenticity discrimination module uses a fully connected layer and a binary mapping function to perform authenticity discrimination on the input feature representation.

7. The task-driven adversarial learning image compression method of claim 1, characterized in that: In step S7, the staged training is as follows: The image compression network is first trained independently, and after the training converges, the multi-branch discriminator network is introduced to perform adversarial joint training.

8. The task-driven adversarial learning image compression method of claim 7, characterized in that: The step S7 comprises: S71, constructing a training sample image set and dividing it into a training set and a validation set, wherein the training set is used in a network training process; S72, independently training the image compression network using a rate-distortion loss function, and adjusting network parameters during training using a back-propagation algorithm and a gradient descent optimizer; S73, iteratively training the image compression network until convergence, or iterating a predetermined number of rounds; S74, training the multi-branch discriminator network using an adversarial discriminant loss function, and adjusting network parameters during training using a back propagation algorithm and a gradient descent optimizer; S75, training the image compression network using a rate-distortion loss function, and adjusting network parameters during training using a back-propagation algorithm and a gradient descent optimizer; S76, repeating alternately step S74 and step S75 until the two networks converge, or iterating for a predetermined number of rounds; S77, using the verification set to verify the performance of the two trained networks; S78. Save the network parameters of the two trained networks.

9. The task-driven adversarial learning image compression method of claim 1, characterized in that: In step S8, compressing the image to be compressed includes: S81, obtaining original data of the image to be compressed; S82, inputting the original data of the image to be compressed into a trained image compression network, and obtaining a first potential representation using an encoder; Alternatively, decompression of a compressed image involves: S83, inputting the compressed image into a trained image compression network, and obtaining prior information through super encoding and decoding and entropy encoding and decoding; S84, quantizing the compressed image, performing entropy encoding and decoding based on prior information, and obtaining a corresponding second potential representation; S85. Decode the obtained second latent representation to generate a reconstructed image.

10. A task-driven adversarial learning image compression system, characterized in that: The system is configured to execute the method according to any one of claims 1 to 9.

Citation Information

Patent Citations

  • Natural image compression method based on generative adversarial network

    CN110225350A

  • Abnormal brain connection prediction system, method and device and readable storage medium

    CN113724880A

  • Image encoding and decoding, video encoding and decoding: methods, systems, and training methods

    CN116584098A

  • Satellite image compression method under extremely low bandwidth condition

    CN116939213A

  • Image compression method based on enhanced multi-scale residual generative adversarial network

    CN117173263A

Cited By

  • Forestry remote sensing data lightweight method based on submerged space optimization

    CN119418081A

  • End-to-end joint optimization method for image compression transmission application

    CN119767041A

  • End-to-end joint optimization method for image compression transmission application

    CN119767041B