A task-driven adversarial learning image compression method and system

Abstract

The present invention discloses a task-driven adversarial learning image compression method and system, which relates to the field of image compression and is used to improve the performance of machine vision tasks while maintaining the visual quality of human eyes. The present invention realizes compression by encoding an image through a trained image compression system; for the compressed image, super-encoding and decoding and entropy encoding and decoding are performed to obtain prior information, and entropy encoding and decoding are performed based on the prior information to generate a second latent representation, and then the reconstructed image is decoded to realize decompression. A trained multi-branch discriminator network is used to distinguish the authenticity of the reconstructed image and identify the category of the original image. The present invention can maintain a high image quality while reducing the size of the image file, retain key details, improve the transmission efficiency, save the storage cost, and improve the accuracy of downstream image classification.

Description

Technical Field

[0001] The present invention relates to the field of image compression, and in particular to a task-driven adversarial learning image compression method and system. Background Art

[0002] In order to efficiently store and transmit the increasingly large image data with the development of Internet technology, image compression technology has emerged. Traditional image compression methods are based on specific mathematical models and compress image data through transformation, quantization, and entropy coding. There is a trade-off between compression efficiency and image quality in such methods, and high compression ratios and high image quality often cannot coexist. In addition, the performance of such methods usually depends on their preset assumptions about image content, which means that for images of different types and characteristics, their compression effects may not be ideal.

[0003] In many practical applications, the compressed images are not only for human eyes to view, but many times for various image analysis tasks. For example, in the fields of security monitoring, medical analysis, and autonomous driving, images are further used for object detection, semantic segmentation, and environmental perception in autonomous driving. However, traditional image compression methods mainly focus on the human visual quality of the reconstructed images, and the design and optimization of the methods often do not consider the final usage scenarios of the images, resulting in performance degradation of the compressed images in subsequent image analysis tasks.

[0004] Image compression methods based on deep learning have brought new breakthroughs to the field of image compression. Deep learning networks are used to capture the essential structure of data and use it to efficiently represent the data, achieving automatic learning and extraction of effective representations of images. In the existing Compress then Analyze (CTA) mode in image compression methods based on deep learning, there is also a problem that machine vision information is biased to be retained during the compression process, and human visual information is removed as redundant information, resulting in poor visual image quality. Summary of the Invention

[0005] The object of the present invention is to provide a task-driven adversarial learning image compression method and system aiming at the above problems, aiming to improve the performance of machine vision tasks while maintaining the human visual quality.

[0006] The technical solution adopted by the present invention is as follows:

[0007] A task-driven adversarial learning image compression method, which includes:

[0008] Constructing an image compression network and a multi-branch discriminator network respectively;

[0009] Among them, the processing of the input image by the image compression network includes:

[0010] S1. Preprocess the original image;

[0011] S2. Encode the preprocessed original image to generate a first latent representation;

[0012] S3. Perform hyper-encoding and decoding and entropy encoding and decoding on the first latent representation to obtain prior information;

[0013] S4. Quantize the first latent representation and perform entropy encoding and decoding based on the prior information to generate a second latent representation;

[0014] S5. Decode the second latent representation to obtain a reconstructed image;

[0015] The processing of the input image by the multi-branch discriminator network includes:

[0016] S6. Input the original image and the reconstructed image into the multi-branch discriminator network to obtain class information and discriminant information;

[0017] S7. Train the image compression network and the multi-branch discriminator network in stages;

[0018] S8. Use the trained image compression network to perform the processing of steps S1 - S2 on the image to be compressed, and use the encoded output first latent representation as the compressed image; or, use the trained image compression network to perform the processing of steps S3 - S5 on the compressed image, and use the decoded output reconstructed image as the decompressed image.

[0019] Further, the step S2 includes:

[0020] S21. Construct a depthwise separable convolution network, and the depthwise separable convolution network includes multiple convolutional layers;

[0021] S22. Add a LeakyReLU activation function and a normalization layer to each convolutional layer;

[0022] S23. Add a residual block to each convolutional layer, and the residual block adds the input of the convolutional layer to the output of the convolutional layer;

[0023] S24. Input the preprocessed original image into the depthwise separable convolution network for downsampling;

[0024] S25. Use the feature map output by the depthwise separable convolution network as the first latent representation.

[0025] Further, the step S3 includes:

[0026] S31. Input the first latent representation into a super encoder for super encoding to obtain a first super-encoded feature map; the super encoder includes convolutional layers, and ReLU is used as the activation function between adjacent convolutional layers;

[0027] S32. Quantize the first super-encoded feature map, then perform entropy encoding and entropy decoding to generate a second super-encoded feature map;

[0028] S33. Input the second super-encoded feature map into a super decoder symmetric to the super encoder to obtain prior information.

[0029] Further, the step S4 includes:

[0030] S41. Quantize the first latent representation;

[0031] S42. Perform entropy encoding on the quantized first latent representation through a probability model generated from the prior information;

[0032] S43. Perform entropy decoding on the entropy-encoded first latent representation through a probability model generated from the prior information to generate a second latent representation.

[0033] Further, the step S5 includes:

[0034] Perform depthwise separable convolution and pixel recombination on the second latent representation to obtain a reconstructed image.

[0035] Further, the step S6 includes:

[0036] S61. Input the original image and the reconstructed image into a feature extraction module to obtain a feature representation; the feature extraction module uses a multi-layer convolutional neural network, followed by a batch normalization layer and a LeakyReLU activation function;

[0037] S62. Input the feature representation into an image classification module to generate class information; the image classification module uses a fully connected layer and a non-linear activation function to classify the input feature representation;

[0038] S63. Input the feature representation into a authenticity discrimination module to generate discrimination information; the authenticity discrimination module uses a fully connected layer and a binary mapping function to discriminate the authenticity of the input feature representation.

[0039] Further, in the step S7, the staged training is as follows:

[0040] First, independently train the image compression network. When the training converges, introduce the multi-branch discriminator network for adversarial joint training.

[0041] Further, the step S7 includes:

[0042] S71, constructing a training sample image set and dividing it into a training set and a validation set, wherein the training set is used for the network training process;

[0043] S72. Independently train the image compression network using a rate-distortion loss function, and adjust network parameters during training using a backpropagation algorithm and a gradient descent optimizer;

[0044] S73, iteratively training the image compression network until convergence, or iterating for a predetermined number of rounds;

[0045] S74, training the multi-branch discriminator network using an adversarial discriminant loss function, and adjusting network parameters during training using a back propagation algorithm and a gradient descent optimizer;

[0046] S75. Training the image compression network using a rate-distortion loss function, and adjusting network parameters during training using a backpropagation algorithm and a gradient descent optimizer;

[0047] S76, repeating alternate steps S74 and S75 until the image compression network converges, or iterating a predetermined number of rounds;

[0048] S77, using the verification set to verify the performance of the trained image compression network;

[0049] S78. Save the network parameters of the trained image compression model.

[0050] Furthermore, in step S8, compressing the image to be compressed includes:

[0051] S81, obtaining original data of the image to be compressed;

[0052] S82, inputting the original data of the image to be compressed into a trained image compression network, and obtaining a first latent representation using an encoder;

[0053] Alternatively, decompressing a compressed image involves:

[0054] S83, inputting the compressed image into a trained image compression network, and obtaining prior information through super encoding and decoding and entropy encoding and decoding;

[0055] S84, quantizing the compressed image, performing entropy encoding and decoding based on prior information, and obtaining a corresponding second latent representation;

[0056] S85. Decode the obtained second latent representation to generate a reconstructed image.

[0057] A task-driven adversarial learning image compression system is configured to perform the above method.

[0058] In summary, due to the adoption of the above technical solutions, the beneficial effects of the present invention are as follows:

[0059] 1. It can maintain high image quality while reducing the size of image files. By optimizing the compression algorithm and sampling strategy, this technical solution can efficiently compress image data, reduce the occupation of storage space, and provide faster speed and lower bandwidth requirements for image transmission and storage.

[0060] 2. Retain key details: Despite compression, the present invention can still retain the key details and important features of the image. By precisely controlling the parameters of the compression algorithm and optimizing the image encoding process, this solution can minimize information loss and maintain important details in the compressed image, making the image still visually clear, natural, and recognizable.

[0061] 3. Improve transmission efficiency: Since the size of the compressed image file is reduced, the present invention can improve the efficiency of image transmission. In network transmission or storage, a smaller file size can reduce transmission time and resource consumption, speed up the image loading speed, and provide a faster response time. This is of great significance for scenarios such as online image display, video communication, and cloud storage.

[0062] 4. Save storage costs: By adopting the image compression technology based on adversarial learning, the present invention can significantly reduce storage costs. A smaller image file size means that more image data can be stored in the same storage space, thus saving the demand and cost of storage devices, especially having obvious advantages in large-scale image storage and processing applications.

[0063] 5. Improve the accuracy of downstream image classification: By adopting the image compression technology based on adversarial learning, the images compressed by the present invention achieve higher accuracy in image classification tasks. This means that the compressed images of this model have obvious advantages compared with other compression schemes when applied to image analysis tasks.

[0064] 6. Through adversarial learning, the present invention retains low-level human visual information and high-level semantic information, allowing humans to intuitively evaluate the reconstructed images while being more adaptable to specific downstream tasks. The present invention maintains the human visual quality of the image while ensuring the performance of machine vision tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The present invention will be described by way of examples with reference to the accompanying drawings, where:

[0066] Figure 1 is the flowchart of the task-driven adversarial learning image compression method according to the embodiment of the present invention;

[0067] Figure 2Schematic diagram of the principle of the task-driven adversarial learning image compression method according to an embodiment of the present invention;

[0068] Figure 3 Flowchart of encoding the original image by the task-driven adversarial learning image compression method according to an embodiment of the present invention;

[0069] Figure 4 Flowchart of obtaining the prior information of the original image by the task-driven adversarial learning image compression method according to an embodiment of the present invention;

[0070] Figure 5 Flowchart of entropy encoding and decoding the original image by the task-driven adversarial learning image compression method according to an embodiment of the present invention;

[0071] Figure 6 Flowchart of the multi-branch discriminator network obtaining class information and discriminant information by the task-driven adversarial learning image compression method according to an embodiment of the present invention;

[0072] Figure 7 Flowchart of optimizing the image compression network and the multi-branch discriminator network by using a loss function through a training set according to an embodiment of the present invention;

[0073] Figure 8 Flowchart of encoding and decoding an image based on a trained network to implement image compression and decompression according to an embodiment of the present invention. Detailed implementation manners

[0074] All features disclosed in this specification, or all steps in the disclosed methods or processes, except for mutually exclusive features and / or steps, can be combined in any manner.

[0075] Any feature disclosed in this specification (including any additional claims, abstract) can be replaced by other equivalent or similar-purpose alternative features unless specifically stated otherwise. That is, unless specifically stated otherwise, each feature is only an example of a series of equivalent or similar features.

[0076] Reference Figure 1 、 Figure 2 A task-driven adversarial learning image compression method provided by an embodiment of the present invention includes the following steps:

[0077] Construct an image compression network and perform the following configurations:

[0078] Step S100: Preprocess the original image, including rotation, scaling, cropping, etc.

[0079] First, we select an image to be processed from the image dataset. Random cropping is to select a random position on the image and crop a part of the image starting from that position according to the given cropping size. This randomness makes the cropping position and size different each time, increasing the diversity of the data and helping to improve the robustness and generalization ability of the model.

[0080] During the cropping process, we can set the cropping size, for example, specify the crop as a square or rectangular area and specify the width and height of the crop. The resulting image area after cropping will be a subset of the input image, retaining important visual features and information. Next, for subsequent calculations and analyses, we convert the cropped image into a tensor format. A tensor is an abstract concept of a multi-dimensional array and is widely used in deep learning. By converting the image into a tensor, we can utilize the parallel processing power of the computer to efficiently perform various image processing and analysis tasks.

[0081] Finally, after cropping and conversion are completed, we obtain the preprocessed image data, represented in the form of a tensor. These processed image data can be used for tasks such as training a deep learning model and image compression.

[0082] Step S200: Input the preprocessed original image x i into the encoder to encode the image. The encoder consists of convolutional layers, Generalized Divisive Normalization (GDN) layers, and residual blocks, and is used to extract the features of the original image to generate the first latent representation z i ;

[0083] Taking an image of size H×W as an example, the following details how the lightweight image compression model compresses the original image into a feature map. As Figure 3 shown, step S200 includes:

[0084] Step S210: Construct a depthwise separable convolutional network structure containing multiple convolutional layers. Feed the original image into this convolutional network, and after convolution, a downsampled feature map is obtained.

[0085] Specifically, first, taking an image of size H×W as an example, we use it as the original input image. After random cropping and tensor conversion in step S100, this image becomes a tensor of shape (H,W,3), where ³ represents the RGB channels of the image. Next, we construct a depthwise separable convolutional network structure. Depthwise separable convolution is a convolution operation that decomposes the standard convolution in a convolutional layer into two steps: depthwise grouped convolution and pointwise convolution.

[0086] In depthwise separable convolution, we perform a convolution operation on the input tensor using a set of convolutional kernels. These convolutional kernels can be regarded as a set of filters for extracting different features in the image. Through the convolution operation, we can slide the convolutional kernel over the image, perform a weighted combination of the pixels at each position with the convolutional kernel, and obtain the output feature map. The purpose of depthwise separable convolution is to group the number of channels of the input feature map to reduce the computational amount and the number of parameters.

[0087] Next, the feature map after depthwise separable convolution is fed into the pointwise convolution layer. Pointwise convolution is a convolution operation with a convolutional kernel of only one pixel, which can be regarded as a linear combination between channels of the feature map. Through pointwise convolution, we can effectively integrate features and transform dimensions, thus obtaining the downsampled feature map.

[0088] Step S220: Add a LeakyReLU activation function and a normalization layer to each convolutional layer. The asymmetry of LeakyReLU helps to capture the asymmetric features in the image and better retain the detailed information of the image. The normalization layer is used to accelerate the convergence of the network and improve the robustness of the model.

[0089] In step S210, the downsampled feature map is obtained. To increase the expressive power of the network and introduce non-linearity, an activation function is added after each convolutional layer. In this embodiment, LeakyReLU (Leaky Rectified Linear Unit) is used as the activation function. LeakyReLU is a variant of the rectified linear unit, which introduces a small slope in the negative value region to avoid the problem of neuron death. Specifically, for the input x, the LeakyReLU function is defined as: f(x) = max(ax, x), where a is a parameter less than 1, usually set to 0.1 or 0.01. The LeakyReLU function retains the positive input and introduces a certain negative slope in the negative value region, thus increasing the non-linear ability of the network. After adding the activation function, a normalization layer can also be added between adjacent convolutional layers to accelerate the convergence of the network and improve the robustness of the model. In this embodiment, GDN is used as the normalization layer.

[0090] GDN is a normalization technique that normalizes by dividing each channel of the feature map by an estimate of the local pixel mean and variance. Specifically, for each channel in the feature map, GDN first calculates the local pixel mean and variance estimate of that channel, and then divides each pixel value of that channel by the corresponding mean and variance. The normalization operation can make the channels of the feature map have relatively consistent scales and dynamic ranges, which helps the training and optimization of the network. At the same time, GDN also has adaptability and can be adaptively adjusted according to the statistical characteristics of the input data, improving the robustness of the model to different data distributions.

[0091] By adding the LeakyReLU activation function and the GDN normalization layer, the non-linear expression ability of the depthwise separable convolutional network is further enhanced in step S220, and the convergence speed and robustness of the model are improved. These operations enable the network to better extract the features of the input image and provide a more accurate and reliable feature representation for subsequent image compression and processing tasks.

[0092] Step S230: Add a residual block structure to each convolutional layer.

[0093] The residual block used in this embodiment mainly consists of two parts: a convolutional layer and a skip connection. The skip connection adds the input of the residual block to the output of the convolutional layer. By enhancing the transmission and learning of features, the network can better retain the information of the image during the compression process.

[0094] Step S240: Through forward propagation, the preprocessed original image is input into the depthwise separable convolutional network for downsampling. The original image undergoes multiple operations of the convolutional layer and gradually extracts higher-level features. The output of the depthwise separable convolutional network is the feature map, where each channel corresponds to a specific feature.

[0095] In step S220, a depthwise separable convolutional network with the LeakyReLU activation function, the GDN normalization layer, and the residual block has been constructed. Here, the original image is input into this network for forward propagation operations to obtain the feature map. First, the original image is fed as input into the first convolutional layer of the network. This convolutional layer has been defined in step S210 and it includes depthwise grouped convolution and pointwise convolution operations. Through this convolutional layer, after the convolutional operation on the original image, a set of downsampled feature maps is obtained.

[0096] Next, this set of feature maps will be passed as input to the next convolutional layer of the network. This convolutional layer is similar to the first convolutional layer and also includes depthwise grouped convolution and pointwise convolution operations. Through the operations of this convolutional layer, the feature maps are further convolved to extract higher-level features. This process will be repeated multiple times. Each time, a new convolutional layer is passed, taking the feature maps of the previous layer as input and performing depthwise grouped convolution and pointwise convolution operations. Each convolutional operation will further extract the features of the image, thus gradually forming a more abstract and higher-level feature representation.

[0097] Finally, when the original image passes through all the operations of the convolutional layers, a set of feature maps is obtained as the output. The size of these feature maps is usually smaller than the original image, and each channel corresponds to a specific feature. These features can capture different structures, textures, or semantic information in the image.

[0098] Step S250: The feature map output by the depthwise separable convolutional network serves as the first latent representation. The feature map, as the encoded representation, is passed to the subsequent image compression process. The feature map has a low spatial resolution and a high channel dimension, containing important feature information of the original image. This encoded representation can be stored and transmitted more efficiently, providing a basis for subsequent image reconstruction.

[0099] In step S240, we obtained a set of feature maps as the output of the depthwise separable convolutional network through forward propagation operations. These feature maps have a low spatial resolution and a high channel dimension, with each channel corresponding to a specific feature. Next, in step S240, we passed these feature maps as the encoded representation to the subsequent image compression process. The encoded representation has the following advantages compared to the original image:

[0100] 1) Low spatial resolution: Due to multiple convolution operations, the spatial resolution of the feature map is usually smaller than that of the original image. This means that the feature map can represent the structural and texture information in the original image with fewer pixels, thus achieving spatial compression.

[0101] 2) High channel dimension: The channel dimension of the feature map represents the presence and intensity of different features. The number of channels is usually much larger than the number of color channels in the original image. By increasing the channel dimension, the feature map can better capture the semantic information and abstract features in the image.

[0102] Step S300: Perform hyper-encoding and decoding and entropy encoding and decoding on the first latent representation. Input the encoded latent representation into the hyperprior branch module, and further compress the first latent representation through a hyper-encoder and decoder as well as an entropy model to obtain the prior information σ i .

[0103] Specifically, as Figure 4 shown, step S300 is as follows:

[0104] Step S310: Input the encoded latent representation z i into the hyper-encoder. The hyper-encoder includes convolutional layers, and ReLU is used as the activation function between the convolutional layers. The output is the first hyper-encoded feature map y i ;

[0105] Step S320: Quantize the first hyper-encoded feature map y i , and then encode and decode the quantized y i using the entropy model to generate the second hyper-encoded feature map

[0106] Step S330: The second hyper-encoded feature map Input super decoder, which is symmetric to the super encoder, includes a series of transposed convolutional layers and ReLU activation functions. The first latent representation z will be generated. i The prior information σ i .

[0107] Step S400: Quantize the first latent representation z i with the assistance of the prior information σ i . After encoding and decoding the quantized first latent representation z i using an entropy model, a second latent representation

[0108] will be generated. Specifically, as Figure 5 shown, step S400 is as follows:

[0109] Step S410: Quantize the first latent representation z i . Quantization is the conversion of continuous eigenvalue into discrete representation to reduce the representation space and storage requirement of data. By mapping the values of the feature map to a set of predefined discrete levels, the quantization operation can be achieved.

[0110] The process of quantization can be divided into the following steps:

[0111] 1) Select quantization levels: First, it is necessary to determine the quantization levels or resolution. The choice of quantization levels determines the degree of discretization of the feature map. Higher quantization levels can provide a more refined representation, but will generate more data volume. Lower quantization levels can reduce the data volume, but will lose some detailed information. When choosing quantization levels, the limitations of storage and transmission, as well as the requirements for the quality of the reconstructed image, need to be considered.

[0112] 2) Establish a quantization table: According to the selected quantization levels, establish a quantization table or mapping function. The quantization table maps the continuous eigenvalue to the discrete quantization levels.

[0113] 3) Apply quantization operation: Map the value of each pixel in the feature map to the corresponding level in the quantization table. This can be achieved by comparing the pixel values in the latent representation with the quantization table and finding the closest quantization level. Replace each pixel value in the latent representation with the corresponding quantization level, thus achieving discretization.

[0114] The quantization operation can reduce the representation space and storage requirement of the latent representation, because the discrete quantization levels can be represented with fewer bits. However, quantization will also introduce information loss, because the continuous eigenvalue is mapped to discrete levels. Therefore, when choosing quantization levels, it is necessary to balance the relationship between compression efficiency and the quality of the reconstructed image. Through step S410, the encoded first latent representation z iQuantify to map continuous eigenvalue to discrete representation, thereby achieving data compression and storage optimization.

[0115] Step S420: Through the probability model generated from prior information, perform entropy encoding on the quantized first latent representation z i Entropy encoding is a lossless data compression technique used to further reduce the size of the encoded data. By leveraging the statistical characteristics in the data and the probability model generated from prior information σ i common patterns and symbols are represented as shorter encoded byte sequences.

[0116] The process of entropy encoding can be divided into the following steps:

[0117] 1) Establish a probability model: First, a probability model needs to be established to describe the occurrence probabilities of different symbols in the feature map. The probability model can be constructed based on the statistical distribution of the latent representation, usually using frequency statistics or probability estimation methods. The probability model of this method is based on prior information σ i to construct.

[0118] 2) Symbol mapping: According to the probability model, map the symbols in the latent representation to the encoding dictionary of the entropy encoding. Each symbol in the encoding dictionary corresponds to a short encoding sequence. Common symbols correspond to shorter encodings, while uncommon symbols correspond to longer encodings. By using the probability information in the probability model, symbols with higher occurrence frequencies can use shorter encodings, thus achieving higher compression efficiency.

[0119] 3) Encoding generation: Traverse the quantized latent representation and replace each symbol with its corresponding short encoding. The process of encoding generation can be achieved by looking up the encoding dictionary or performing dynamic encoding using the probability model. According to the probability model, symbols with higher occurrence frequencies are represented by shorter encodings, thereby reducing the size of the encoded data.

[0120] 4) Output the encoded data: Output the generated entropy-encoded data as the final compressed encoding sequence. The encoded data generated by entropy encoding usually has a shorter length, which can further reduce the representation space and storage requirements of the latent representation.

[0121] Step S430: Through the probability model generated from prior information, the decoding end performs entropy decoding on the entropy-encoded first latent representation z i Entropy decoding is the process of decoding the entropy-encoded data back to the original data. By using the probability model and decoding algorithm generated from prior information σ i the entropy-encoded data is converted back to the quantized latent representation.

[0122] During the decoding process, a probability model is needed to restore the state information in the encoding process. The encoded bytes are read one by one from the entropy-encoded data.

[0123] According to the probability model and the decoding algorithm, the read encoded bytes are converted back to the quantized latent representation. After decoding is completed, the decoded second latent representation is obtained as the final decoding result. This feature map should be exactly the same as the quantized latent representation before encoding, achieving lossless data decoding.

[0124] Step S500: Input the second latent representation into the decoder for decoding. The decoder is symmetric to the encoder and includes a series of transposed convolutional layers and inverse GDN layers. The latent representation is restored to a reconstructed image i with the same size as the original image x

[0125] Specifically, step S500 is as follows:

[0126] Using depthwise separable convolution technology and pixel rearrangement, the second latent representation is restored to the original image. Pixel rearrangement is to rearrange and combine the pixels in the latent representation to restore the spatial structure and pixel values of the original image. The specific application of pixel rearrangement and depthwise separable convolution technology is as follows:

[0127] 1) Pixel rearrangement: In the feature map, pixels are usually rearranged and combined to restore the spatial structure of the original image. This can be achieved by remapping the pixels of the latent representation back to the pixel positions of the original image. Through pixel rearrangement, the pixels of the latent representation can be restored to their positions in the original image, thus restoring the spatial structure of the original image.

[0128] 2) Depthwise separable convolution: Depthwise separable convolution is a convolution operation that can effectively reduce the number of parameters and improve the computational efficiency. Applying depthwise separable convolution to the latent representation can further extract features and enhance the details of the image. In this way, while maintaining efficiency, the pixel values and features of the original image can be restored.

[0129] Through the application of depthwise separable convolution technology and pixel rearrangement, the latent representation can be restored to the original image. Pixel rearrangement restores the spatial structure of the image, while depthwise separable convolution helps to restore the pixel values and features of the image. In this way, the finally obtained image will be closer to the original image, with higher quality and accuracy.

[0130] Construct a multi-branch discriminator network and configure it as follows:

[0131] Step S600: Input the original image xi and the reconstructed image Input the multi-branch discriminator network to obtain class information and discrimination information.

[0132] Specifically, as Figure 6 shown, step S600 specifically includes:

[0133] Step S610: Input the original image x i and the reconstructed image into the feature extraction module, which adopts a multi-layer convolutional neural network structure and introduces a batch normalization layer and a LeakyReLU activation function to obtain multi-scale and multi-level feature representations of the input image.

[0134] The input of the feature extraction module includes the original image and the image reconstructed by the graphics compression network. Through multi-layer convolution and downsampling operations, the original 3×H×W-dimensional image is mapped to a 512×H×W-dimensional feature space.

[0135] Step S620: Input the extracted feature representation into the image classification module to generate class information.

[0136] The image classification module aims to evaluate the impact of the reconstructed image on the downstream classification task at the feature level. This module uses the feature representation output by the feature extraction module as input, and adopts a fully connected layer to integrate the features and perform a non-linear function mapping to achieve the classification of the image content. During the training process, the classification loss generated by this image classification module will be fed back to the image compression network to guide it to retain the semantic information related to the task decision during the compression process and maintain the recognizability of the image content.

[0137] Step S630: Input the extracted features into the authenticity discrimination module to generate discrimination information.

[0138] The authenticity discrimination module focuses on the closeness of the reconstructed image to the original image at the pixel level, making up for the deficiency of the image classification module in terms of perceptual quality. Based on a fully connected layer network, this module maps the feature representation output by the feature extraction module through a binary mapping function into a binary classification label of 0 or 1 to determine whether the input is the original image or the reconstructed image. The discrimination loss generated by the authenticity discrimination module prompts the image compression network to generate a more realistic and higher-quality reconstructed image, thus ensuring the good visual quality of the reconstruction result.

[0139] Through the task-driven multi-branch discriminator network, the image compression network can learn to generate reconstructed images that can be correctly recognized and have high visual quality during the training process.

[0140] Step S700: Train the image compression network and the multi-branch discriminator network in stages. First, independently train the image compression network using the rate-distortion loss function in deep learning image compression. When the performance of the image compression network reaches convergence, introduce the multi-branch discriminator network to start adversarial joint training. First, load the previously trained weights of the image compression network, and then use the loss function to alternately train the image compression network and the multi-branch discriminator network.

[0141] Preferably, as Figure 7 shown, the step S700 includes:

[0142] Step S710: Prepare a training sample image set and divide it into a training set and a validation set. The training sample image set includes a series of diverse and representative images.

[0143] Preparing a training sample image set is an important step in training the network. By preparing diverse and representative image samples and reasonably dividing the training set and the validation set, it can ensure that the network has good generalization ability and the accuracy of performance evaluation. In this way, the trained network can be better applied to the processing and compression of unknown data in subsequent tasks.

[0144] Step S720: Independently train the image compression network using the rate-distortion loss function in deep learning image compression. Through the backpropagation algorithm and the gradient descent optimizer, adjust the network parameters to minimize the loss function.

[0145] Preferably, define the rate-distortion loss function: The rate-distortion loss function is an index that comprehensively considers the compression quality and the compression bit rate. It measures the distortion between the compressed image and the original image and combines the compression bit rate. Initialize the image compression network parameters: Before starting the training, it is necessary to initialize the parameters of the image compression network. Random initialization or pre-trained parameters can be used as the starting point. Input the training sample images into the image compression network for forward propagation to generate the compressed images. Then, calculate the rate-distortion loss function to measure the difference between the compressed image and the original image. Through the backpropagation algorithm, calculate the gradient of the loss function with respect to the network parameters. Then, use the gradient descent optimizer (such as stochastic gradient descent) to update the network parameters to gradually reduce the loss function. Repeat the training process of forward propagation, loss calculation, backpropagation, and parameter update.

[0146] By using the rate-distortion loss function as the objective function to train the image compression network, the network parameters can be optimized so that it can minimize the distortion between the reconstructed image and the original image while maintaining a relatively high compression bit rate. In this way, the trained network can provide better quality and efficiency in subsequent image compression tasks.

[0147] Step S730: Repeat step S720 until the network converges or reaches a predetermined number of training epochs.

[0148] Step S740: Train the multi-branch discriminator network using the adversarial discriminative loss function. Adjust the network parameters to minimize the loss function through the backpropagation algorithm and the gradient descent optimizer.

[0149] The multi-branch discriminator network needs to perform two tasks in this embodiment: one is to distinguish whether the input image is the original image or the image generated by the image compression network G, that is, the discriminative information generation task; the other is to accurately identify its category when the input image is the original image, that is, the category information generation task; the information of the two tasks is backpropagated to the image compression network for network parameter optimization. To optimize the visual quality of the reconstructed image and the performance of the discriminator network, the proposed adversarial discriminative loss consists of two parts: the discriminative loss of the discriminator network for the original image and the reconstructed image, and the classification loss of the discriminator network for the original image.

[0150] Step S750: Train the image compression network using the rate-distortion loss function. Adjust the network parameters to minimize the loss function through the backpropagation algorithm and the gradient descent optimizer.

[0151] Step S760: Repeat steps S740 and S750 alternately until the two networks converge or reach a predetermined number of training epochs.

[0152] Step S770: Evaluate the performance of the two trained networks on the validation set. Use a set of independent image samples to calculate the image classification accuracy, peak signal-to-noise ratio (PSNR), and multi-scale structural similarity (MS-SSIM) of the compressed images to measure the compression effect and image classification performance of the network.

[0153] Calculate the evaluation metrics for all images on the validation set, and conduct statistics and analysis. The average value, standard deviation, and distribution can be calculated to understand the overall performance of the network on the validation set. Evaluating the performance of the trained network on the validation set is important, as it can help evaluate the generalization ability and compression effect of the network. Based on the evaluation results, the network design or training strategy can be further adjusted to improve the performance of the network and the image quality.

[0154] Step S780: Save the trained network models, that is, the network parameters of the two networks. Save the parameters of the network (such as weights and configurations) in a file for subsequent image compression tasks. This can avoid retraining the network every time an image is compressed, improving the efficiency and speed of compression.

[0155] Specifically, step S780 includes:

[0156] 1) Determine the saving path: Select a suitable path to save the network model file. This path should be accessible to ensure that the model can be correctly loaded in subsequent tasks.

[0157] 2) Save the network parameters: Save the weights and parameters of the network to a file. Generally, modern deep learning frameworks provide functions and tools for saving and loading models. You can use the corresponding functions to save the network parameters to the specified file.

[0158] 3) Save the training configuration: In addition to the weights and parameters of the network, the training configuration information can also be saved, including training parameters, the state of the optimizer, etc. This information can help ensure that the state at the end of training can be restored when the model is loaded.

[0159] After saving the trained network model, when you need to perform image compression tasks, you can directly load the model and use it for image compression and reconstruction operations. This can save time and computing resources and ensure the use of a consistent network model in subsequent tasks.

[0160] Use the trained image compression network to process the image to be compressed in steps S100 - S200, and use the first latent representation output by encoding as the compressed image; or, use the trained image compression network to process the compressed image in steps S300 - S500, and use the reconstructed image output by decoding as the decompressed image.

[0161] Step S800: When receiving the image to be compressed, based on the trained image compression network, encode the image to achieve image compression. The compressed image can be sent to the target device or stored locally for subsequent use. When receiving the compressed image to be decompressed, based on the trained image compression network, decode the image to achieve image decompression.

[0162] Preferably, as Figure 8 shown, the step S800 specifically includes:

[0163] Compression method:

[0164] Step S810: Obtain the original data of the image to be compressed. This data can include pixel values, color channel information, etc. of the image. The goal is to effectively encode this data to reduce the size of the image while trying to maintain the image quality.

[0165] Step S820: Input the obtained original data into the trained image compression network, and use the encoder to compress and obtain the first latent representation. Through the encoder part of the network, the original data is converted into the first latent representation, which contains the important features of the image.

[0166] Decompression method:

[0167] Step S830: Input the compressed image into the trained image compression network, and obtain the prior information through super encoding and decoding and entropy encoding and decoding.

[0168] Step S840: Quantize the compressed image, use the prior information for assistance, and perform entropy encoding and decoding using the entropy model to obtain the corresponding second latent representation.

[0169] Step S850: Based on the pixel recombination technology, decode the second latent representation to generate a reconstructed image. This process is the inverse process of encoding, aiming to restore a reconstructed image as close as possible to the original image from the latent representation.

[0170] Step S860: Deliver the reconstructed image. Send the decoded reconstructed image to the target device or store it locally. The compressed image can be transmitted to a remote device through a network or stored in a local storage medium for subsequent use.

[0171] Embodiment 2

[0172] This embodiment proposes a task-driven adversarial learning image compression system, which is configured to execute the task-driven adversarial learning image compression method in Embodiment 1.

[0173] Specifically, the system includes a processor and a memory. The memory stores a computer program. Running this computer program can execute the method of Embodiment 1. The processor is connected to the memory to run the computer program therein, and thus execute the method of Embodiment 1. In addition, the training sample image set is also stored in this memory.

[0174] The present invention is not limited to the foregoing specific embodiments. The present invention extends to any new feature disclosed in this specification or any new combination, as well as any new method or process step disclosed or any new combination.

Claims

1. A task-driven adversarial learning image compression method, characterized in that Including: Construct an image compression network and a multi-branch discriminator network respectively; Among them, the processing of the input image by the image compression network includes: S1. Preprocess the original image; S2. Encode the preprocessed original image to generate a first latent representation; S3. Perform super-encoding and decoding and entropy encoding and decoding on the first latent representation to obtain prior information; S4. Quantize the first latent representation and perform entropy encoding and decoding based on the prior information to generate a second latent representation; S5. Decode the second latent representation to obtain a reconstructed image; The processing of the input image by the multi-branch discriminator network includes: S6. Input the original image and the reconstructed image into the multi-branch discriminator network to obtain class information and discriminant information; S7. Train the image compression network and the multi-branch discriminator network in stages; In step S7, the staged training is as follows: first independently train the image compression network, and when the training converges, introduce the multi-branch discriminator network for adversarial joint training; Step S7 includes: S71. Construct a training sample image set and divide it into a training set and a validation set, and the training set is used for the training process of the network; S72. Independently train the image compression network using the rate-distortion loss function, and adjust the network parameters through the backpropagation algorithm and the gradient descent optimizer during training; S73. Iteratively train the image compression network until convergence, or iterate a predetermined number of rounds; S74. Train the multi-branch discriminator network using the adversarial discrimination loss function, and adjust the network parameters through the backpropagation algorithm and the gradient descent optimizer during training; S75. Train the image compression network using the rate-distortion loss function, and adjust the network parameters through the backpropagation algorithm and the gradient descent optimizer during training; S76. Repeat steps S74 and S75 alternately until both networks converge, or iterate a predetermined number of rounds; S77. Use the validation set to verify the performance of the two trained networks; S78. Save the network parameters of the two trained networks; S8. Use the trained image compression network to perform the processing of steps S1 - S2 on the image to be compressed, and use the encoded output first latent representation as the compressed image; or, use the trained image compression network to perform the processing of steps S3 - S5 on the compressed image, and use the decoded output reconstructed image as the decompressed image.

2. The task-driven adversarial learning image compression method according to claim 1, characterized in that, Step S2 includes: S21. Construct a depthwise separable convolutional network, and the depthwise separable convolutional network includes multiple convolutional layers; S22. Add a LeakyReLU activation function and a normalization layer to each convolutional layer; S23. Add a residual block to each convolutional layer, and the residual block adds the input of the convolutional layer to the output of the convolutional layer; S24. Input the preprocessed original image into the depthwise separable convolutional network for downsampling; S25. The feature map output by the depthwise separable convolutional network is used as the first latent representation.

3. The task-driven adversarial learning image compression method according to claim 1, wherein Step S3 includes: S31. Input the first latent representation into a super encoder for super encoding to obtain a first super-encoded feature map. The super encoder includes convolutional layers, and ReLU is used as the activation function between adjacent convolutional layers. S32. Quantize the first super-encoded feature map, then perform entropy encoding and entropy decoding to generate a second super-encoded feature map. S33. Input the second super-encoded feature map into a super decoder symmetric to the super encoder to obtain prior information.

4. The task-driven adversarial learning image compression method according to claim 1, characterized in that, The step S4 includes: S41. Quantize the first latent representation. S42. Perform entropy encoding on the quantized first latent representation through a probability model generated from the prior information. S43. Perform entropy decoding on the entropy-encoded first latent representation through a probability model generated from the prior information to generate a second latent representation.

5. The task-driven adversarial learning image compression method according to claim 2, wherein The step S5 includes: Perform depthwise separable convolution and pixel rearrangement on the second latent representation to obtain a reconstructed image.

6. The task-driven adversarial learning image compression method according to claim 1, characterized in that The step S6 includes: S61. Input the original image and the reconstructed image into a feature extraction module to obtain a feature representation. The feature extraction module uses a multi-layer convolutional neural network, followed by a batch normalization layer and a LeakyReLU activation function. S62. Input the feature representation into an image classification module to generate class information. The image classification module uses a fully connected layer and a non-linear activation function to classify the input feature representation. S63. Input the feature representation into a authenticity discrimination module to generate discrimination information. The authenticity discrimination module uses a fully connected layer and a binary mapping function to discriminate the authenticity of the input feature representation.

7. The task-driven adversarial learning image compression method according to claim 1, characterized in that, In the step S8, the compression of the image to be compressed includes: S81. Obtain the original data of the image to be compressed. S82. Input the original data of the image to be compressed into a trained image compression network, and use the encoder to obtain a first latent representation. Alternatively, the decompression of the compressed image includes: S83. Input the compressed image into a trained image compression network, and obtain prior information through super encoding / decoding and entropy encoding / decoding. S84. Quantize the compressed image, perform entropy encoding / decoding based on the prior information to obtain the corresponding second latent representation. S85. Decode the obtained second latent representation to generate a reconstructed image.

8. A task-driven adversarial learning image compression system, characterized in that, The system is configured to execute the method according to any one of claims 1-7.

Images