Deep learning image compression method and system based on semantic discriminator

Abstract

The invention discloses a deep learning image compression method and system based on a semantic discriminator, and mainly solves the problem that the visual task performance of a downstream machine is remarkably reduced due to serious semantic information loss under a high compression rate in the conventional image compression method. According to the implementation scheme, a group of images are selected from an existing image data set and are divided into a training set, a verification set and a test set, and the training set, the verification set and the test set are preprocessed respectively; constructing an image compression network comprising an image codec, a semantic extraction network and a semantic guide discriminator under a Pytorch framework; inputting the training set into an image compression network, carrying out two-stage iterative training, and verifying through a verification set; and inputting the test set into the trained image compression network, and only calling the image codec to output the compressed reconstructed image. According to the method, the structural integrity and semantic consistency of the reconstructed image are remarkably improved, higher visual quality and task accuracy can be kept at a low code rate, better image compression performance is embodied, and the method can be used for efficient transmission and storage of image data.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and specifically relates to a deep learning image compression method and system, which can be used for efficient transmission and storage of image data. Background Technology

[0002] The explosive growth of image and video data across all sectors of society today has created a continuous and urgent demand for efficient compression technologies. Traditional image compression standards, such as JPEG, JPEG2000, and AVC / HEVC derived from video coding, aim to minimize pixel-level distortion, such as mean square error (MSE), and use metrics like PSNR and SSIM to measure rate-distortion performance at the encoding end, striving to achieve the highest possible reconstruction fidelity for human vision at a given bitrate.

[0003] In recent years, with breakthroughs in deep learning technology, end-to-end image compression methods based on neural networks have made significant progress. Among them, architectures such as Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) have been successfully applied to image compression, surpassing traditional encoders in subjective visual quality and even some objective metrics. However, in-depth research reveals that these advanced deep learning methods have largely not escaped the paradigm of "pixel-level reconstruction." The loss functions they employ, such as MSE, MS-SSIM, and discriminator design, are fundamentally based on pursuing a high degree of similarity between the reconstructed image and the original image at the pixel level or in shallow features. This optimization orientation has significant limitations: the network tends to smooth out subtle textures and structures that are difficult to effectively describe with a bitstream, resulting in severe loss of crucial semantic information in the image.

[0004] This deficiency is particularly severe in machine vision applications. In modern intelligent systems, the downstream users of compressed images are often no longer humans, but various machine vision algorithms, such as object detection, image classification, or instance segmentation models. For these algorithms, the contours of specific objects in an image, key feature points, and their semantic relationships are more important than background texture. Existing traditional and deep learning-based image compression methods, due to their lack of semantic awareness, cannot selectively retain this crucial information for machine decision-making during the encoding process. Specifically, at high compression ratios, object boundaries in the image become blurred, discriminative features are weakened, ultimately leading to a significant performance degradation in downstream machine vision tasks using the compressed image.

[0005] Furthermore, discriminators in existing GAN-based compression frameworks are typically limited to real / fake identification at the pixel level or through features extracted from shallow networks. This design fails to introduce and utilize the external semantic knowledge inherent in the image itself, which is relevant to higher-level tasks, to guide and constrain the compression process. Therefore, while pursuing extremely high compression efficiency, existing technologies struggle to achieve an optimal balance among the three key dimensions of visual fidelity, compression ratio, and machine-oriented semantic fidelity, constituting a technical bottleneck that urgently needs to be addressed.

[0006] Patent document with application number 202311173090.X discloses an "image compression system for machine vision and human vision", which proposes to use two decoders to restore the compressed bitstream, and the restored image is used for machine vision and human vision respectively, so as to effectively balance the performance of machine vision tasks and the effect of human vision.

[0007] Patent document with application number 202411211529.8 discloses an "image encoding, decoding method and compression method for machine and human vision", which proposes to adaptively adjust the quantization step size of feature map quantization to form a balanced rate-distortion performance for human vision and different machine vision tasks.

[0008] The two methods mentioned above are less flexible and difficult to adapt to multi-task dynamic switching scenarios because they require pre-specifying machine vision tasks and the compression network needs to be trained in conjunction with the task network. Furthermore, when task requirements change, the entire model needs to be retrained, which increases deployment costs and time, resulting in poor system portability and versatility.

[0009] Patent document with application number 202310207420.6 discloses an "image feature compression and decompression method that combines human vision and machine vision". Although it can effectively complete target detection and image classification tasks, it requires the use of an additional task network at the encoding end to generate an importance mask map as prior information for the task and transmit it to the decoding end. Therefore, it increases the computational complexity and transmission bandwidth overhead at the encoding end, which limits its practical application scenarios. Summary of the Invention

[0010] The purpose of this invention is to address the shortcomings of the prior art by proposing a deep learning image compression method and system based on a semantic discriminator, which can significantly improve the structural integrity and semantic consistency of reconstructed images, improve machine vision adaptability and reduce bandwidth costs while eliminating task network binding and auxiliary information dependence.

[0011] The technical approach to achieve the objective of this invention is as follows: design a general semantic guidance discriminator that is decoupled from the compression network; optimize the structural integrity and semantic consistency of the reconstructed image through the generative adversarial mechanism; strengthen the key features of machine vision through adversarial learning to achieve downstream task adaptability improvement without task network binding; and reduce transmission bandwidth costs by using single independent encoding and decoding in the inference stage to achieve zero auxiliary information transmission.

[0012] Based on the above technical concept, the technical solution of the present invention includes the following:

[0013] 1. A deep learning-based image compression method based on a semantic discriminator, characterized in that it includes:

[0014] (1) Select a set of images from the existing image dataset, divide them into training set, validation set and test set, and preprocess them respectively;

[0015] (2) Construct a deep learning image compression network based on a semantic discriminator in the PyTorch framework, and select an existing loss function. As the loss function of the network;

[0016] (3) Input the training set into the constructed deep learning image compression network, perform two-stage iterative training and verify it through the validation set to obtain the trained image compression network;

[0017] (4) Input the test set into the trained deep learning image compression network and output the compressed reconstructed image.

[0018] Furthermore, in step (2), a deep learning image compression based on a semantic discriminator is constructed within the PyTorch framework.

[0019] A network, the implementation of which includes:

[0020] (2a) Establish an image encoder including an encoder, a hyper-prior encoder, a hyper-prior decoder, a decoder, and an entropy unit.

[0021] Decoder;

[0022] (2b) Establish a semantic extraction network for extracting high-level semantic information;

[0023] (2c) Establish a semantic guidance discriminator consisting of a cascaded semantic embedding unit and a downsampling unit;

[0024] (2d) The image codec is cascaded with the semantic guidance discriminator, the input end of the semantic extraction network is connected to the input end of the image codec, and the output end of the semantic extraction network is connected to the semantic embedding unit of the semantic guidance discriminator to form a deep learning image compression network based on the semantic discriminator.

[0025] 2. A deep learning image compression system based on a semantic discriminator, characterized in that it comprises:

[0026] The data processing module is used to select images from the image dataset, divide them into training set, validation set and test set, and perform independent preprocessing operations on each set;

[0027] A network construction module is used to build an image compression network based on a semantic discriminator within a deep learning framework, and to configure a rate-distortion loss function for the network.

[0028] The network training module is used to iteratively train the image compression network using the training set and validation set through an optimization algorithm to minimize the loss function and obtain a trained image compression network.

[0029] The image compression module is used to input the test set images to be compressed into the trained image compression network and output the compressed reconstructed images.

[0030] Compared with the prior art, the present invention has the following advantages:

[0031] First, it achieved semantic-level quality optimization, significantly improving the performance of downstream tasks.

[0032] Existing deep learning-based image compression methods are mostly limited to pixel-level fidelity or visual perception quality, lacking targeted protection of the high-level semantic structure of images, resulting in severe loss of semantic information at high compression rates. This invention innovatively introduces a semantically guided discriminator, which performs semantic-level discrimination and adversarial training in the feature space, enabling the compression network to selectively retain semantic information that is crucial to machine vision tasks during the encoding process. This results in reconstructed images exhibiting higher accuracy and robustness in downstream tasks such as object detection and instance segmentation.

[0033] Second, it adopts an efficient training-inference architecture, ensuring convenience and efficiency in practical applications.

[0034] Existing compression schemes that combine semantic tasks often still rely on additional semantic models or complex inputs during the inference phase, causing inconvenience in deployment and additional computational overhead. This invention adopts a unique training-inference separation design. During the training phase, a semantic extraction network and a semantic-guided discriminator are used to assist in the optimization of the backbone image codec network. In the final deployment and application, only the lightweight image codec needs to be retained, without any additional network parameters. This ensures both high semantic fidelity and inference efficiency and ease of engineering application.

[0035] Third, it has strong architectural versatility and portability, and does not depend on a specific compressed network.

[0036] The semantic enhancement mechanism of this invention interacts with the backbone compression network through a standardized module interface. Its design does not depend on the specific implementation inside the codec. Therefore, this semantic guidance framework can seamlessly adapt to and enhance a variety of mainstream end-to-end compression architectures, such as VAE structures based on advanced priors or GAN structures based on generative adversarial systems, demonstrating excellent versatility and portability, and greatly expanding its application scope.

[0037] Experimental results show that, compared with traditional image compression standards and existing deep learning methods, this invention can significantly improve the semantic fidelity of reconstructed images while maintaining high compression efficiency, achieves better performance on a variety of machine vision tasks, and realizes the best balance between compression efficiency and task performance. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating the implementation of the deep learning image compression method based on a semantic discriminator according to the present invention.

[0039] Figure 2 This is a schematic diagram of the deep learning image compression network structure based on a semantic discriminator constructed in the method of the present invention.

[0040] Figure 3 for Figure 2 A schematic diagram of the image codec network structure in the image codec.

[0041] Figure 4 for Figure 2 A schematic diagram of the semantic guidance discriminator structure in the document;

[0042] Figure 5 for Figure 4 A schematic diagram of the semantic embedding unit structure in the diagram;

[0043] Figure 6 This is the framework of the deep learning image compression system based on a semantic discriminator according to the present invention;

[0044] Figure 7This is a comparison chart of the perceptual quality of image reconstruction performed by the present invention and existing image compression methods at different bitrates.

[0045] Figure 8 This is a comparison chart showing the accuracy of downstream tasks for reconstructing images at different bitrates using the methods of this invention and existing image compression methods.

[0046] Figure 9 This is a subjective comparison chart showing the image compression effects of the present invention and existing image compression methods. Detailed Implementation

[0047] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0048] Example 1: Deep Learning Image Compression Method Based on Semantic Discriminator

[0049] This example uses a semantic discriminator to perform lossy image compression at low bitrates. First, the input image is fed into an image codec to obtain a reconstructed image. Simultaneously, the original image is input into an image semantic extraction network to extract semantic information. This semantic information, along with the reconstructed and original images, is then used to train a semantic-guided discriminator, updating the parameters of both the image codec and the semantic-guided discriminator. After training, the semantic extraction network and the semantic-guided discriminator are removed, and the image codec is used to compress and decompress the image.

[0050] Reference Figure 1 The implementation steps for this example are as follows:

[0051] Step 1: Acquire image data, classify it, and preprocess it.

[0052] 1.1) Dataset Acquisition and Partitioning:

[0053] Select a set of images from the existing image dataset and divide them into training data according to official settings or predetermined proportions.

[0054] The training set, validation set, and test set are selected from image datasets including but not limited to MSCOCO, ImageNet, and OpenImages. The MSCOCO2017 set of images is divided into training set, validation set, and test set according to the official settings or predetermined ratio. The training set contains 118,287 images, the validation set contains 5,000 images, and the test set contains 40,000 images.

[0055] 1.2) Data Preprocessing

[0056] This example performs different preprocessing steps on different datasets, including:

[0057] For the training set, the pixel values ​​are first converted from the 0-255 range to the 0-1 range, and then the image size is uniformly adjusted to 256×256 pixels by random cropping.

[0058] For the validation set, the pixel values ​​are first converted to the 0-1 range, and then the image size is uniformly adjusted to 256×256 pixels by center cropping.

[0059] For the test set, pixel values ​​are first converted to the 0-1 range, and then the image size is aligned according to the encoder downsampling factor. In this embodiment, for the case of a downsampling factor of 16, when the image width and height are not integer multiples of 16, edge padding is used to pad them to the smallest integer multiple of 16.

[0060] Step 2: Construct a deep learning image compression network based on a semantic discriminator within the PyTorch framework.

[0061] The PyTorch framework, which is currently used to provide dynamic computation graphs, is an open-source machine learning framework that supports efficient GPU-accelerated computing. Its core dependencies include the deep learning foundation library torch and the image processing tool library torchvision.

[0062] Reference Figure 2 The implementation of this step includes the following:

[0063] 2.1) Constructing the codec network:

[0064] This example uses the ELIC architecture as the image codec network. This architecture integrates a transform encoder, a hyper-prior encoder, a hyper-prior decoder, an entropy unit, and a transform decoder into a single unit. Its overall structure is as follows: Figure 3 As shown.

[0065] The transform encoder has its input connected to the original image data. Its internal structure is cascaded and includes: a first convolutional layer, a first set of residual bottleneck blocks, a second convolutional layer, a second set of residual bottleneck blocks, a first attention layer, a third convolutional layer, a third set of residual bottleneck blocks, a fourth convolutional layer, and a second attention layer. The first, second, and third sets of residual bottleneck blocks are each composed of multiple stacked residual bottleneck units. The output of the second attention layer serves as the output of the main transform encoder, outputting the underlying latent representation.

[0066] The super-prior encoder has its input connected to the output of the transform encoder. Its internal structure is a cascaded convolutional network, which includes a preprocessing convolutional layer with a kernel size of 3×3, and two downsampling convolutional layers with ReLU activation functions. The output of the last downsampling convolutional layer serves as the output of the super-prior encoder, outputting the super-prior latent representation.

[0067] The super-prior decoder has its input connected to the output of the super-prior encoder. Its internal structure is a cascaded deconvolutional network, consisting of two deconvolutional layers with ReLU activation functions, and a final convolutional layer with a 3×3 kernel. The output of the final convolutional layer is connected to the entropy unit.

[0068] The entropy unit connects the transform encoder and the transform decoder, and receives signals from the super-prior decoder. Internally, this unit includes a quantizer, a Gaussian conditional model, and a slice-based multi-branch context network. The multi-branch context model sets up parallel channel context transform branches, spatial context prediction branches, and parameter aggregation branches for each slice group. The input of the parameter aggregation branch is connected to the output of the super-prior decoder, the output of the channel context transform branch, and the output of the spatial context prediction branch, respectively, and its output is used to generate entropy parameters.

[0069] The transform decoder, whose input is connected to the output of the entropy unit, has an internal structure consisting of an attention layer, an upsampling deconvolution layer, and multiple sets of feature reconstruction layers cascaded alternately, and its structure is mirror-symmetric to the main transform encoder. Specifically, the transform decoder sequentially includes: a third attention layer, a first deconvolution layer, a fourth set of residual bottleneck blocks, a second deconvolution layer, a fourth attention layer, a fifth set of residual bottleneck blocks, a third deconvolution layer, a sixth set of residual bottleneck blocks, and a fourth deconvolution layer. The output of the fourth deconvolution layer serves as the output of the entire network, outputting the reconstructed image.

[0070] 2.2) Constructing a semantic extraction network:

[0071] This example uses a pre-trained Efficient-SAM model as the semantic extraction network to extract multi-scale high-level semantic information. The Efficient-SAM model consists of a lightweight image encoder and a cue-guided mask decoder cascaded together. The mask decoder includes four upsampling operations, and the multi-scale feature map obtained after each upsampling requires the following operations:

[0072] The feature map is reorganized from a four-dimensional tensor containing batch, channel, height, and width dimensions into a five-dimensional tensor containing batch, cue point, channel, height, and width dimensions. It is then averaged along the query dimension to eliminate the dependency on specific cue points such as points or boxes.

[0073] The processed multi-scale semantic features are stored in a semantic feature list in hierarchical order and used as input to the semantically guided discriminator.

[0074] 2.3) Constructing a semantically guided discriminator:

[0075] The semantic guidance discriminator consists of two parts: a semantic downsampling unit and a semantic embedding unit. Its overall structure is as follows: Figure 4 As shown.

[0076] 2.3.1) Constructing downsampling units:

[0077] The downsampling unit constructed in this example consists of a feature extraction backbone, a three-level fusion interface, and a discrimination output.

[0078] The feature extraction backbone consists of five sequentially cascaded convolutional stages. The first convolutional stage contains a convolutional layer with a kernel size of 4×4, a stride of 2, and padding of 1. This convolutional layer is configured with spectral normalization, followed by a batch normalization layer and a LeakyReLU activation layer. The second convolutional stage contains a convolutional layer with a kernel size of 4×4, a stride of 2, and padding of 1, followed by an instance normalization layer and a LeakyReLU activation layer. The third convolutional stage contains a convolutional layer with a kernel size of 4×4, a stride of 2, and padding of 1, followed by an instance normalization layer and a LeakyReLU activation layer. The fourth convolutional stage contains a convolutional layer with a kernel size of 4×4, a stride of 1, and padding of 1, followed by an instance normalization layer and a LeakyReLU activation layer. The fifth convolutional stage contains a convolutional layer with a kernel size of 4×4, a stride of 1, and padding of 1.

[0079] The three-level fusion interface is respectively set at the output of the second, third, and fourth convolutional stages of the feature extraction backbone. Each fusion interface is connected to the input of the semantic embedding unit and the output of the semantic embedding unit is back-connected to the input of the next level convolutional stage.

[0080] The discrimination output terminal is connected to the output terminal of the fifth convolutional stage in the feature extraction backbone, and outputs the final single-channel discrimination result.

[0081] 2.3.2) Constructing semantic embedding units:

[0082] The semantic embedding unit in this example is built on the Transformer architecture. Internally, it contains interconnected multi-head transposed self-attention components, gated deep convolutional feedforward network components, and residual integration components. The overall structure is as follows: Figure 5 As shown.

[0083] The multi-head transpose attention component has its input connected to the semantic extraction network and downsampling unit via three parallel 1×1 convolutional layers, corresponding to the input channels of the query vector, key vector, and value vector, respectively. Internally, this component includes a feature segmentation unit, a dot product similarity calculation unit, a scaling factor unit, and a Softmax normalization unit. The output of the Softmax normalization unit is connected to the feature weighting unit.

[0084] The gated depthwise convolutional feedforward network component includes two independent 1×1 convolutional layer branches. The outputs of the two branches are respectively connected to GELU activation layers. The outputs of the two GELU activation layers are connected to element-wise multiplication nodes. The output of these nodes is connected to the 1×1 convolutional layer that restores the dimension.

[0085] The residual integration component has its input terminals connected to the original input terminal of the semantic embedding unit and the output terminal of the gated deep convolutional feedforward network component, respectively. Internally, it consists of a cascaded channel splicing layer, a convolutional block composed of a 3×3 convolutional layer and an ELU activation function, and a terminal 1×1 convolutional layer.

[0086] 2.4) Configure the loss function:

[0087] This example uses a composite loss function as the network training objective. This loss function consists of the generator loss. With discriminator loss It consists of two parts. All loss terms are implemented in the PyTorch framework through a custom loss class, supporting backpropagation gradient calculation. The configuration process is as follows:

[0088] 2.4.1) Establish a generator loss consisting of a weighted sum of three terms: base rate-distortion loss, perceptual loss, and adversarial loss. :

[0089] ,

[0090] in, , Representing the underlying potential representation The code rate loss is calculated by using entropy units to compute the probability distribution of each element of the underlying latent representation and estimate the number of bits required for arithmetic coding. Representing the super-prior latent representation The bit rate loss is calculated by using a hyper-prior entropy model to calculate the probability distribution of the hyper-latential representation; Represents the original image and reconstructed images The MSE distortion loss between the two values; 𝜆 is the distortion weight parameter, used to adjust the balance between bitrate and distortion. The larger the value, the smaller the compression ratio.

[0091] In order to perceive loss, Represents the AlexNet network's... The layer feature extraction function, in this embodiment, selects, but is not limited to, the high-level feature maps of the 3rd, 4th, and 5th layers for distance measurement;

[0092] To combat the losses, Represented by coded symbols For conditional information, for the reconstructed image The semantically guided discriminator function performs the discrimination. The reconstructed image distribution output by the generator.

[0093] 2.4.2) Establish an adversarial loss mechanism that includes generated samples. Adversarial loss against real samples Discriminator loss :

[0094] ,

[0095] in Represented by coded symbols For conditional information, for the original image The semantically guided discriminator function performs the discrimination.

[0096] Step 3: Iteratively train the deep learning image compression network based on the semantic discriminator.

[0097] This step employs a two-stage progressive training strategy. In the first stage, only the image codec is enabled, while in the second stage, all networks are enabled, but the parameters of the semantic extraction network are kept frozen. The implementation includes the following:

[0098] 3.1) Perform the first stage of basic pre-training:

[0099] 3.1.1) Input the training data into the network, use the Adam optimizer, set the image batch size to 32, and the initial learning rate to 1×10⁻ 4 During training, the learning rate is gradually reduced to 1×10⁻ using a cosine annealing strategy. 6 Set the total number of training steps to 500,000.

[0100] 3.1.2) Perform forward propagation on the codec of the training image at each step to obtain the latent representation and reconstructed image, and calculate the base rate-distortion loss for the current step. Then, automatic differential backpropagation is initiated, and all trainable parameters in the image codec are updated once using the Adam optimizer.

[0101] 3.1.3) Repeat step 3.1.2) When the number of training steps reaches an integer multiple of 3,000, perform forward inference on the validation set using the current network parameters, accumulating the base rate-distortion loss term for each validation image. The average value is then calculated and written to the convergence curve log. Simultaneously, the loss from this validation is compared to the historical minimum.

[0102] If the value is lower than the historical minimum, save the model checkpoint file containing all weights of the image codec, optimizer state, current step number and minimum loss value, overwrite the old checkpoint, and ensure that the disk only retains the single model instance with the lowest validation set loss.

[0103] Otherwise, continue repeating step 3.1.2 without performing any disk write operations; the disk will remain unchanged with the original model file that has the lowest loss.

[0104] 3.1.4) Repeat steps 3.1.2) and 3.1.3) until 500,000 steps are completed, thus completing the first phase of training.

[0105] 3.2) Conduct the second phase of joint combat training:

[0106] 3.2.1) After the first stage convergence, load the checkpoint of the pre-trained model with the lowest loss on the validation set, introduce a semantic extraction network and a semantic-guided discriminator, keep the parameters of the semantic extraction network frozen, adjust the batch size to 16, and reset the learning rate of the image codec to 1×10⁻. 4 The discriminator learning rate is set to 1×10⁻ 5 The total number of steps in training was 44,000.

[0107] 3.2.2) With fixed image codec parameters, the training batch is input into the network for forward propagation to obtain latent representations and reconstructed images. The discriminator then distinguishes between the reconstructed images and the real images, and calculates the discriminator loss for the current step. Then, automatic differential backpropagation is initiated to update the discriminator parameters; then, the discriminator parameters are fixed, and the generator loss for the current step is calculated. Then, automatic differential backpropagation is initiated to update the image codec parameters;

[0108] 3.2.3) Repeat step 3.2.2) until the number of training steps reaches an integer multiple of 3000. Then, using the current network parameters, perform forward inference on the validation set and accumulate the generator loss for each validation image. The average value is then calculated and written to the convergence curve log. Simultaneously, the loss from this validation is compared to the historical minimum.

[0109] If the value is lower than the historical minimum, save the model checkpoint file containing all weights of the image codec and discriminator, optimizer state, current step number and minimum loss value, overwrite the old checkpoint, and ensure that the disk only retains the single model instance with the lowest validation set loss.

[0110] Otherwise, continue repeating step 3.2.2), without performing any disk write operations, and the disk will remain unchanged with the original model file with the lowest loss.

[0111] 3.2.4) Repeat steps 3.2.2) and 3.2.3) until the training steps reach 44,000, thus completing the second phase of training;

[0112] 3.3) Derive the final weights:

[0113] Once the two-stage training is complete, delete all optimizer state dictionaries, gradient buffers, and temporary checkpoints. Extract only the image codec parameters from the checkpoints of the models with the lowest loss on the validation set, write them to a read-only storage file, and obtain a static parameter file containing only inference weights. This will give you the trained codec, which will be used for subsequent image compression.

[0114] Step 4: Input the test set into the trained image codec and output the reconstructed image result.

[0115] It should be noted that the step numbers in this example and the claims are only for the purpose of clearly describing the embodiments of the present invention and facilitating understanding, and their order is not limited.

[0116] Example 2: Deep Learning Image Compression System Based on Semantic Discriminator

[0117] This embodiment provides a method for achieving high-fidelity image reconstruction under low bitrate conditions, and significantly improves the semantic consistency and subjective quality of the reconstructed image by introducing a semantic discrimination mechanism.

[0118] Reference Figure 6 The system includes: a data processing module 1, a network construction module 2, a network training module 3, and an image compression module 4. The network construction module 2 further includes an image encoding / decoding submodule 21, a semantic extraction submodule 22, a semantic embedding submodule 23, and a semantic downsampling submodule 24. The working principle of the entire system is as follows:

[0119] The data processing module 1 is used to acquire, divide and preprocess the input image data, and provide the network training module 3 and the image compression module 4 with the required image data during the training and inference phases of the system, respectively, so as to ensure that the images entering the training and inference process have consistent distribution characteristics and standardized format.

[0120] The network construction module 2 is used to construct an image compression network structure based on a semantic discriminator within a deep learning framework, providing an end-to-end optimizable network component for the network training module 3. The image encoding / decoding submodule 21 defines the structure of the basic compression path, including a transform encoder, a hyper-prior encoder, a hyper-prior decoder, an entropy unit, and a transform decoder. The input to this submodule is a preprocessed image tensor, and the output is a basic latent representation. Super-prior latent representation Probability parameters used for entropy coding and reconstructed images The semantic extraction submodule 22 is used to define the network structure for extracting multi-level semantic features from the original image. Its input is the preprocessed original image tensor, and its output is a set of multi-scale semantic feature maps. This submodule specifies the scale, number of channels, and output format of the semantic features during the network construction stage, and provides a semantic feature query interface for the semantic embedding submodule 23 to call. The semantic embedding submodule 23 and the semantic downsampling submodule 24 together constitute the semantic guided discriminator. The semantic embedding submodule 23 is used to define the structural interface for the fusion of semantic features and image features. It receives semantic features and image features at the corresponding level from the downsampling submodule 23 as input, and outputs semantic enhancement features aligned by level for the semantic downsampling submodule 24 to call. The semantic embedding submodule 24 is used to define the structural interface for the discrimination backbone to downsample the original image and the reconstructed image, and receives features including the original image features, the reconstructed image features, and the semantic enhancement features from the semantic embedding submodule 23. The output is the discrimination result in vector form.

[0121] The network training module 3 is used to receive training set and validation set data provided by the data processing module 1, iteratively optimize the model parameters provided by the network construction module 2 based on a predefined optimization strategy, and save the inference weights of the image encoding and decoding submodule 21 after the training reaches the predetermined termination condition for use by the image compression module 4.

[0122] The image compression module 4 is used for actual encoding and decoding work in the inference and deployment phases. It receives the test set provided by the data processing module 1, performs encoding and decoding operations through the image encoding and decoding submodule weights 21 trained by the network training module 3, and outputs the reconstructed image.

[0123] It should be noted that the above functional modules can be implemented, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented, in whole or in part, as program instruction products. A program instruction product includes one or a set of program instructions. When the program instructions are loaded and executed on a computer, the described process or function is generated, in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The program instructions can be stored in a computer-readable and writable storage medium, or transferred from one computer's readable and writable storage medium to another.

[0124] In this embodiment, the direct coupling or communication connection between the modules can be achieved through indirect coupling or communication connection via interfaces, devices, or modules. Furthermore, each functional module and submodule can dynamically reside within a single processing unit, or each module can exist physically independently, or two or more modules can dynamically reside within a single processing unit. When these dynamic components are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable and writable storage medium. This storage medium can be a memory, disk, or optical disc, etc.

[0125] The effectiveness of this invention is further illustrated by the following simulation tests:

[0126] I. Test Conditions

[0127] Dataset: The existing MSCOCO dataset will be used;

[0128] Experimental platform: CPU is Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz, operating system is Ubuntu 22.04, graphics card is NVIDIA GeForce RTX 4090, PyTorch version is 2.4.1;

[0129] II. Simulation Test Content

[0130] Test 1: Under the above conditions, the MSCOCO VAL2017 dataset was compressed using the present invention and existing methods such as VVC, ELIC, PO-ELIC, and MS-ILLM, respectively, to obtain reconstructed images at different compression bitrates. Multiple perceptual quality metrics, including LPIPS, FID, and KID, were calculated to evaluate the image perceptual quality. The results are as follows: Figure 7 As shown, the horizontal axis represents the bitrate, measured in bits per pixel (bpp); the vertical axis represents the distortion, with lower distortion values ​​indicating better compression performance.

[0131] Figure 7(a) Evaluation results of reconstructed images obtained by several compression methods on the LPIPS index;

[0132] Figure 7 (b) Evaluation results of the reconstructed images obtained by several compression methods on the FID index;

[0133] Figure 7 (c) Evaluation results of reconstructed images obtained by several compression methods on the KID index;

[0134] from Figure 7 As can be seen, this invention achieves optimal performance on the LPIPS metric, with its curve consistently below that of the comparison method, indicating that at the same bitrate, the image reconstructed by this invention has the smallest difference from the real image perceived by the human eye. Simultaneously, this invention also demonstrates strong competitiveness on the FID and KID metrics, especially in the low bitrate range, with the distribution of the reconstructed image most closely resembling the distribution of the real image. This proves that the semantic guidance mechanism proposed in this invention can effectively preserve global and local semantic information crucial for visual perception.

[0135] Test 2: Under the above conditions, the MSCOCO VAL2017 dataset was compressed using the present invention and existing methods such as VVC, ELIC, PO-ELIC, and MS-ILLM, respectively, to obtain reconstructed images at different compression bitrates. The reconstructed images obtained by different methods were then input into the task networks YOLOv8 and Cascade R-CNN for object detection and instance segmentation, respectively, to evaluate their performance retention capabilities in downstream tasks. The results are as follows: Figure 8 As shown, the horizontal axis represents the bitrate, and the vertical axis represents the task accuracy (mAP50). A higher mAP50 value indicates better compression performance.

[0136] Figure 8 (a) Results of object detection using YOLOv8 on reconstructed images obtained by several compression methods;

[0137] Figure 8 (b) The results of instance segmentation of reconstructed images obtained by several compression methods using YOLOv8;

[0138] Figure 8 (c) Results of object detection using Cascade R-CNN on reconstructed images obtained by several compression methods;

[0139] Figure 8 (d) shows the results of instance segmentation of reconstructed images obtained by several compression methods using Cascade R-CNN;

[0140] Figure 8Experimental data clearly show that the present invention achieves optimal task performance across all four task and model combinations. Specifically, under low bitrate conditions, when the reconstructed image of the contrast method leads to a sharp drop in task accuracy, the method of the present invention maintains significantly higher task accuracy, which fully demonstrates that the present invention can still effectively preserve discriminative features crucial to machine vision tasks even at high compression ratios.

[0141] Test 3: Under the above conditions, the MSCOCO VAL2017 dataset was compressed using the present invention and existing methods such as VVC, ELIC, PO-ELIC, and MS-ILLM to obtain reconstructed images. The reconstructed images obtained by different methods were then input into the YOLOv8 network for object detection, yielding reconstructed images and corresponding detection results for each method, as shown below. Figure 9 As shown, where:

[0142] Figure 9 (a) shows the detection results in the original images of the dataset and the dataset annotations;

[0143] Figure 9 (b) shows the original images in the dataset and the detection results obtained using the task network;

[0144] Figure 9 (c) For the purpose of using the method of the present invention to Figure 9 (a) Reconstruction and detection results of the compressed image;

[0145] Figure 9 (d) Using the PO-ELIC method for Figure 9 (a) Reconstruction and detection results of the compressed image;

[0146] Figure 9 (e) For using the MS-ILLM method to Figure 9 (a) Reconstruction and detection results of the compressed image;

[0147] Figure 9 (f) For using the VVC method to Figure 9 (a) Reconstruction and detection results of the compressed image;

[0148] from Figure 9 The subjective comparison chart clearly shows that, compared with existing methods, the images reconstructed by this invention visually retain clearer object contours and texture details. Especially at low bitrates, images reconstructed by the comparison methods exhibit severe blurring and artifacts, leading to missed and false detections in downstream detectors. In contrast, the images reconstructed using this invention produce object detection and segmentation results closest to the original uncompressed image and the original dataset annotations, without any missed or false detections. This fully verifies the dual advantages of this invention in terms of subjective quality and task utility.

Claims

1. A deep learning image compression method based on a semantic discriminator, characterized in that, include: (1) Select a set of images from the existing image dataset, divide them into training set, validation set and test set, and preprocess them respectively; (2) Construct a deep learning image compression network based on a semantic discriminator in the PyTorch framework, and select an existing loss function. As the loss function of the network; (3) Input the training set into the constructed deep learning image compression network, perform two-stage iterative training and verify it through the validation set to obtain the trained image compression network; (4) Input the test set into the trained deep learning image compression network and output the compressed reconstructed image.

2. The method according to claim 1, characterized in that, In step (1), the selected set of images is divided into a training set, a validation set, and a test set, and preprocessed separately. This process includes: (1a) Divide the selected images into training set, validation set and test set according to a predetermined ratio; (1b) Perform corresponding preprocessing on the training set, validation set, and test set respectively, wherein: The training set is normalized, and the images are adjusted to a predetermined size by random cropping. The validation set is normalized, and the image is adjusted to a predetermined size by center cropping; The test set is normalized, and when the image width and height are not integer multiples of the encoder downsampling factor, they are padded to the smallest integer multiple of the factor.

3. The method according to claim 1, characterized in that, In step (2), a deep learning image compression network based on a semantic discriminator is constructed under the PyTorch framework. Its implementation includes: (2a) Establish an image codec including a transform encoder, a priori encoder, a priori decoder, a transform decoder, and an entropy unit; (2b) Establish a semantic extraction network for extracting high-level semantic information; (2c) Establish a semantic guidance discriminator consisting of a cascaded semantic embedding unit and a downsampling unit; (2d) The image codec is cascaded with the semantic guidance discriminator, the input end of the semantic extraction network is connected to the input end of the image codec, and the output end of the semantic extraction network is connected to the semantic embedding unit of the semantic guidance discriminator to form a deep learning image compression network based on the semantic discriminator.

4. The method according to claim 3, characterized in that, The structure of each device unit constituting the image codec in (2a) is as follows: The transform encoder has its input connected to the original image and is used to compress the image into a basic latent representation. The super-prior encoder, whose input is connected to the output of the transform encoder, is used to analyze the basic latent representation to generate a super-prior latent representation. The super-prior decoder has its input connected to the output of the super-prior encoder and is used to decode the entropy unit parameters from the super-prior latent representation. The entropy unit has its input connected to the output of the transform encoder and the output of the super-prior decoder, respectively, and is used to entropy encode and entropy decode the basic latent representation using the entropy unit parameters. The transform decoder, whose input is connected to the output of the entropy unit, is used to reconstruct the underlying latent representation after entropy decoding into an image.

5. The method according to claim 3, characterized in that, The semantic extraction network in (2b) adopts a pre-trained semantic segmentation network or a visual basic network, which outputs semantic feature maps at three spatial resolution levels, which are respectively used as the first semantic feature, the second semantic feature and the third semantic feature. Its parameters are set to either frozen or fine-tunable during training.

6. The method according to claim 3, characterized in that, The semantic embedding unit in (2c) includes: Three parallel 1×1 convolutional layers are used to generate query feature maps from semantic features provided by the semantic extraction network, and to generate key feature maps and value feature maps from features extracted from the cascaded downsampling units, respectively. The feature segmentation and similarity calculation subunit is used to segment the query, key, and value features into multiple subspaces according to a preset number of heads, and calculate the dot product similarity between the query and the key in each subspace, and perform normalization processing using a scaling factor. The attention weight generation subunit is used to perform a softmax operation on the similarity in the head number dimension to obtain normalized attention weights. The feature fusion and restoration subunit is used to weight and fuse the value features using attention weights, concatenate the output features of all heads, and restore the number of channels to the original dimension through a 1×1 convolution. The residual integration subunit is used to concatenate the original input features and attention-weighted features through the residual connection mechanism. The final feature integration is completed by passing the convolution block consisting of 3×3 convolution with reflection padding and ELU activation function, and 1×1 convolution layer.

7. The method according to claim 3, characterized in that, The downsampling unit in (2c) includes: The feature extraction backbone consists of five sequentially connected convolutional layers, each using the LeakyReLU activation function to extract multi-level feature representations of the input image; The three-level fusion interface is set after the second, third, and fourth convolutional layers of the feature extraction backbone, respectively. It is used to send the features output by the corresponding convolutional layer and the semantic features of the corresponding level into the semantic embedding unit for fusion, and return the fusion result to the feature extraction backbone. The discriminant output is used to take the features output by the fifth convolutional layer of the feature extraction backbone as the discriminant result of the semantically guided discriminator.

8. The method according to claim 1, characterized in that, The rate distortion loss function in (2) Including generator loss and discriminator loss They are represented as follows: ， ， in: , The bitrate loss representing the underlying latent representation. The bit rate loss representing the super-prior latent representation. represents the MSE distortion loss of the original image and the reconstructed image, D(·) represents the semantically guided discriminator, and 𝜆 is the distortion weight parameter; For perceptual loss, it represents the distance metric between the original image and the reconstructed image in the high-level feature space of the pre-trained deep convolutional neural network. For sensing weight parameters; For the generator's adversarial loss term, This represents the expectation operation on the distribution of the training data; This is the discriminative loss term of the discriminator with respect to the reconstructed image; This is the discriminative loss term of the discriminator with respect to the original real image.

9. The method according to claim 1, characterized in that, In step (3), the training set is input into the image compression network for two-stage iterative training and validation using a validation set. This includes basic pre-training and joint adversarial training, wherein: The first stage of basic pre-training involves enabling only the image codec and using the Adam optimizer to minimize the generator loss. The base rate distortion loss term Update network parameters for the target until the specified time. It converges on the validation set; The second phase of joint adversarial training, after the first phase, involves simultaneously activating the image codec and the semantic-guided discriminator, fixing or fine-tuning the parameters of the semantic extraction network, using an asymmetric learning rate strategy to set different learning rates for the image codec and discriminator, and alternately optimizing the generator loss. With the loss of the discriminator The network parameters of the image codec and discriminator are updated until the loss function converges and the performance of the validation set is stable, thus obtaining a trained image compression network.

10. A deep learning image compression system based on a semantic discriminator, characterized in that, include: The data processing module is used to select images from the image dataset, divide them into training set, validation set and test set, and perform independent preprocessing operations on each set; A network building module is used to build an image compression network based on a semantic discriminator within a deep learning framework, and to configure a loss function for the network. The network training module is used to iteratively train the image compression network using the training set and validation set through an optimization algorithm to minimize the loss function and obtain a trained image compression network. The image compression module is used to input the test set images to be compressed into the trained image compression network and output the compressed reconstructed images.

11. The system according to claim 10, characterized in that, The network construction module includes: The image encoding / decoding submodule is used to encode, perform advanced prior encoding / decoding, entropy estimation, and decoding / reconstruction of the input image. The semantic extraction submodule is used to extract multi-level high-level semantic features from the original image; The semantic embedding submodule is used to interactively fuse the semantic features provided by the semantic extraction submodule with the corresponding level image features extracted by the semantic downsampling submodule, and to feed back the fused semantic enhancement features to the semantic downsampling submodule. The semantic downsampling submodule is used to extract image features from the original input image and the reconstructed image, and during the extraction process, it receives semantic enhancement features from the semantic embedding submodule to generate the final identification result.

Images