Man-machine co-friendly image compression method and system based on interested features

Abstract

The invention provides a man-machine co-friendly image compression method and system based on interested features, and relates to the technical field of image compression. Extracting original multi-scale interest features based on an original image, and performing adaptive modulation of a spatial domain, a channel domain and a frequency domain on potential features generated by encoding through an encoder feature adaptive module to obtain task-oriented potential features; inputting the potential features into a hyper-prior network, and performing adaptive quantization based on the original multi-scale interest features to obtain quantization parameters, mean values and variances of the potential feature elements; performing quantization and entropy coding on the potential features by using quantization parameters to generate a compressed code stream; and according to the inverse quantization potential features and the multi-scale prior guidance interest features, a reconstruction process is modulated through a feature adaptive module of a decoder, and finally a reconstructed image is obtained. On the premise that the calculation complexity is not increased basically, the performance of the compressed image in a machine vision task is effectively improved, and meanwhile better code rate-distortion balance is achieved.

Description

Technical Field

[0001] This invention relates to the field of image compression technology, and in particular to a human-computer collaborative image compression method and system based on features of interest. Background Technology

[0002] Image compression technology aims to balance data storage and transmission efficiency with visual quality under limited bitrates. Traditional solutions are mostly designed around human visual needs, using a classic architecture of encoders, quantizers, decoders, and entropy models to map images into latent features and compress them into a bitstream. Some methods also introduce advanced prior networks to improve coding efficiency. With the popularization of intelligent vision applications, images need to adapt to both human and machine vision tasks, forming a new demand for compression for dual-vision targets, that is, to balance subjective viewing effect and downstream model recognition accuracy.

[0003] However, machine vision tasks exhibit significant selectivity in their utilization of image features, relying only on a few key spatial regions, feature channels, and specific frequency components, rather than uniformly depending on all image information. This contradicts the design logic of traditional compression techniques. Existing solutions, whether classical standards or learning-based frameworks, aim for overall reconstruction quality and employ uniform allocation strategies in quantization and distortion control, failing to specifically meet the differentiated needs of machine vision.

[0004] Although some machine vision image compression solutions attempt to adapt to downstream tasks by retraining models, preprocessing images, or adjusting features, they lack systematic optimization of quantization mechanisms and feature distortion distribution. This makes it difficult to allocate sufficient bit resources to task-related features under bitrate constraints, resulting in a significant loss of machine vision task performance and an inability to efficiently balance human visual reconstruction effects with machine vision adaptation capabilities. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a human-computer collaborative image compression method and system based on features of interest. Through task-oriented adaptive feature modulation and quantization, the performance of compressed images in machine vision tasks is effectively improved without significantly increasing computational complexity, while also achieving a better bitrate-distortion tradeoff.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a human-computer collaborative image compression method based on features of interest, comprising: The original image is input into the multi-scale interest feature extraction module to obtain the original multi-scale interest features; Based on the original multi-scale interest features, the latent features generated during the encoding process are adaptively modulated in the spatial domain, channel domain, and frequency domain through the feature adaptation module in the encoder to obtain task-oriented latent features. The task-oriented latent features are input into the super-prior network, and adaptive quantization is performed based on the original multi-scale interest features to predict the quantization parameters, mean, and variance of each latent feature element. The latent features are quantized and entropy encoded using the quantization parameters to generate a compressed bitstream; At the decoding end, the latent features are dequantized using the quantization parameters, and the prior information in the compressed bitstream is transformed to obtain multi-scale prior guided interest features. The reconstruction process is then modulated by the feature adaptive module of the decoder to finally obtain the reconstructed image.

[0007] Secondly, the present invention provides a human-computer collaborative image compression system based on features of interest, comprising: The feature extraction module is used to input the original image into the multi-scale interest feature extraction module to obtain the original multi-scale interest features; The coding and modulation module is used to perform adaptive modulation of the latent features generated during the coding process in the spatial domain, channel domain, and frequency domain based on the original multi-scale interest features, through the feature adaptation module in the encoder, to obtain task-oriented latent features. The quantization parameter prediction module is used to input the task-oriented latent features into the super-prior network and perform adaptive quantization based on the original multi-scale interest features to predict the quantization parameters, mean and variance of each latent feature element. The quantization encoding module is used to quantize and entropy encode the latent features using the quantization parameters to generate a compressed bitstream. The decoding and reconstruction module is used at the decoding end to dequantize the latent features using the quantization parameters, transform the prior information in the compressed bitstream to obtain multi-scale prior guided interest features, and modulate the reconstruction process through the feature adaptive module of the decoder to finally obtain the reconstructed image.

[0008] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the human-computer collaborative image compression method based on features of interest described in the first aspect.

[0009] Fourthly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the human-computer collaborative image compression method based on features of interest described in the first aspect.

[0010] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) This invention introduces a multi-scale interest feature extraction module, enabling the compression system to perceive and focus on image content crucial to downstream machine vision tasks (such as classification and detection). Based on this, the encoding and modulation module adaptively refines the modulation of potential features in three dimensions: space, channel, and frequency, making their distribution more suitable for specific task requirements. The quantization parameter prediction module further utilizes interest features to dynamically generate quantization step size and distribution parameters that match the task content, realizing intelligent non-uniform distribution of distortion in the spatial and feature dimensions. This task-content adaptive joint optimization mechanism enables the compression system to accurately allocate limited bitrate resources to the regions and features that have the greatest impact on task performance. Ultimately, this invention effectively improves the accuracy and robustness of reconstructed images for subsequent machine vision analysis while maintaining encoding efficiency and without significantly increasing model complexity and training costs, while also reducing the overall bitrate, achieving a better comprehensive optimization effect of bitrate-distortion-task performance.

[0011] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0012] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute a limitation thereof.

[0013] Figure 1 A main flowchart of a human-computer collaborative image compression method based on features of interest provided in this embodiment of the invention; Figure 2 An architecture diagram of a human-computer collaborative image compression method based on features of interest provided in an embodiment of the present invention; Figure 3 Detailed diagram of the feature adaptation module provided in the embodiments of the present invention; Figure 4 These are test experimental results in image classification tasks provided by embodiments of the present invention; Figure 5 These are the test experimental results in the target detection task provided by the embodiments of the present invention; Figure 6 The test results provided in the instance segmentation task according to the embodiments of the present invention. Detailed Implementation

[0014] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0015] Example 1 like Figure 1As shown, this embodiment discloses a human-computer collaborative image compression method based on features of interest, including the following steps: S1: Input the original image into the multi-scale interest feature extraction module to obtain the original multi-scale interest features; S2: Based on the original multi-scale interest features, the latent features generated during the encoding process are adaptively modulated in the spatial domain, channel domain, and frequency domain through the feature adaptation module in the encoder to obtain task-oriented latent features; S3: Input the task-oriented latent features into the super-prior network, and perform adaptive quantization based on the original multi-scale interest features to predict the quantization parameters, mean and variance of each latent feature element; S4: Use the quantization parameters to quantize and entropy encode the latent features to generate a compressed bitstream; S5: At the decoding end, the latent features are dequantized using the quantization parameters, and the prior information in the compressed bitstream is transformed to obtain multi-scale prior guided interest features. The reconstruction process is modulated through the feature adaptive module of the decoder to finally obtain the reconstructed image.

[0016] Next, combined Figure 2 This embodiment provides a detailed description of a human-computer collaborative image compression method based on features of interest.

[0017] First, interest feature information is obtained based on a lightweight interest feature extraction module, including a first interest feature extraction module and a second interest feature extraction module; the first interest feature extraction module consists of a Transformer and multiple convolutional layers, and the second interest feature extraction module consists of multiple convolutional layers.

[0018] The first interest feature extraction module is located at the encoding end. It obtains the original multi-scale interest features based on the input original image, which is used to capture the multi-scale key information of the original image and provide core data support for subsequent feature compression and quantization.

[0019] The second interest feature extraction module is set at the decoding end. It takes the intermediate features of the super-prior network decoding end as input to obtain multi-scale prior guided interest features. These features are used to combine the compressed prior information transmitted from the encoding end to restore the key detail features of the image and provide accurate feature references for the high-quality reconstruction of the final image.

[0020] Subsequently, the original image and the resulting multi-scale interest features are input into the encoder feature adaptation module based on these features. Specifically, the original image is downsampled and then input into a sub-module that performs Transformer, FOI-FA, and downsampling. The original multi-scale interest features are also input into this sub-module for adaptive alignment and fusion. This process of Transformer, FOI-FA, and downsampling is repeated to progressively extract higher-level, more abstract image coding features. Through the feature adaptive alignment mechanism, the original multi-scale interest features are deeply fused with the image coding features at each level, ultimately resulting in a coding feature representation that combines downstream task-oriented detail information with global semantics.

[0021] Specifically, the interest-based encoder feature adaptation is achieved through the interest-based feature adaptation module (FOI-FA in the figure). The module consists of spatial domain adaptation, channel adaptation, and frequency domain adaptation. Figure 2 The structure of the feature adaptation module is introduced. Spatial domain adaptation and channel adaptation are combined into spatial domain-channel adaptation, which processes the input features in parallel with frequency domain adaptation, and is linked to the input features through skip links.

[0022] (1) Spatial adaptation based on interest features Spatial domain attention is generated through spatial domain adaptation based on interest features. Input features are obtained by processing the original image using a pre-trained encoder, such as... Figure 2 The input features are generated from pre-trained, fixed gray Transformer blocks. The input features are first processed by max pooling and average pooling, and the pooling results are concatenated along the channel dimension. A convolutional network is then used to generate spatial attention. This spatial attention is applied to the input features while the input features skip connections. This process can be described as follows:

[0023]

[0024] in , and These represent the original multi-scale interest features, input features, and modulated output features, respectively. Since the adaptation process is the same at different stages of the codec, the stage labels are omitted. This represents the resulting spatial attention map; This represents a convolutional neural network with a 7×7 kernel. and These represent average pooling and max pooling applied to different channels at the same location, respectively. and These represent the ReLU activation function and element-wise multiplication, respectively.

[0025] (2) Channel domain feature adaptation based on interest features Channel-domain adaptation based on interest features emphasizes attention between different channels. Similar to spatial adaptation, channel-domain adaptation generates channel attention based on the current input features. Specifically, interest feature information is first processed through average pooling and max pooling along the channel dimension; then, it passes through down-projection linear layers and up-projection linear layers to generate attention between different channels. This process can be described as follows:

[0026]

[0027] in Represents the channel attention map; and These represent max pooling and average pooling at different positions of the same channel, respectively. This indicates the output of the channel domain adaptation module; This indicates the lower projection linear layer and the upper projection linear layer.

[0028] (3) Frequency domain feature adaptation based on interest features Frequency-domain adaptation based on features of interest emphasizes the frequency components relevant to the current task while suppressing task-irrelevant frequency components. Unlike the spatiotemporal adaptation module, the features of interest are first mapped into the latent space and then subjected to a Fast Fourier Transform (FFT). In the frequency domain, attention to different frequencies and channels is obtained through linear layers and depthwise convolutions. The attention map and the FFT-processed input features are multiplied element-wise, and then subjected to an Inverse Fourier Transform (IFFT) and a mapping layer to obtain the frequency-domain adapted features. This process can be described as follows:

[0029]

[0030] in and These represent linear layers and depthwise convolutions, respectively. and This indicates FFT and IFFT; This represents the attention map in the frequency domain.

[0031] After spatial-channel adaptation and frequency-domain adaptation, the input features The features modulated by each domain are added together to obtain the output features of the feature adaptation module. This process can be represented as:

[0032] Through multiple rounds of iterative modulation and deep feature fusion using Transformer, FOI-FA, and downsampling, a task-oriented potential feature y is finally formed, which combines downstream task-oriented detailed information with global semantics.

[0033] Furthermore, through the proposed adaptive quantization, the task-oriented latent feature y is quantized and entropy-encoded using the predicted mean, variance, and quantization step size.

[0034] Distortion of latent features primarily arises from the quantization process. A more direct way to adjust the distribution of latent features is through adaptive quantization of different elements of the features. Therefore, interest-based adaptive quantization is applied to the super-prior network to adaptively generate different quantization parameters for different tasks. Interest-based adaptive quantization integrates an interest-based feature adaptation block (FOI-QA in the figure) into the pre-trained super-prior encoder. The frozen super-prior decoder generates the mean, variance, and quantization parameters. Based on this, the feature adaptation module is inserted into the super-prior encoder to dynamically generate quantization parameters based on the interest feature information.

[0035] Specifically, the latent features generated by the encoder The latent features are modulated by a downsampling and feature adaptation module (FOI-QA module in the diagram); the modulated features are then transformed by a Transformer block, and the intermediate features are further processed by the feature adaptation module and downsampling to generate smaller-sized hyperprior information. After encoding (AE) at the encoder, the resulting bitstream is decoded (AD) at the decoder. The decoded prior information is then upsampled, processed by a Transformer block, and subjected to an entropy model to obtain the mean, variance, and quantization parameters. Similar to feature adaptation in encoders based on interest features, adaptive quantization based on interest features also modulates the prior features from three perspectives: spatial domain, channel domain, and frequency domain. This process can be represented as:

[0036] in This represents a hyper-prior encoder based on interest feature adaptation. Let E represent the super-prior decoder, and let E represent the parameters estimated by the entropy model. These are the predicted quantization parameters, mean, and variance. The quantization parameters are then used to quantize and dequantize the latent features. Simultaneously, the mean and variance are adjusted to align with the quantized latent features. This dynamic quantization at the encoding end can be represented as:

[0037] in, This represents the adjustment value for the latent features after quantization. This represents the adjusted value of the mean. This represents the adjustment value for variance.

[0038] On the decoding end, Dequantized : By utilizing super-prior quantization adaptation based on features of interest, latent features can generate different distortions for different tasks, thereby improving the performance of downstream tasks.

[0039] Finally, the reconstructed image is obtained through the synthetic transformation corresponding to the network structure. While other modules in the encoding and decoding networks are fixed, the lightweight trainable modules in the encoding and decoding networks are trained together to decode latent features and advanced prior information from the received bitstream, as well as multi-scale prior-guided interest features of the decoding network. The reconstructed image is obtained through the decoding network with a feature adaptation module based on interest features.

[0040] Specifically, the decoder extracts the quantized latent features and prior information from the received bitstream. The latent features are then dequantized to obtain... The multi-scale prior-guided interest features of the decoding network are obtained by scaling the intermediate features of the entropy model through convolution. Similar to the encoding network, the decoding network dequantizes the latent features. The input is processed by a sub-module that performs upsampling, FOI-FA, and Transformer operations. Multi-scale prior-guided interest features are input into this module for adaptive feature alignment and fusion at the decoding end. After repeating the upsampling, FOI-FA, and Transformer processes at different scales, the downstream task-oriented RGB reconstructed image is obtained through upsampling.

[0041] To verify the effectiveness of this embodiment, such as Figures 4-6 The diagrams shown are, in order, comparative results between the proposed method and other methods in image classification, object detection, and instance segmentation tasks. CD-ICMH, Trans TIC, ICMH-Net, Adapt-ICMH, and TTFTICM are all existing models modified from human visual codec models.

[0042] The test results are presented in the form of bitrate-accuracy. The accuracy for image classification tasks uses Top-1 accuracy, while the accuracy for object detection and instance segmentation tasks uses mAP@50-95. Bitrate is uniformly measured in bits per pixel (bpp).

[0043] from Figures 4-6As can be seen, the method proposed in this embodiment has significantly better accuracy and mAP than existing models at different bpp bitrates, achieving a dual optimization of improving machine vision task accuracy and reducing bitrate, and achieving a better balance between compression efficiency and task performance.

[0044] This specific embodiment is based on the adaptive encoder-decoder feature based on interest features and the adaptive quantization based on interest features. By adaptively adjusting the latent features in the spatial, channel, and frequency dimensions and designing them together with adaptive quantization, the accuracy of machine vision tasks is significantly improved and the bit rate is effectively reduced without significantly increasing the computational complexity and training cost of the model, thereby obtaining better overall optimization results.

[0045] Example 2 This embodiment provides a human-computer collaborative image compression system based on features of interest, including: The feature extraction module is used to input the original image into the multi-scale interest feature extraction module to obtain the original multi-scale interest features; The coding and modulation module is used to perform adaptive modulation of the latent features generated during the coding process in the spatial domain, channel domain, and frequency domain based on the original multi-scale interest features, through the feature adaptation module in the encoder, to obtain task-oriented latent features. The quantization parameter prediction module is used to input the task-oriented latent features into the super-prior network and perform adaptive quantization based on the original multi-scale interest features to predict the quantization parameters, mean and variance of each latent feature element. The quantization encoding module is used to quantize and entropy encode the latent features using the quantization parameters to generate a compressed bitstream. The decoding and reconstruction module is used at the decoding end to dequantize the latent features using the quantization parameters, transform the prior information in the compressed bitstream to obtain multi-scale prior guided interest features, and modulate the reconstruction process through the feature adaptive module of the decoder to finally obtain the reconstructed image.

[0046] Example 3 This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a human-computer collaborative image compression method based on features of interest as described in Embodiment 1 above.

[0047] Example 4 This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the human-computer collaborative image compression method based on features of interest described in Embodiment 1 above.

[0048] The steps or modules involved in Embodiments 2 to 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0049] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A human-computer collaborative image compression method based on features of interest, characterized in that, include: The original image is input into the multi-scale interest feature extraction module to obtain the original multi-scale interest features; Based on the original multi-scale interest features, the latent features generated during the encoding process are adaptively modulated in the spatial domain, channel domain, and frequency domain through the feature adaptation module in the encoder to obtain task-oriented latent features. The task-oriented latent features are input into the super-prior network, and adaptive quantization is performed based on the original multi-scale interest features to predict the quantization parameters, mean, and variance of each latent feature element. The latent features are quantized and entropy encoded using the quantization parameters to generate a compressed bitstream; At the decoding end, the latent features are dequantized using the quantization parameters, and the prior information in the compressed bitstream is transformed to obtain multi-scale prior guided interest features. The reconstruction process is then modulated by the feature adaptive module of the decoder to finally obtain the reconstructed image.

2. The human-computer collaborative image compression method based on features of interest as described in claim 1, characterized in that, The multi-scale interest feature extraction module includes an encoding-end extraction network and a decoding-end extraction network. The encoding-end extraction network takes the original image as input and extracts the original multi-scale interest features through a series of connected Transformer layers and multiple convolutional layers. The decoding-end extraction network takes the intermediate features of the decoding end of the super-prior network as input and obtains the multi-scale prior guided interest features required by the decoding end through convolutional layers.

3. The human-computer collaborative image compression method based on features of interest as described in claim 1, characterized in that, The feature adaptation module in the encoder includes a spatial-channel domain adaptation submodule and a frequency domain adaptation submodule that are configured in parallel. The spatial-channel adaptation submodule sequentially performs spatial attention modulation and channel attention modulation on the input features; The frequency domain adaptation submodule performs frequency domain attention modulation on the input features; The outputs of each submodule are added to the input features through skip connections to achieve adaptive modulation and obtain task-oriented latent features.

4. The human-computer collaborative image compression method based on features of interest as described in claim 1, characterized in that, In the aforementioned prior network, adaptive quantization specifically refers to: In the encoder of the super-prior network, a feature adaptation block is integrated, which modulates the input latent features in the spatial domain, channel domain, and frequency domain based on the original multi-scale interest features. After the modulated features are processed by the decoder of the super-prior network, the quantization step size, mean, and variance corresponding to each potential feature element are predicted.

5. The human-computer collaborative image compression method based on features of interest as described in claim 1, characterized in that, The step of quantizing and entropy encoding the latent features using the quantization parameters to generate a compressed bitstream specifically includes: dividing each element value, mean, and variance of the latent features by the corresponding quantization step size to complete numerical normalization; and entropy encoding the normalized feature values ​​to generate a compressed bitstream.

6. The human-computer collaborative image compression method based on features of interest as described in claim 1, characterized in that, In the decoding end, the reconstruction process is specifically as follows: Decode the quantized latent features, quantization parameters, and multi-scale features of interest from the compressed bitstream; The multi-scale prior-guided interest features are used to modulate the decoding process through the feature adaptation module in the decoder; The decoded latent features are multiplied element-wise with the corresponding quantization step size to complete dequantization, and then the image is synthesized and reconstructed through the decoder network.

7. A human-computer collaborative image compression system based on features of interest, characterized in that, include: The feature extraction module is used to input the original image into the multi-scale interest feature extraction module to obtain the original multi-scale interest features; The coding and modulation module is used to perform adaptive modulation of the latent features generated during the coding process in the spatial domain, channel domain, and frequency domain based on the original multi-scale interest features, through the feature adaptation module in the encoder, to obtain task-oriented latent features. The quantization parameter prediction module is used to input the task-oriented latent features into the super-prior network and perform adaptive quantization based on the original multi-scale interest features to predict the quantization parameters, mean and variance of each latent feature element. The quantization encoding module is used to quantize and entropy encode the latent features using the quantization parameters to generate a compressed bitstream. The decoding and reconstruction module is used at the decoding end to dequantize the latent features using the quantization parameters, transform the prior information in the compressed bitstream to obtain multi-scale prior guided interest features, and modulate the reconstruction process through the feature adaptive module of the decoder to finally obtain the reconstructed image.

8. The human-computer collaborative image compression system based on features of interest as described in claim 7, characterized in that, The multi-scale interest feature extraction module includes an encoding-end extraction network and a decoding-end extraction network. The encoding-end extraction network takes the original image as input and extracts the original multi-scale interest features through a series of connected Transformer layers and multiple convolutional layers. The decoding-end extraction network takes the intermediate features of the decoding end of the super-prior network as input and obtains the multi-scale prior guided interest features required by the decoding end through convolutional layers.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the human-computer collaborative image compression method based on features of interest as described in any one of claims 1-6.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the human-computer collaborative image compression method based on features of interest as described in any one of claims 1-6.

Images