Extensible image coding method based on collaboration of human eyes and machine vision

Abstract

The invention discloses an expandable image coding method based on human eye and machine vision collaboration. The invention provides a task-flexible human eye-machine collaborative expandable image coding network. The task-flexible human eye-machine collaborative expandable image coding network comprises an encoder, an adaptive feature division network, a super encoder, a conditional context model, a super decoder, a potential feature transformation network, a decoder and a machine vision task downstream network. In the loss function of the expandable coding network, a multi-stage learning loss function is added. According to the method, the optimal rate-distortion performance is realized.

Description

Technical Field

[0001] The present invention belongs to the fields of machine learning, image coding and computer vision, and aims to achieve scalable image coding that collaborates with human vision and machine vision, and optimizes high-accuracy machine vision tasks and high-fidelity human vision reconstruction. Background Art

[0002] In recent years, with the rapid increase in the proportion of images and videos as the main media forms in the Internet, the explosive growth of visual data has become one of the main driving forces of Internet traffic. However, the limited resources such as storage devices and transmission bandwidth make it impractical to directly store source data, so efficient compression of images and videos has become an urgent problem to be solved. In particular, with the rise of 5G and Internet of Things technologies, object-to-object communication is gradually surpassing and replacing human-to-human communication, making the amount of video generated by edge devices, transmitted through the network and processed by machines rapidly surpass the amount of video processed by humans.

[0003] At present, image coding algorithms based on deep learning are generally optimized for the human visual system and suitable for human viewing applications, but they are not optimized for other application scenarios, such as the very common image processing and machine vision applications. For these applications, it is often necessary to fully decode the image to obtain the reconstructed image before the reconstructed image can be used as the input of these applications. This process is complex and time-consuming, and the lossy image after decoding often has a negative impact on the accuracy of machine vision tasks. Reducing the bit rate while maintaining the subjective quality of the image unchanged and providing a representation suitable for direct use as machine vision input data has become an urgent problem to be solved. Therefore, traditional video coding methods no longer meet the needs of machine vision. It is necessary to study new image compression methods based on the characteristics of machine vision to improve efficiency and performance. Research in this field not only helps to solve the challenges faced in practical applications, but also promotes the integrated development of machine vision and image processing. Efficient machine vision coding (Video Voding for Machines, VCM) has become an important topic in academia and industry. VCM technology is committed to connecting the two independent research directions of image compression and feature compression to a certain extent, while jointly optimizing high-accuracy machine vision tasks and high-fidelity human vision reconstruction as much as possible.

[0004] In the existing scalable coding framework, the latent space is divided into two parts, the base layer and the enhancement layer. The base layer is used to support machine vision tasks, such as target detection. However, the latent space in the coding task does not match the features of target detection. It is necessary to use the latent space transformation or knowledge distillation network to transform the features of the coding domain into the feature space required for target detection, and then input it into the subsequent network of target detection for processing to obtain the final output result Map. Therefore, how to solve the transformation of latent space features is a key problem that needs to be solved. Summary of the invention

[0005] To address the problem of how to efficiently convert the potential features of the coding space into downstream features of machine vision tasks in a scalable coding framework to obtain the results of machine vision tasks, the present invention proposes a new feature space transformation network and a new scalable image coding method that collaborates with human eyes and machine vision.

[0006] The structure of the Task-Flexible Scalable Human-Machine Image Compression Net (TFSIC) proposed in the present invention is as follows: Figure 1 As shown, it includes encoder, adaptive channel partitioning module (ACPM), super encoder, conditional context model, super decoder, latent feature transform network (LST), decoder and machine vision task downstream network. In the loss function of the scalable encoding network, a multi-stage learning loss function is added.

[0007] In the TFSIC network structure, the Task-FlexibleScalable Human-Machine Image CompressionNet (TFSIC) structure includes an encoder, an adaptive channel partitioning module (ACPM), a super encoder, a conditional context model, a super decoder, a latent feature transform network (LST), a decoder, and a downstream network for machine vision tasks. It includes eight parts: the encoder extracts the latent features of the image The adaptive feature partitioning network adaptively divides the potential features into the base layer and enhancement layer The enhancement layer and the base layer are successively passed through the super encoder to obtain the super prior features and The super-prior feature is passed through the arithmetic encoder to obtain a bit stream, and the bit stream is passed through the arithmetic decoder and the super decoder to obtain the entropy parameters of the potential features; the entropy parameters and the base layer and enhancement layer features are decoded in the conditional context model. First, the enhancement layer super-prior feature z1 and the enhancement layer y1 are encoded and decoded in the entropy parameter model of the first stage to obtain the enhancement layer code stream and the decoded enhancement layer The base layer super prior feature z2 and base layer y2 are encoded and decoded in the second stage entropy parameter model to obtain the base layer code stream and the decoded base layer Decoded base layer The features are obtained through the latent feature transformation network feature Input to the downstream network of the machine vision task to obtain the machine vision task result; the decoded base layer and the decoded enhancement layer The latent feature decoding after decoding is obtained by merging operation Decoding of latent features after decoding Input to the decoder to get the decoded image

[0008] (a) encoder;

[0009] In the encoder network, three down-sampling convolution residual blocks and a set of convolution residual blocks are used to extract the image Extracting latent features As the input of the adaptive feature segmentation module; it is expressed as:

[0010] y=g a (X)

[0011] Where X represents the input image, g a represents the encoder and y represents the latent feature y.

[0012] (b) Adaptive feature segmentation module;

[0013] The adaptive feature partitioning network adaptively divides the potential features into the base layer and enhancement layer like Figure 3 shown.

[0014] All potential features Input the adaptive pooling layer, adjust y to the form of 1×1×192, and get F2. After the convolution operation, get F3 with the shape of 1×1×192×2. After the Gumbel-softmax operation, get the mask M2 corresponding to the base layer and the mask M1 corresponding to the enhancement layer. The form of M1 may be 10011..., the form of M2 may be 01100..., 1 represents the selected channel, and 0 represents the unselected channel. Multiply y by M1 to get the enhancement layer, and multiply y by M2 to get the base layer. Through the rearrangement operation, according to the order of index 1, the channels of the base layer and the enhancement layer that are all 0 are transformed to the front and the back respectively, to obtain the continuous non-zero base layer y2 and enhancement layer y1, which can ensure a smaller bit rate for subsequent arithmetic coding. Expressed as:

[0015]

[0016] in, It is an element-wise multiplication operation. Trans represents channel rearrangement by index. y1 and y2 represent the enhancement layer and the base layer respectively.

[0017] (c) Supercoder / decoder;

[0018] In the super encoder / decoder, the enhancement layer and the base layer are successively passed through the super encoder to obtain the super prior features. and The super-prior feature is passed through the arithmetic encoder to obtain a bit stream, and the bit stream is passed through the arithmetic decoder and the super decoder to obtain the entropy parameters of the potential features; the entropy parameters and the base layer and enhancement layer features are decoded in the conditional context model. First, the enhancement layer super-prior feature z1 and the enhancement layer y1 are encoded and decoded in the entropy parameter model of the first stage to obtain the enhancement layer code stream and the decoded enhancement layer The base layer super prior feature z2 and base layer y2 are encoded and decoded in the second stage entropy parameter model to obtain the base layer code stream and the decoded base layer It is expressed as:

[0019]

[0020] in represents the base layer super encoder, z2 represents the obtained base layer super potential feature, represents the enhancement layer super encoder, z1 represents the obtained enhancement layer super decoder, represents the base layer super decoder, ψ2 represents the entropy parameter of the base layer, ψ1 represents the entropy parameter of the enhancement layer, G 2 represents the entropy model at the base layer stage, G 1 represents the entropy model of the enhancement layer stage, represents the decoded enhancement layer, Represents the decoded base layer.

[0021] (d) Latent feature transformation network;

[0022] In the latent feature transformation network, the encoded base layer is first transformed through an upper convolution layer After zooming in, the image is weighted by the attention module we designed, and then downsampled four times to learn the 5th, 8th, 12th, and 13th layers of FPN, and finally input into the FPN subsequent network to obtain the target detection result. It can be expressed as:

[0023]

[0024] In addition, the loss function is

[0025] L=R+λD

[0026]

[0027] Where x represents the input data, For all potential representations (including base layer and enhancement layer), is the super latent representation of all . The distortion D is calculated as

[0028]

[0029] Among them, F1, F2, F3, and F4 are the features of the 5th, 8th, 12th, and 13th layers of FPN.

[0030] (e) decoder;

[0031] The encoder uses three upsampling convolutional residual blocks and a set of convolutional residual blocks, where the decoded enhancement layer and the decoded base layer Concatenate into the decoded latent feature layer The decoded image is obtained by decoding It is expressed as:

[0032]

[0033] in, represents the decoded image, g s Represents a decoder, represents the decoded enhancement layer, Represents the decoded base layer. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is the eye-machine collaborative scalable image coding network (TFSIC).

[0035] Figure 2 It's an encoder.

[0036] Figure 3 It is an adaptive partitioning module.

[0037] Figure 4 It is a super codec.

[0038] Figure 5 is the latent feature transformation network.

[0039] Figure 6 It's a decoder.

[0040] Figure 7 This is a comparison of the target detection performance of the framework of the present invention and the baseline network on the COCO2014 validation set. DETAILED DESCRIPTION

[0041] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0042] like Figure 1 As shown in Figure 2, the structure of Task-FlexibleScalable Human-Machine Image CompressionNet (TFSIC) is as follows: Figure 1 As shown in the figure, it includes encoder, adaptive channel partitioning module (ACPM), super encoder, conditional context model, super decoder, latent space transform network (LST), decoder and machine vision task downstream network. It includes eight parts: encoder extracts image latent features The adaptive feature partitioning network adaptively divides the potential features into the base layer and enhancement layer The enhancement layer and the base layer are successively passed through the super encoder to obtain the super prior features and The super-prior feature is passed through the arithmetic encoder to obtain a bit stream, and the bit stream is passed through the arithmetic decoder and the super decoder to obtain the entropy parameters of the potential features; the entropy parameters and the base layer and enhancement layer features are decoded in the conditional context model. First, the enhancement layer super-prior feature z1 and the enhancement layer y1 are encoded and decoded in the entropy parameter model of the first stage to obtain the enhancement layer code stream and the decoded enhancement layer The base layer super prior feature z2 and base layer y2 are encoded and decoded in the second stage entropy parameter model to obtain the base layer code stream and the decoded base layer Decoded base layer The features are obtained through the latent feature transformation network feature Input to the downstream network of the machine vision task to obtain the machine vision task result; the decoded base layer and the decoded enhancement layer The latent feature decoding after decoding is obtained by merging operation Decoding of latent features after decoding Input to the decoder to get the decoded image

[0043] (a) Encoder

[0044] The network result of the encoder is as follows Figure 2 As shown:

[0045] The encoder uses three down-sampling convolutional residual blocks and a set of convolutional residual blocks to extract the image Extracting latent features As the input of the adaptive feature segmentation module.

[0046] The process can be expressed as:

[0047] y=g a (X)

[0048] Where X represents the input image, g a represents the encoder and y represents the latent feature y.

[0049] (b) Adaptive feature segmentation module

[0050] The adaptive feature partitioning network adaptively divides the potential features into the base layer and enhancement layer like Figure 3 shown.

[0051] All potential features Input the adaptive pooling layer, adjust y to the form of 1×1×192, and get F2. After the convolution operation, get F3 with the shape of 1×1×192×2. After the Gumbel-softmax operation, get the mask M2 corresponding to the base layer and the mask M1 corresponding to the enhancement layer. The form of M1 may be 10011..., the form of M2 may be 01100..., 1 represents the selected channel, and 0 represents the unselected channel. Multiply y by M1 to get the enhancement layer, and multiply y by M2 to get the base layer. Through the rearrangement operation, according to the order of index 1, the channels of the base layer and the enhancement layer that are all 0 are transformed to the front and the back respectively, to get the continuous non-zero base layer y2 and enhancement layer y1, which can ensure a smaller bit rate for subsequent arithmetic coding.

[0052] The process can be expressed as:

[0053]

[0054] in, It is an element-wise multiplication operation. Trans represents channel rearrangement by index. y1 and y2 represent the enhancement layer and the base layer respectively.

[0055] (c) Supercoder / Decoder

[0056] Supercoder / decoder such as Figure 4 (a) and (b) indicate that the enhancement layer and the base layer are successively passed through the super encoder to obtain the super prior features. and The super-prior feature is passed through the arithmetic encoder to obtain a bit stream, and the bit stream is passed through the arithmetic decoder and the super decoder to obtain the entropy parameters of the potential features; the entropy parameters and the base layer and enhancement layer features are decoded in the conditional context model. First, the enhancement layer super-prior feature z1 and the enhancement layer y1 are encoded and decoded in the entropy parameter model of the first stage to obtain the enhancement layer code stream and the decoded enhancement layer The base layer super prior feature z2 and base layer y2 are encoded and decoded in the second stage entropy parameter model to obtain the base layer code stream and the decoded base layer

[0057] The process can be expressed as:

[0058]

[0059] in represents the base layer super encoder, z2 represents the obtained base layer super potential feature, represents the enhancement layer super encoder, z1 represents the obtained enhancement layer super decoder, represents the base layer super decoder, ψ2 represents the entropy parameter of the base layer, ψ1 represents the entropy parameter of the enhancement layer, G 2 represents the entropy model at the base layer stage, G 1 represents the entropy model of the enhancement layer stage, represents the decoded enhancement layer, Represents the decoded base layer.

[0060] (d) Latent Feature Transformation Network

[0061] The latent feature transformation network is as follows Figure 5 As shown, the encoded base layer is firstly transformed through an upper convolution layer After zooming in, the image is weightedly processed by the attention module we designed, and then downsampled four times to learn the 5th, 8th, 12th, and 13th layers of FPN, and finally input into the FPN subsequent network to obtain the target detection result.

[0062] The process can be expressed as:

[0063]

[0064] In addition, our loss function is

[0065] L=R+λD

[0066]

[0067] Where x represents the input data, For all potential representations (including base layer and enhancement layer), is the super latent representation of all . The distortion D is calculated as

[0068]

[0069] Among them, F1, F2, F3, and F4 are the features of the 5th, 8th, 12th, and 13th layers of FPN.

[0070] (e) Decoder

[0071] The decoder network results are as follows Figure 6 As shown:

[0072] The encoder uses three upsampling convolutional residual blocks and a set of convolutional residual blocks, where the decoded enhancement layer and the decoded base layer Concatenate into the decoded latent feature layer The decoded image is obtained by decoding

[0073] The process can be expressed as:

[0074]

[0075] in, represents the decoded image, g s Represents a decoder, represents the decoded enhancement layer, Represents the decoded base layer.

[0076] The present invention selects 300,000 images from the OpenImages dataset and randomly crops them into a size of 256*256. The present invention sets the batch size to 16 and uses the Adam optimizer on an RTX 2080 GPU equipped with 11GB of memory to train the model for multiple cycles with an initial learning rate of 1×10-4. When the validation loss does not decrease in three consecutive cycles, indicating that overfitting may occur, the present invention decreases the learning rate by a factor of 0.8 until the learning rate drops below 1×10-5, at which point training is stopped. The present invention selects six λ values

[0077] {0.0018, 0.0035, 0.0067, 0.013, 0.025, 0.0483}. The number of channels for all λ is set to C=192.

[0078] The latent space transformer module transforms the basic layer Map to In YOLOv3, the 13th layer consists of a 3×3 convolutional layer, batch normalization, and a Leaky ReLU activation function. Since the pre-trained YOLOv3 weights contain prior knowledge obtained from the training data, the output of the 13th convolutional layer of the target detection network YOLOv3 is defined as the true feature (where the residual connection counts as one layer).

[0079] The network task of the present invention includes input reconstruction that supports human visual perception, which is evaluated on the Kodak dataset.

[0080] Benchmarks: The performance of our network is compared to multiple benchmarks, including standard codecs HEVC, VVC, and DNN-based codecs optimized for MSE. The reported results for JPEG, HEVC, and VVC are from CompressAI, using HM16.20 and VTM-12.3 for HEVC and VVC, respectively. When evaluating, RGB images are first converted to YUV444 format and then directly fed into the codec. The decoded YUV444 format images are then converted back to RGB format for PSNR and MS-SSIM calculations.

[0081] Performance Evaluation: Table 1 shows the BD-rate comparison between different codecs on the Kodak dataset, which is a common metric to measure the average difference in distortion-bitrate curves or accuracy-bitrate curves. A negative BD-rate value indicates an average bitrate saving at the same accuracy.

[0082] Table 1 Comparison of the method of the present invention with other traditional methods on Kodak dataset at BD-rate

[0083]

[0084] Figure 7 The object detection performance of the present invention is shown in comparison with various benchmarks in terms of bitrate per pixel (bpp) and mean average precision (mAP). HEVC and VVC are encoded using reference software HM-16.20 and VTM-11.1. The bpp is calculated by dividing the base layer bitrate and the hyper-prior bitrate by the number of input pixels. The present method achieves the best rate-distortion performance. Figure 4-6In the figure, the black dashed line indicates the default mAP performance of 62.8%, which is the performance when the images in the COCO2014 validation set are used as input and the YOLOv3 pre-trained weights are used.

Claims

1. A scalable image coding method based on the collaboration of human eyes and machine vision, characterized in that: The scalable image coding network TFSIC for collaboration between human vision and machine vision includes an encoder, Adaptive feature partitioning network ACPM, super encoder, conditional context model, super decoder, latent feature transformation network LST, decoder and downstream network for machine vision tasks; In the loss function of the scalable coding network, a multi-stage learning loss function is added; in the TFSIC network structure, the human eye-machine collaborative scalable image coding network includes an encoder, Adaptive feature partitioning network ACPM, super encoder, conditional context model, super decoder, latent feature transformation network LST, decoder and downstream network for machine vision tasks; It consists of eight parts: the encoder extracts the potential features of the image The adaptive feature partitioning network adaptively divides the potential features into the base layer and enhancement layer The enhancement layer and the base layer are successively passed through the super encoder to obtain the super prior features and The super-prior feature is passed through the arithmetic encoder to obtain a bit stream, and the bit stream is passed through the arithmetic decoder and the super decoder to obtain the entropy parameters of the potential features; the entropy parameters and the base layer and enhancement layer features are decoded in the conditional context model. First, the enhancement layer super-prior feature z1 and the enhancement layer y1 are encoded and decoded in the entropy parameter model of the first stage to obtain the enhancement layer code stream and the decoded enhancement layer The base layer super prior feature z2 and base layer y2 are encoded and decoded in the second stage entropy parameter model to obtain the base layer code stream and the decoded base layer Decoded base layer The features are obtained through the latent feature transformation network feature Input into the downstream network of the machine vision task to obtain the machine vision task result; Decoded base layer and the decoded enhancement layer The latent feature decoding after decoding is obtained by merging operation Decoding of latent features after decoding Input to the decoder to get the decoded image 2. The scalable image coding method based on the collaboration of human eye and machine vision according to claim 1, characterized in that: In the encoder network, three down-sampling convolution residual blocks and a set of convolution residual blocks are used to extract the image Extracting latent features As the input of the adaptive feature segmentation module; it is expressed as: y=g a (X) Where X represents the input image, g a represents the encoder and y represents the latent feature y.

3. The scalable image coding method based on the collaboration of human eye and machine vision according to claim 1, characterized in that: The adaptive feature partitioning network adaptively divides the potential features into the base layer and enhancement layer All potential features Input the adaptive pooling layer, adjust y to the form of 1×1×192, and get F2; after the convolution operation, get F3 with the shape of 1×1×192×2; after the Gumbel-softmax operation, get the mask M2 corresponding to the base layer and the mask M1 corresponding to the enhancement layer, M1 may be in the form of 10011..., M2 may be in the form of 01100..., 1 represents the selected channel, 0 represents the unselected channel; y point multiply M1 to get the enhancement layer, y point multiply M2 to get the base layer, and through the rearrangement operation, according to the order of index 1, the channels of the base layer and the enhancement layer that are all 0 are transformed to the front and the back respectively, to get the continuous non-zero base layer y2 and enhancement layer y1, which can ensure that the subsequent arithmetic coding bit rate is smaller; expressed as: in, It is an element-wise multiplication operation. Trans represents channel rearrangement by index. y1 and y2 represent the enhancement layer and the base layer respectively.

4. The scalable image coding method based on the collaboration of human eye and machine vision according to claim 1, characterized in that: In the super encoder / decoder, the enhancement layer and the base layer are successively passed through the super encoder to obtain the super prior features. and The super-prior feature is passed through the arithmetic encoder to obtain a bit stream, and the bit stream is passed through the arithmetic decoder and the super decoder to obtain the entropy parameters of the potential features; the entropy parameters and the base layer and enhancement layer features are decoded in the conditional context model. First, the enhancement layer super-prior feature z1 and the enhancement layer y1 are encoded and decoded in the entropy parameter model of the first stage to obtain the enhancement layer code stream and the decoded enhancement layer The base layer super prior feature z2 and base layer y2 are encoded and decoded in the second stage entropy parameter model to obtain the base layer code stream and the decoded base layer It is expressed as: in represents the base layer super encoder, z2 represents the obtained base layer super potential feature, represents the enhancement layer super encoder, z1 represents the obtained enhancement layer super decoder, represents the base layer super decoder, ψ2 represents the entropy parameter of the base layer, ψ1 represents the entropy parameter of the enhancement layer, G 2 represents the entropy model at the base layer stage, G 1 represents the entropy model of the enhancement layer stage, represents the decoded enhancement layer, Represents the decoded base layer.

5. The scalable image coding method based on the collaboration of human eye and machine vision according to claim 1, characterized in that: In the latent feature transformation network, the encoded base layer is first transformed through an upper convolution layer After zooming in, the weighted processing is performed by the attention module we designed, and then four downsampling processes are performed to downsample and learn the 5th, 8th, 12th, and 13th layers of FPN respectively, and finally input into the FPN subsequent network to obtain the target detection result; it is expressed as: In addition, the loss function is L = R + λD Where x represents the input data, For all potential representations, including base layer and enhancement layer, is the super potential representation of all ; the distortion D is calculated as: Among them, F1, F2, F3, and F4 are the features of the 5th, 8th, 12th, and 13th layers of FPN.

6. The scalable image coding method based on the collaboration of human eye and machine vision according to claim 1, characterized in that: The encoder uses three upsampling convolutional residual blocks and a set of convolutional residual blocks, where the decoded enhancement layer and the decoded base layer Concatenate into the decoded latent feature layer The decoded image is obtained by decoding It is expressed as: in, represents the decoded image, gs represents the decoder, represents the decoded enhancement layer, Represents the decoded base layer.

Images