An end-to-end image coding method for machine vision

Abstract

The application discloses an end-to-end image coding method for machine vision. The end-to-end image coding and decoding device is composed of a neural network and is optimized through back propagation of a loss function. An encoder network transforms an input image through analysis to obtain a compact high-dimensional latent representation, and a decoder network restores the original image through synthesis transformation of the latent representation. The method comprises the following steps: (1) taking a target detection algorithm as a typical task of machine vision, inputting an uncompressed image into a target detector YOLO to obtain an uncompressed detection result and precision; (2) performing saliency analysis of machine vision to obtain a saliency map of target detection; (3) adjusting the channel number of the latent representation after analysis transformation according to the obtained saliency map of machine vision; (4) using a loss function combining machine vision and human eye vision distortion to train the encoder and decoder network in an end-to-end mode; and (5) using the obtained model to compress and decompress actual images. The application fully considers the characteristics of machine vision, so that the final coding effect can maintain higher target detection precision under the condition of consuming the same bit rate.

Description

Technical Field

[0001] This invention relates to the field of end-to-end image compression, and in particular to an image coding method for machine vision. Background Technology

[0002] Ubiquitous cameras are constantly generating new images and videos, and this visual data accounts for the largest proportion of data traffic. Compact representation of visual data is crucial, and image compression can alleviate the storage and transmission burden. Furthermore, advancements in machine vision enable humans to process visual data quickly and easily, obtaining the information they need. This frees up human labor, making machines increasingly important consumers of visual data.

[0003] Previous image compression standards were quite mature; however, these methods primarily optimized for human vision. For example, the human eye is more sensitive to changes in luminance than to changes in chromaticity, so encoders allocate more coding resources to areas with drastic brightness variations. Machine vision, on the other hand, is more sensitive to task-related information. Existing encoders, while allocating more coding resources to what human vision focuses on, neglect information important to machine vision. Therefore, when machine vision receives images compressed by existing encoders, its detection performance deteriorates significantly. Considering machine vision characteristics in the encoder would clearly be beneficial for compression methods geared towards machine vision.

[0004] Current image compression methods can be divided into two categories: traditional block-based encoders and end-to-end neural network-based encoders. First, in traditional encoders, each encoding module is relatively independent. Utilizing machine vision characteristics often results in intelligent optimization of parameters in only a few modules, offering limited improvement to overall encoding efficiency. Second, with the development of neural networks, end-to-end image compression methods based on neural networks can extract more high-level semantic information relevant to the visual task. However, existing end-to-end image compression methods, when considering machine vision, often rely solely on the distortion of machine vision task accuracy caused by the loss function to indirectly make the encoder focus more on the parts important to machine vision. Furthermore, while the encoder converts the compressed image into a high-dimensional tensor, considerable redundancy still exists in these channels.

[0005] Introducing neural network interpretability methods into end-to-end encoders allows for direct access to the regions of interest for machine vision, resulting in a more compact latent representation obtained after encoder analysis and transformation. Existing work, under traditional encoding methods, can only optimize local modules, making it difficult to achieve an overall improvement in encoding efficiency. Furthermore, in the end-to-end encoding framework, existing work does not directly consider the characteristics of machine vision, thus retaining a significant amount of redundant information for machine vision purposes. Summary of the Invention

[0006] The technical problem this invention aims to solve is to provide an end-to-end image compression scheme based on machine vision saliency analysis. This scheme, within the end-to-end image compression framework, more fully considers the characteristics of machine vision and allocates encoding resources according to the importance of image content to machine vision. It improves the accuracy of the compressed reconstructed image under machine vision conditions with the same bitrate overhead, and the reconstructed image signal can be extended to other tasks.

[0007] To address the aforementioned technical problems, this invention provides an end-to-end image encoding method for machine vision, comprising the following steps:

[0008] (1) Taking the target detection algorithm as a typical task of machine vision, the uncompressed image is input into the target detector YOLO to obtain the uncompressed detection results and accuracy;

[0009] (2) Perform saliency analysis of machine vision to obtain the saliency map of target detection;

[0010] (3) Based on the obtained machine vision saliency map, adjust the number of channels in the transformed latent representation;

[0011] (4) Combine the distortions of human vision and machine vision, and train the encoder and decoder networks end-to-end in the form of a loss function;

[0012] (5) Use the obtained model to compress and decompress the actual image.

[0013] Preferably, in step (1), obtaining the detection results and detection accuracy of the uncompressed image specifically includes the following steps:

[0014] (1-1) Input the uncompressed image into YOLO to obtain the detection result Pred. gt :

[0015] Pred gt (cls, bbox, conf) = YOLO(x)

[0016] Where x represents the input uncompressed image, Pred gt It represents the detection results on the uncompressed image and includes three parts of information: category information cls, detection box position bbox, and confidence information conf;

[0017] (1-2) Under the tag information, YOLO will calculate the detection accuracy mAP@0.5 and mAP@0.5:0.95.

[0018] Preferably, in step (2), the detection results are subjected to machine vision saliency analysis based on the interpretability of the neural network, and the final saliency map of the target detection is obtained in the form of a heatmap; specifically, the following steps are included:

[0019] (2-1) Obtain the saliency map S of each target in the image. k :

[0020]

[0021] p ij σ represents the raw saliency value of the pixel located in the i-th row and j-th column of the image, ranging from 0 to 1. k This represents the significance threshold of target k, when p ij Less than the threshold σ k When the value is zero, the significance value at that position is set to zero; otherwise, the original significance value is retained.

[0022] (2-2) All saliency maps S k The images are fused to form a final saliency map S representing the saliency of all targets in the image:

[0023]

[0024] S(p ij () represents the saliency at each spatial location in the final saliency map S, which is composed of multiple saliency maps S. k The sum is obtained by adding the values. If the region is greater than 1, it will be set to 1.

[0025] Preferably, in step (3), based on the final saliency map S obtained in step (2), the feature channels obtained from the encoder network output are adjusted according to S. The number of channels after adjustment is calculated as follows:

[0026] C = C min +S(p ij (C) max -C min )

[0027] C represents the adjusted number of channels. min C is the number of basic channels corresponding to the position with the weakest significance. max This refers to the number of feature channels before adjustment;

[0028] Preferably, in step (4), the rate-distortion optimization problem in traditional image compression is extended to machine vision distortion. Combining human vision and machine vision distortion, the encoder and decoder are trained end-to-end in the form of a loss function. Specifically, the steps are as follows:

[0029] (4-1) The rate-distortion optimization problem after overall expansion is calculated using the following formula:

[0030] L=R+λ signal D signal +λ task D task

[0031] Where D signal It is the distortion of signal reconstruction, while D task This is the distortion in machine vision tasks, λ signal and λ task These are the Lagrange coefficients used to control the relationship between human visual distortion and machine visual distortion and bit rate;

[0032] (4-2) The distortion of computer vision tasks is calculated using the following formula:

[0033] D task =αD cls +βD iou +γD conf

[0034] Where D task The target detection result consists of three elements: category information distortion (D) cls Detection frame distortion D iou And confidence level information distortion D conf ;

[0035] (4-3) Calculate the category information distortion D cls The following calculation formula is used:

[0036]

[0037] When the category information is consistent, D cls =0, otherwise, detection box distortion D cannot be considered. iou And confidence level information distortion D conf Then the whole D task =D cls ;

[0038] (4-4) Calculate the detection frame distortion D iou The following calculation formula is used:

[0039]

[0040] iou() calculates the intersection-union ratio (CUP) between bounding boxes before and after compression, bbox. These represent the positions of the detection boxes before and after compression, respectively.

[0041] (4-5) Calculate the confidence level information distortion D conf Cross-entropy is used for calculation.

[0042] Preferably, step (5) uses the trained model to complete the image compression and decompression process.

[0043] The beneficial effects of this invention are as follows: This invention fully considers the characteristics of machine vision within the end-to-end image coding framework, and uses the saliency map obtained from the interpretability of the neural network as auxiliary information of the encoder, so that the final coding effect can maintain higher detection accuracy of the compressed image in machine vision tasks while consuming less bits. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0045] An end-to-end image coding method for machine vision, such as Figure 1 As shown, the specific steps include the following:

[0046] Step (1) Obtain the detection results and detection accuracy of the uncompressed image;

[0047] Step (1-1) Input the uncompressed image into YOLO to obtain the detection result Pred. gt :

[0048] Pred gt (cls, bbox, conf) = YOLO(x)

[0049] Where x represents the input uncompressed image, Pred gt It represents the detection results on the uncompressed image and includes three parts of information: category information cls, detection box position bbox, and confidence information conf;

[0050] In steps (1-2), under the tag information, YOLO will calculate the detection accuracy mAP@0.5 and mAP@0.5:0.95;

[0051] Step (2) Perform saliency analysis using machine vision to obtain the final saliency map for object detection;

[0052] Step (2-1) Obtain the saliency map S of each target in the image. k :

[0053]

[0054] p ij σ represents the raw saliency value of the pixel located in the i-th row and j-th column of the image, ranging from 0 to 1. k This represents the significance threshold of target k, when p ij Less than the threshold σ k When the value is zero, the significance value at that position is set to zero; otherwise, the original significance value is retained.

[0055] Step (2-2) will include all saliency maps Sk The images are fused to form a final saliency map S representing the saliency of all targets in the image:

[0056]

[0057] S(p ij () represents the saliency at each spatial location in the final saliency map S, which is composed of multiple saliency maps S. k The sum is obtained by adding the values; regions with values ​​greater than 1 will be set to 1.

[0058] Step (3) Based on the final saliency map S obtained in step (2), adjust the feature channels obtained from the encoder network output according to S. The number of channels after adjustment is calculated as follows:

[0059] C = C min +S(p ij (C) max -C min )

[0060] C represents the adjusted number of channels. min C is the number of basic channels corresponding to the position with the weakest significance. max This refers to the number of feature channels before adjustment;

[0061] Step (4) extends the rate-distortion optimization problem in traditional image compression to machine vision distortion, combining human vision and machine vision distortion, and trains the encoder and decoder end-to-end in the form of a loss function;

[0062] The rate-distortion optimization problem after the overall expansion in step (4-1) is calculated using the following formula:

[0063] L=R+λ signal D signal +λ task D task

[0064] Where D signal It is the distortion of signal reconstruction, while D task This is the distortion in machine vision tasks, λ signal and λ task These are the Lagrange coefficients used to control the relationship between human visual distortion and machine visual distortion and bit rate;

[0065] Step (4-2) calculates the distortion of the machine vision task using the following formula:

[0066] D task =αD cls +βD iou +γD conf

[0067] Where Dtask The target detection result consists of three elements: category information distortion (D) cls Detection frame distortion D iou And confidence level information distortion D conf ;

[0068] Step (4-3) Calculate the category information distortion D cls The following calculation formula is used:

[0069]

[0070] When the category information is consistent, D cls =0, otherwise, detection box distortion D cannot be considered. iou And confidence level information distortion D conf Then the whole D task =D cls ;

[0071] Step (4-4) Calculate the detection frame distortion D iou The following calculation formula is used:

[0072]

[0073] iou() calculates the intersection-union ratio (CUP) between bounding boxes before and after compression, bbox. These represent the positions of the detection boxes before and after compression, respectively.

[0074] Step (4-5) Calculate the confidence information distortion D conf Cross-entropy is used for calculation;

[0075] Step (5) Use the trained model to complete the image compression and decompression process.

[0076] This invention addresses machine vision optimization within an end-to-end image coding framework. It leverages the interpretability of neural networks to obtain saliency analysis results from machine vision, representing them as heatmaps. Hotter regions indicate that the content in that region is more important to machine vision, and vice versa. This heatmap serves as auxiliary information, guiding the end-to-end encoder to allocate bitrate according to spatial content, assigning more bitrate to important regions while suppressing other regions. Furthermore, by incorporating distortions in both human and machine vision, the encoder and decoder are trained end-to-end using a loss function.

[0077] This invention aims to minimize performance distortion in machine vision detection. It fully considers the characteristics of machine vision and, in the absence of task labels, combines loss functions that address both machine vision and human visual distortion to train the encoder-decoder end-to-end. Furthermore, the encoding scheme can be extended to other tasks for image reconstruction. This invention improves the machine vision performance of reconstructed images while reducing bit overhead in the image encoding process.

Claims

1. An end-to-end image coding method for machine vision, comprising the following steps: (1) Taking object detection algorithms as a typical task in machine vision, uncompressed raw images are input into the object detector YOLO to obtain the detection results and detection accuracy of the uncompressed images; among which, Input the uncompressed image into YOLO to obtain the detection result Pred. gt Pred gt (cls, bbox, conf) = YOLO(x); where x represents the input uncompressed image, Pred gt The detection results on the uncompressed image are represented by three parts: category information (cls), bounding box position (bbox), and confidence information (conf). Guided by the label information, YOLO calculates the detection accuracy (mAP@0.5) and mAP@0.5 = 0.

95. (2) Based on the interpretability of the neural network, perform machine vision saliency analysis on the detection results to obtain the final saliency map of the target detection in the form of a heatmap; specifically including: (2-1) obtaining the saliency map S of each target in the image. k : Where, p ij σ represents the raw saliency value of the pixel located in the i-th row and j-th column of the image, ranging from 0 to 1; k This represents the significance threshold of target k, when p ij Less than the threshold σ k When the significance value at that position is zero, the significance value is set to zero; otherwise, the original significance value is retained. (2-2) All significance plots S k The saliency is fused into a final saliency map S representing the saliency of all targets in the image: Among them, S(p ij ) represents the saliency at each spatial location in the final saliency map S, which is composed of multiple saliency maps S. k By adding them together, regions where the sum of the saliency maps is greater than 1 will be set to 1. (3) Based on the final saliency map S obtained in step (2), adjust the number of feature channels obtained from the encoder network output. The adjusted number of channels is calculated as follows: C = C min +S(p ij (C) max -C min ); where C represents the adjusted number of channels, C min C is the number of basic channels corresponding to the position with the weakest significance. max This refers to the number of feature channels before adjustment; (4) Use a loss function that combines machine vision and human visual distortion to train the encoder and decoder networks end-to-end; (5) Use the obtained model to compress and decompress the actual image.

2. The end-to-end image coding method for machine vision as described in claim 1, characterized in that, In step (4), combining the distortions of human vision and machine vision, the encoder and decoder are trained end-to-end in the form of a loss function, specifically including the following steps: (4-1) The rate-distortion optimization problem after overall expansion is calculated using the following formula: L=R+λ signal D signal +λ task D task Where L is the total loss, R is the bitrate required for compression, and D... signal It is the distortion of signal reconstruction, while D task This is the distortion in machine vision tasks, λ signal and λ task These are the Lagrange coefficients used to control the relationship between human visual distortion and machine visual distortion and bit rate; (4-2) The distortion of computer vision tasks is calculated using the following formula: D task =αD cls +βD iou +γD conf Where D task The target detection result consists of three elements: category information distortion (D) cls Detection frame distortion D iou And confidence level information distortion D conf ; (4-3) Calculate the category information distortion D cls The following calculation formula is used: When the category information is consistent, D cls =0, otherwise, detection box distortion D cannot be considered. iou And confidence level information distortion D conf Then the whole D task =D cls ; (4-4) Calculate the detection frame distortion D iou The following calculation formula is used: iou() calculates the intersection-union ratio (CUP) between bounding boxes before and after compression, bbox. These represent the positions of the detection boxes before and after compression, respectively. (4-5) Calculate the confidence level information distortion D conf Cross-entropy loss is used for calculation.

Images