A machine-oriented image coding method with edge enhancement

By using Grad-Cam and F3Net to generate saliency maps and boundary maps, and combining them with an adaptive QP adjustment scheme, image coding is optimized, solving the problem of low coding efficiency in existing machine vision technologies and achieving bit rate savings while maintaining the same level of precision.

CN118053155BActive Publication Date: 2026-11-17HOHAI UNIV
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202211430531.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-11-15
Publication Date
2026-11-17
Estimated Expiration
2042-11-15

AI Technical Summary

Technical Problem

Existing image/video coding schemes are geared towards human vision and cannot effectively address the irregular shapes of objects and the varying importance of different parts in machine vision tasks, resulting in low coding efficiency.

Method used

The Grad-Cam algorithm is used to generate saliency maps and perform thresholding. The boundary maps are obtained by combining F3Net and Canny operators. The adaptive QP adjustment scheme dynamically allocates bits to each CTU to optimize image coding.

Benefits of technology

While ensuring image quality, it saves bit rate and improves the coding efficiency and effectiveness of machine vision tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118053155B_ABST
    Figure CN118053155B_ABST
Patent Text Reader

Abstract

This invention discloses a machine-oriented image coding method with edge enhancement, comprising the following steps: (1) generating a saliency map using the Grad-Cam algorithm and performing thresholding; (2) obtaining a boundary map based on F3Net and the Canny operator, and correcting the boundary map based on the saliency map; (3) dynamically adjusting the QP for each CTU based on the generated saliency map and boundary map according to an adaptive QP adjustment scheme; (4) encoding the image using the final CTU-level QP value. This invention fully considers the coding quality of machine vision regions of interest, enabling the final coding effect to reduce the bit rate and achieve higher accuracy at the same bit rate while maintaining the same accuracy in machine vision analysis tasks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of machine vision image coding technology, and more specifically to an image coding optimization algorithm based on machine vision saliency. Background Technology

[0002] Vision is one of the five human senses and a crucial way to understand and perceive the world. People are more receptive to visual information than to verbal instruction. In the era of big data, visual data such as images and videos have become the largest source of data. The goal of traditional image / video encoding is to minimize the amount of data occupied by images / videos while ensuring visual quality, thus facilitating their transmission and storage.

[0003] However, with the rapid development of deep learning, more and more intelligent image / video applications have emerged, such as: surveillance video, license plate detection, face detection, medical diagnosis, online teaching, etc.

[0004] In traditional image / video encoders such as High Efficiency Video Coding (HEVC), pixel-level distortion metrics, such as MSE (Mean Separation Scale), are widely used as distortion measures. These pixel-level distortion metrics treat every pixel equally, without considering that the importance of each pixel is different. Based on this, to provide a better viewing experience for the human eye, researchers have proposed many methods for detecting regions of interest (ROIs) and, based on ROIs, many rate-distortion optimization methods to improve the viewing experience.

[0005] However, with the development of the aforementioned intelligent applications, a large amount of image / video data is being produced for machine semantic analysis. The amount of images / videos consumed by machines far exceeds the amount consumed by humans. Existing image / video encoding schemes are designed for humans and are not machine-friendly. Machine semantic information distortion is not the same as pixel-level distortion or human visual perception distortion. When consuming images / videos, machines pay different levels of attention to different regions of the image / video. For networks with specific visual tasks, regions containing feature information receive more attention than redundant background regions.

[0006] Most current image coding methods for machine vision adopt an "analysis-then-encode" approach. First, high-precision network or neural network feature interpretation methods are used to analyze and identify the regions of interest to the machine, i.e., semantic information regions. Then, by adjusting quantization parameters (QP), more bits are allocated to these semantic information regions, integrating semantic distortion into the traditional image / video encoder.

[0007] However, these works often use candidate bounding boxes to generate semantic information regions. The candidate bounding box is a regular rectangular region and can only be contained within the candidate bounding box region or not contained within the candidate bounding box region. This cannot well reflect the irregular shapes of real-world objects and the fact that different parts of an object have different importance. Summary of the Invention

[0008] To address the shortcomings of existing technologies, the present invention aims to provide an edge-enhanced machine-oriented image coding method. In the traditional image / video coding framework, this method considers salient regions that affect machine vision tasks and boundary regions of significant subject categories. Based on these regions, more bits are allocated to them according to an adaptive QP adjustment formula, ultimately achieving the effect of saving more bit rate while maintaining the same accuracy.

[0009] A machine-oriented image coding method with edge enhancement according to the present invention includes the following steps:

[0010] (1) Use the Grad-Cam algorithm to generate a saliency map and perform thresholding;

[0011] (2) Obtain the boundary map based on F3Net and Canny operators, and correct the boundary map based on the saliency map;

[0012] (3) Based on the generated saliency map and boundary map, QP is dynamically adjusted for each CTU according to the adaptive QP adjustment scheme;

[0013] (4) Encode the image using the final CTU-level QP value;

[0014] Preferably, in step (1), the Grad-Cam algorithm is used to generate a saliency map and perform thresholding, as illustrated in the diagram below. Figure 3 As shown, the specific steps include the following:

[0015] (1-1) Generate the saliency map S of each image using the Grad-Cam algorithm, and normalize the saliency map.

[0016] (1-2) Since the saliency map region generated by Grad-Cam is often much larger than the actual class subject region, in order to save as many bits as possible, the pixels of the generated saliency map are thresholded. The threshold is set to the average value of the normalized saliency map of each image. If the value of the pixel is lower than the average value, it is set to zero; if the value of the pixel is greater than the average value, it remains unchanged.

[0017] Preferably, in step (2), the boundary map is obtained based on F3Net and the Canny operator, as shown in the schematic diagram below. Figure 4 As shown, the boundary map is corrected based on the saliency map, and its schematic diagram is as follows. Figure 5As shown, the specific steps include the following:

[0018] (2-1) The original image is segmented using the image segmentation network F3Net to obtain the segmentation map and obtain the main body region of the image. Compared with traditional methods such as directly using the Canny operator, which are prone to introducing too much background redundant information, it can obtain the boundary of the main body of the image more efficiently and accurately.

[0019] (2-2) Use the Canny operator on the segmentation map to find the specific image boundary and generate boundary map B;

[0020] (2-3) Perform intersection processing on the boundary map and the saliency map. The boundary map is only valid if it overlaps with the saliency map.

[0021] Preferably, in step (3), the saliency map and boundary map generated by the backbone are dynamically adjusted for each CTU according to the adaptive QP adjustment scheme, specifically including the following steps:

[0022] (3-1) Considering the distortion of semantically informational regions, define the weighted distortion of the k-th CTU. As shown in the following formula:

[0023]

[0024] Where α and β are parameters used to weigh the weights of the saliency map and the boundary map, S k B is the significance weight of the k-th CTU. k D is the significance weight of the k-th CTU. k It is the distortion of the k-th CTU.

[0025] (3-2) The quantization parameter QP of the k-th CTU is derived. k As shown in the following formula:

[0026] QP k =QP base -4.2005×ln(1+α*S k +β*B k )

[0027] Among them, QP base It is the QP corresponding to the global Lagrange factor λ.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. This invention provides a method for measuring the importance of semantic information in an image. It posits that pixels located in category feature regions and subject edge regions are more important than those in other locations. Grad-Cam is used to generate saliency maps of category feature regions, and the F3Net segmentation network and Canny operator are used to generate boundary maps of subject edge regions. The importance of semantic information is then measured using these saliency maps and boundary maps.

[0030] 2. The importance of semantic information is reflected in image coding. This invention defines a new distortion metric based on saliency maps and boundary maps, and derives a CTU-level adaptive QP adjustment scheme based on rate-distortion optimization techniques, allocating more bits to semantic information regions. Attached Figure Description

[0031] Figure 1 This is a general framework diagram of the present invention.

[0032] Figure 2 This is a schematic diagram of the coding framework based on saliency maps and boundary maps of the present invention.

[0033] Figure 3 This is a schematic diagram comparing the original image, Grad-Cam, and saliency map of the dataset used in this invention.

[0034] Figure 4 This is a schematic diagram comparing the Canny operator, F3Net, and boundary graph of this invention.

[0035] Figure 5 This is a schematic diagram comparing the original image, Grad-Cam, and boundary map of the dataset of this invention. Detailed Implementation

[0036] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are only for illustrating the present invention and are not intended to limit the scope of the present invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art fall within the scope defined by the appended claims.

[0037] According to the present invention, a machine-oriented image coding method with edge enhancement is provided, such as... Figure 1 As shown, it includes:

[0038] (1) The Grad-Cam algorithm is used to generate a saliency map and perform thresholding. A schematic diagram is shown below. Figure 3 As shown;

[0039] (1-1) Generate the saliency map S of each image using the Grad-Cam algorithm, and normalize the saliency map.

[0040] (1-2) Since the saliency map region generated by Grad-Cam is often much larger than the actual class subject region, in order to save as many bits as possible, the pixels of the generated saliency map are thresholded. The threshold is set to the average value of the normalized saliency map of each image. If the value of the pixel is lower than the average value, it is set to zero; if the value of the pixel is greater than the average value, it remains unchanged.

[0041] (2) Obtain the boundary map based on F3Net and the Canny operator, as shown in the schematic diagram below. Figure 4 As shown, the boundary map is corrected based on the saliency map, and its schematic diagram is as follows. Figure 5 As shown;

[0042] (2-1) The original image is segmented using the image segmentation network F3Net to obtain the segmentation map and obtain the main body region of the image. Compared with traditional methods such as directly using the Canny operator, which are prone to introducing too much background redundant information, it can obtain the boundary of the main body of the image more efficiently and accurately.

[0043] (2-2) Use the Canny operator on the segmentation map to find the specific image boundary and generate boundary map B;

[0044] (2-3) Perform intersection processing on the boundary map and the saliency map. The boundary map is only valid if it overlaps with the saliency map.

[0045] (3) Based on the generated saliency map and boundary map, QP is dynamically adjusted for each CTU according to the adaptive QP adjustment scheme;

[0046] (3-1) Considering the distortion of semantically informational regions, define the weighted distortion of the k-th CTU. As shown in the following formula:

[0047]

[0048] Where α and β are parameters used to weigh the weights of the saliency map and the boundary map, S k B is the significance weight of the k-th CTU. k D is the significance weight of the k-th CTU. k It is the distortion of the k-th CTU.

[0049] (3-2) The quantization parameter QP of the k-th CTU is derived. k As shown in the following formula:

[0050] QP k =QP base -4.2005×ln(1+α*S k +β*B k )

[0051] Among them, QPbase It is the QP corresponding to the global Lagrange factor λ.

[0052] (4) Encode the image using the final CTU-level QP value;

[0053] This invention proposes an image coding framework for machine vision based on saliency maps and boundary maps, and provides an adjustment scheme for the quantization parameter (QP) of each coding tree unit (CTU). First, to measure the importance of semantic information, the deep learning visualization method Grad-Cam is used to generate pixel-level saliency maps, and subject detection and edge detection are used to generate boundary maps of the image subject, thus enhancing the boundaries of the saliency maps. Then, based on the generated boundary-enhanced saliency maps, more bits are allocated to semantically important regions of the image using the derived CTU-level adaptive QP adjustment formula.

Claims

1. A method of encoding a machine-oriented image with edge enhancement, characterized by, Includes the following steps: (1) Use the Grad-Cam algorithm to generate a saliency map and perform thresholding; (2) Obtain the boundary map based on F3Net and Canny operators, and correct the boundary map based on the saliency map; (3) Based on the generated saliency map and boundary map, dynamically adjust the QP for each CTU according to the adaptive QP adjustment scheme, which includes the following steps: (3-1) Considering distortion with semantic information region, define the weighted distortion located at the k-th block CTU As shown in the following formula: wherein a and b are parameters used to weigh the weight size of the saliency map and the boundary map, S k is the saliency weight of the k-th CTU, B k is the boundary map weight of the k-th CTU, D k is the distortion of the k-th CTU; (3-2) According to the rate-distortion optimization relationship, the quantization parameter QP of the kth CTU is derived k As shown in the following formula: QP k = QP base - 4.2005 x ln(l + a * S k + b * B k ) where QP base is the QP corresponding to the global Lagrangian factor λ; (4) Encode the image using the final CTU-level QP value.

2. The edge-enhanced machine-oriented image encoding method according to claim 1, wherein In step (1), the Grad-Cam algorithm is used to generate a saliency map and perform thresholding, specifically including the following steps: (1-1) Generate the saliency map S of each image using the Grad-Cam algorithm, and normalize the saliency map. (1-2) Since the saliency map region generated by Grad-Cam is often much larger than the actual class subject region, in order to save as many bits as possible, the pixels of the generated saliency map are thresholded. The threshold is set to the average value of the normalized saliency map of each image. If the value of the pixel is lower than the average value, it is set to zero; if the value of the pixel is greater than the average value, it remains unchanged.

3. The edge-enhanced machine-oriented image coding method according to claim 1, characterized in that, In step (2), the boundary map is obtained based on F3Net and the Canny operator, and the boundary map is corrected based on the saliency map. Specifically, the steps are as follows: (2-1) The original image is segmented using the image segmentation network F3Net to obtain the segmentation map and obtain the main body region of the image. Compared with traditional methods such as directly using the Canny operator, which are prone to introducing too much background redundant information, it can obtain the boundary of the main body of the image more efficiently and accurately. (2-2) Use the Canny operator on the segmentation map to find the specific image boundary and generate boundary map B; (2-3) Perform intersection processing on the boundary map and the saliency map. The boundary map is only effective when it overlaps with the saliency map.

Citation Information

Patent Citations

  • Image encoding device, image encoding method, image decoding device, and image decoding method

    JP2022129683A