A method and system for inter-frame image rate-distortion optimization for machine vision

Abstract

The application discloses a kind of interframe image coding rate distortion optimization method and system for machine vision, it is related to image processing technical field, including steps: extracting the feature map of current video frame original image and corresponding reconstruction image under each scale;According to the feature map of original image and reconstruction image, obtain the distortion measure based on feature similarity;According to the point coordinate pixel value of original image and reconstruction image, obtain the distortion measure based on square error;By mixing respectively based on feature similarity, based on square error distortion measure, rate distortion optimization under the control of mixed distortion measure is carried out.The application can better meet the sensitivity of machine vision system to video content features by mapping video frames to feature space and extracting multi-scale features using residual deep neural network, which not only improves the coding efficiency, but also ensures that the key features of the encoded video content are preserved during machine processing, thereby improving the performance and task completion of the algorithm.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a frame inter-image coding rate distortion optimization method and system for machine vision. BACKGROUND

[0002] In recent years, video coding for machine vision has become an important issue in research and industry, which is closely related to the rapid development of 5G and Internet of Things technologies. With the rise of these technologies, the amount of video content generated by edge devices is increasing, and most of it is ultimately consumed by machines, which is even larger than the video for human viewing. Machine vision tasks, such as object detection, image segmentation, and target tracking, have different requirements for video coding than video content for human vision, thus leading to the development needs of video coding for machines (VCM) technology.

[0003] In July 2019, the International Organization for Standardization MPEG established a VCM group to focus on potential standardization work for machine vision video coding. In this context, VVC (Versatile Video Coding) as one of the video coding standards, its rate distortion optimization (RDO) method focuses on improving the balance between coding efficiency and video quality. VVC reduces the amount of data required while ensuring video quality through innovative technologies and algorithms, including sample-based gradual edge and angle prediction, and intra-frame bit allocation scheme based on saliency detection, which combines Canny edge detection and color contrast saliency detection algorithms to improve rate control.

[0004] However, traditional video coding rate distortion optimization is mainly aimed at human visual experience, and does not fully consider the specific sensitivity of video features for machine vision tasks. Therefore, a feature-based rate distortion optimization (RDO) method has emerged, which converts video blocks to feature space for optimization, aiming to maintain high fidelity of features rather than pixel-level fidelity. However, this method may lead to non-optimal coding decisions due to differences in distortion metric ranges and the complexity of neural network processing of coded blocks. SUMMARY

[0005] In view of the above, the present application aims to preserve important features during the encoding process to ensure more efficient encoding without compromising the performance of machine vision algorithms. Based on this, the present application proposes a frame inter-image coding rate distortion optimization method for machine vision, including the steps of:

[0006] S1: Extracting feature maps of the current video frame original image and the corresponding reconstructed image at each scale;

[0007] S2: Obtain a distortion measure based on feature similarity from the feature maps of the original image and the reconstructed image;

[0008] S3: Obtain a distortion measure based on squared error from the pixel values ​​of the point coordinates in the original image and the reconstructed image;

[0009] S4: Rate-distortion optimization is performed under hybrid distortion metric control by mixing distortion metrics based on feature similarity and squared error respectively.

[0010] Furthermore, in step S1, the feature map is extracted using a residual neural network, expressed as the following formula:

[0011]

[0012] In the formula, For the coded block at the current scale, It is a residual neural network. The obtained feature map.

[0013] Furthermore, in step S2, a distortion measure based on feature similarity is obtained through a structural similarity index, expressed as the following formula:

[0014]

[0015] In the formula, The feature map of the original image. To reconstruct the feature map of the image, The mean of the feature map. The variance of the feature map. The covariance of the feature maps between the original image and the reconstructed image. It is a constant. This is a distortion measure based on feature similarity.

[0016] Furthermore, in step S3, the distortion metric based on the squared error is obtained using the following formula:

[0017]

[0018] In the formula, For located The pixel values ​​of the original image. For located The pixel values ​​of the reconstructed image. Image resolution size, This is a distortion measure based on squared error.

[0019] Furthermore, in step S4, the hybrid distortion metric is obtained using the following formula:

[0020]

[0021] wherein, is a distortion measure weight coefficient dynamically adjusted by a deep neural network, is a distortion measure based on feature similarity, is a distortion measure based on square error, is a code rate, is a Lagrange multiplier for balancing distortion and code rate, QP is a quantization parameter, and k is a constant.

[0022] The present application also provides a machine vision-oriented inter-frame image coding rate-distortion optimization system, comprising:

[0023] a feature extraction module configured to extract feature maps of a current video frame original image and a corresponding reconstructed image at each scale;

[0024] a first measurement module configured to obtain a feature similarity-based distortion measure according to the feature maps of the original image and the reconstructed image;

[0025] a second measurement module configured to obtain a square error-based distortion measure according to the point coordinate pixel values of the original image and the reconstructed image;

[0026] a hybrid measurement module configured to perform rate-distortion optimization under hybrid distortion measure control by mixing the feature similarity-based and square error-based distortion measures.

[0027] Further, in the feature extraction module, the feature maps are extracted by a residual neural network, represented by the following formula:

[0028]

[0029] wherein, is a current scale coding block, is a residual neural network, is an obtained feature map.

[0030] Further, in the first measurement module, the feature similarity-based distortion measure is obtained by a structural similarity index, represented by the following formula:

[0031]

[0032] wherein, is a feature map of the original image, is a feature map of the reconstructed image, is a mean value of the feature map, is a variance of the feature map, is a covariance of the feature map between the original image and the reconstructed image, is a constant, is a feature similarity based distortion measure.

[0033] Further, in the second measure module, the square error based distortion measure is obtained by the following formula:

[0034]

[0035] wherein, is a pixel value of the original image at is a pixel value of the reconstructed image at is a pixel value of the original image at is a pixel value of the reconstructed image at is an image resolution size, is a square error based distortion measure.

[0036] Further, in the hybrid measure module, the hybrid distortion measure is obtained by the following formula:

[0037]

[0038] wherein, is a distortion measure weight coefficient dynamically adjusted by a deep neural network, is a feature similarity based distortion measure, is a square error based distortion measure, is a code rate, is a Lagrange multiplier for balancing distortion and code rate, QP is a quantization parameter, and k is a constant.

[0039] Compared with the prior art, the present application has at least the following beneficial effects:

[0040] (1) The frame inter-image coding rate distortion optimization method and system for machine vision can better meet the sensitivity of the machine vision system to video content features by mapping the video frame to the feature space and extracting multi-scale features using the residual deep neural network, which not only improves the coding efficiency, but also ensures that the key features of the coded video content are preserved during machine processing, thereby improving the performance and task completion of the algorithm.

[0041] (2) The feature distortion measure calculated in combination with the structural similarity index forms a hybrid video coding rate distortion cost function together with the traditional square error distortion measure, more accurately reflects the distortion types of interest to machine vision, and avoids the performance degradation that may be caused by simply relying on pixel-level distortion measures.

[0042] (3) By controlling the rate-distortion, the balance between code rate and video quality can be achieved, which can adapt to the coding needs of different scenes, realize the optimal balance between code rate and quality, and thus enhance the overall working efficiency of the machine vision system;

[0043] (4) In the case of limited network bandwidth or storage resources, the video content can still maintain high recognizability and analysis accuracy, which is crucial for machine vision applications that rely on high-quality video input. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 A flowchart of a frame inter-image coding rate-distortion optimization method for machine vision;

[0045] Figure 2 A module diagram of a frame inter-image coding rate-distortion optimization system for machine vision. DETAILED DESCRIPTION

[0046] The following is a specific embodiment of the present application and further describes the technical solutions of the present application in conjunction with the drawings, but the present application is not limited to these embodiments. Embodiment one

[0047] The present application proposes a frame inter-image coding rate-distortion optimization method for machine vision optimization, aiming to preserve important features during the encoding process and ensure more efficient encoding without compromising the performance of machine vision algorithms. By mapping the encoding block and the original block to the feature space defined by the pre-trained neural network and evaluating the distortion in this space, the present application can more accurately guide the encoding decision, thereby maintaining a high video quality at low bit rates and meeting the special needs of machine vision application scenarios. Specifically, as shown in Figure 1 The present application proposes a frame inter-image coding rate-distortion optimization method for machine vision, including the steps of:

[0048] S1: Extracting the feature maps of the current video frame original image and the corresponding reconstructed image at each scale;

[0049] S2: Obtaining the distortion measure based on feature similarity according to the feature maps of the original image and the reconstructed image;

[0050] S3: Obtaining the distortion measure based on squared error according to the pixel values of the point coordinates of the original image and the reconstructed image;

[0051] S4: Performing rate-distortion optimization under mixed distortion measure control by mixing the distortion measures based on feature similarity and squared error respectively.

[0052] Considering that the rate-distortion optimization (RDO) method in traditional VVC mainly focuses on improving the balance between coding efficiency and video quality, and is biased towards human visual effects, that is, pixel fidelity rather than feature fidelity, and does not take into account the different sensitivities of machine vision tasks to video features. Therefore, in the present application, first, the residual neural network (ResNet) is used to extract the feature maps in each coding block under multiple scales of the current video frame. These feature maps contain feature information such as texture, shape and semantic information. Here, we use ResNet layers of different levels to capture multi-level feature information from low to high levels, thereby helping machines to fully understand video content. Specifically, it can be expressed as follows:

[0053]

[0054] In the formula, is the coding block under the current scale, is the residual neural network, is the obtained feature map.

[0055] Then, we select the structural similarity index (SSIM) for feature similarity-based distortion measurement. SSIM is a method for evaluating image quality, which can take into account the brightness, contrast and structural information of the image, and is therefore closer to the visual quality perception of the machine vision system. Here, the feature maps obtained from the original image and the reconstructed image (constructed by the encoder, which is equivalent to the reference frame) are compared by SSIM, thereby realizing feature similarity-based distortion measurement in the feature space:

[0056]

[0057] In the formula, is the feature map of the original image, is the feature map of the reconstructed image, is the mean of the feature map, is the variance of the feature map, is the covariance of the feature map between the original image and the reconstructed image, is a constant, is the feature similarity-based distortion measurement.

[0058] Compared with the conventional VVC method of performing inter-frame image rate-distortion optimization by square error distortion measurement , the present application proposes a method of combining and to find the optimal code rate and quality balance under machine vision by adjusting the weights of different distortion measurements. The square error distortion measurement can be obtained by the following formula:

[0059]

[0060] wherein, is the pixel value of the original image at , is the pixel value of the reconstructed image at , is the image resolution size, is the distortion measure based on the square error.

[0061] The improved rate-distortion cost function is expressed as the following formula, and the rate-distortion optimization under the mixed distortion measure control can be performed through the formula to obtain a suitable code rate.

[0062]

[0063] wherein, is the distortion measure weight coefficient dynamically adjusted by the deep neural network, is the distortion measure based on the feature similarity, is the distortion measure based on the square error, is the code rate, is the Lagrange multiplier for balancing distortion and code rate, QP is the quantization parameter, and k is a constant (determined by the coding standard and the frame type).

[0064] In order to adaptively adjust the weight coefficient to optimize the video compression effect, the application dynamically adjusts the value of by the method of machine learning, especially by the deep neural network (DNN). Specifically, in the process of adjusting by the DNN, more attention needs to be paid to the performance in the video encoding process, and needs to focus on the structural similarity distortion. Based on different emphases, the corresponding feature information is extracted from the previous coded frames of the current video frame, so as to input the DNN for deep learning and output the corresponding optimal weight coefficient. Among them, the dynamic adjustment of needs to pay attention to the texture complexity, motion estimation parameter average code rate average distortion, etc., and the dynamic adjustment of needs to pay attention to the difference features (such as inter-frame difference statistical data) of the current frame and the previous frame, and the context information (such as QP value, encoding quality, etc.) of the current encoding setting. Embodiment two

[0065] In order to better understand the technical content of the application, the embodiment describes the application in the form of system structure, as shown in Figure 2 a frame inter-image coding rate-distortion optimization system for machine vision, comprising:

[0066] a feature extraction module configured to extract feature maps of a current video frame original image and a corresponding reconstructed image at each scale;

[0067] a first metric module configured to obtain a feature similarity based distortion metric according to the feature maps of the original image and the reconstructed image;

[0068] a second metric module configured to obtain a squared error based distortion metric according to the point coordinate pixel values of the original image and the reconstructed image;

[0069] a hybrid metric module configured to perform rate-distortion optimization under hybrid distortion metric control by mixing the feature similarity based distortion metric and the squared error based distortion metric.

[0070] Further, in the feature extraction module, the feature maps are extracted by a residual neural network, represented by the following formula:

[0071]

[0072] wherein, is a current scale coding block, is the residual neural network, is the obtained feature map.

[0073] Further, in the first metric module, the feature similarity based distortion metric is obtained by a structural similarity index, represented by the following formula:

[0074]

[0075] wherein, is the feature map of the original image, is the feature map of the reconstructed image, is the mean of the feature map, is the variance of the feature map, is the covariance of the feature map between the original image and the reconstructed image, is a constant, is the feature similarity based distortion metric.

[0076] Further, in the second metric module, the squared error based distortion metric is obtained by the following formula:

[0077]

[0078] wherein, is the pixel value of the original image at is the pixel value of the reconstructed image at is the image resolution size, ​​is a distortion metric based on square error.

[0079] Further, in the hybrid metric module, the hybrid distortion metric is obtained by the following formula:

[0080]

[0081] wherein, is a distortion metric weight coefficient dynamically adjusted by a deep neural network, is a distortion metric based on feature similarity, is a distortion metric based on square error, is a code rate, is a Lagrange multiplier for balancing distortion and code rate, QP is a quantization parameter, and k is a constant.

[0082] In summary, the inter-frame image coding rate and distortion optimization method and system for machine vision can better meet the sensitivity of the machine vision system to video content features by mapping video frames to a feature space and extracting multi-scale features using a residual deep neural network. This not only improves coding efficiency, but also ensures that key features of the encoded video content are preserved during machine processing, thereby improving the performance and task completion of the algorithm.

[0083] The feature distortion metric calculated in combination with the structural similarity index is combined with the traditional square error distortion metric to form a hybrid video coding rate and distortion cost function, which more accurately reflects the distortion types of interest to machine vision and avoids performance degradation caused by relying solely on pixel-level distortion metrics.

[0084] Balancing code rate and video quality through hybrid rate distortion control can adapt to coding needs in different scenarios, achieve optimal balance between code rate and quality, and thus enhance the overall work efficiency of the machine vision system.

[0085] In the case of limited network bandwidth or tight storage resources, the video content can still maintain high recognizability and analysis accuracy, which is crucial for machine vision applications that rely on high-quality video input.

[0086] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present application are only used to explain the relative positional relationship, motion condition, etc. between components in a certain posture (as shown in the drawings), and if the certain posture changes, the directional indications will also change accordingly.

[0087] In addition, in the present application, the description such as "first", "second", "one" and the like are only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the technical features or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly specified and limited.

[0088] In the present application, unless otherwise explicitly specified and limited, the terms "connection", "fixing" and the like should be understood broadly, for example, "fixing" can be fixed connection, or detachable connection, or integral; can be mechanical connection, or electrical connection; can be directly connected, or indirectly connected through intermediate medium, can be the internal communication of two elements or the interaction relationship between two elements, unless otherwise explicitly limited. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0089] In addition, the technical solutions of various embodiments of the present application can be combined with each other, but it must be based on the realization of the ordinary skilled in the art, when the combination of technical solutions appears contradictory or cannot be realized, it should be considered that the combination of technical solutions does not exist, nor within the scope of protection required by the present application.

Claims

1. A method for optimizing inter-frame image coding rate-distortion for machine vision, characterized in that, Including the following steps: S1: Extract the feature maps of the original image of the current video frame and the corresponding reconstructed image at various scales; S2: Obtain a distortion measure based on feature similarity from the feature maps of the original image and the reconstructed image; S3: Obtain a distortion measure based on squared error from the pixel values ​​of the point coordinates in the original image and the reconstructed image; S4: Rate-distortion optimization is performed under hybrid distortion metric control by mixing distortion metrics based on feature similarity and squared error respectively; In step S1, the feature map is a multi-level feature map containing texture, shape, and semantic information from low to high level. The feature map is extracted using a residual neural network, as shown in the following formula: In the formula, For the coded block at the current scale, It is a residual neural network. To obtain the feature map; In step S4, the hybrid distortion metric is obtained using the following formula: In the formula, The distortion metric weights are dynamically adjusted by a deep neural network. For distortion measurement based on feature similarity, For distortion measures based on squared error, For bitrate, QP is the Lagrange multiplier used to balance distortion and bit rate, k is the quantization parameter, and k is a constant. During the dynamic adjustment process Focus on performance during the video encoding process. It focuses on structural similarity distortion.

2. The inter-frame image coding rate-distortion optimization method for machine vision as described in claim 1, characterized in that, In step S2, a distortion measure based on feature similarity is obtained through a structural similarity index, expressed as the following formula: In the formula, The feature map of the original image. To reconstruct the feature map of the image, The mean of the feature map. The variance of the feature map. The covariance of the feature maps between the original image and the reconstructed image. It is a constant. This is a distortion measure based on feature similarity.

3. The inter-frame image coding rate-distortion optimization method for machine vision as described in claim 1, characterized in that, In step S3, the distortion metric based on squared error is obtained using the following formula: In the formula, For located The pixel values ​​of the original image. For located The pixel values ​​of the reconstructed image. Image resolution size, This is a distortion measure based on squared error.

4. A frame-to-frame image coding rate-distortion optimization system for machine vision, characterized in that, include: The feature extraction module is used to extract feature maps of the original image of the current video frame and the corresponding reconstructed image at various scales; The first measurement module is used to obtain a distortion measurement based on feature similarity from the feature maps of the original image and the reconstructed image; The second measurement module is used to obtain a distortion measurement based on the squared error according to the pixel values ​​of the point coordinates of the original image and the reconstructed image; The hybrid metric module is used to perform rate-distortion optimization under hybrid distortion metric control by mixing distortion metrics based on feature similarity and squared error respectively. In the feature extraction module, the feature map is a multi-level feature information from low to high level, containing texture, shape, and semantic information. The feature map is extracted through a residual neural network, as shown in the following formula: In the formula, For the coded block at the current scale, It is a residual neural network. To obtain the feature map; In step S4, the hybrid distortion metric is obtained using the following formula: In the formula, The distortion metric weights are dynamically adjusted by a deep neural network. For distortion measurement based on feature similarity, For distortion measures based on squared error, For bitrate, QP is the Lagrange multiplier used to balance distortion and bit rate, k is the quantization parameter, and k is a constant. During the dynamic adjustment process Focus on performance during the video encoding process. It focuses on structural similarity distortion.

5. The inter-frame image coding rate-distortion optimization system for machine vision as described in claim 4, characterized in that, In the first measurement module, a distortion measure based on feature similarity is obtained through the structural similarity index, expressed as the following formula: In the formula, The feature map of the original image. To reconstruct the feature map of the image, The mean of the feature map. The variance of the feature map. The covariance of the feature maps between the original image and the reconstructed image. It is a constant. This is a distortion measure based on feature similarity.

6. The inter-frame image coding rate-distortion optimization system for machine vision as described in claim 4, characterized in that, In the second measurement module, the distortion measure based on squared error is obtained using the following formula: In the formula, For located The pixel values ​​of the original image. For located The pixel values ​​of the reconstructed image. Image resolution size, This is a distortion measure based on squared error.

Images