A just-identifiable distortion prediction model and method suitable for machine vision tasks

Abstract

The application belongs to the field of video image compression, and discloses a just recognizable distortion prediction model and method suitable for a machine vision task, which comprises a JRD image generation network G, a discrimination network D and a residual guide JRD regression network R; the JRD image generation network G takes an uncompressed original image as input and takes a false JRD image as output; then, the residual of the generated image and the original image is input into the residual guide JRD regression network R as compensation information of the original image, and finally, a predicted JRD score is output, wherein the discrimination network D is trained in an adversarial manner together with the generation network G to help G generate a generated result closer to a true JRD image. In the no-reference model proposed in the application, the original image does not need to be compressed and decoded for several times, so that the prediction efficiency is greatly improved. Meanwhile, in the regression network, the residual of the generated image is input as supplementary information, so that the final prediction result of the model is more accurate and reliable.

Description

Technical Field

[0001] This invention belongs to the field of video image compression, and in particular relates to a identifiable distortion prediction model and method suitable for machine vision tasks. Background Technology

[0002] Under the influence of various psychological and physiological factors, the human visual system (HVS) cannot perceive certain changes in images and videos; that is, the HVS exhibits just-perceptible distortion (JND). Exploring the threshold of JND is instructive for improving image and video coding methods and bitrate control. Image and video coding based on a reasonable JND model can significantly reduce the bitrate required for storage and transmission without affecting the HVS's perceptual level.

[0003] However, image and video encoding methods and bitrate control based on JND thresholding are only optimized for HVS characteristics. With the improvement of hardware performance and the development of deep learning algorithms, the performance of machine vision tasks is constantly improving, and video and image data oriented towards machine vision algorithms rather than HVS is growing exponentially. Furthermore, it is well known that the accuracy of machine vision tasks depends not only on the sophistication of hardware and algorithms but also on the quality of compressed images or videos. Therefore, studying the mapping relationship between the accuracy of machine vision tasks and the quality of compressed data is of great significance for guiding image and video compression. Thus, to reduce redundancy in the transmission and storage of video and image data, distortion threshold prediction methods suitable for machine vision tasks urgently need to be studied.

[0004] In recent years, research on the relationship between machine vision performance and image or video frame quality has begun to advance. Some researchers have optimized the encoding for specific machine vision tasks, including methods for generating realistic adversarial images that can simultaneously fool machine vision algorithms and humans, and semantically guided navigators (JNDs) for image classification tasks through unsupervised learning. However, these works rely on innovations specific to machine vision tasks and are not universally applicable to multiple types of machine vision tasks.

[0005] Inspired by the concept of Just-Identifiable Distortion (JND), some researchers have further explored whether machine vision possesses a similar distortion threshold. Some researchers conducted comprehensive experiments on various machine vision tasks, demonstrating that machine vision also has a threshold similar to JND in HVS, and proposed a new concept called Just-Identifiable Distortion (JRD) to describe the maximum distortion in image and video encoding at an acceptable level of machine vision performance. Assuming I0 is the original image, its compressed variants are I1, I2, I3, ..., I... n Where a larger n indicates a worse quality image, for a machine vision model M, the JRD of I0 is defined as:

[0006]

[0007] Where j is the quantization parameter (QP) for encoding selection, ε is any positive integer, and M specifically refers to the object detection task. The concept of Machine Satisfaction Ratio (SMR) has also been proposed to evaluate image quality from the perspective of machine satisfaction, and a deep learning-based model has been proposed to predict the SMR of compressed images or video frames. These studies were all conducted using standard video coding methods such as HEVC. The concepts of JRD and SMR are defined to describe the acceptable level of image and video coding distortion for machine vision tasks. However, these studies also failed to propose an accurate and efficient framework for no-reference prediction of JRD.

[0008] In summary, existing work has preliminarily explored the relationship between machine vision task accuracy and the quality of compressed images and videos described by concepts similar to JND (such as JRD and SMR). However, related research is still insufficient, mainly in the following aspects: 1) Most existing JRD or SMR models are full-reference or semi-reference prediction models, which means that these methods require at least one encoding compression of the original image as a reference, resulting in low efficiency. 2) Image and video distortion threshold prediction usually depends on specific machine vision task algorithms, and the versatility of the JRD prediction framework needs to be further improved to adapt to different machine vision tasks. 3) Existing JRD prediction models have low accuracy performance and cannot meet the needs of practical applications. Summary of the Invention

[0009] The purpose of this invention is to provide a recognizable distortion prediction method suitable for machine vision tasks, so as to solve the above-mentioned technical problems.

[0010] To address the aforementioned technical problems, the present invention provides a specific technical solution for a precisely identifiable distortion prediction model and method suitable for machine vision tasks, as follows:

[0011] A just-identifiable distortion prediction model suitable for machine vision tasks includes a JRD image generation network G, a discriminant network D, and a residual-guided JRD regression network R. The JRD image generation network G takes an uncompressed original image as input and a fake JRD image as output. Then, the residual between the generated image and the original image is used as compensation information for the original image and input into the residual-guided JRD regression network R. Finally, the predicted JRD score is output, which is the optimal quantization parameter for image compression. The discriminant network D is trained together with the generation network G in an adversarial manner to help G generate a result that is closer to the true JRD image.

[0012] Furthermore, the model uses the residual image between the original image and the corresponding pseudo JRD image as supplementary information, and compares it with the original image. Figure 1The input residual guides the JRD regression network, which ultimately produces the predicted JRD score, which is the optimal quantization parameter for image compression.

[0013] Furthermore, the generator part of the JRD image generation network G is implemented using the basic structure of U-Net. In the U-Net network, the input passes through a series of layers and is downsampled step by step until the bottleneck layer. After that, transposed convolution is performed to achieve upsampling. A large amount of low-level information between the input and output is transmitted through the skip connection structure.

[0014] Furthermore, the discriminant network D uses a multi-scale discriminator, employing two discriminators with identical structures but operating at different image scales. These two discriminators operating at different scales are referred to as D1 and D2, respectively operating at the original scale of the real image and the generated image, and at a scale that is downsampled by 2 times.

[0015] Furthermore, the main function of the residual-guided JRD regression network R is to evaluate the degree of distortion of the fake JRD image relative to the original image;

[0016] We choose to input the original image and the residual into the feature extractor separately, and then perform linear regression calculation on their fused features. The feature extractor used is VGG16, and the network parameters of VGG16 are the network parameters pre-trained on the ImageNet dataset. During training, we only need to learn and optimize the network parameters of the linear regression part.

[0017] Furthermore, the model includes a loss function L. R During training, if the predicted JRD score φ(i) is greater than the true value Φ(i), MSE loss is used; otherwise, L1 loss is used.

[0018]

[0019] N is the total number of images, φ(i) is the JRD prediction score of the i-th image, and Φ(i) is the true JRD of the image, where the predicted value φ(i) is rounded down.

[0020] This invention also discloses a training method for a just-identifiable distortion prediction model suitable for machine vision tasks, comprising the following steps:

[0021] Step 1: Create a dataset containing uncompressed raw image or video frames and their corresponding JRD images;

[0022] Step 2: Generate the adversarial network.

[0023] Furthermore, step 1 includes the following steps:

[0024] For machine vision tasks involving object detection using YOLOv7, a dataset of 130,000 uncompressed raw images or video frames and their corresponding JRD images was created, along with annotations of their JRD values. The raw images in the dataset are from COCO2017. The YOLOv7 algorithm is used to infer the detected object bounding boxes from the raw images or video frames. The cropped image I0 and its compressed variants I1, I2, I3, ..., I... are presented. n Inference was performed using the YOLOv7 algorithm with the same parameters, and the JRD image I corresponding to I0 was finally selected. j .

[0025] Furthermore, step 2 includes the following steps:

[0026] The adversarial learning framework applies the basic network structure of cGAN.

[0027]

[0028] Where x is the original image, y is the true JRD image, z is noise, and G(x,z) is the output of G with x and z as inputs, which is the generated fake JRD image. In the above formula, G attempts to minimize L cGAN To counteract D's attempt to maximize this objective, it is expressed as:

[0029]

[0030] Introducing a multi-scale feature matching loss function L fm and perceptual loss function L vgg Optimize the loss function of cGAN.

[0031] Among them, the multi-scale feature matching loss function L fm Represented as:

[0032]

[0033] Discriminators D representing different scales k The i-th layer feature extractor, This represents the features extracted by the k-th discriminator at the i-th layer, using the original image and the true JRD image as input. The original image and the fake JRD image are then used as input;

[0034] Introducing the VGG loss function as a perceptual loss, L vgg The calculation formula is shown below:

[0035]

[0036] i represents the i-th image out of n images, F represents the feature map of the generated fake JRD image, R represents the feature map of the real JRD image, and C, H, and W represent the number of channels, height, and width of the feature map, respectively.

[0037] The ultimate goal of adversarial learning is as follows:

[0038]

[0039] The exactly-identifiable distortion prediction model and method of the present invention, applicable to machine vision tasks, has the following advantages:

[0040] The invention decomposes the difficult task of predicting JRD scores into two relatively simple tasks: a generation task and a regression task. In the referenceless model proposed in this invention, the original image does not require multiple compression and decoding operations, significantly improving prediction efficiency. Simultaneously, in the regression network, the residuals of the generated image are input as supplementary information, making the model's final prediction results more accurate and reliable. Attached Figure Description

[0041] Figure 1 This is a diagram showing the main structure of the prediction framework of this invention.

[0042] Figure 2 This is a flowchart illustrating the creation process of the JRD dataset for the object detection task in this invention.

[0043] Figure 3 This is a schematic diagram illustrating the principle of adversarial training of the JRD image generation network G and the discriminant network D in this invention.

[0044] Figure 4 This is a structural diagram of the multi-scale discriminator of the present invention. Detailed Implementation

[0045] To better understand the purpose, structure, and function of this invention, the following detailed description, in conjunction with the accompanying drawings, provides an appropriately identifiable distortion prediction model and method suitable for machine vision tasks.

[0046] like Figure 1 , Figure 3As shown, a just-identifiable distortion prediction model for machine vision tasks according to the present invention includes a JRD image generation network G, a discriminant network D, and a residual-guided JRD regression network R. The JRD image generation network G takes an uncompressed original image as input and a fake JRD image as output. Then, the residual between the generated image and the original image is used as compensation information for the original image and input into the residual-guided JRD regression network R, finally outputting the predicted JRD score, which is the optimal quantization parameter for image compression. The discriminant network D is trained adversarially with the generation network G to help G generate results that more closely resemble true JRD images.

[0047] The model of this invention uses the residual image between the original image and the corresponding pseudo JRD image as supplementary information, and compares it with the original image. Figure 1 The input residual guides the JRD regression network, ultimately generating the predicted JRD score, which is the optimal quantization parameter for image compression. Using the method proposed in this invention to guide image and video compression can minimize data redundancy during transmission and storage while maintaining high accuracy in machine vision tasks.

[0048] JRD Image Generation Network G:

[0049] In this invention, the generator part is implemented using the basic architecture of U-Net, and its network structure is as follows: Figure 1 As shown, U-Net is a fully convolutional neural network architecture. Compared to common encoder-decoder networks that first downsample to a low dimension and then upsample to the original resolution, U-Net differs in its unique skip-connection structure. Skip connections allow the feature layers in the encoding process to be directly concatenated with the feature layers of the same dimension after decoding in a channel-stacked manner, in order to preserve pixel-level detail information at different scales.

[0050] A characteristic of image-to-image translation problems is that they map a high-resolution input mesh to a high-resolution output mesh. The input and output are not identical in content, but are both rendered at different qualities based on the same underlying content. Therefore, the structure in the input is roughly aligned with the structure in the output. In the U-Net network, the input passes through a series of layers, progressively downsampled until the bottleneck layer, after which transposed convolutions are used for upsampling. This network design ensures that all information flows through all layers. For the image-to-image translation problem that this invention aims to solve, there is a significant amount of low-level information shared between the input and output; therefore, directly transmitting this information through skip connections is effective.

[0051] Therefore, U-Net's structural design allows it to combine information from both deep and shallow features. Shallow features provide more refined detail for image generation, improving the finer details of the generated image. Deep features, on the other hand, provide contextual semantic information about the image content within the overall image. Since the input and output images in this invention are corresponding, this contextual information helps improve the quality of the generated images under the same underlying structure.

[0052] Discriminant network D:

[0053] Images with different semantic meanings correspond to different JRD values. To generate content that more closely resembles the quality of images with true JRD, the generative network relies heavily on a discriminator. This presents a significant challenge to the design of the GAN discriminator. To better distinguish between generated and real images, this invention employs a multi-scale discriminator, structured as follows: Figure 4 As shown, this invention uses two discriminators with identical structures, but they operate at different image scales. These two discriminators operating at different scales are designated D1 and D2. D1 and D2 operate at the original scale of the real image and the generated image, and at a scale that is 2 times downsampled, respectively. The D2 discriminator, operating at a coarse-grained scale, has a larger receptive field, enabling it to discriminate images from a more global perspective and guiding the generator to produce more globally consistent images. Meanwhile, the D1 discriminator, operating at a finer-grained scale, can make judgments at a more refined scale, helping the generator to produce finer details. Because this multi-scale discriminator structure does not use excessively deep network structures or excessively large convolutional kernels, it consumes fewer computational resources and reduces the risk of overfitting.

[0054] Residual-guided JRD regression network R:

[0055] The main function of the residual-guided JRD regression network is to evaluate the degree of distortion of the spurious JRD image relative to the original image. Directly predicting the JRD score of an image with only the original image as input is a very difficult task. Therefore, the design strategy of this invention is to introduce this network module, using the residual image between the generated JRD image and the original image as supplementary information. Its structure is as follows: Figure 1 As shown.

[0056] Since the JRD score of an image is highly correlated with its semantic content, this invention selects to input the original image and the residual separately into a feature extractor, and then perform linear regression calculation on their fused features. The feature extractor chosen is VGG16, which has a large number of parameters (138M) and strong fitting ability. Its structure, which increases network depth through convolution, can effectively extract semantic information from the image. The network parameters of VGG16 are those pre-trained on the ImageNet dataset. Therefore, during training, only the network parameters for optimizing the linear regression part need to be learned, greatly reducing the training difficulty.

[0057] To prioritize the accuracy of the image compressed with the prediction score φ(i) for machine vision tasks, this invention designs a loss function L for the network. R As shown in the following equation. During training, if the predicted JRD score φ(i) is greater than the true value Φ(i), MSE loss is used; otherwise, L1 loss is used. The proposed loss function L... R The predicted JRD score, which is greater than the actual JRD score Φ(i), is penalized more, which ensures that the compressed video frames and images still have high accuracy for machine vision tasks under the guidance of the method of this invention.

[0058]

[0059] N is the total number of images, φ(i) is the JRD prediction score of the i-th image, and Φ(i) is the true JRD of the image. The predicted value φ(i) is rounded down.

[0060] The present invention provides a training method for just-identifiable distortion prediction suitable for machine vision tasks, comprising the following steps:

[0061] Step 1: Create a dataset containing uncompressed raw image or video frames and their corresponding JRD images;

[0062] Because adversarial learning is often used to learn complex correspondences, theoretically, as long as there is a sufficient amount of data, a GAN network can learn the mapping relationship between machine vision task accuracy and image distortion. Therefore, this invention believes that a GAN network, driven by a large amount of data, can learn the JRD scores corresponding to different images under a specific machine vision task. Thus, the image generation effect of the generative network is highly dependent on the dataset. In the experiments of this invention, the machine vision model M is defined as the YOLOv7 object detection algorithm. This invention creates a dataset containing 130,000 uncompressed original images or video frames and their corresponding JRD images for the YOLOv7 object detection machine vision task, and labels the corresponding JRD values. The original images in the dataset are from COCO2017. This invention uses the YOLOv7 algorithm to infer from the original images or video frames and extract the detected object boxes. The extracted image I0 and its variants I1, I2, I3, ..., I2 after compression with different quantization parameters are used. n Inference was performed using the YOLOv7 algorithm with the same parameters, and the JRD image I corresponding to I0 was finally selected. j The process of creating the JRD dataset is as follows: Figure 2 As shown.

[0063] Step 2: Generate the adversarial network;

[0064] The adversarial learning framework of this invention mainly applies the basic network structure of cGAN (conditional-GAN), such as... Figure 3 As shown, cGAN is an improved version of Generative Adversarial Networks (GANs). Unlike traditional GANs that generate images directly from noise, cGANs provide additional conditional information to the generator (G) and discriminator (D) of the original GAN, achieving conditional and stable generation. The core operation of cGAN is adding conditional information to G and D, which can be image data, labels, or other auxiliary information. In other words, cGAN can be trained using the original image and its corresponding labeled image, enabling the generative network to transform the original image into an image with specified content and specific changes.

[0065]

[0066] like Figure 3 As shown, in this invention, x is the original image, y is the true JRD image, z is noise, and G(x,z) is the output of G with x and z as inputs, which is the generated fake JRD image. In the above formula, G attempts to minimize L cGAN To counteract D's attempt to maximize this objective, it is expressed as:

[0067]

[0068] To enhance the stability of network training, this invention proposes to introduce a multi-scale feature matching loss function L. fm and perceptual loss function L vgg The loss function of cGAN is optimized. Specifically, the network extracts features from the multi-level discriminator, learns the correspondence between feature parameters at different levels, and achieves multi-level feature matching between the original image and the fake JRD image.

[0069] Among them, the multi-scale feature matching loss function L fm Represented as:

[0070]

[0071] Discriminators D representing different scales k The i-th layer feature extractor, This represents the features extracted by the k-th discriminator at the i-th layer, using the original image and the true JRD image as input. The original image and the fake JRD image are then used as input.

[0072] Furthermore, this invention introduces the VGG loss function as a perceptual loss to ensure that the quality of the generated image is closer to that of a true JRD image across multiple dimensions. vgg The calculation formula is shown below:

[0073]

[0074] i represents the i-th image out of n images, F represents the feature map of the generated fake JRD image, and R represents the feature map of the real JRD image. C, H, and W represent the number of channels, height, and width of the feature map, respectively.

[0075] Therefore, the ultimate goal of adversarial learning is as shown in equation (6):

[0076]

[0077] The present invention proposes a no-reference just-identifiable distortion prediction method for machine vision tasks, which is applicable to guiding most image and video data compression tasks for machine vision algorithms (such as image classification, object detection, etc.).

[0078] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A training method for a just-identifiable distortion prediction model suitable for machine vision tasks, characterized in that, Includes the following steps: Step 1: Create a dataset containing uncompressed raw image or video frames and their corresponding JRD images; Step 2: Generate the adversarial network; The adversarial learning framework applies the basic network structure of cGAN. Where x is the original image, y is the true JRD image, z is noise, and G(x,z) is the output of G with x and z as inputs, which is the generated fake JRD image. In the above formula, G attempts to minimize L cGAN To counteract D's attempt to maximize the aforementioned objective function, it can be expressed as: Introducing a multi-scale feature matching loss function L fm and perceptual loss function L vgg Optimize the loss function of cGAN. Among them, the multi-scale feature matching loss function L fm Represented as: Discriminators D representing different scales k The i-th layer feature extractor, This represents the features extracted by the k-th discriminator at the i-th layer, using the original image and the true JRD image as input. The original image and the fake JRD image are then used as input; Introducing the VGG loss function as a perceptual loss, L vgg The calculation formula is shown below: i represents the i-th image out of n images, F represents the feature map of the generated fake JRD image, R represents the feature map of the real JRD image, and C, H, and W represent the number of channels, height, and width of the feature map, respectively. The ultimate goal of adversarial learning is as follows:

2. The training method according to claim 1, characterized in that, Step 1 includes the following steps: For machine vision tasks involving object detection using YOLOv7, a dataset of 130,000 uncompressed raw images or video frames and their corresponding JRD images was created, along with annotations of their JRD values. The raw images in the dataset are from COCO2017. The YOLOv7 algorithm is used to infer the detected object bounding boxes from the raw images or video frames. The cropped image I0 and its compressed variants I1, I2, I3, ..., I... are presented. n Inference was performed using the YOLOv7 algorithm with the same parameters, and the JRD image I corresponding to I0 was finally selected. j .

3. A just-recognizable distortion prediction model suitable for machine vision tasks for performing the training method as described in claim 1 or 2, characterized in that, The system includes a JRD image generation network G, a discriminant network D, and a residual-guided JRD regression network R. The JRD image generation network G takes the uncompressed original image as input and a fake JRD image as output. Then, the residual between the generated image and the original image is used as compensation information for the original image and input into the residual-guided JRD regression network R. Finally, the predicted JRD score is output, which is the optimal quantization parameter for image compression. The discriminant network D is trained together with the generation network G in an adversarial manner to help G generate a result that is closer to the real JRD image.

4. The exactly-recognizable distortion prediction model for machine vision tasks according to claim 3, characterized in that, The model uses the residual image between the original image and the corresponding pseudo JRD image as supplementary information, and inputs it along with the original image into the residual to guide the JRD regression network, ultimately generating the predicted JRD score, which is the optimal quantization parameter for image compression.

5. The exactly-recognizable distortion prediction model for machine vision tasks according to claim 3, characterized in that, The generator part of the JRD image generation network G is implemented using the basic structure of U-Net. In the U-Net network, the input passes through a series of layers and is downsampled step by step until the bottleneck layer. After that, transposed convolution is performed to achieve upsampling. A large amount of low-level information between the input and output is transmitted through the skip connection structure.

6. The exactly-recognizable distortion prediction model for machine vision tasks according to claim 3, characterized in that, The discriminant network D uses a multi-scale discriminator, employing two discriminators with identical structures but operating at different image scales. These two discriminators operating at different scales are referred to as D1 and D2, respectively operating at the original scale of the real image and the generated image, and at a scale that is downsampled by 2 times.

7. The exactly identifiable distortion prediction model for machine vision tasks according to claim 3, characterized in that, The main function of the residual-guided JRD regression network R is to evaluate the degree of distortion of the fake JRD image relative to the original image; We choose to input the original image and the residual into the feature extractor separately, and then perform linear regression calculation on their fused features. The feature extractor used is VGG16, and the network parameters of VGG16 are the network parameters pre-trained on the ImageNet dataset. During training, we only need to learn and optimize the network parameters of the linear regression part.

8. The exactly identifiable distortion prediction model for machine vision tasks according to claim 3, characterized in that, The model includes the loss function L R During training, if the predicted JRD score φ(i) is greater than the true value Φ(i), MSE loss is used; otherwise, L1 loss is used. N is the total number of images, φ(i) is the JRD prediction score of the i-th image, and Φ(i) is the true JRD of the image, where the predicted value φ(i) is rounded down.

Images