Multi-level full RGB just noticeable perceptual coding distortion prediction method for video compression

Abstract

The invention discloses a multi-level full RGB just perceptible perception coding distortion prediction method for video compression, and the method comprises the steps: firstly obtaining a public video set, processing videos in the video set into RGB videos, and obtaining a perception coding distortion label; then extracting visual perception features based on the RGB video; and finally, constructing an RGB-JNCD-NET prediction model, and outputting a coding distortion prediction result by the visual perception features through the RGB-JNCD-NET prediction model. According to the method, high-efficiency and accurate distortion coding prediction can be realized without influencing the perception quality.

Description

Technical Field

[0001] The invention belongs to the field of video perceptual coding optimization, and in particular relates to a multi-level full RGB just perceptible perceptual coding distortion prediction method for video compression. Background Art

[0002] The explosive growth of video data has brought great challenges to data compression. The continuous optimization of video coding technology is an effective way to solve this challenge. The visual perception mechanism reveals the perception ability of the human visual system (HVS) to distortion in compressed videos. The effective measurement of perceptible distortion in compressed videos is one of the prerequisites for guiding the optimization of video perceptual coding.

[0003] The Just Noticeable Distortion (JND) modeling method defines the minimum distortion that can be perceived by HVS and applies it to video perceptual coding optimization, effectively improving the efficiency of video compression. For a series of compressed videos of different qualities, the human visual system can only perceive a limited number of quality levels. At the same quality level, different compressed videos have different distortions but no significant difference in subjective quality. Therefore, the maximum coding distortion at the same quality level can be defined as the Just Noticeable Coding Distortion (JNCD) at that quality level. The effective prediction of JNCD can be used to guide coding optimization under the same quality level constraints to achieve maximum bitrate savings.

[0004] At present, the research on JNCD prediction model is still in its infancy, and compressed video datasets with different coding modes have been constructed. Since the prediction of JNCD involves visual perception mechanism and video coding mechanism, the relevant research progress is relatively slow, and the proposed prediction model is generally only for the first quality level single color channel video JNCD prediction.

[0005] Just perceptible distortion prediction is generally divided into pixel domain prediction model and transform domain prediction model. Traditional JND model mainly relies on manual features and is modeled by nonlinear fusion, which generally has the problem of overestimation or underestimation of threshold. At the same time, images and videos in daily life are usually compressed, and the noise in them mainly comes from coding distortion. Due to the complexity of the video coding process, the JND modeling process generally does not consider the video coding mechanism, and the JND prediction model is generally not accurate in predicting perceptual coding distortion.

[0006] Aiming at the problem of predicting just perceptible coding distortion, a neural network-based JNCD prediction model is explored based on published datasets. Studies have shown that HVS can only perceive a limited number of quality levels for a series of compressed videos of different qualities. Therefore, the prediction of multi-level JNCD has become a research hotspot in this field. At present, the multi-level JNCD prediction problem is usually transformed into the prediction problem of user satisfaction (Satisfied-User-Ratio, SUR), and the JNCD thresholds of images and videos are derived by constructing a SUR prediction model mapping. However, the existing JNCD or SUR prediction models are generally targeted at color images and cannot be directly used to predict the JNCD thresholds of each color component, resulting in technical defects in the existing models in guiding multi-channel perceptual coding of videos. Summary of the invention

[0007] To address this problem, the present invention proposes a multi-level full RGB just perceptual coding distortion prediction method for video compression, which provides perceptual distortion estimation for video coding and guides video coding to improve perceptual video compression efficiency.

[0008] Multi-channel JNCD threshold prediction involves video encoding mechanism and multi-channel visual perception mechanism. How to analyze and extract multi-channel visual perception features is the primary technical problem faced by the present invention. Secondly, how to design a JNCD prediction model to meet multi-level and multi-channel threshold prediction is another technical problem that the present invention needs to solve.

[0009] Multi-channel visual perception feature analysis: Considering that the brightness component contains most of the video information, three visual features, namely brightness adaptive masking, entropy masking and saliency, are extracted for the brightness component; contrast masking is the decisive visual feature that determines the JNCD threshold of the video, so the contrast masking effect is extracted for the three RGB channels respectively.

[0010] RGB multi-level JNCD prediction: Based on the analysis and extraction of multi-channel visual perception features, a JNCD prediction model with autoencoder as the backbone network is designed. This model takes three-channel video and six visual perception features as network input, uses autoencoder as feature extractor, and uses the weighted sum of image gradient and L1 norm as loss function to train the RGB-JNCD-NET prediction network.

[0011] A multi-level full RGB just perceptual coding distortion prediction method for video compression, comprising the following steps:

[0012] S1. Obtain a public video set, process the videos in the video set into RGB videos, and obtain the RGB video perceptual coding distortion label

[0013] S2. Extract visual perception features based on RGB video.

[0014] S3. Build an RGB-JNCD-NET prediction model. Visual perception features are passed through the RGB-JNCD-NET prediction model to output coding distortion prediction results.

[0015] Furthermore, the extracted visual perception features in step S2 include a brightness adaptive masking feature map, a visual attention map, a space-time entropy masking map, and an RGB three-channel contrast masking feature map. The specific process is as follows:

[0016] S2.1, brightness adaptive masking feature map M lum :

[0017]

[0018] Where B(x,y) represents the average background brightness of the n×n area near the video frame (x,y).

[0019] S2.2 Visual Attention Map M fd :The video frame passes through the PoolNet network model to obtain the visual attention map M fd S2.3, Space-time domain entropy masking map M em :

[0020] M em (x,y)=θ·|O(x,y)-P(x,y)|

[0021] Among them, O(x,y) represents the pixel value of the original visual signal at (x,y), P(x,y) is the predicted visual signal of the autoregressive prediction model at the corresponding position, and θ is a constant.

[0022] S2.4, RGB three-channel contrast mask feature map i∈(r,g,b), respectively represents the contrast masking feature map of the red, green and blue channels. The three-channel contrast masking feature map is obtained by fitting the brightness contrast:

[0023]

[0024] α i , β i is the fitting parameter of the current channel, Cont i (x, y) is the contrast of the m×m area near the current position of the channel.

[0025] Beneficial effects of the present invention: The RGB-JNCD graph obtained by the present invention is applied to video coding to eliminate the perceptual distortion during the coding process, which can reduce the time cost by an average of 8.1% in terms of operating efficiency without affecting the perceived quality, thereby achieving efficient and accurate distortion coding prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a multi-level full RGB JNCD prediction model;

[0027] Figure 2 This is the block diagram of the ED autoencoder;

[0028] Figure 3 Graph showing performance comparison of the model of the present invention. DETAILED DESCRIPTION

[0029] A multi-level full RGB just-perceptible perceptual coding distortion prediction method for video compression, with specific examples:

[0030] The present invention first defines multi-level RGB-JNCD: different quantization parameters (QP) are selected under the RGB channels to encode the compressed video, and the quality is evaluated, and then the QP values ​​corresponding to different quality levels under each channel are obtained. After the video is encoded with the boundary QP corresponding to the same quality level, the difference between each pixel is the RGB-JNCD threshold of the quality level. Under such a threshold, any changes in the three RGB channels will not be perceived by HVS. The present invention obtains multi-level RGB-JNCD labels through mathematical and statistical methods, thereby guiding the prediction of the RGB-JNCD threshold of the video frame in the construction of the RGB-JNCD-NET model training. The overall framework is as follows Figure 1 shown.

[0031] (1) Perceptual coding distortion label determination.

[0032] A video sequence consists of a series of video frames. Under the encoding of the same quantization parameters, there are certain differences in the subjective and objective qualities of different video frames. The compression quality of all video frames jointly determines the subjective quality of the compressed video. The present invention uses the public video set MCL-JCV. The data set is encoded in the x.264 format and contains a total of 30 source videos with a resolution of 1920×1080. The present invention processes the video set into RGB videos, with a total of 90 videos. Each video has 51 compressed versions in each of the three RGB channels, corresponding to video encoding quantization parameters QP of 1-51. Generally speaking, there are three quality levels for videos, and each level has a just-perceptible QP. The coding distortion corresponding to the just-perceptible QP of each chrominance channel at each level is used as the perceptual coding distortion label of the current channel at the current video quality level. Guide the prediction of video frames in building RGB-JNCD-NET model training Threshold.

[0033] (2) Extract visual perception features based on RGB video.

[0034] Extract the visual perception feature map M of the sampled frame lum 、Mfd , M em .

[0035] 1. Brightness adaptive masking feature map M lum : Brightness adaptive masking effect refers to the masking effect of HVS on background brightness. Usually, visual sensitivity is lower in darker or brighter areas. For 8-bit images, the specific expression is as follows:

[0036]

[0037] Where B(x,y) represents the average background brightness of the 5×5 area near (x,y).

[0038] 2. Visual Attention Map M fd

[0039] Visual perception redundancy is strongly correlated with visual attention distribution. Specifically, compared with the edge area of ​​the image, the focus area has higher visual sensitivity and the corresponding JNCD threshold is smaller. The present invention uses the PoolNet network to obtain the visual attention map M fd , the network has a good segmentation effect on the target contour and can accurately capture the exact location of the salient object:

[0040] M fd =PoolNet(O)

[0041] Where O represents the original video frame and PoolNet(·) represents the PoolNet algorithm.

[0042] 3. Space-time entropy masking graph M em :HVS predicts structured information as accurately as possible, while ignoring detailed visual information in the visual perception process. In other words, the noise in the area containing more uncertain information is not easily perceived, and it will be ignored together with the uncertain information. Therefore, the JNCD threshold of the entropy masking effect can be estimated based on the difference between the input signal and the predicted signal. According to an autoregressive prediction model proposed in the current research, which spontaneously simulates the prediction process of HVS, the JNCD threshold of the entropy masking effect can be briefly expressed as:

[0043] M em (x,y)=θ·|O(x,y)-P(x,y)|

[0044] Among them, O(x,y) represents the pixel value of the original visual signal at (x,y), P(x,y) is the predicted visual signal of the autoregressive prediction model at the corresponding position, and θ is a constant, usually set to 0.1.

[0045] 4. RGB three-channel contrast masking feature map Where i∈(r,g,b) represents the contrast masking feature map of the red, green and blue channels respectively. The present invention intends to obtain the contrast masking feature map of the three channels by fitting the brightness contrast:

[0046]

[0047] α i , β i is the fitting parameter of the current channel, Cont i (x, y) is the contrast of the 5×5 area near the current position of the channel.

[0048] (3) RGB-JNCD-NET network model construction: Based on the analysis of visual perception effects, the present invention designs a neural network model to learn the quantitative mapping relationship between visual perception features and RGB-JNCD, constructs an RGB-JNCD-NET prediction model, inputs visual perception features, and outputs coding distortion prediction results, such as Figure 1 As shown in the figure, the entire prediction model consists of the following three parts:

[0049] 1. Input layer: original video RGB channel content O i ,i∈(r,g,b). i The six feature maps M obtained according to (2) lum 、M fd 、M n , The element-by-element bipolar addition is used as the input layer of RFB-JNCD-NTE, that is, the network input is a 9×H×W vector. That is:

[0050]

[0051] where f represents an element-wise bipolar add operation.

[0052] 2.ED autoencoder: The autoencoder contains two subnets E and D, representing the encoding and decoding parts respectively. Figure 2 As shown in the figure, the encoding module includes four convolutional layers, and the decoding module includes two convolutional layers and two upsampling layers. The numbers around each layer, i.e. (x, y, z), respectively indicate that the layer has x feature channels, the convolution size y×y, and the step size z. z controls the upsampling multiple, and the step size of the transposed convolution is uniformly set to 1.

[0053] 3. Loss function: According to research, the size of JNCD is proportional to the complexity of the visual content. However, when the image has a simple texture, small noise is obvious. The method of using the gradient of the image to represent the texture complexity of the image is widely used in image and video processing.

[0054]

[0055] in:

[0056]

[0057] i is the color channel, h and w are the width and height of the video frame, g1 and g2 represent the vertical and horizontal gradients at the pixel (h, w) in channel c, respectively. G is the gradient of the image, which is used to represent the texture complexity of the image.

[0058]

[0059] P is the sum of the absolute values ​​of the pixels p in the three channels of RGB. t is a constant to avoid the denominator being 0.

[0060] During the optimization process, L1 reaches its minimum value when P is close to G, which ensures that the size of the generated RGB-JNCD is proportional to the gradient of the image. In other words, the above formula realizes the quantitative modeling of RGB-JNCD based on the visual content of the three RGB channels.

[0061]

[0062] Where i∈(r,g,b), α+β+γ=1. The prediction value JNCD generated by the decoder is i With label value Perform an absolute value subtraction operation, and then add the weighted differences of the three channels to obtain another loss function L2 to guide JNCD i generate.

[0063] The total loss function of the RGB-JNCD-NET network model is the weighted addition of the above sub-loss functions:

[0064] L=λL1+ηl2

[0065] Example:

[0066] The performance of the proposed model is compared with the most advanced model Wu2017. The model performance evaluation is divided into subjective evaluation and objective evaluation. First, the starting frames of the videos encoded after eliminating JNCD are evaluated by subjective perception effect comparison. Then, the IQA method is used in Table 1 for objective evaluation.

[0067] Table 1. Performance comparison of models under different IQA

[0068]

[0069] like Figure 3As shown, the leftmost column is the starting frame of the original video, the middle column is the starting frame encoded after eliminating the jncd generated by Wu2017, and the rightmost column is the starting frame encoded after eliminating the jncd generated by the model of the present invention. In the Wu2017 model, only the pattern complexity and masking features are combined, resulting in obvious distortion in complex areas. The model of the present invention automatically mines the mapping relationship between the RGB three-channel JNCD and the masking effect based on a neural network, and takes into account the above-mentioned factors affecting the quality of visual perception, and the model performance is significantly improved.

[0070] A total of four IQA methods were selected to compare the prediction performance of the model of the present invention with other models, namely MAD, NLPD, VIF and VSI. The CSIQ dataset was used for comparative experiments, and the average results of all images are shown in Table 1. The closer the scores of MAD and NLPD methods are to zero, the better, while the closer the scores of VIF and VSI methods are to 1, the better the performance. It can be seen from the table that the model proposed by the present invention has better effect, which is due to the loss function carefully designed by the present invention for the core problem of JNCD threshold prediction.

Claims

1. A multi-level full RGB just-perceptible perceptual coding distortion prediction method for video compression, It is characterized in that The following steps are involved: S1. Obtain a public video set, process the videos in the video set into RGB videos, and obtain the RGB video perceptual coding distortion label S2, extract visual perception features based on RGB video; S3. Build an RGB-JNCD-NET prediction model. Visual perception features are passed through the RGB-JNCD-NET prediction model to output coding distortion prediction results.

2. The multi-level full RGB just perceptible perceptual coding distortion prediction method for video compression according to claim 1, It is characterized in that The perceptual coding distortion label is obtained Specifically, each video has 51 compressed versions in the RGB channels, corresponding to the video coding quantization parameters QP of 1-51; there are three quality levels for the video, each level has a just perceptible QP, and the coding distortion corresponding to the just perceptible QP of each chroma channel at each level is used as the perceptual coding distortion label of the current channel at the current video quality level 3. The multi-level full RGB just perceptible perceptual coding distortion prediction method for video compression according to claim 1, It is characterized in that The extracted visual perception features include brightness adaptive masking feature map, visual attention map, space-time domain entropy masking map and RGB three-channel contrast masking feature map. The specific process is as follows: S2.1, brightness adaptive masking feature map M lum : Where B(x,y) represents the average background brightness of the n×n area near the video frame (x,y); S2.2 Visual Attention Map M fd :The video frame passes through the PoolNet network model to obtain the visual attention map M fd ; S2.

3. Space-time entropy masking map M em : M em (x,y)=θ·|O(x,y)-P(x,y)| Among them, O(x,y) represents the pixel value of the original visual signal at (x,y), P(x,y) is the predicted visual signal of the autoregressive prediction model at the corresponding position, and θ is a constant; S2.4, RGB three-channel contrast mask feature map i∈(r,g,b), respectively represents the contrast masking feature map of the red, green and blue channels. The three-channel contrast masking feature map is obtained by fitting the brightness contrast: α i , β i is the fitting parameter of the current channel, Cont i (x, y) is the contrast of the m×m area near the current position of the channel.

4. The multi-level full RGB just perceptible perceptual coding distortion prediction method for video compression according to claim 3, It is characterized in that The RGB-JNCD-NET prediction model consists of three parts: input layer, ED autoencoder and loss function, as follows: S3.1, input layer: original video RGB channel content O i ,i∈(r,g,b); O i According to the six feature maps M lum 、M fd 、M n , The element-by-element bipolar addition is used as the input layer of RFB-JNCD-NTE, that is, the network input is a 9×H×W vector: Where f represents an element-by-element bipolar addition operation; S3.2, ED autoencoder: ED autoencoder contains encoding and decoding parts based on convolutional networks; S3.3, loss function: i is the color channel, h and w are the width and height of the video frame, g 1 , g 2 They represent the vertical and horizontal gradients at the pixel (h, w) in channel c, respectively. G is the gradient of the image, which indicates the texture complexity of the image. P is the sum of the absolute values ​​of pixel p in the three RGB channels; t is a constant; Where i∈(r,g,b), α+β+γ=1; the prediction value JNCD generated by the decoder i With label value Perform absolute value subtraction, and then weighted add the differences of the three channels to obtain another loss function L 2 , to guide JNCD i generate; The total loss function of the RGB-JNCD-NET network model is the above sub-loss function L 1 and L 2 The weighted addition of .

Images