HDR video coding method based on perceptual optimization
By converting HDR video to YUV video and constructing a CTU-level quantization model for visual perception saliency, the problems of visual redundancy and uneven coding bitrate are solved, achieving more efficient coding performance and better visual perception effect.
Patent Information
- Application Number
- CN202410873569.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-01
- Publication Date
- 2024-11-08
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing HDR video coding methods fail to fully consider the perceptual characteristics of the human visual system, resulting in excessive visual redundancy, uneven coding bitrate distribution, and poor visual perception.
HDR video is converted to YUV video using a perceptual quantization transfer function, perceptual lossless reconstruction is performed using a perceptual lossless preprocessing model, and a CTU-level quantization model based on visual perceptual saliency is constructed to guide CTU-level bit allocation, eliminate visual redundancy, and optimize human eye perception during the encoding process.
It achieves improved coding performance, saving 22.67% of coding bits, 26.91% of time, and 5.07% of peak signal-to-noise ratio. The encoded video has low distortion, high quality, and better subjective visual perception.
Smart Images

Figure CN118921464A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a video encoding method, in particular to an HDR video encoding method based on perception optimization. Background Art
[0002] Compared with traditional low dynamic range videos, high dynamic range (HDR) videos can more accurately record the wide dynamic range color information of real scenes. They contain a large color space and rich layers. They can accurately record the full dynamic range color information of real scenes and present more realistic visual effects than traditional low dynamic range videos. They are widely used in satellite remote sensing, medical imaging, virtual reality, digital imaging and other fields. However, while HDR videos bring better visual experience, their huge amount of data also brings new challenges to video compression and coding technology.
[0003] Traditional coding schemes do not take into account the perceptual characteristics of the human visual system (HVS) and cannot provide the best visual experience under bit rate constraints. For this reason, perceptual optimization coding combined with HVS characteristics has received widespread attention in recent years. Depending on whether the internal structure of the encoder has changed, visual perceptual coding combined with HVS characteristics can be divided into preprocessing-based methods and embedded coding methods. The preprocessing-based method starts from the visual perception mechanism, considers spatial masking effects, brightness and contrast sensitivity, attention and other characteristics, and preprocesses the video before encoding. However, most of the JND models constructed in the preprocessing method start from the image itself and do not fully consider the HVS characteristics, especially the visual perception of the human eye. There are some disadvantages in this method. For example, the JND model after mixing different masking effects has a greater improvement in encoding performance, but the subjective effect of the decoded video is poor.
[0004] Embedded coding methods improve coding performance by directly modifying the internal structure of the encoder, among which rate control is the main research direction of embedded coding. However, scholars focus on finding new RD models or new bit allocation strategies, and do not fully consider the connection between different CTU blocks and the relationship between bit allocation strategies and human vision, especially the degree of attention paid by the human eye to different CTU blocks, which ultimately reduces people's visual experience. Summary of the invention
[0005] The technical problem to be solved by the present invention is to provide an HDR video encoding method based on perception optimization, which solves the problems of excessive visual redundancy, uneven coding bit rate distribution and poor visual perception effect caused by directly encoding HDR video, and has better subjective visual perception effect.
[0006] The technical solution adopted by the present invention to solve the above technical problem is: a HDR video encoding method based on perception optimization, comprising the following steps:
[0007] Step ①, obtaining the HDR video to be encoded, and converting the HDR video to be encoded into a YUV video through a perceptual quantization transfer function;
[0008] Step ②, input the YUV video frame by frame into the perceptual lossless preprocessing model for perceptual lossless reconstruction to obtain perceptual lossless video;
[0009] Step ③, input the perceptual lossless video frame by frame into a coding platform embedded with a CTU-level quantization model based on visual perception saliency for coding and compression, obtain a coded and compressed HDR video, and complete HDR video coding;
[0010] In step ③, the specific operation process of obtaining the encoded and compressed HDR video is as follows:
[0011] Step ③-1, taking the first frame of the perceptual lossless video as the current frame;
[0012] Step ③-2, divide the current frame into O CTUs of size 128×128 to be encoded, obtain and fuse the spatial perceptual saliency and temporal perceptual saliency of each CTU to be encoded, and obtain the spatiotemporal saliency of each CTU to be encoded in the current frame;
[0013] Step ③-3, using rate-distortion optimization technology to allocate a target number of bits for each CTU to be encoded in the current frame, and constructing a CTU-level quantization model based on visual perception saliency corresponding to the current frame according to the spatiotemporal saliency and target number of bits of each CTU to be encoded in the current frame;
[0014] Step ③-4, taking the next frame in the perceptual lossless video as the current frame, returning to step ③-2 and continuing to execute until all frames in the perceptual lossless video are constructed to obtain corresponding CTU-level quantization models based on visual perceptual saliency;
[0015] Step ③-5, embed the CTU-level quantization model based on visual perception saliency corresponding to each frame into the original encoding platform to obtain an encoding platform embedded with a CTU-level quantization model based on visual perception saliency, and input the perceptual lossless video frame by frame into the encoding platform embedded with a CTU-level quantization model based on visual perception saliency to obtain the encoded and compressed HDR video.
[0016] Compared with the prior art, the advantage of the present invention is that the HDR video to be encoded is first converted into a YUV video through a perceptual quantization transfer function, and then the YUV video is perceptually losslessly reconstructed through a perceptual lossless preprocessing model. On the basis of the above steps, a specific CTU-level quantization model based on visual perception saliency is constructed to guide the bit allocation at the CTU level, so that the visual perception redundancy of the HDR video itself is eliminated in the encoding process, the encoding performance is improved, and the encoding bit rate is evenly distributed, and finally the bit rate redistribution that meets the perception of the human eye is achieved, thereby improving the encoding efficiency. The experimental results obtained by encoding different video test sequences show that compared with the VVC reference encoding platform VTM 17.0, the method of the present invention saves an average of 22.67% of encoding bits, an average of 26.91% of encoding time, and an average improvement of 5.07% of the peak signal-to-noise ratio. Therefore, the method of the present invention has small distortion, good quality and high compression efficiency in encoding videos, and has better subjective visual perception effects.
[0017] Furthermore, in step ②, the specific operation process of obtaining the perceptually lossless video is as follows:
[0018] Step ②-1, taking the first frame of the YUV video as the current frame;
[0019] Step ②-2, record the current frame as I, its size as M×N, and divide I into A set of non-overlapping image sub-blocks, and expressed in matrix form I = [I1, I2, ..., I s ,…I S ]∈R K×S , where I s ∈R K×1 represents the s-th image sub-block vector, 1≤s≤S, K represents the area of the image sub-block after segmentation, S represents the total number of image sub-blocks after segmentation in the current frame, and the covariance matrix of I is defined as CovI, Where I represents the average row vector of I and T represents the transpose;
[0020] Step ②-3, calculate the eigenvalues and eigenvectors of the covariance matrix CovI, and arrange the eigenvectors in descending order of eigenvalue size to obtain the eigenvector matrix P, P = [p1, p2, ..., p k ,…,p K ]∈R K×K , where p k ∈R K×1 represents the kth eigenvector, 1≤k≤K;
[0021] Step ②-4, perform KLT transformation on I, which is expressed as: Y = P T I, where Y = [ y 1,y2,…,yk , …, y K ] T ∈R K×S represents the KLT coefficient matrix after I transformation, y k =(p k ) T ×I represents the kth spectral component of the coefficient matrix, and T represents the transpose;
[0022] Step ②-5, calculate the KLT coefficient energy Ex of the kth spectral component,
[0023] Step ②-6, normalize the KLT coefficient energy of the kth spectral component to obtain the corresponding normalized coefficient energy Ep k ,
[0024] Step ②-7, accumulate the energy of the first k normalized coefficients to obtain the accumulated energy distribution EP k , E.P. k =Ep1+Ep2+…Ep k , 1≤k≤K;
[0025] Step ②-8, according to the accumulated energy distribution EP k And the first k spectral components, get the perceptual lossless threshold L of the current frame, L = argmink, in, Indicates the average value of the preset KLT coefficient, Indicates the range of variation;
[0026] Step ②-9, combined with the perceptual lossless threshold L, the first L spectral components are used to reconstruct the video image through the KLT inverse transform to obtain the perceptual lossless image of the current frame. This operation is expressed as:
[0027] I (L) =PY (L)
[0028] Y (L) ∈R K×S
[0029] Among them, I (L) Represents the perceptual lossless image of the current frame, Y (L) Represents the reconstruction of the KLT coefficient matrix;
[0030] Step ②-10, take the next frame in the YUV video as the current frame, return to step ②-2 and continue to execute until each frame in the YUV video has a corresponding perceptual lossless image, and all perceptual lossless images are used as perceptual lossless videos. Based on the JND theory, a perceptual lossless preprocessing model based on KLT transform is constructed to preprocess the input YUV video to improve the encoding performance; and the visual redundancy can be eliminated based on the KLT orthogonal transform combined with the perceptual lossless threshold.
[0031] Furthermore, in step ③-2, the specific operation process of obtaining the spatiotemporal saliency of each CTU to be encoded in the current frame is as follows:
[0032] Step ③-2-1, record the co-occurrence matrix of the current frame as H, H = [h(a, b)], 1≤a, b≤J, where J represents the quantization level, h(a, b) represents the co-occurrence histogram, and a and b represent the grayscale values respectively;
[0033] Step ③-2-2, normalize H to obtain the probability mass function P(a, b);
[0034] Step ③-2-3, calculate the spatial visual saliency S of the th pixel in the current frame s (p),
[0035] S s (p) = ∑invP(V HDR (p), V HDR (p′)), where invP represents the inverse probability mass function, V HDR (p) represents the gray value of the pth pixel, 1≤p≤A, A represents the total number of pixels in the current frame, V HDR (p′) represents the gray value of point p′, where p′ represents a point in a circle with a radius of r and the center being the pth pixel, where r=4;
[0036] Step ③-2-4, calculate the spatial saliency Svs(o) of the oth CTU to be encoded in the current frame, Where d represents the size of the oth CTU to be encoded, S s (p) represents the spatial saliency of the p-th pixel in the o-th CTU to be encoded, 1≤ o≤0;
[0037] Step ③-2-5, calculate the spatial perceptual saliency of the oth CTU to be encoded in the current frame Among them, Svs max Indicates the maximum spatial saliency among all CTU spatial saliencies, Svs min Indicates the minimum spatial significance among all CTU spatial significances;
[0038] Step ③-2-6, calculate the temporal saliency S of the th pixel in the current frame t (p), Among them, Y p(i,j) Represents the brightness component of the pth pixel in the current frame, Y cp′(i,j) Represents the brightness component of the p′th pixel at the same position of the reference frame of the current frame;
[0039] Step ③-2-7, calculate the temporal saliency Tvs(o) of the oth CTU to be encoded in the current frame, Where d represents the size of the oth CTU to be encoded, S t (p) represents the temporal saliency of the p-th pixel in the o-th CTU to be encoded;
[0040] Step ③-2-8, calculate the temporal perceptual saliency of the oth CTU to be encoded in the current frame Among them, TVs max Indicates the maximum temporal saliency among the temporal saliencies of all CTUs to be encoded, Tvs min Indicates the minimum temporal saliency among the temporal saliencies of all CTUs to be encoded;
[0041] Step ③-2-9, the spatial perceptual saliency of the oth CTU to be encoded in the current frame and the temporal perceptual saliency of the oth CTU to be encoded in the current frame are merged to obtain the spatiotemporal saliency of the oth CTU to be encoded in the current frame. Wherein, θ1 represents the first constant parameter, θ2 represents the second constant parameter, and θ3 represents the third constant parameter.
[0042] Further, the specific operation process of step ③-3 is as follows:
[0043] Step ③-3-1, using rate-distortion optimization technology to allocate a target bit number T for the mth CTU to be encoded in the current frame CTU (m), Among them, T f Indicates the target number of bits for the current frame, R CTUm,C Indicates the actual total number of bits used by the encoded CTU in the current frame. Indicates the bit weight of the mth CTU to be encoded in the current frame, represents the spatiotemporal saliency of the mth CTU to be encoded in the current frame, represents the spatiotemporal saliency of the i-th CTU to be encoded in the current frame, 1≤m≤O;
[0044] Step ③-3-2, construct a CTU-level quantization model based on visual perception saliency, expressed as:
[0045] QP VTS =4.2005lnλ VPS (m)+13.7122+0.5,
[0046] Limit to
[0047] Among them, QP VPS represents the quantization parameter, λ VPS (m) represents the Lagrangian factor of the mth CTU to be encoded, λ VPS_AdjoinCTU represents the Lagrangian factor of the CTU adjacent to the mth CTU to be encoded, Among them, α and β represent model parameters related to video content characteristics, N pixel Indicates the total number of pixels of the mth CTU to be encoded. The purpose of bit allocation is to allocate the optimal target number of bits to each coding unit so that the total distortion after video encoding is minimized.
[0048] Furthermore, in step ③-2-9, the values of θ1, θ2, and θ3 are 0.5, 0.5, and 0.25, respectively.
[0049] Furthermore, the YUV video described in step ① is a 10-bit bit-depth video that complies with the HDR10 standard; and the original encoding platform described in step ③-5 is VTM17.0. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 It is a schematic diagram of the overall process of the present invention;
[0051] FIG2( a ) is a schematic diagram showing the results of subjective comparative analysis of the test sequence FoodMarket4 by the VTM17.0 standard algorithm, the KI comparison algorithm and the method of the present invention;
[0052] FIG2( b ) is a schematic diagram showing the results of subjective comparative analysis of the test sequence Tango by the VTM17.0 standard algorithm, the KI comparison algorithm and the method of the present invention;
[0053] FIG2( c ) is a schematic diagram showing the results of subjective comparative analysis of the test sequence CatRobot by the VTM17.0 standard algorithm, the KI comparison algorithm and the method of the present invention;
[0054] FIG2( d ) is a schematic diagram showing the results of subjective comparative analysis of the test sequence DaylightRoad by the VTM17.0 standard algorithm, the KI comparison algorithm and the method of the present invention;
[0055] FIG2( e ) is a schematic diagram showing the results of subjective comparative analysis of the test sequence ParkRunning by the VTM17.0 standard algorithm, the KI comparison algorithm and the method of the present invention;
[0056] FIG3( a ) is a schematic diagram showing the comparison results of the subjective quality of the method of the present invention and the original platform VTM17.0 standard algorithm when the encoding bit rate is 500 kbps;
[0057] FIG3( b ) is a schematic diagram showing the comparison results of the subjective quality of the method of the present invention and the original platform VTM17.0 standard algorithm when the encoding bit rate is 1000 kbps;
[0058] FIG3( c ) is a schematic diagram showing the comparison results of the subjective quality of the method of the present invention and the original platform VTM17.0 standard algorithm when the encoding bit rate is 2000 kbps;
[0059] FIG3( d ) is a schematic diagram showing the comparison results of the subjective quality of the method of the present invention and the original platform VTM17.0 standard algorithm when the encoding bit rate is 4000 kbps;
[0060] FIG4( a ) is a schematic diagram of the Bitrate and MOS performance results of the VTM17.0 standard algorithm and the method of the present invention when the test sequence is FoodMarket4;
[0061] FIG4( b ) is a schematic diagram of the Bitrate and MOS performance results of the VTM17.0 standard algorithm and the method of the present invention when the test sequence is Tango;
[0062] FIG4( c ) is a schematic diagram of the Bitrate and MOS performance results of the VTM17.0 standard algorithm and the method of the present invention when the test sequence is CatRobot;
[0063] FIG4( d ) is a schematic diagram of the Bitrate and MOS performance results of the VTM17.0 standard algorithm and the method of the present invention when the test sequence is DaylightRoad;
[0064] FIG4( e ) is a schematic diagram of the Bitrate and MOS performance results of the VTM17.0 standard algorithm and the method of the present invention when the test sequence is ParkRunning. DETAILED DESCRIPTION
[0065] The present invention is further described in detail below with reference to the accompanying drawings.
[0066] like Figure 1 As shown, a HDR video encoding method based on perceptual optimization includes the following steps:
[0067] Step ①, obtain the HDR video to be encoded, and convert the HDR video to be encoded into a 10-bit YUV video that complies with the HDR10 standard through a perceptual quantization transfer function;
[0068] Step ②, input the YUV video frame by frame into the perceptual lossless preprocessing model for perceptual lossless reconstruction to obtain perceptual lossless video;
[0069] In step ②, the specific operation process of obtaining the perceptually lossless video is as follows:
[0070] Step ②-1, taking the first frame of the YUV video as the current frame;
[0071] Step ②-2, record the current frame as I, its size as M×N, and divide I into A set of non-overlapping image sub-blocks, and expressed in matrix form I = [I1, I2, ..., I s ,…I S ]∈R K×S , where I s ∈R K×1 represents the s-th image sub-block vector, 1≤s≤S, K represents the area of the image sub-block after segmentation, S represents the total number of image sub-blocks after segmentation in the current frame, and the covariance matrix of I is defined as CovI, in, represents the average row vector of I, and T represents the transpose;
[0072] Step ②-3, calculate the eigenvalues and eigenvectors of the covariance matrix CovI, and arrange the eigenvectors in descending order of eigenvalue size to obtain the eigenvector matrix P, P = [p1, p2, ..., p k ,…,p k ]∈R K×K , where p k ∈R K×1 represents the kth eigenvector, 1≤k≤K;
[0073] Step ②-4, perform KLT transformation on I, which is expressed as: Y = P T I, where Y = [y1, y2, ..., y k , …, y K ] T ∈R K×S represents the KLT coefficient matrix after I transformation, y k =(p k ) T ×I represents the kth spectral component of the coefficient matrix, and T represents the transpose;
[0074] Step ②-5, calculate the KLT coefficient energy E of the kth spectral component k ,
[0075] Step ②-6, normalize the KLT coefficient energy of the kth spectral component to obtain the corresponding normalized coefficient energy Ep k ,
[0076] Step ②-7, accumulate the energy of the first k normalized coefficients to obtain the accumulated energy distribution EP k , E.P. k =Ep1+Ep2+…Ep k , 1≤k≤K;
[0077] Step ②-8, according to the accumulated energy distribution EP k And the first k spectral components, get the perceptual lossless threshold L of the current frame, L = argmink, in, Indicates the average value of the preset KLT coefficient, Indicates the range of variation; the average value of the preset KLT coefficient is the average value of the energy distribution of all reconstructed images used when the reconstructed image is perceptually lossless;
[0078] Step ②-9, combined with the perceptual lossless threshold L, the first L spectral components are used to reconstruct the video image through the KLT inverse transform to obtain the perceptual lossless image of the current frame. This operation is expressed as:
[0079] I (L) =PY (L)
[0080] Y (L) ∈R K×S
[0081] Among them, I (L) Represents the perceptual lossless image of the current frame, Y (L) Represents the reconstruction of the KLT coefficient matrix;
[0082] Step ②-10, taking the next frame in the YUV video as the current frame, returning to step ②-2 and continuing to execute until each frame in the YUV video obtains a corresponding perceptual lossless image, and taking all perceptual lossless images as perceptual lossless videos;
[0083] Step ③, input the perceptual lossless video frame by frame into a coding platform embedded with a CTU-level quantization model based on visual perception saliency for coding and compression, obtain a coded and compressed HDR video, and complete HDR video coding;
[0084] In step ③, the specific operation process of obtaining the encoded and compressed HDR video is as follows:
[0085] Step ③-1, taking the first frame of the perceptual lossless video as the current frame;
[0086] Step ③-2, divide the current frame into O CTUs of size 128×128 to be encoded, obtain and fuse the spatial perceptual saliency and temporal perceptual saliency of each CTU to be encoded, and obtain the spatiotemporal saliency of each CTU to be encoded in the current frame;
[0087] In step ③-2, the specific operation process of obtaining the spatiotemporal saliency of each CTU to be encoded in the current frame is as follows:
[0088] Step ③-2-1, record the co-occurrence matrix of the current frame as H, H = [h(a, b)], 1≤a, b≤J, where J represents the quantization level, h(a, b) represents the co-occurrence histogram, and a and b represent the grayscale values respectively;
[0089] Step ③-2-2, normalize H to obtain the probability mass function P(a, b);
[0090] Step ③-2-3, calculate the spatial visual saliency S of the p-th pixel in the current frame s (p),
[0091] S s (p) = ∑invP(V HDR (p), V HDR (p′)), where invP represents the inverse probability mass function, V HDR (p) represents the gray value of the pth pixel, 1≤p≤A, A represents the total number of pixels in the current frame, V HDR (p′) represents the gray value of point p′, where p represents a point in a circle with a radius of r and the center being the pth pixel, and r=4;
[0092] Step ③-2-4, calculate the spatial saliency Svs(o) of the oth CTU to be encoded in the current frame, Where d represents the size of the oth CTU to be encoded, S s (p) represents the spatial saliency of the th pixel in the oth CTU to be encoded, 1≤o≤O;
[0093] Step ③-2-5, calculate the spatial perceptual saliency of the oth CTU to be encoded in the current frame Among them, Svs max Indicates the maximum spatial saliency among all CTU spatial saliencies, Svs min Indicates the minimum spatial significance among all CTU spatial significances;
[0094] Step ③-2-6, calculate the temporal saliency S of the p-th pixel in the current frame t (p), Among them, Y p(i,j) Represents the brightness component of the pth pixel in the current frame, Y cp′(i,j) Represents the brightness component of the p′th pixel at the same position of the reference frame of the current frame;
[0095] Step ③-2-7, calculate the temporal saliency Tvs(o) of the oth CTU to be encoded in the current frame, Where d represents the size of the oth CTU to be encoded, S t (p) represents the temporal saliency of the p-th pixel in the o-th CTU to be encoded;
[0096] Step ③-2-8, calculate the temporal perceptual saliency of the oth CTU to be encoded in the current frame Among them, TVs max Indicates the maximum temporal saliency among the temporal saliencies of all CTUs to be encoded, Tvs min Indicates the minimum temporal saliency among the temporal saliencies of all CTUs to be encoded;
[0097] Step ③-2-9, the spatial perceptual saliency of the oth CTU to be encoded in the current frame and the temporal perceptual saliency of the oth CTU to be encoded in the current frame are merged to obtain the spatiotemporal saliency of the oth CTU to be encoded in the current frame. Wherein, θ1 represents the first constant parameter, θ2 represents the second constant parameter, and θ3 represents the third constant parameter; the values of θ1, θ2, and θ3 are 0.5, 0.5, and 0.25 respectively;
[0098] Step ③-3, using rate-distortion optimization technology to allocate a target number of bits for each CTU to be encoded in the current frame, and constructing a CTU-level quantization model based on visual perception saliency corresponding to the current frame according to the spatiotemporal saliency and target number of bits of each CTU to be encoded in the current frame;
[0099] The specific operation process of step ③-3 is as follows:
[0100] Step ③-3-1, using rate-distortion optimization technology to allocate a target bit number T for the mth CTU to be encoded in the current frame CTU (m), Among them, T f Indicates the target number of bits for the current frame, R CTUm,C Indicates the actual total number of bits used by the encoded CTU in the current frame. Indicates the bit weight of the mth CTU to be encoded in the current frame, represents the spatiotemporal saliency of the mth CTU to be encoded in the current frame, represents the spatiotemporal saliency of the i-th CTU to be encoded in the current frame, 1≤m≤O;
[0101] Step ③-3-2, construct a CTU-level quantization model based on visual perception saliency, expressed as:
[0102] QP VPS =4.2005lnλ VPS (m)+13.7122+0.5,
[0103] Limit to
[0104] Among them, QP VPS represents the quantization parameter, λ VPS (m) represents the Lagrangian factor of the mth CTU to be encoded, λ VPS_AdjoinCTU represents the Lagrangian factor of the CTU adjacent to the mth CTU to be encoded, Among them, α and β represent model parameters related to video content characteristics, N pixel Indicates the total number of pixels of the mth CTU to be encoded;
[0105] Step ③-4, taking the next frame in the perceptual lossless video as the current frame, returning to step ③-2 and continuing to execute until all frames in the perceptual lossless video are constructed to obtain corresponding CTU-level quantization models based on visual perceptual saliency;
[0106] Step ③-5, embedding the CTU-level quantization model based on visual perception saliency corresponding to each frame into the original encoding platform to obtain an encoding platform embedded with the CTU-level quantization model based on visual perception saliency, and inputting the perceptual lossless video frame by frame into the encoding platform embedded with the CTU-level quantization model based on visual perception saliency to obtain the encoded and compressed HDR video;
[0107] In this embodiment, the original encoding platform is VTM17.0;
[0108] The present invention adopts four evaluation indicators, namely Peak Signal to Noise Ratio (PSNR), Structure Similarity Index Measure (SSIM), coding bit rate (Bitrate) and coding time (Time), to evaluate the method of the present invention and the VTM17.0 standard algorithm, wherein PSNR reflects the degree of distortion of the video by comparing the difference in pixel gray values; SSIM measures the similarity between the encoded video and the original video from three aspects of brightness, contrast and structure; the saved coding bits (Bitrate Saving, BS) and the saved coding time (Time Saving, TS) are obtained by comparing with the VTM17.0 standard algorithm.
[0109] Table 1 shows the experimental results obtained by encoding different test video sequences (FoodMarket4, Tango, Campfire, CatRobot, DaylightRoad and ParkRunning) of the original platform VTM17.0 standard algorithm and the present invention. Four quantization parameters (QP) are selected for each test sequence in the table for testing. Due to the particularity of HDR video, the time required to encode one frame of HDR video is several times that of encoding ordinary LDR video sequence, so only the first 50 frames of each video sequence are encoded and compressed. The four objective evaluation indicators of the original platform VTM17.0 standard algorithm and the gain data of the present invention are recorded in Table 1. The larger the PSNR, the smaller the distortion of the encoded video and the better the quality; the SSIM value is between 0 and 1, and the larger its value is, the more similar the encoded video is to the original video and the higher the quality is; the smaller the Bitrate rate is, the higher the compression efficiency is. Proposed-Pre represents the method of the present invention.
[0110] It can be seen from Table 1 that compared with the original platform VTM17.0 standard algorithm, the PSNR and SSIM of the present invention are improved to a certain extent, and the Bitrate and Time are saved, which fully demonstrates the effectiveness of the present invention. The present invention saves an average of 22.67% of coding bits, and the compression efficiency is improved. In terms of time, 51.79% of the coding time is saved at the Campfire test sequence, and the average coding time saving reaches 26.91%, and the coding efficiency is also improved.
[0111] Table 1 Comparison of experimental results of the present invention and the original platform VTM17.0 standard algorithm
[0112]
[0113]
[0114] In order to further verify the subjective performance of the present invention, the VTM17.0 standard algorithm, the KI comparison algorithm and the method of the present invention were used to conduct subjective comparative analysis on different sequence test videos. Specifically, 50 subjects were selected to observe the same video sequence encoded by three algorithms to obtain encoded videos, and the distortion degree of the video encoded by each algorithm was evaluated by comparing and analyzing with the original video. Figure 2(a) to Figure 2(e) The pie charts show the visual perceptual distortion of the videos encoded by the three algorithms under four different quantization parameters QP = 22, 27, 32, and 37. The larger the area of the pie chart, the closer the perceptual quality of the video encoded by the algorithm is to the original video, that is, the distortion caused by this method is smaller and has a higher visual quality. Figure 2(a) to Figure 2(e) It can be seen that the present invention achieves better visual perception under different QPs of the five HDR test sequences, which shows the advancement of the present invention. In addition, as the quantization parameter QP decreases, the observer gradually cannot distinguish the difference between the encoding results of the three algorithms, that is, when the quantization parameter is small enough, any preprocessing algorithm has the same improvement on the subjective quality of the video.
[0115] Select the test sequence as CatRobot, Figure 3(a) to Figure 3(d) A detailed comparison of the subjective quality of the method of the present invention and the original platform VTM17.0 standard algorithm under the same encoding bit rate is given, and Bitrate = 100kbps, 2000kbps, and 4000kpbs are selected for encoding respectively. Figure 3(a) to Figure 3(d) It can be seen that under the condition of a certain total bit rate, the present invention allocates more bits to the visually important area based on the human visual perception mechanism, and uses a smaller quantization parameter to encode the area to achieve a better visual effect, which reflects the feasibility of the present invention.
[0116] The existing objective evaluation algorithms do not take the visual perception of the human eye into consideration. In order to further verify the good visual effect of the method proposed in the present invention, a subjective comparison test is conducted between the present invention and the original platform VTM17.0 standard algorithm. The experimental environment refers to the ITU-TP.913 audio and video subjective evaluation standard, adopts a single stimulus excitation method, and uses a percentage-based MOS scoring system to subjectively score the video under test. The higher the MOS score, the higher the quality of the video, and the better the subjective perception effect of the human eye. Specifically, randomly play videos encoded by VTM17.0 and the present invention, watch each video for 10 seconds, and immediately rate the video after the end. When sorting and analyzing the data, remove the testers with a Pearson coefficient less than 0.75, and finally get the MOS score for each video. Figure 4(a) to Figure 4(e)The subjective MOS scores of different sequence videos after being encoded by VTM17.0 and the present invention at the same encoding bit rate are shown. It can be seen that under low encoding bit rate conditions, the present invention has higher visual quality, and gradually tends to a saturation state as the encoding bit rate increases.
[0117] Explanation of terms in this invention:
[0118] YUV: It is a color encoding method. "Y" represents brightness (Luminance or Luma), which is the grayscale value. "U" and "V" represent chrominance (Chrominance or Chroma). It is used to describe the color and saturation of the image and is used to specify the color of the pixel.
[0119] CTU: (Coding Tree Unit) coding tree unit;
[0120] KLT transform: (Karhunen-Loeve Transform) Karhunen-Loeve transform;
[0121] QP: (Quantization Parameter) quantization parameter.
Claims
1. A HDR video encoding method based on perceptual optimization, comprising the following steps: Step ①, obtaining the HDR video to be encoded, and converting the HDR video to be encoded into a YUV video through a perceptual quantization transfer function; Step ②, input the YUV video frame by frame into the perceptual lossless preprocessing model for perceptual lossless reconstruction to obtain perceptual lossless video; Step ③, input the perceptual lossless video frame by frame into a coding platform embedded with a CTU-level quantization model based on visual perception saliency for coding and compression, obtain a coded and compressed HDR video, and complete HDR video coding; It is characterized in that, in step ③, the specific operation process of obtaining the encoded and compressed HDR video is as follows: Step ③-1, taking the first frame of the perceptual lossless video as the current frame; Step ③-2, divide the current frame into O CTUs of size 128×128 to be encoded, obtain and fuse the spatial perceptual saliency and temporal perceptual saliency of each CTU to be encoded, and obtain the spatiotemporal saliency of each CTU to be encoded in the current frame; Step ③-3, using rate-distortion optimization technology to allocate a target number of bits for each CTU to be encoded in the current frame, and constructing a CTU-level quantization model based on visual perception saliency corresponding to the current frame according to the spatiotemporal saliency and target number of bits of each CTU to be encoded in the current frame; Step ③-4, taking the next frame in the perceptual lossless video as the current frame, returning to step ③-2 and continuing to execute until all frames in the perceptual lossless video are constructed to obtain corresponding CTU-level quantization models based on visual perceptual saliency; Step ③-5, embed the CTU-level quantization model based on visual perception saliency corresponding to each frame into the original encoding platform to obtain an encoding platform embedded with a CTU-level quantization model based on visual perception saliency, and input the perceptual lossless video frame by frame into the encoding platform embedded with a CTU-level quantization model based on visual perception saliency to obtain the encoded and compressed HDR video.
2. The HDR video encoding method based on perceptual optimization according to claim 1, characterized in that In step ②, the specific operation process of obtaining the perceptually lossless video is as follows: Step ②-1, taking the first frame of the YUV video as the current frame; Step ②-2, record the current frame as I, its size as M×N, and divide I into A set of non-overlapping image sub-blocks, and expressed in matrix form I = [I1, I2, ..., I s ,…I S ]∈R K×S , where I s ∈R K×1 represents the s-th image sub-block vector, 1≤s≤S, K represents the area of the image sub-block after segmentation, S represents the total number of image sub-blocks after segmentation in the current frame, and the covariance matrix of I is defined as CovI, in, represents the average row vector of I, and T represents the transpose; Step ②-3, calculate the eigenvalues and eigenvectors of the covariance matrix CovI, and arrange the eigenvectors in descending order of eigenvalue size to obtain the eigenvector matrix P, p = [p1, p2, ..., p k ,…,p k ]∈R K×K , where p k ∈R K×1 represents the kth eigenvector, 1≤k≤K; Step ②-4, perform KLT transformation on I, which is expressed as: Y = P T I, where Y = [y1, y2, ..., y k , …, y K ] T ∈R K ×S represents the KLT coefficient matrix after I transformation, y k =(p k ) T ×I represents the kth spectral component of the coefficient matrix, and T represents the transpose; Step ②-5, calculate the KLT coefficient energy E of the kth spectral component k , Step ②-6, normalize the KLT coefficient energy of the kth spectral component to obtain the corresponding normalized coefficient energy Ep k , Step ②-7, accumulate the energy of the first k normalized coefficients to obtain the accumulated energy distribution EP k , E.P. k =Ep1+Ep2+…Ep k , 1≤k≤K; Step ②-8, according to the accumulated energy distribution EP k and the first k spectral components, and obtain the perceptual lossless threshold L of the current frame, L = arg min k, in, Indicates the average value of the preset KLT coefficient, Indicates the range of variation; Step ②-9, combined with the perceptual lossless threshold L, the first L spectral components are used to reconstruct the video image through the KLT inverse transform to obtain the perceptual lossless image of the current frame. This operation is expressed as: IN (L) =PY (L) Among them, I (L) Represents the perceptual lossless image of the current frame, Y (L) Represents the reconstruction of the KLT coefficient matrix; Step ②-10, taking the next frame in the YUV video as the current frame, returning to step ②-2 to continue executing, until each frame in the YUV video obtains a corresponding perceptual lossless image, and all perceptual lossless images are taken as perceptual lossless videos.
3. The HDR video encoding method based on perceptual optimization according to claim 1, characterized in that In step ③-2, the specific operation process of obtaining the spatiotemporal saliency of each CTU to be encoded in the current frame is as follows: Step ③-2-1, record the co-occurrence matrix of the current frame as H, H = [h(a, b)], 1≤a, b≤J, where J represents the quantization level, h(a, b) represents the co-occurrence histogram, and a and b represent the grayscale values respectively; Step ③-2-2, normalize H to obtain the probability mass function P(a, b); Step ③-2-3, calculate the spatial visual saliency S of the p-th pixel in the current frame s (p), S s (p) = ∑invP(V HDR (p), V HDR (p′)), where invP represents the inverse probability mass function, V HDR (p) represents the gray value of the pth pixel, 1≤p≤A, A represents the total number of pixels in the current frame, V HDR (p′) represents the grayscale value of point p, where p represents a point in a circle with a radius of r and the center being the pth pixel, where r=4; Step ③-2-4, calculate the spatial saliency Svs(o) of the oth CTU to be encoded in the current frame, Where d represents the size of the oth CTU to be encoded, S s (p) represents the spatial saliency of the th pixel in the oth CTU to be encoded, 1≤o≤O; Step ③-2-5, calculate the spatial perceptual saliency of the oth CTU to be encoded in the current frame Among them, Svs max Indicates the maximum spatial saliency among all CTU spatial saliencies, Svs min Indicates the minimum spatial significance among all CTU spatial significances; Step ③-2-6, calculate the temporal saliency S of the th pixel in the current frame t (p), Among them, Y p(i,j) Represents the brightness component of the pth pixel in the current frame, Y cp′(i,j) Represents the brightness component of the p′th pixel at the same position of the reference frame of the current frame; Step ③-2-7, calculate the temporal saliency Tvs(o) of the oth CTU to be encoded in the current frame, Where d represents the size of the oth CTU to be encoded, S t (p) represents the temporal saliency of the p-th pixel in the o-th CTU to be encoded; Step ③-2-8, calculate the temporal perceptual saliency of the oth CTU to be encoded in the current frame Among them, TVs max Indicates the maximum temporal saliency among the temporal saliencies of all CTUs to be encoded, Tvs min Indicates the minimum temporal saliency among the temporal saliencies of all CTUs to be encoded; Step ③-2-9, the spatial perceptual saliency of the oth CTU to be encoded in the current frame and the temporal perceptual saliency of the oth CTU to be encoded in the current frame are merged to obtain the spatiotemporal saliency of the oth CTU to be encoded in the current frame. Wherein, θ1 represents the first constant parameter, θ2 represents the second constant parameter, and θ3 represents the third constant parameter.
4. The HDR video encoding method based on perceptual optimization according to claim 3 is characterized in that The specific operation process of step ③-3 is as follows: Step ③-3-1, using rate-distortion optimization technology to allocate a target bit number T for the mth CTU to be encoded in the current frame CTU (m), Among them, T f Indicates the target number of bits for the current frame, R CTUm,C Indicates the actual total number of bits used by the encoded CTU in the current frame. Indicates the bit weight of the mth CTU to be encoded in the current frame, represents the spatiotemporal saliency of the mth CTU to be encoded in the current frame, represents the spatiotemporal saliency of the i-th CTU to be encoded in the current frame, 1≤m≤O; Step ③-3-2, construct a CTU-level quantization model based on visual perception saliency, expressed as: QP VPS =4.2005lnλ VPS (m)+13.7122+0.5, Limit to Among them, QP VPS represents the quantization parameter, λ VPS (m) represents the Lagrangian factor of the mth CTU to be encoded, λ VPS_AdjoinCTU represents the Lagrangian factor of the CTU adjacent to the mth CTU to be encoded, Among them, α and β represent model parameters related to video content characteristics, N pixel Indicates the total number of pixels of the mth CTU to be encoded.
5. The HDR video encoding method based on perceptual optimization according to claim 3, characterized in that In step ③-2-9, the values of θ1, θ2, and θ3 are 0.5, 0.5, and 0.25, respectively.
6. The HDR video encoding method based on perceptual optimization according to claim 1, characterized in that In step ①, the YUV video is a 10-bit video that complies with the HDR10 standard; the original encoding platform described in step ③-5 is VTM17.0.
