A method for improving the quality of compressed video based on the NVIDIA Jetson TX2 platform

Abstract

This invention discloses a method for improving the quality of compressed video based on the NVIDIA Jetson TX2 platform. The main steps include: first, effectively utilizing the spatiotemporal correlation between adjacent video frames through a motion compensation subnetwork, and extracting fused features containing spatiotemporal information of the compressed video; finally, inputting the fused features into a quality improvement subnetwork to obtain enhanced video frames. Experimental results show that the method of this invention can be successfully deployed and run smoothly on edge computing platforms such as the NVIDIA Jetson TX2, effectively suppressing video compression effects and improving video quality.

Description

Technical Field

[0001] This invention relates to compressed video quality improvement technology, specifically to a method for improving compressed video quality based on the NVIDIA Jetson TX2 platform, belonging to the field of image communication. Background Technology

[0002] With the development of the times, video has gradually become one of the main ways for people to obtain and exchange information. However, the huge amount of data in video brings storage and transmission problems. Video codec standards such as HEVC can effectively remove temporal and spatial redundancy in video, reducing the amount of data. However, due to quantization, transformation, and other techniques in the encoding process, video inevitably suffers from compression effects, leading to a decrease in the quality of the compressed video after decoding. Currently, the mainstream methods for improving the quality of compressed video are mainly divided into loop filtering methods and post-processing methods. Among them, loop filtering methods operate inside the video codec, such as deblocking filtering and sample adaptive compensation methods. However, due to the limitations of the codec, loop filtering methods cannot be designed to be too complex, resulting in limited performance. Post-processing methods, on the other hand, are independent of the codec. They are performed after the compressed video has been reconstructed by the video codec standard, and then the quality is improved. Therefore, they do not need to change the video codec structure, have fewer restrictions, and relatively better performance.

[0003] The NVIDIA Jetson TX2, developed by NVIDIA, is a powerful AI edge computing device well-suited as a development platform for edge devices, enabling the deployment of tasks such as image processing. The Jetson TX2 features an integrated 256-core NVIDIA Pascal GPU, a hexadecimal-core ARMv8 64-bit CPU complex, 8GB of LPDDR4 memory, and a 128-bit interface. Due to its portability and high performance, deploying the Jetson TX2 in the Internet of Things (IoT) allows for many controls to be implemented locally without relying on the cloud, with processing completed at the local edge computing layer. This significantly improves processing efficiency and reduces the load on the cloud. Being closer to the user also provides faster response times, addressing needs at the edge. Therefore, researching methods for improving the quality of compressed video based on the NVIDIA Jetson TX2 platform is of great significance.

[0004] Deep learning has achieved remarkable success in computer vision and image processing. Currently, deep learning-based methods are also being applied to improve the quality of decoded videos. However, some algorithms only utilize the spatial information of video sequences, resulting in limited performance improvements; others use complex and redundant network structures, which, while enhancing performance, are difficult to deploy on edge computing platforms for industrial applications. Therefore, this invention aims to research an algorithm that is easy to deploy on edge computing platforms and significantly improves the quality of compressed videos. Summary of the Invention

[0005] The purpose of this invention is to improve the quality of HEVC compressed video on the NVIDIA Jetson TX2 platform.

[0006] The present invention proposes a method for improving the quality of HEVC compressed video by combining spatial and temporal information, which mainly includes the following steps:

[0007] (1) Input the compressed video sequence into the motion compensation subnetwork to obtain the motion offset.

[0008] (2) Combine the motion offset obtained in (1) to perform spatiotemporal fusion processing on the input video sequence.

[0009] (3) Input the output of (2) into the quality enhancement subnetwork, and finally obtain the quality-enhanced compressed video sequence.

[0010] Specifically, in steps (1) and (2) shown, assume a compressed video is F, composed of n frames, i.e. {F1, F2, F3, ..., F...} n}∈F. When the target frame F is lifted at time t (t∈[1,n]), t When considering quality, select the adjacent previous frame F. t-1 With the next frame F t+1 Use it as a reference frame and obtain time-domain information. (F) t F t-1 With F t+1 As input frame input Figure 2 The motion compensation subnetwork structure is shown in the diagram. The subnetworks are connected via a "Concat" operation to incorporate spatiotemporal correlations. Subsequently, F is predicted through a network based on a U-net structure. t-1 With F t+1 To F t Motion offset δ m U-net networks, such as Figure 2The lower left section shows the network's input, which consists of three frames after "concat" fusion processing. The network primarily comprises three consecutive downsampling convolutional layers and three consecutive deconvolutional upsampling layers. Furthermore, channels with the same feature size as the downsampling and upsampling outputs are concatenated and added using skip connections, reusing features while accelerating network convergence. The ordinary convolutional layers have a stride of 1, a convolutional size of 3×3, and use linear rectified units as the activation function. The downsampling layers consist of convolutional layers with a stride of 2 and a convolutional size of 3×3, while the upsampling layers consist of deconvolutional layers with a stride of 2 and a convolutional size of 4×4. Each downsampling or upsampling layer is followed by an ordinary convolutional layer. The output feature maps from the upsampling and downsampling processes of the same size are "concatted" together using skip connections and then input into the next layer. Finally, the output of this U-net network is the predicted motion offset δ. m This process is represented by equation (1-1).

[0011] δ m =U(F t-1 ,F t ,F t+1 (1-1)

[0012] Where U represents the U-net network.

[0013] Subsequently, the obtained motion offset δ m With F t-1 F t+1 and F t When input together into a depthwise separable convolutional layer, F can be made t-1 and F t+1 Align F t This process reduces motion errors and extracts spatiotemporal fusion features, and is represented by equation (1-2).

[0014] f st =D c (δ m ,F t-1 ,F t ,F t+1 (1-2)

[0015] Among them, D c This represents depthwise separable convolution, f st This indicates the spatiotemporal fusion characteristics.

[0016] The entire motion compensation subnetwork can be simplified as equation (1-3):

[0017] f st =H MC (F t-1 ,Ft ,F t+1 (1-3)

[0018] Among them, H MC This represents the entire motion compensation subnetwork. Therefore, according to equations (1-2) and (1-3), H MC It can also be expressed by equation (1-4), as follows:

[0019] H MC (F t-1 ,F t ,F t+1 ) = D c (U(F t-1 ,F t ,F t+1 ),F t-1 ,F t ,F t+1 (1-4)

[0020] Finally, the spatiotemporal fusion feature f st The video is then fed into the quality enhancement subnetwork to reconstruct it.

[0021] The quality enhancement subnetwork in step (3) consists of three quality enhancement modules and one convolutional layer. The input spatiotemporal fusion feature block is continuously enhanced and mapped through three consecutive quality enhancement modules, and then a 3×3 convolutional layer is used to reconstruct the image residual. Finally, the global residual learning method is used in conjunction with F... t The summation yields the improved output frame F. e This process can be represented by formula (1-5).

[0022] F e =H QI (f st )+F t (1-5)

[0023] Among them, H QI This represents the quality improvement subnetwork.

[0024] The structure of the quality improvement module in the network is also shown in Figure 2The architecture primarily consists of six 3×3 convolutional layers. Each pair of convolutional layers is grouped together. The input feature map first enters the first group of convolutional layers, and after a "split" operation to control the number of kernels, it is fed into the second and third groups. The output of the first group is then "split" and "concat"ed with the initial input before being fed into the second group. Similarly, the output of the second group is "split," "concat"ed with the initial input and the output of the first group, and then fed into the third group. Finally, the initial input, the output of the first group, and the output of the second group are "concatted," and then added to the third group using local residuals. The final output is then passed to the next quality enhancement module or convolutional layer.

[0025] The quality improvement subnetwork can also be represented by formula (1-6).

[0026] H QI (f st ) = C onv (h QI (h QI (h QI (f st (1-6)

[0027] Among them, h QI Represents the quality improvement module, C onv This indicates a convolutional layer. Attached Figure Description

[0028] Figure 1 This is a block diagram of a method for improving the quality of compressed video based on the NVIDIA Jetson TX2 platform according to the present invention.

[0029] Figure 2 This is a diagram of the motion compensation subnetwork structure in this invention.

[0030] Figure 3 This is a diagram of the quality improvement sub-network structure in this invention.

[0031] Figure 4The image shows a comparison of the subjective visual quality of the BQMall_832×480 sequence at QP=42, using the HEVC standard, the method of this invention, and two comparative methods. Figure (a) is the first frame of the original BQMall sequence, Figure (b) is the first frame of the sequence after compression using the HEVC standard (PSNR is 28.05dB), Figure (c) is the same frame of the sequence after compression using the HEVC standard and then processing using the comparative method [1] (PSNR is 28.60dB), Figure (d) is the same frame of the sequence after compression using the HEVC standard and then processing using the comparative method [2] (PSNR is 28.54dB), and Figure (e) is the same frame of the sequence after compression using the HEVC standard and then processing using the method of this invention (PSNR is 28.68dB).

[0032] Figure 5 This is a comparison of the subjective visual quality of the HEVC standard, the method of this invention, and the two comparative methods when the QP=37 of the BQSquare_416×240 sequence. To better illustrate the subjective visual effect, Figures (b)-(e) enlarge the right side of the face, which is... Figure 5 (a) The area in the box is observed. Figure (a) is a frame of the original BQSquare sequence. Figure (b) is the same frame of the sequence after HEVC standard compression, with a PSNR of 28.55dB. Figure (c) is the same frame of the sequence after HEVC standard compression and then processing by the comparison method [1], with a PSNR of 29.21dB. Figure (d) is the same frame of the sequence after HEVC standard compression and then processing by the comparison method [2], with a PSNR of 29.17dB. Figure (e) is the same frame of the sequence after HEVC standard compression and then processing by the present invention, with a PSNR of 29.33dB. Detailed Implementation

[0033] The present invention will be further described in detail below with reference to the embodiments. It should be noted that the following embodiments are only used to further illustrate the present invention and should not be construed as limiting the scope of protection of the present invention. Those skilled in the art can make some non-essential improvements and adjustments to the present invention based on the above-described invention, and these improvements and adjustments should still fall within the scope of protection of the present invention.

[0034] Figure 1 Specifically, this involves a method for improving the quality of compressed video based on the NVIDIA Jetson TX2 platform, which can be divided into the following steps:

[0035] (1) Input the compressed video sequence into the motion compensation subnetwork to obtain the motion offset.

[0036] (2) Combine the motion offset obtained in (1) to perform spatiotemporal fusion processing on the input video sequence.

[0037] (3) Input the output of (2) into the quality enhancement subnetwork, and finally obtain the quality-enhanced compressed video sequence.

[0038] As described in steps (1), (2), and (3) above, experiments have verified that this algorithm can run well and quickly on the NVIDIA Jetson TX2 platform and has a good quality enhancement effect on compressed video.

[0039] To better illustrate the effectiveness of this invention, two test sequences, "BQMall_832×480" and "BQSquare_416×240", were selected and compared with the compression standard HEVC and other methods. The subjective visual effects are as follows: Figure 4 and Figure 5 As shown in Table 1, the PSNR improvement (in dB) of the method of this invention and the comparative method compared to HEVC standard compressed video under four representative QPs (32, 37, 42, and 47) demonstrates that the present invention outperforms the classic convolutional neural network method.

[0040] Table 1. Improvement effect of different methods at QP values ​​of 32, 37, 42, and 47.

[0041]

[0042] The comparison method is as follows:

[0043] Method 1: The method proposed by Deng J, Wang L, Pu S et al., reference "Spatio-temporaldeformable convolution for compressed video quality enhancement[C].Proceedings of the AAAI Conference on Artificial Intelligence(AAAI).2020,34(07):10696-10703."

[0044] Method 2: The method proposed by Lu M, Chen T, Liu H et al., reference "A new framework based on spatio-temporal information for enhancing compressed video[C]. 2021 4th International Conference on Information Communication and Signal Processing (ICICSP). IEEE, 2021: 571-576."

Claims

1. A method for improving the quality of compressed video based on the NVIDIA Jetson TX2 platform, characterized in that, Based on the specific limitations of the NVIDIA Jetson TX2 edge computing platform, a framework for a spatiotemporal compression video quality improvement method incorporating motion compensation is constructed, and the steps are as follows: (1) Input the compressed video sequence into the motion compensation subnetwork to obtain the motion offset; The specific method is to divide three adjacent frames F t F t-1 With F t+1 The networks are connected using the "Concat" operation to combine spatiotemporal correlations. Subsequently, F is predicted using a network based on a U-net structure. t-1 With F t+1 To F t Motion offset δ m The network mainly consists of three consecutive downsampling convolutional layers and three consecutive upsampling convolutional layers. Each downsampling convolutional layer is formed by a downsampling layer followed by a regular convolutional layer, and each upsampling convolutional layer is formed by an upsampling layer followed by a regular convolutional layer. The regular convolutional layer has a stride of 1, a convolutional size of 3×3, and uses a linear rectified unit as the activation function. The downsampling layer is formed by a convolutional layer with a stride of 2 and a convolutional size of 3×3. The upsampling layer is formed by a deconvolutional layer with a stride of 2 and a convolutional size of 4×4. The output feature maps from the upsampling and downsampling processes of the same size are concatenated using a skip connection ("Concat") and then input into the next layer. Finally, the output of this U-net network is the predicted motion offset δ. m ; (2) Combine the motion offset obtained in (1) to perform spatiotemporal fusion processing on the input video sequence; The specific method is to obtain the motion offset δ m With F t-1 F t+1 and F t When input together into a depthwise separable convolutional layer, F can be made t-1 and F t+1 Align F t This reduces motion errors and extracts spatiotemporal fusion features; (3) Input the output of (2) into the quality enhancement subnetwork, and finally obtain the quality-enhanced compressed video sequence; The quality enhancement subnetwork mainly consists of three quality enhancement modules and one convolutional layer. Each quality enhancement module comprises six 3×3 convolutional layers, grouped into pairs. The input feature map first enters the first group of convolutional layers, and after the number of kernels is controlled by a "Split" operation, it is fed into the second and third groups. The output of the first group of convolutional layers is then "Split" again, followed by a "Concat" operation to fuse the features with the initial input before being fed into the second group of convolutional layers. Similarly, the output of the second group of convolutional layers is "Split" again... After the "t" operation, the features are fused with the initial input and the output of the first group ("Concat") and then fed into the third convolutional layer. Finally, the initial input, the output of the first convolutional layer, and the output of the second convolutional layer are "Concat" and added to the third convolutional layer using local residuals. The final output is then passed to the next quality enhancement module or convolutional layer. The spatiotemporal fusion feature block is continuously enhanced and mapped through three consecutive quality enhancement modules. Then, a 3×3 convolutional layer is used to reconstruct the image residuals. Finally, the global residual learning method is combined with F... t The sums are used to obtain the output frame with improved quality.

Images