A joint optimization method for compression and reconstruction of ultra-high-definition video

Abstract

The present invention discloses a joint optimization method for compression and reconstruction of ultra-high-definition video, and belongs to the technical field of ultra-high-definition video compression and reconstruction. The technical problem of blind reconstruction caused by the separation of ultra-high-definition video compression and reconstruction process due to the inability to obtain effective information of ultra-high-definition video at the decoding end is solved. The technical scheme is as follows: It includes the following steps: S1, establishing a bidirectional mapping relationship between ultra-high-definition video compression and reconstruction based on mathematical modeling; S2, designing an ultra-high-definition video spatiotemporal feature map information enhancement interaction mechanism based on Transformer; S3, constructing an ultra-high-definition video compression and reconstruction integrated framework based on the spatiotemporal feature map information enhancement interaction mechanism. The beneficial effects of the present invention are as follows: The present invention is oriented to ultra-high-definition video, breaks through the original video encoding and decoding framework, and realizes the joint optimization of ultra-high-definition video compression and reconstruction from the perspective of architectural design.

Description

Technical Field

[0001] The present invention relates to the technical field of ultra-high definition video compression and reconstruction, and in particular to a compression and reconstruction joint optimization method for ultra-high definition video. Background Art

[0002] Video compression aims to reduce the amount of data required to store or transmit the video while keeping the video quality basically unchanged. Traditional video compression methods divide video frames into coding blocks of different sizes based on modern video compression standards, and remove the spatiotemporal redundancy of video sequences through various manually designed inter-frame and intra-frame coding algorithms, thereby reducing the amount of video data. However, these algorithms that rely on manually formulated decision rules cannot effectively represent complex video scenes. With the rapid development and widespread application of deep learning, some researchers have begun to try to achieve video compression through deep learning methods. Among them, most methods use deep learning networks to replace specific modules in the video codec framework, such as intra-frame prediction, mode decision, entropy coding, and post-processing modules. However, since these methods are not separated from the original video codec framework, they still face the problem of video compression distortion caused by lossy coding and transmission.

[0003] As an important link after video compression, video reconstruction can restore the original video based on the bitstream information after video compression. Current video reconstruction methods can be divided into traditional methods based on hybrid coding strategies and deep learning methods based on generative models. Traditional methods mainly use intra-frame, inter-frame prediction and residual frames to eliminate spatiotemporal redundancy or use super-resolution reconstruction technology to improve the quality of video frames, but this method is difficult to reconstruct when the video scene is highly complex. Deep learning methods mainly use variational autoencoders, generative adversarial networks and autoregressive models for video reconstruction, which can be divided into sliding window types and recursive types. However, due to the limitation of window size, the former has limitations in information utilization. When processing longer video sequences, it is difficult to obtain accurate alignment information. When processing longer video sequences, the latter is also prone to the problem of video scene details not being able to be restored due to error accumulation.

[0004] Aiming at the joint optimization of video compression and reconstruction, Guo Lu and others built a deep video compression framework and proposed an end-to-end learning video compression method for the first time. This framework imitates the video compression coding framework in H.265 / HEVC, replacing all key modules in the traditional framework with deep learning networks. The entire video compression framework is composed of neural networks and can be jointly trained. In order to further enhance the performance of the deep video compression framework, the team used a more superior compression network based on the original framework and added a post-processing network to further improve the quality of the reconstructed video.

[0005] Although the above-mentioned joint optimization technology of compression and reconstruction provides a feasible solution for the joint optimization of UHD video compression and reconstruction, the above-mentioned method is still limited to the traditional video codec architecture, and it is difficult to overcome the contradiction between the expansion of transmission bandwidth and storage space requirements caused by the surge in the amount of UHD video data and the current insufficient video compression capabilities. Therefore, driven by deep learning, it is of great significance to build an integrated framework for the entire process of joint "compression-reconstruction" for UHD video to achieve the joint optimization of UHD video compression and reconstruction from the perspective of architectural design. Summary of the invention

[0006] The purpose of the present invention is to provide a joint optimization method for compression and reconstruction of ultra-high-definition video, so as to solve the technical problem of blind reconstruction caused by the separation of ultra-high-definition video compression and reconstruction process, which is caused by the decoding end being unable to obtain effective information of the ultra-high-definition video. Ultimately, for ultra-high-definition video, it breaks through the original video encoding and decoding framework, realizes the joint optimization of ultra-high-definition video compression and reconstruction from the architectural design, and proves the effectiveness of the method of the present invention in terms of subjective performance indicators and subjective visual quality.

[0007] The inventive idea of ​​the present invention is: the present invention provides a joint optimization method for compression and reconstruction of ultra-high-definition video, firstly, a bidirectional mapping relationship between ultra-high-definition video compression and reconstruction based on mathematical modeling is established, then, the frame rate and resolution feature maps of the ultra-high-definition video spatiotemporal visual are extracted, and then, an ultra-high-definition video spatiotemporal feature map information enhancement interaction mechanism based on Transformer is designed, and finally, an ultra-high-definition video compression and reconstruction integrated framework based on the spatiotemporal feature map information enhancement interaction mechanism is constructed.

[0008] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is specifically: a method for joint optimization of compression and reconstruction for ultra-high-definition video, comprising the following steps:

[0009] 1.1. Establish a bidirectional mapping relationship between ultra-high-definition video compression and reconstruction based on mathematical modeling;

[0010] 1.2. Extract the frame rate and resolution feature maps of ultra-high-definition video spatiotemporal vision, and design an interactive mechanism for enhancing the information of ultra-high-definition video spatiotemporal feature maps based on Transformer;

[0011] 1.3. Construct an integrated framework for ultra-high-definition video compression and reconstruction based on the interactive mechanism of spatiotemporal feature map information enhancement.

[0012] Furthermore, the step 1.1 specifically includes the following steps:

[0013] 2.1. Model the compression process of ultra-high-definition video. The formula is: Among them, Io Represents the original ultra-high-definition video, I c represents the compressed ultra-high-definition video, C represents the compressed network, c Indicates that I is obtained by compressing the network c The network mainly consists of two parts: convolution kernel k c And downsampling operation↓ S ;

[0014] 2.2. Model the reconstruction process of ultra-high-definition video, and its formula is: Among them, I r represents the reconstruction of ultra-high-definition video, R represents the reconstruction network, r Indicates that I is obtained by reconstructing the network r The network mainly consists of two parts: convolution kernel k r And upsampling operation↑ S ;

[0015] 2.3. Set the joint optimization goal of the ultra-high-definition video compression and reconstruction process, and its formula is: in, and is a set of optimal network training parameters.

[0016] Furthermore, the step 1.2 specifically includes the following steps:

[0017] 3.1. Send the ultra-high-definition video into the Transformer module to extract the temporal visual frame rate feature map and spatial visual resolution feature map of the ultra-high-definition video;

[0018] 3.2. Vertically concatenate the frame rate feature map and the resolution feature map, fuse the spatiotemporal feature map information, and assign new weights to each element. The formula is: Where x represents the original element in the integrated feature map, y represents the weighted element in the interactive feature map, C(x) is the normalization parameter, g(·) is the single point mapping function, which maps a point to a vector, and f(·) is the similarity measurement function;

[0019] 3.3. Embedded Gaussian function is used as the similarity measurement function, and its formula is: Among them, θ(·) and φ(·) are also single-point mapping functions.

[0020] Furthermore, the step 1.3 specifically includes the following steps:

[0021] 4.1. Use the basic degradation types of blur degradation and noise degradation to degrade the original ultra-high-definition video to obtain a low-data-volume supervised video;

[0022] 4.2. Use the compression network to map the features of the original ultra-high-definition video from the time and space dimensions respectively, and generate a compressed ultra-high-definition video that meets the transmission bandwidth constraint;

[0023] 4.3. Use the reconstruction network to reconstruct the feature information of the original ultra-high-definition video from the time and space dimensions respectively, and obtain the reconstructed ultra-high-definition video;

[0024] 4.4. During the integrated framework training process, the low-data-volume supervision video obtained in step 4.1 is used to supervise the compressed ultra-high-definition video generated in step 4.2, and the original ultra-high-definition video is used to supervise the reconstructed ultra-high-definition video.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] (1) The present invention is oriented to ultra-high-definition video. Based on mathematical modeling, a bidirectional mapping relationship between ultra-high-definition video compression and reconstruction is established, and then an integrated framework for the entire process of joint compression and reconstruction is constructed, thereby achieving joint optimization of ultra-high-definition video compression and reconstruction from the perspective of architectural design;

[0027] (2) The integrated framework constructed by the present invention starts from the frame rate in the time dimension and the resolution in the spatial dimension of ultra-high-definition video, and improves the traceability and compression reconstruction efficiency of the framework through the formulated spatial-temporal feature map information enhancement interaction mechanism;

[0028] (3) The integrated framework constructed by the present invention breaks through the traditional video encoding and decoding architecture, effectively alleviating the contradiction between the expansion of transmission bandwidth and storage space requirements caused by the surge in ultra-high-definition video data and the insufficient current video encoding capabilities;

[0029] (4) The combined optimization method for ultra-high-definition video compression and reconstruction of the present invention effectively alleviates the problem of blind reconstruction caused by the separation of ultra-high-definition video compression and reconstruction processes, which is caused by the inability of the decoding end to obtain effective information of the ultra-high-definition video;

[0030] (5) The present invention applies the joint optimization method of ultra-high-definition video compression and reconstruction to the field of ultra-high-definition video compression and reconstruction, compresses and reconstructs the original ultra-high-definition video in the temporal and spatial dimensions, and achieves good results;

[0031] (6) By comparing the present invention with the traditional video processing method of first extracting frames, reducing resolution, then encoding, transmitting, decoding, and finally restoring the ultra-high-definition video through a series of reconstruction technologies at a 2x frame rate conversion and resolution conversion scale, the subjective quality of the reconstructed video is improved, confirming the application potential of this method in the field of ultra-high-definition video compression and reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0033] Figure 1 A flowchart of a joint optimization method for compression and reconstruction of ultra-high-definition video provided by the present invention.

[0034] Figure 2 Schematic diagram of the ultra-high-definition video spatiotemporal feature map information enhancement interaction mechanism designed in the present invention.

[0035] Figure 3 Schematic diagram of the integrated framework for ultra-high-definition video compression and reconstruction based on the spatial-temporal feature map information enhancement interaction mechanism constructed in the present invention.

[0036] Figure 4 The results of subjective visual quality comparison between the present invention and the comparative method on the video sequence Cactus at 2 times the frame rate and resolution conversion scale.

[0037] Figure 5 The subjective visual quality comparison results of the present invention and the comparison method on the video sequence KristenAndSara at 2 times the frame rate and resolution conversion scale. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Of course, the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0039] Example 1

[0040] See also Figures 1 to 3 The technical solution provided in this embodiment is a joint optimization method for compression and reconstruction of ultra-high-definition video, comprising the following steps:

[0041] 1. Establish a bidirectional mapping relationship between ultra-high-definition video compression and reconstruction based on mathematical modeling;

[0042] 2. Extract the frame rate and resolution feature maps of ultra-high-definition video spatiotemporal vision, and design an interactive mechanism for enhancing the information of ultra-high-definition video spatiotemporal feature maps based on Transformer;

[0043] 3. Construct an integrated framework for ultra-high-definition video compression and reconstruction based on the interactive mechanism of spatiotemporal feature map information enhancement.

[0044] Specifically, in step 1, a bidirectional mapping relationship between ultra-high-definition video compression and reconstruction based on mathematical modeling is established, including the following steps:

[0045] 1) Model the compression process of ultra-high-definition video, and the formula is: Among them, I o Represents the original ultra-high-definition video, I c represents the compressed ultra-high-definition video, C represents the compressed network, c Indicates that I is obtained by compressing the network c The network mainly consists of two parts: convolution kernel k c And downsampling operation↓ S ;

[0046] 2) Model the reconstruction process of ultra-high-definition video, and the formula is: Among them, I r represents the reconstruction of ultra-high-definition video, R represents the reconstruction network, r Indicates that I is obtained by reconstructing the network r The network mainly consists of two parts: convolution kernel k r And upsampling operation↑ S ;

[0047] 3) Set the joint optimization goal of the ultra-high-definition video compression and reconstruction process, and its formula is: in, and is a set of optimal network training parameters.

[0048] Specifically, in step 2, refer to Figure 2 , extract the frame rate and resolution feature maps of ultra-high-definition video spatiotemporal vision, and design an interactive mechanism for enhancing the information of ultra-high-definition video spatiotemporal feature maps based on Transformer, including the following steps:

[0049] 1) Send the ultra-high-definition video to the Transformer module to extract the temporal visual frame rate feature map and spatial visual resolution feature map of the ultra-high-definition video;

[0050] 2) Vertically concatenate the frame rate feature map and the resolution feature map, fuse the spatiotemporal feature map information, and assign new weights to each element. The formula is: Where x represents the original element in the integrated feature map, y represents the weighted element in the interactive feature map, C(x) is the normalization parameter, g(·) is the single point mapping function, which maps a point to a vector, and f(·) is the similarity measurement function;

[0051] 3) Using embedded Gaussian function as similarity measurement function, its formula is: Among them, θ(·) and φ(·) are also single-point mapping functions.

[0052] Specifically, in step 3, refer to Figure 3, an integrated framework for ultra-high-definition video compression and reconstruction based on the information enhancement interaction mechanism of spatiotemporal feature maps is constructed, including the following steps:

[0053] 1) Use the basic degradation types of blur degradation and noise degradation to degrade the original ultra-high-definition video to obtain a low-data-volume supervised video;

[0054] 2) Using the compression network to map the features of the original ultra-high-definition video from the time and space dimensions, and generate a compressed ultra-high-definition video that meets the transmission bandwidth constraint;

[0055] 3) Using the reconstruction network to reconstruct the feature information of the original ultra-high-definition video from the two dimensions of time and space, respectively, to obtain the reconstructed ultra-high-definition video;

[0056] 4) During the integrated framework training process, the low-data-volume supervision video obtained in step 1) is used to supervise the compressed ultra-high-definition video generated in step 2), and the original ultra-high-definition video is used to supervise the reconstructed ultra-high-definition video.

[0057] In order to test the performance of the method proposed in this embodiment, the method of this embodiment is compared with the traditional video processing method of first extracting frames, reducing resolution, then encoding, transmitting, decoding, and finally restoring ultra-high-definition video through a series of reconstruction technologies under the scale of 2x frame rate conversion and resolution conversion. The data set in the experiment consists of official standard video sequences under five different resolution categories of HEVC.

[0058] For the video sequence Cactus with a resolution of 1920×1080, the experimental comparison results of the present invention and the comparison method on the subjective performance index under the scale of 2x frame rate conversion and resolution conversion are shown in Fig. Figure 4 From the perspective of subjective visual quality, the method of this embodiment can effectively reconstruct the original video, and is superior to the original video in terms of information retention and detail presentation.

[0059] Example 2

[0060] Based on Example 1, for the video sequence Traffic with a resolution of 2560×1600, the experimental comparison results of the present invention and the comparison method on the subjective performance index under the scale of 2x frame rate conversion and resolution conversion are shown in FIG. Figure 5 From the perspective of subjective visual quality, the method of this embodiment can effectively reconstruct the original video, and is superior to the original video in terms of information retention and detail presentation.

[0061] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A joint optimization method for compression and reconstruction of ultra-high-definition video, characterized in that: The following steps are involved: S1. Establish a bidirectional mapping relationship between ultra-high-definition video compression and reconstruction based on mathematical modeling; The step S1 comprises the following steps: S11. Model the compression process of ultra-high-definition video, and its formula is: Among them, I o Represents the original ultra-high-definition video, I c represents the compressed ultra-high-definition video, C represents the compressed network, c Indicates that I is obtained by compressing the network c , the network mainly consists of two parts: convolution kernel k c And downsampling operation↓ S ; S12. Model the reconstruction process of ultra-high-definition video, and its formula is: Among them, I r represents the reconstruction of ultra-high-definition video, R represents the reconstruction network, r Indicates that I is obtained by reconstructing the network r , the network consists of two parts: convolution kernel k r And upsampling operation↑ S ; S13. Set the joint optimization goal of the ultra-high-definition video compression and reconstruction process, and its formula is: in, and is a set of optimal network training parameters; S2, extract the frame rate and resolution feature maps of ultra-high-definition video spatiotemporal vision, and design an interactive mechanism for enhancing the information of ultra-high-definition video spatiotemporal feature maps based on Transformer; The step S2 comprises the following steps: S21, sending the ultra-high-definition video to the transformer, extracting the temporal visual frame rate feature map and the spatial visual resolution feature map of the ultra-high-definition video; S22, vertically cascade the frame rate feature map and the resolution feature map, fuse the spatiotemporal feature map information, and assign new weights to each element. The formula is: Where x represents the original element in the integrated feature map, y represents the weighted element in the interactive feature map, C(x) is the normalization parameter, g(·) is the single point mapping function, which maps a point to a vector, f(·) is the similarity measurement function, and x i represents the original element of the frame rate feature map, and xj represents the original element of the resolution feature map; S23, using embedded Gaussian function as similarity measurement function, its formula is: Among them, θ(·) and φ(·) are also single-point mapping functions; S3. Construct an integrated framework for ultra-high-definition video compression and reconstruction based on the information enhancement interaction mechanism of spatiotemporal feature maps; Step S3 includes the following steps: S31, using blur degradation and noise degradation of basic degradation types to perform degradation operations on the original ultra-high-definition video to obtain a low-data-volume supervision video; S32, using a compression network to perform feature mapping on the original ultra-high-definition video from two dimensions of time and space, to generate a compressed ultra-high-definition video that meets the transmission bandwidth constraint; S33, using the reconstruction network to reconstruct feature information of the original ultra-high-definition video from two dimensions of time and space, to obtain a reconstructed ultra-high-definition video; S34. During the integrated framework training process, the low-data-volume supervision video obtained in step S31 is used to supervise the compressed ultra-high-definition video generated in step S32, and the original ultra-high-definition video is used to supervise the reconstructed ultra-high-definition video.

Images