A progressively refined method for generating virtual reference frames

Abstract

The present invention discloses a method for generating a progressively refined virtual reference frame, comprising: constructing a progressively refined reference frame generation network consisting of a disparity-guided frame-level synthesis module and a transformer-based block-level aggregation module; generating a disparity map of a temporal and inter-viewpoint reference frame using a disparity estimation module; and converting the inter-viewpoint reference frame F into a frame-level synthesis module. i Align to the target viewpoint; introduce parallax attention to adaptively fuse the aligned inter-viewpoint reference frames F in the parallax direction i and the temporal reference frame F t , generate the primary reference frame F at the frame level p ; Use the transformer-based block-level aggregation module to model the temporal reference frame F t and the primary reference frame F p The long-range dependencies between pixels are analyzed, and finally a refined high-quality reference frame is generated through a U-Net-based reconstruction block. This method realizes the generation of progressively refined virtual reference frames, thereby reducing the storage pressure and transmission bandwidth requirements of multi-viewpoint video.

Description

Technical Field

[0001] The present invention relates to the field of virtual reference frame generation, and in particular to a progressively refined virtual reference frame generation method. Background Art

[0002] To provide an immersive visual experience, multi-view video captures scene objects from multiple viewpoints, significantly increasing the amount of data. This massive amount of data places enormous pressure on data storage and transmission, making the exploration of efficient multi-view video coding methods crucial. To this end, 3D-HEVC (3D High Efficiency Video Coding) integrates effective multi-view video coding tools such as viewpoint compensation prediction and advanced residual prediction. These tools significantly improve multi-view video coding performance.

[0003] In multi-view video coding, high-quality reference frames can effectively remove data redundancy and improve the coding quality of subsequent frames. Therefore, relevant research has explored temporal correlation to generate high-quality reference frames. For example, some traditional hand-crafted methods use optical flow estimation or frame rate up-conversion techniques to interpolate bidirectional temporal reference frames to generate new reference frames, or map temporal reference frames to target frames through affine mapping strategies. In recent years, deep learning-based methods have achieved success in the field of computer vision, and some researchers have focused on using convolutional neural networks to learn and generate high-quality reference frames. Zhu et al. proposed an interpolation network to generate additional reference frames for inter-frame prediction. Katayama et al. proposed a dynamic learning strategy for generating reference frames. Ding et al. proposed a collaborative reconstruction enhancement and reference generation network to generate high-quality reference blocks for inter-frame prediction. Zhao et al. proposed a deep learning-based frame rate up-conversion algorithm for CTU (coding tree unit)-level inter-frame prediction. However, these temporal reference frame generation methods ignore the inter-view correlation in multi-view video coding.

[0004] Based on the above temporal reference frame generation method, Lei et al. further explored the disparity correlation between inter-view reference frames and temporal reference frames to generate high-quality virtual reference frames. However, existing methods do not fully utilize the useful information of temporal and inter-view reference frames and lack refined designs to effectively synthesize temporal and inter-view reference frames to generate high-quality reference frames. Summary of the Invention

[0005] This paper designs a progressively refined reference frame generation method. Considering that existing methods do not fully utilize the useful information of temporal and inter-viewpoint reference frames, this method aims to design a refined network framework to fuse the complementary information of temporal and inter-viewpoint disparity at the frame level and compensate for local complex texture details at the block level, thereby achieving progressively refined virtual reference frame generation, thereby reducing the storage pressure and transmission bandwidth requirements of multi-viewpoint video. The details are described below:

[0006] A method for generating a progressively refined virtual reference frame, the method comprising:

[0007] Construct a progressively refined reference frame generation network consisting of a disparity-guided frame-level synthesis module and a transformer-based block-level aggregation module;

[0008] The disparity estimation module is used to generate the disparity map of the time domain and the inter-view reference frame, and the disparity map is used to convert the inter-view reference frame F i Align to the target viewpoint;

[0009] Introducing disparity attention to adaptively fuse the aligned inter-viewpoint reference frames F in the disparity direction w and the temporal reference frame F t , generate the primary reference frame F at the frame level p ;

[0010] Modeling the temporal reference frame F using a transformer-based block-level aggregation module t and the primary reference frame F p The long-range dependencies between pixels are analyzed, and finally a refined high-quality reference frame is generated through a U-Net based reconstruction block.

[0011] Among them, the aligned inter-view reference frame F w for:

[0012] F w =Warp(d,F i )

[0013] Warp refers to the warping operation.

[0014] Furthermore, the method further includes: constructing a parallax attention module model F t and F w The correlation C between them is:

[0015] C=Softmax(conv(f t )×f w )

[0016] Among them, conv(·) represents a 1×1 convolutional layer, Softmax(·) represents a Softmax operation, × represents a matrix multiplication operation, and f w Indicates F w characteristics.

[0017] Wherein, the primary reference frame F p for:

[0018] Using the correlation C to f t Weighted, the weighted features and F w The characteristic fw Cascade, fusion generation:

[0019]

[0020] in, represents the weighted features of the temporal reference frame, Indicates channel cascading.

[0021] Furthermore, the transformer-based block-level aggregation module generates a refined high-quality reference frame as follows:

[0022] Use multi-head attention to model the correlation between primary reference features and temporal reference features, and use FFN to generate high-quality reference features f b :

[0023]

[0024] Among them, [·] represents the normalization layer, f m express Based on the output of multi-head attention, d k represents the dimension of the key vector k.

[0025] The method further comprises: using a step-by-step synthesis loss to constrain the generation of high-quality reference frames, including: a frame-level reconstruction loss and a block-level reconstruction loss,

[0026] The step-by-step synthesis loss is:

[0027]

[0028] Where λ represents the weight of the frame-level reconstruction loss, n represents the n-th pixel value of the corresponding frame, and N represents the total number of pixels in a frame.

[0029] The beneficial effects of the technical solution provided by the present invention are:

[0030] 1. Based on aligned inter-viewpoint reference frames and temporal reference frames, the present invention generates primary reference frames using disparity attention fusion at the frame level, and generates high-quality reference frames by aggregating local complex texture details based on transformers.

[0031] 2. The present invention adds the generated reference frame to the reference list for inter-frame prediction, effectively improving the compression performance of multi-view video coding;

[0032] 3. Compared with the 3D-HEVC reference software HTM16.2, the reference frames generated by the present invention effectively improve the reconstruction quality of non-base viewpoint B frames, reduce bit consumption, and thus reduce the storage pressure and transmission bandwidth requirements of multi-viewpoint videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Flowchart of a method for generating a reference frame with progressive refinement. DETAILED DESCRIPTION

[0034] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0035] To improve the coding performance of multi-view video, the present invention utilizes inter-view correlation and long-range dependencies between pixels to meticulously design a reference frame generation network. This method progressively fuses temporal reference frames and inter-view reference frames through a frame-level to block-level generation paradigm to generate high-quality virtual reference frames for multi-view video coding. The method includes the following steps:

[0036] 1. Constructing a progressively refined reference frame generation network framework

[0037] In order to improve the performance of multi-view video coding, the embodiment of the present invention designs a progressively refined reference frame generation network. The input of the network is the temporal reference frame F t and the inter-view reference frame F i , the output is a high-quality reference frame F b The network consists of a disparity-guided frame-level synthesis module and a transformer-based block-level aggregation module. In the disparity-guided frame-level synthesis module, F t and F i Then, the disparity estimation module is used to generate the disparity map of the time domain and the inter-view reference frame, and the disparity map is used to map the inter-view reference frame F i Align to the target viewpoint. Then, the parallax attention is introduced to adaptively fuse the aligned inter-viewpoint reference frames F in the parallax direction. w and the temporal reference frame F t , generate the primary reference frame F at the frame level p .

[0038] In order to further compensate the primary reference frame F p Complex texture details, using a transformer-based block-level aggregation module to model the temporal reference frame F t and the primary reference frame F p The long-range dependencies between pixels are analyzed, and finally a refined high-quality reference frame is generated through a U-Net based reconstruction block.

[0039] Among them, U-Net is a network well known to those skilled in the art, and will not be described in detail in the embodiment of the present invention.

[0040] 2. Building a parallax-guided frame-level synthesis module

[0041] In practical applications, the parallax variation between viewpoints leads to mismatch in the information of the same position in different viewpoints. This mismatch hinders the effective use of complementary information between viewpoints. Therefore, a parallax-guided frame-level synthesis module is designed to fuse the complementary information between viewpoints by exploring the parallax relationship at the frame level. Given an inter-view reference frame F i and the temporal reference frame F t , feature extraction is performed through two convolutional layers to obtain the features f corresponding to the time domain and the reference frame between viewpoints. t With f i . Using the reference feature f t and f i Disparity estimation is performed to obtain the disparity map d. Among them, the disparity estimation module branch learns the cascade cost volume and the group correlation cost volume, and uses the two cost volumes cascaded as the input of the 3D aggregation network to perform cost aggregation to generate the disparity map d. The disparity map d is used to align F t and F i , generate the aligned inter-view reference frame F w :

[0042] F w =Warp(d,F i )

[0043] Warp refers to the warping operation.

[0044] Then, considering that the temporal reference frame and the aligned inter-view reference frame contain complementary image priors, in order to explore the correlation between viewpoints, a disparity attention module is constructed to model F t and F w The correlation C between them is calculated as follows:

[0045] C=Softmax(conv(f t )×f w )

[0046] Among them, conv(·) represents a 1×1 convolutional layer, Softmax(·) represents a Softmax operation, × represents a matrix multiplication operation, and f w Indicates F w characteristics.

[0047] Finally, use the correlation C to t Weighted, the weighted features and F w The characteristic f w Cascade and fusion generate the primary reference frame f p :

[0048]

[0049] in, represents the weighted features of the temporal reference frame, Indicates channel cascading.

[0050] 3. Building a Transformer-based Block-Level Aggregation Module

[0051] Considering that the primary reference frame still contains inaccurate textures in complex texture areas, the embodiment of the present invention further focuses on local texture details to refine the reference frame generation. Specifically, a transformer-based block-level aggregation module is designed to aggregate block-level reference frames. The transformer block is introduced to compensate for local inaccurate textures by modeling the long-range dependencies between pixels. This module converts the primary reference frame F p Mapping to feature f p , and f t Cut together into block-level features, that is, In this way, local texture details are aggregated. Transform to get the query vector q, Transform to get the key vector k and value vector v:

[0052]

[0053] Here, Linea(·) represents a linear layer.

[0054] Then, multi-head attention is used to model the correlation between the primary reference features and the temporal reference features, and then FFN (feedforward neural network) is used to generate high-quality reference features f b :

[0055]

[0056] Among them, [·] represents the normalization layer, f m express Based on the output of multi-head attention, d k represents the dimension of the key vector k.

[0057] Finally, the reconstruction generates a high-quality reference frame F b :

[0058] F b =Recon(f b )

[0059] Among them, Recon(·) represents the reconstruction block based on U-Net.

[0060] 4. Constructing a Loss Function for Gradually Refining the Reference Frame Generation Network

[0061] The step-by-step synthesis loss L is used to constrain the generation of high-quality reference frames. L consists of two parts. The first part is the frame-level reconstruction loss, which is used to constrain the aligned inter-view reference frame F w and the original frame F gt , and F p and the original frame F gt The second part is the block-level reconstruction loss, which is used to constrain the high-quality reference frame F b With the original frame F gt The minimum prediction error between .

[0062] Among them, the step-by-step synthesis loss L is:

[0063]

[0064] Where λ represents the weight of the frame-level reconstruction loss and is set to 0.1. n represents the nth pixel value of the corresponding frame, and N represents the total number of pixels in a frame.

[0065] 5. Training the Gradually Refined Reference Frame Generation Network

[0066] The reference frame generation network was trained on KITTI. The reconstructed dataset was obtained by compressing KITTI using 3D-HEVC. The reconstructed dataset was then cropped into 128×128 blocks for training. To improve the network's generalization, data augmentation was performed using horizontal and vertical flipping. Training and testing were performed using PyTorch. The network's initial learning rate was 2e-4, halved every five epochs, for a total of 30 epochs. The batch size was set to 8, and the network was optimized using the Adam algorithm.

[0067] 6. Integration into 3D-HEVC Platform

[0068] The network proposed in this embodiment of the present invention generates additional candidate frames for the reference list of inter-frame prediction. To verify the effectiveness of the proposed method, the network was integrated into 3D-HEVC and tested using the first 9 frames of the reference sequence, acting on all B-frames of non-base viewpoints. First, a separable convolution is used to generate a temporal reference frame. Then, a progressive learning paradigm from the frame level to the block level is used to synthesize the temporal reference frame and the inter-viewpoint reference frame into a high-quality reference frame. The generated high-quality reference frame is added as an additional reference to the forward reference list List0 and the backward reference list List1.

[0069] VII. The high-quality virtual reference frames generated by this invention provide high-quality reference information for inter-frame prediction, effectively improving the efficiency of multi-view video coding. Compared with the 3D-HEVC reference software HTM16.2, the proposed method improves the reconstruction quality of non-base view B-frames in multi-view video coding, achieving a 6.41% bit reduction, thereby reducing the storage pressure and transmission bandwidth requirements of multi-view video.

[0070] Those skilled in the art will understand that the accompanying drawings are only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0071] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for generating a progressively refined virtual reference frame, characterized in that: The method comprises: Construct a progressively refined reference frame generation network consisting of a disparity-guided frame-level synthesis module and a transformer-based block-level aggregation module; The disparity estimation module is used to generate the disparity map of the time domain and the inter-view reference frame, and the disparity map is used to convert the inter-view reference frame Align to the target viewpoint; Introducing parallax attention to adaptively fuse aligned inter-viewpoint reference frames in the parallax direction and temporal reference frame , generating primary reference frames at the frame level ; Modeling temporal reference frames using a transformer-based block-level aggregation module and primary reference frame The long-range dependencies between pixels are analyzed, and finally a refined high-quality reference frame is generated through a U-Net based reconstruction block.

2. The method for generating a progressively refined virtual reference frame according to claim 1, wherein: Aligned inter-view reference frames for: ； in, For warping operations, Disparity map.

3. The method for generating a progressively refined virtual reference frame according to claim 2, wherein: The method also includes: constructing a parallax attention module model and The correlation between for: ； in, represents a 1×1 convolutional layer, Represents the Softmax operation, × represents the matrix multiplication operation, express Features, express characteristics.

4. The method for generating a progressively refined virtual reference frame according to claim 3, wherein: The primary reference frame for: Leverage correlation right Weighted, the weighted features and Features Cascade, fusion generation: ； in, represents the weighted features of the temporal reference frame, Indicates channel cascading.

5. The method for generating a progressively refined virtual reference frame according to claim 3, wherein: The transformer-based block-level aggregation module generates a refined high-quality reference frame as follows: Use multi-head attention to model the correlation between primary reference features and temporal reference features, and use FFN to generate high-quality reference features. : ； in, represents the normalization layer, express Based on the output of multi-head attention, Represents the key vector Dimensions, primary reference frame Mapping to features ,and Cut together into block-level features, that is, ; in, , represents a linear layer.

6. The method for generating a progressively refined virtual reference frame according to claim 1, wherein: The method further includes: using a step-by-step synthesis loss to constrain the generation of a high-quality reference frame, including a frame-level reconstruction loss and a block-level reconstruction loss, where the step-by-step synthesis loss is: ； in, represents the weight of the frame-level reconstruction loss, Represents the nth pixel value of the corresponding frame, Indicates the total number of pixels in a frame, represents the original frame, Represents a high-quality reference frame.

Images