Virtual viewpoint rendering hole filling method based on multi-layer feature fusion

Abstract

The application discloses a virtual viewpoint rendering hole filling method based on multi-layer feature fusion, which transmits space-time information from a known area outside a hole area, obtains a local feature block and a non-local feature block by performing feature refinement on local continuous frames and non-local random frames through multi-layer encoding, performs optical flow completion on the local continuous frames, obtains optical flow features of adjacent frames with holes, and is used for detecting moving objects in a virtual viewpoint sequence to guide feature transmission; the feature transmission combines the local feature block, transmits and updates information of the moving objects, and alleviates a front background aliasing problem; and the updated feature and the non-local feature block are subjected to content reconstruction, and a hole repair frame is obtained by decoding and reconstruction, a multi-head space-time self-attention module and a fusion feedforward neural network are added, information of different features is aggregated, features of local and non-local adjacent frames are fully utilized, and a false image problem is better handled.

Description

Technical Field

[0001] The present invention belongs to the field of virtual viewpoint rendering hole filling, and in particular relates to a virtual viewpoint rendering hole filling method based on multi-level feature fusion. Background Art

[0002] Emerging three-dimensional (3D) multimedia visual services, such as free-viewpoint video, stereoscopic television, and virtual reality, offer users an immersive and interactive visual experience and are gaining increasing attention and popularity. However, representing interactive 3D video requires a large amount of viewpoint information. Due to acquisition costs and bandwidth limitations, only a limited number of viewpoints can be captured and transmitted in practical applications. Currently, the multi-view plus depth (MVD) encoding format is the mainstream format for compressed 3D and free-viewpoint video. At the decoding end, it uses depth image based rendering (DIBR) technology to render the required virtual viewpoints from the MVD video to compensate for the insufficient number of viewpoints. However, when rendering virtual viewpoints, problems such as foreground and background occlusion between different viewpoints can result in missing areas such as holes and cracks in the rendered image, necessitating repair of these missing areas.

[0003] Traditional hole filling techniques for virtual viewpoint rendering are mainly based on spatial consistency and temporal consistency. Spatial consistency-based techniques mainly use filters and patch-based methods, while temporal consistency-based techniques use background modules to construct background hole areas.

[0004] The existing technology CN201310017391.3 proposes a new method for virtual viewpoint synthesis based on depth map rendering technology. First, a virtual viewpoint depth image is obtained through 3D image transformation, and the depth image is optimized; then, a reverse 3D image transformation is performed based on the depth map obtained through optimization to obtain a virtual viewpoint color image; finally, holes are filled by an image restoration algorithm based on depth information. Through the reverse 3D image transformation, cracks can be avoided on the virtual viewpoint color image and the quality of the virtual viewpoint image can be improved; in addition, for the non-occluded hole areas in the virtual viewpoint image, the image restoration method is used to fill them, which can ensure that the rendered image produces the best display effect. The above scheme is still based on the patch method of searching a single view for spatial consistency for filling, which easily gives the same weight to the foreground and background, and the artifact phenomenon is serious.

[0005] However, the filters in spatial consistency technology can only repair cracks and small baseline holes that appear during the drawing process. Patch-based methods search and match similar patches. Due to the low accuracy of the search algorithm, it is easy to assign the same weight to the foreground and background. The method based on temporal consistency uses a reconstructed background image to fill the virtual view, which has many steps. Moreover, the scene contains moving objects, which can easily cause the foreground to be modeled as the background, resulting in aliasing of foreground and background pixels and artifacts. Summary of the Invention

[0006] In order to address the shortcomings of the existing technology and achieve the goal of not only considering spatial information but also performing consistency processing on temporal information during the process of filling holes in virtual viewpoint rendering, the present invention adopts the following technical solutions:

[0007] The virtual viewpoint rendering hole filling method based on multi-level feature fusion includes the following steps:

[0008] Step S1: local continuous frames T l and non-local random frame T nl ,The features of each layer are refined through multi-layer coding, and spatiotemporal information is propagated from the known regions outside the hole region in each frame, and local feature blocks and non-local feature blocks are obtained;

[0009] Step S2: For local continuous frames T l Performing optical flow completion to obtain optical flow features of adjacent frames with holes is used to detect moving objects in a virtual viewpoint sequence to guide feature propagation. Feature propagation combines the local feature blocks to transfer and update information about the moving objects. Since the presence of large motion in non-local frames makes optical flow prediction accuracy low or even ineffective, the present invention only applies an optical flow-based model to features extracted from local continuous frames.

[0010] Step S3: reconstruct the content of the updated features and non-local feature blocks, and then reconstruct the picture by the decoder to obtain a hole repair frame; in the embodiment of the present invention, since the hole area after the virtual viewpoint is drawn is large, it is not enough to only use the local time neighboring information provided by the optical flow completion and feature propagation model. The local adjacent hole content may appear in the non-local content, where the non-local content is randomly sampled from the video sequence. Therefore, the information in the non-local time neighbor can be regarded as an important reference for these missing areas in the local neighbor, in order to effectively combine the information from the local and non-local time neighbors to perform content reconstruction.

[0011] Furthermore, in the multi-level encoding in step S1, as the receptive field increases, the first layer features are always cascaded to the feature maps of other layers:

[0012]

[0013] Among them, T t Represents the number of video frames, j represents the number of multi-level layers, L represents the total number of multi-level layers, Conv represents the convolution operation with a convolution kernel size of 3×3, LReLU or LeakyReLU represents the activation function, and Concat represents the cascade along the channel dimension. Represents the j-th layer feature map The next layer of feature maps obtained after convolution and activation function processing, Represents the first-level feature map and Feature map after cascading.

[0014] Furthermore, in step S2, the optical flow prediction between adjacent frames i and j is performed by the optical flow completion model Calculation yields:

[0015]

[0016] in, and represents the original adjacent frames with holes, Represents the optical flow features from the i-th frame to the j-th frame.

[0017] In step S2, since the missing areas drawn by the virtual viewpoints will affect the quality of optical flow prediction, it is necessary to constrain the forward and backward flows before using them for feature propagation to improve the ability of optical flow prediction. The optical flow loss L flow Reconstruct bidirectional flow through L1 loss:

[0018]

[0019] Among them, F t→t+1 and F t→t-1 Represent the forward and backward flow features of the original sequence frames, and They are the forward flow and backward flow features calculated by the optical flow completion model, T t Indicates the number of video frames.

[0020] Furthermore, in step S2, the forward flow feature after optical flow completion Helps capture the motion of the hole region from the t-th frame to the t+1-th frame. Once the pixels of the hole region of the t-th content feature are known in the valid region of the t+1-th feature, the forward flow feature is used The back propagation feature of the t+1th local feature block Draw to the current local feature block for update, the backward flow features are similar, and finally the local features that fuse the forward and backward propagation features are obtained.

[0021] Furthermore, in order to alleviate the problem of irrelevant information brought about during the propagation process, the present invention uses deformable convolution to further index and weight the candidate feature points, so that the sampling points are more inclined to the background features of the moving object. The step S2 specifically includes the following steps:

[0022] First, calculate the weight mask M t→t+1 and the offset ΔF in the optical flow prediction t→t+1 :

[0023]

[0024] Among them, C b (·) represents multiple cascaded convolutional layers, warp(·) represents the spatial rendering operation based on optical flow, and the weight mask M t→t+1 and offset ΔF t→t+1 are the parameters of the deformable convolution, both of size H / 4×W / 4×K 2 ×G, where K and G represent the size of the deformable convolution kernel and the number of groups, respectively. Based on experience, K is set to 3 and G is set to 16. Add an offset ΔF to t→t+1 , generating multiple (K 2 ×G) candidate feature points, more flexible sampling locations can well compensate for inaccurate traffic completion and make it easier to find valid content in the surrounding environment;

[0025] Then, using the deformable convolutional layer D b (·) to plot the backpropagation features The features processed by deformable convolution are fused with the current local features Et and propagated through the backpropagation function P b (·) To update:

[0026]

[0027] in, represents the back propagation feature of the t-th time length, P b (·) represents the propagation function, which is two convolutional layers with a LeakyReLU activation function, and the weight mask M t→t+1 The value of is normalized by the Sigmoid function;

[0028] Finally, a learnable convolutional layer is used to adaptively fuse the forward and backward propagated features:

[0029]

[0030] Among them, I(·) represents a 1×1 convolutional layer, represents the fused local features, forward-propagated features, backward-propagated features.

[0031] Further, multiple multi-head spatio-temporal self-attention Transformer modules are stacked, and a single module can be represented as:

[0032] Z' n = MTSA(LN1(Z n-1 ))+Z n-1

[0033] Z n = F3N(LN2(Z' n ))+Z' n

[0034] wherein Z n-1 represents an input token, wherein n e [1, N], N represents a stacking number of multi-head spatio-temporal self-attention Transformer modules, N = 8, Z' n represents an intermediate variable value of the input token, MTSA(·) and LN(·) represent multi-head spatio-temporal self-attention and layer normalization in the Transformer respectively, the subscript i represents the i-th layer, and F3N(·) represents a feed-forward neural network of fusion soft segmentation operation;

[0035] The multi-head spatio-temporal self-attention module is a window sliding-based self-attention module, which splits Z n-1 into sub-window tokens t of s h x s w x s n to calculate the query vector Q n and the key-value pair vector {K n , V l} of self-attention, T nl represents a local continuous frame number, T p represents a selected non-local frame number, M x N is a spatial token dimension of embedding, and C0 represents a feature dimension after a soft segmentation operation;

[0036] First, a linear projection layer f p is used to perform sub-window pooling on the sub-window tokens in space, and information tokens far away from the query vector are coarsely focused, thereby obtaining coarse-grained sub-window tokens:

[0037]

[0038] wherein f p (·) represents a linear projection operation.

[0039] Then, the query set of the middle block is used to represent the sliding window, that is, the local time information token. This local window is not subjected to sub-window pooling processing and is used as the calculation of the query vector. Therefore, it is also a focus on fine-grainedness. Then, three linear projection layers f are used respectively. q 、f k and f v To calculate Q n , K n and V n :

[0040]

[0041] in, and Represent the key vectors of the local window and the coarse-grained sub-window, V l n and Represent the value vectors of the local window and the coarse-grained window respectively, f q (·),f k (·) and f v (·) is a 1×1 convolutional layer. The purpose of combining local and coarse-grained windows is to calculate the attention of local-global interactions. Therefore, the multi-head spatiotemporal self-attention is calculated according to the above conditions:

[0042]

[0043] Among them, Attention represents multi-head self-attention, softmax is the normalization function, and T is the transpose symbol.

[0044] Furthermore, in step S3, a fused feedforward neural network (F3N) is used to connect the connections between embedded tokens. F3N inserts a soft-segmentation (SS) operation between the two MLP layers of the feedforward neural network (FFN). The soft-segmented overlapping patches are then folded (SC) according to their original spatial positions to generate a new feature map. This facilitates aggregating information from different tokens and fully utilizing local and non-local adjacent features. The SS and SC operations are inserted into F3N. SS divides each frame into k×k overlapping patches (patches), such as 1×1, 2×2, and 3×3 in the example shown, and unfolds them into one-dimensional tokens. SC folds the soft-segmented overlapping patches according to their original spatial positions to form a new feature map of the same size as the original feature map. However, due to the presence of overlapping areas, SC adds the values ​​of pixels overlapping at the same spatial position. When the soft-synthesized patches are restored to their original positions after Transformer processing, the overlapping positions aggregate information from different tokens, helping to smooth patch boundaries and expand their receptive field by fusing information from adjacent patches.

[0045] The soft segmentation operation embeds the concatenated local and non-local temporal features in overlapping blocks:

[0046]

[0047] Among them, T l Indicates the number of local continuous frames, T nl Indicates the number of selected non-local frames, E nl represents the encoded features of all non-local neighbors, represents the local time fusion feature, SS represents the soft segmentation operation, Z 0 represents the embedded token containing local and non-local temporal information, M×N is the embedded spatial token dimension, where p represents the padding size, k represents the convolution kernel size, s represents the step size, k>s is required, and C0 represents the feature dimension after the soft segmentation operation.

[0048] Furthermore, the FFN formula is as follows:

[0049] FFN(x)=max(0,xW1+b1)W2+b2

[0050] Among them, x represents the features of the multi-head spatiotemporal self-attention output, W1 and W2 represent the dimensions of the two-dimensional matrix, b1 and b2 represent the bias, and max(·) represents the ReLU nonlinear function. That is, FFN is first linearly transformed, then filtered by the nonlinear function ReLU, and then linearly transformed back to the original dimension.

[0051] Furthermore, the multi-head spatiotemporal attention Transformer module is optimized by fusing reconstruction loss, spatiotemporal loss and optical flow loss, wherein the reconstruction loss is used to confirm the difference between the output image of the generated network and the real image; the discriminator of the spatiotemporal loss is used to assist in training the generator of the multi-head spatiotemporal attention Transformer module, so that it can simultaneously focus on the global and local features of all temporal neighbors, thereby obtaining better synthetic realism and temporal consistency.

[0052] Furthermore, the reconstruction loss L rec , using L1 loss to measure the synthetic video The distance between the original video Y is as follows:

[0053]

[0054] The space-time loss L adv , first by using hinge loss as the objective function:

[0055]

[0056] Among them, E(·) represents the expectation function, D(Y) and denote the output of the original video sequence features and the synthetic video sequence features of the spatiotemporal loss discriminator respectively; then the loss function of the generator of the multi-head spatiotemporal attention Transformer module is defined as:

[0057]

[0058] Finally, construct the total loss function:

[0059] L total =λ rec L rec +λ adv L adv +λ flow L flow

[0060] Among them, λ rec ,λ adv and λ flow Represents reconstruction loss, spatiotemporal loss and optical flow loss L respectively flow The weight parameter of .

[0061] The advantages and beneficial effects of the present invention are:

[0062] The present invention reduces labor costs compared to traditional technologies through end-to-end deep learning technology; uses a multi-level encoder to refine features at each level, which is used to propagate spatiotemporal information from known areas outside the hole area in each frame. At the same time, an optical flow completion model and a feature propagation module are added to alleviate the foreground-background aliasing problem through the transmission and update of forward and backward flows; the addition of a multi-head spatiotemporal self-attention module and a fused feedforward neural network is conducive to aggregating information of different features, making full use of the features of local and non-local adjacent frames, and better handling the problem of artifacts. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a flow chart of the method of the present invention.

[0064] Figure 2 It is a specific flow chart of the method in the embodiment of the present invention.

[0065] Figure 3 4 is a flow chart of feature propagation in an embodiment of the present invention.

[0066] Figure 4 This is a flow chart of the soft segmentation operations SS and SC in the fused feedforward neural network F3N in an embodiment of the present invention.

[0067] Figure 5 2 is a flow chart of multi-head spatiotemporal self-attention (MTSA) in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The following describes the specific embodiments of the present invention in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention and are not intended to limit the present invention.

[0069] like Figure 1 、 Figure 2 As shown in FIG, a virtual viewpoint rendering hole filling method based on multi-level feature fusion includes the following steps:

[0070] Step S1: local continuous frames T l and non-local random frame T nl ,The features of each layer are refined through multi-layer coding, and spatiotemporal information is propagated from the known regions outside the hole region in each frame, and local feature blocks and non-local feature blocks are obtained;

[0071] In the embodiment of the present invention, a multi-layer encoder is used, and group convolution is employed at each layer to refine features at each layer, thereby propagating spatiotemporal information from known regions outside the hole region in each frame. Table 1 shows the network structure of the multi-layer encoder.

[0072] First, the input original video sequence is downsampled to 1 / 4 resolution. Then, as the receptive field increases, the first-level features are always cascaded to the feature maps of other layers. Therefore, the first-level feature map is set to This process is described as follows:

[0073]

[0074] Among them, T t Represents the number of video frames, j represents the number of multi-level layers, L represents the total number of multi-level layers, Conv represents the convolution operation with a convolution kernel size of 3×3, LReLU or LeakyReLU represents the activation function, and Concat represents the cascade along the channel dimension. Represents the j-th layer feature map The next layer of feature maps obtained after convolution and activation function processing, express and Feature map after cascading.

[0075] Table 1 Multi-layer encoder network structure

[0076]

[0077] Step S2: For local continuous frames T l Perform optical flow completion, and the obtained optical flow map is combined with the local feature block, and the information of the moving object is transmitted and updated through feature propagation;

[0078] In the embodiment of the present invention, the local block and the optical flow map obtained by the optical flow completion module are input into the feature propagation module to transfer and update the information of the moving object.

[0079] First, an end-to-end optical flow completion model based on the Spatial Pyramid Network (SpyNet) is used. The generated optical flow features are then used as guidance for feature propagation. This method jointly learns optical flow completion and feature propagation. In another embodiment, PWCNet, a recursive optical flow completion model, or the TransFlow model can also be used instead.

[0080] The purpose of the optical flow completion module is to detect moving objects in a virtual viewpoint sequence using information from previous and subsequent frames, thereby guiding feature propagation. This paper uses a deep learning-based optical flow completion model. Table 2 shows the network structure of SpyNet in the optical flow completion model.

[0081] Table 2 SpyNet network structure

[0082]

[0083] Before performing optical flow prediction, the original frame with holes is first The downsampling is performed to 1 / 4 resolution to match the resolution of the multi-level encoder feature map. The downsampled frame is recorded as The optical flow prediction between adjacent frames i and j is done by the optical flow completion model Calculation yields:

[0084]

[0085] in, and express The original frame with holes obtained by downsampling, Represents the optical flow features from the i-th frame to the j-th frame, where SpyNet uses pre-trained weights to initialize the network to utilize its rich information about optical flow.

[0086] Since the missing areas drawn by the virtual viewpoints will affect the quality of optical flow prediction, it is necessary to constrain the forward and backward flows before using them for feature propagation to improve the ability of optical flow prediction. flow Reconstruct bidirectional flow through L1 loss:

[0087]

[0088] Among them, F t→t+1 and F t→t-1 Represent the forward and backward flows of the original sequence frames respectively, and They are the forward flow and backward flow calculated by the optical flow completion model.

[0089] For the forward and backward flow features generated by the optical flow completion model, the present invention transfers and updates the effective features of the hole area of ​​the virtual viewpoint through feature propagation. Since the existence of large motion in non-local frames makes the optical flow prediction accuracy low or even invalid, the present invention only applies the optical flow-based model on the features extracted from local continuous frames. Assume {E t ∈R H / 4×W / 4×C |t=1,...,T l} is a local time-continuous feature block extracted from the multi-level encoder, where T l Indicates the length of local continuous frames. Figure 3 As shown, the forward flow For example, it helps capture the motion of the hole region from the tth frame to the t+1th frame. Once the pixels of the hole region of the tth content feature are known in the valid region of the t+1th feature, the forward flow The t+1th back-propagation feature Draw to the current timeframe and use this useful information intuitively.

[0090] In order to alleviate the problem of irrelevant information during the propagation process, the present invention uses deformable convolution to further index and weight the candidate feature points, so that the sampling points are more inclined to the background features of the moving object. First, calculate the weight mask M t→t+1 and the offset ΔF in the optical flow prediction t→t+1 :

[0091]

[0092] Among them, C b (·) represents multiple cascaded convolutional layers, warp(·) represents the spatial rendering operation based on optical flow, and the weight mask M t→t+1 and offset ΔF t→t+1 are the parameters of the deformable convolution, both of size H / 4×W / 4×K 2 ×G, where K and G represent the size and number of groups of the deformable convolution kernel respectively. In the embodiment of the present invention, K is set to 3 and G is set to 16 based on experience. Add an offset ΔF to t→t+1 , K can be further generated for each spatial position 2 ×G candidate feature points, more flexible sampling positions can well compensate for inaccurate traffic completion and make it easier to find valid content in the surrounding environment.

[0093] Then, using the deformable convolutional layer D b (·) to plot the backpropagation features The features processed by deformable convolution are fused with the current content features Et and propagated through the backpropagation function P b (·) To update:

[0094]

[0095] in, Represents the backward propagation feature of the t-th time length, the propagation function P b (·) are two convolutional layers with one LeakyReLU activation function, and the weight mask M t→t+1 The value of is normalized by the Sigmoid function.

[0096] Finally, a learnable convolutional layer is used to adaptively fuse the forward and backward propagated features:

[0097]

[0098] Among them, I(·) represents a 1×1 convolutional layer, represents the fused features, represents the forward propagation feature, represents the back-propagation feature.

[0099] In another embodiment, a dual-domain propagation module (image propagation and feature propagation) may be used instead of the feature propagation module.

[0100] Step S3: reconstruct the content of the updated features and non-local feature blocks, and then use the decoder to reconstruct the image to obtain a hole repair frame.

[0101] In an embodiment of the present invention, since the hole area after the virtual viewpoint is drawn is large, it is not enough to only use the local time neighboring information provided in the optical flow completion and feature propagation model. The local neighboring hole content may appear in the non-local content, where the non-local content is randomly sampled from the video sequence. Therefore, the information in the non-local time neighbor can be regarded as an important reference for these missing areas in the local neighbor. In order to effectively combine the information from local and non-local time neighbors to reconstruct the content, the present invention stacks multiple multi-head spatiotemporal self-attention Transformer modules and adopts a fusion feedforward neural network (F3N) in the model. The difference between it and the standard feedforward neural network is that F3N inserts SS and SC operations between the two layers of MLP, which is conducive to aggregating information from different tokens and making full use of local and non-local neighboring features.

[0102] Assume T nl is the number of selected non-local frames, represents the encoded features of all non-local neighbors, Denotes the local time fusion feature obtained by formula (7). A soft segmentation operation is used to embed overlapping blocks of the cascaded local and non-local time features:

[0103]

[0104] Among them, SS represents the soft segmentation operation, Z 0 Represents the embedded token containing local and non-local temporal information, M×N is the spatial token dimension of the embedding, and its value is 22×22, where p represents the padding size, k represents the convolution kernel size, s represents the step size, and k>s is required. C0 represents the feature dimension after the soft segmentation operation, and its value is 512.

[0105] SS and SC are implemented in a fusion feed forward network (F3N). F3N consists of a feed forward network (FFN), SS, and SC. FFN is essentially a two-layer MLP. Therefore, the FFN formula is as follows:

[0106] FFN(x)=max(0,xW1+b1)W2+b2 (11)

[0107] Among them, x represents the features of the multi-head spatiotemporal self-attention output, W1 and W2 represent the dimensions of the two-dimensional matrix, b1 and b2 represent the bias, and max(·) represents the ReLU nonlinear function. That is, FFN is first linearly transformed, then filtered by the nonlinear function ReLU, and then linearly transformed back to the original dimension.

[0108] like Figure 4 As shown in the figure, SS and SC operations are inserted into F3N. SS divides each frame into overlapping patches of size k×k, such as 1×1, 2×2, and 3×3 in the example in the figure, and unfolds them into one-dimensional tokens. SC folds the soft-segmented overlapping blocks according to their original spatial positions to form a new feature map of the same size as the original feature map. However, due to the existence of overlapping areas, SC adds the pixel values ​​overlapping at the same spatial position. When the soft-synthesized blocks are restored to their original positions after being processed by the Transformer, the overlapping positions aggregate information from different tokens, which helps to smooth the block boundaries and expand their receptive field by fusing information from adjacent blocks.

[0109] Assume the input token is Z n-1 , where n∈[1,N], N represents the number of stacked Transformer blocks, N=8, then a Transformer block can be expressed as:

[0110] Z′ n =MTSA(LN1(Z n-1 ))+Z n-1 (12)

[0111] Z n =F3N(LN2(Z′ n ))+Z′ n (13)

[0112] Among them, Z′ n represents the intermediate variable value of the input token, MTSA(·) and LN(·) respectively represent the multi-head temporal self-attention and layer normalization in Transformer, the subscript i represents the i-th layer, and F3N is used to connect the connections between embedded tokens to represent the feedforward neural network that integrates soft segmentation operations, such as Figure 5 shown.

[0113] The multi-head spatiotemporal self-attention module is a self-attention module based on window sliding. Therefore, Z n-1 Split into pieces of size s t ×s h ×s w The child window token To calculate the query vector Q of self-attention n and key-value pair vector {K n ,V n}, T l Indicates the number of local continuous frames, T nl represents the number of selected non-local frames, M×N is the embedding space token dimension, and C0 represents the feature dimension after the soft segmentation operation. First, a linear projection layer f is used p Sub-window pooling is performed on the sub-window tokens in space, and the information tokens far away from the query vector are coarse-grainedly focused to obtain coarse-grained sub-window tokens:

[0114]

[0115] Among them, f p (·) represents a linear projection operation.

[0116] Then, the sliding window is represented by the query set of the middle block, that is, the local time information token ( Figure 5local window in ), this local window is not processed by sub-window pooling, and is taken as a query vector for calculation, so it is also a fine-grained attention. Then, Q q , K k and V v are calculated by three linear projection layers f n , f n and f n respectively:

[0117]

[0118] where, and and represent the key vectors of local window and coarse-grained sub-window respectively, and V l n and represent the value vectors of local window and coarse-grained window respectively, f q (·), f k (·) and f v (·) are 1x1 convolution layers, and the combination of local and coarse-grained windows is to calculate the attention of local-global interaction. Therefore, the multi-head spatio-temporal self-attention is calculated according to the above conditions:

[0119]

[0120] where, Attention represents the multi-head self-attention in Figure 5 , softmax is a normalization function, and T is a transpose symbol.

[0121] In another embodiment, a mask-based sparse self-attention can also be used to replace the multi-head spatio-temporal self-attention in the content reconstruction module.

[0122] The three losses are fused to optimize the multi-head spatio-temporal attention Transformer model:

[0123] (1) Reconstruction Loss

[0124] The reconstruction loss is also called generation loss, which is to confirm the difference between the output image of the generation network and the real image. Therefore, the reconstruction loss L rec is calculated as follows:

[0125]

[0126] (2) Spatio-temporal PatchGAN Loss (T-PatchGAN Loss)

[0127] ​The spatiotemporal PatchGAN loss, also known as the Adversarial Loss, is conducive to generating high-quality and realistic content. The present invention uses a discriminator based on T-PatchGAN to assist in training the Transformer generator, so that the model can focus on the global and local features of all temporal neighbors at the same time, thereby obtaining better synthetic realism and temporal consistency. The discriminator takes real video and synthetic video as input and outputs a scalar range in [0, 1], where 0 represents fake and 1 represents real. T-PatchGAN consists of 6 3D convolutional layers with a convolution kernel size of 3×5×5 and a step size of 1×2×2. The proposed spectral normalization is applied to the generator and discriminator to enhance training stability. In addition, the spatiotemporal loss L adv , first by using Hinge Loss as the objective function:

[0128]

[0129] Among them, E(·) represents the expectation function, D(Y) and They represent the output of the original video sequence features and the synthetic video sequence features of the T-PatchGAN discriminator respectively; then the loss function of the Transformer generator is defined as:

[0130]

[0131] (3) Flow Loss

[0132] The bidirectional flow is reconstructed using the L1 loss as shown in the aforementioned formula (4).

[0133] To sum up, the total loss function expression is:

[0134] L total =λ rec L rec +λ adv L adv +λ flow L flow (twenty two)

[0135] Among them, λ rec ,λ adv and λ flow Represents the reconstruction loss, spatiotemporal PatchGAN loss and optical flow loss L respectively. flow The weight parameter λ is set through experiments rec =1,λ adv =0.01,λ flow =1.

[0136] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A virtual viewpoint rendering hole filling method based on multi-level feature fusion, characterized by The steps include: Step S1: The local continuous frames and non-local random frames are subjected to feature refinement through multi-layer coding, and spatiotemporal information is propagated from the known areas outside the hole area to obtain local feature blocks and non-local feature blocks; Step S2: Perform optical flow completion on local continuous frames to obtain optical flow features of adjacent frames with holes, which are used to detect moving objects in the virtual viewpoint sequence to guide feature propagation. Feature propagation combines the local feature blocks to transfer and update the information of moving objects; the optical flow prediction between adjacent frames i and j is performed by the optical flow completion model Calculation yields: in, and represents the original adjacent frames with holes, Represents the optical flow features from the i-th frame to the j-th frame, Represents the optical flow completion operation; In step S2, the optical flow loss L flow Reconstruct bidirectional flow through L1 loss: Among them, F t→t+1 and F t→t-1 Represent the forward and backward flow features of the original sequence frames, and They are the forward flow and backward flow features calculated by the optical flow completion model, T t Indicates the number of video frames; Forward flow features after optical flow completion Helps capture the motion of the hole region from the t-th frame to the t+1-th frame. Once the pixels of the hole region of the t-th content feature are known in the valid region of the t+1-th feature, the forward flow feature is used The back propagation feature of the t+1th local feature block Draw to the current local feature block for update, and finally obtain the local feature that integrates the forward and backward propagation features; Step S3: reconstruct the content of the updated features and non-local feature blocks, and then decode and reconstruct them to obtain the hole repair frame.

2. The method for filling holes in virtual viewpoint rendering based on multi-level feature fusion according to claim 1, characterized in that: In the multi-layer encoding in step S1, as the receptive field increases, the first layer features are always cascaded to the feature maps of other layers: Among them, T t Represents the number of video frames, j represents the number of multi-level layers, L represents the total number of multi-level layers, Conv represents the convolution operation, LReLU represents the activation function, and Concat represents the cascade along the channel dimension. Represents the j-th layer feature map The next layer of feature maps obtained after convolution and activation function processing, Represents the first-level feature map and Feature map after cascading.

3. The method for filling holes in virtual viewpoint rendering based on multi-level feature fusion according to claim 1, characterized in that: The step S2 specifically includes the following steps: First, calculate the weight mask M t→t+1 and the offset ΔF in the optical flow prediction t→t+1 : Among them, C b (·) represents multiple cascaded convolutional layers, warp(·) represents the spatial rendering operation based on optical flow, and the weight mask M t→t+1 and offset ΔF t→t+1 is the parameter of the deformable convolution, which is completed by Add an offset ΔF to t→t+1 , generate multiple candidate feature points for each spatial location; Then, using the deformable convolutional layer D b (·) to plot the backpropagation features Combine the deformable convolution processed features with the current local features E t Fusion, and through the back propagation function P b (·) To update: in, represents the back propagation feature of the t-th time length, P b (·) represents the propagation function; Finally, the forward and backward propagation features are adaptively fused: Among them, I(·) represents the convolution layer operation, represents the fused local features, represents the forward propagation feature, represents the back-propagation feature.

4. The method for filling holes in virtual viewpoint rendering based on multi-level feature fusion according to claim 1, characterized in that: Multiple multi-head spatiotemporal self-attention modules are stacked, and a single module is represented as: Z′ n =MTSA(LN1(Z n-1 ))+Z n-1 WITH n =F3N(LN2(Z′ n ))+Z′ n Among them, Z n-1 represents the input token, where n∈[1,N], N represents the number of stacks of multi-head spatiotemporal self-attention modules, and Z′ n represents the intermediate variable value of the input token, MTSA(·) represents the multi-head spatiotemporal self-attention module, LN i (·) denotes layer normalization, subscript i denotes the i-th layer, and F3N(·) denotes a feedforward neural network integrating soft segmentation operations; The multi-head spatiotemporal self-attention module is a self-attention module based on window sliding, which n-1 Split into pieces of size s t ×s h ×s w The child window token To calculate the query vector Q of self-attention n and key-value pair vector {K n ,V n }, T l Indicates the number of local continuous frames, T nl represents the number of selected non-local frames, M×N is the dimension of the embedded spatial token, and C0 represents the feature dimension after the soft segmentation operation; First, use the linear projection layer f p Sub-window tokens are spatially pooled, and information tokens far away from the query vector are coarsely focused to obtain coarse-grained sub-window tokens: Among them, f p (·) represents linear projection operation; Then, the query set of the middle block is used to represent the sliding window, that is, the local time information token. This local window is not subjected to sub-window pooling processing and is used as the calculation of the query vector. It is then passed through three convolutional layers f q 、f k and f v To calculate Q n , K n and V n : in, and Represent the key vectors of the local window and the coarse-grained sub-window, V l n and Represent the value vectors of the local window and the coarse-grained window respectively, f q (·),f k (·) and f v (·) is a convolutional layer; Compute multi-head spatiotemporal self-attention: Among them, Attention represents multi-head self-attention, Softmax is the normalization function, and T is the transpose symbol.

5. The method for filling holes in virtual viewpoint rendering based on multi-level feature fusion according to claim 4, characterized in that: In step S3, a fused feedforward neural network F3N is used to connect the connections between the embedded tokens. F3N inserts a soft segmentation SS operation between the two MLP layers of the feedforward neural network FFN, and then folds the overlapping blocks of the soft segmentation according to their original spatial positions to obtain a new feature map. The soft segmentation operation embeds overlapping blocks of the cascaded local and non-local temporal features: Among them, T l Indicates the number of local continuous frames, T nl Indicates the number of selected non-local frames, E nl represents the encoded features of all non-local neighbors, represents the local time fusion feature, SS(·) represents the soft segmentation operation, Z 0 represents the embedded token containing local and non-local temporal information, M×N is the embedded spatial token dimension, where p represents the padding size, k represents the convolution kernel size, and s represents the step size, requiring k>s.

6. The method for filling holes in virtual viewpoint rendering based on multi-level feature fusion according to claim 5, characterized in that: The FFN formula is as follows: FFN(x)=max(0,xW1+b1)W2+b2 Among them, x represents the features of the multi-head spatiotemporal self-attention output, W1 and W2 represent the dimensions of the two-dimensional matrix, b1 and b2 represent the bias, and max(·) represents the nonlinear function. That is, FFN is first linearly transformed, then filtered by the nonlinear function, and then linearly transformed back to the original dimension.

7. The method for filling holes in virtual viewpoint rendering based on multi-level feature fusion according to claim 4, characterized in that: The multi-head spatiotemporal attention module is optimized by fusing reconstruction loss, spatiotemporal loss and optical flow loss. The reconstruction loss is used to confirm the difference between the output image of the generated network and the real image. The discriminator of spatiotemporal loss is used to assist in training the generator of the multi-head spatiotemporal attention module, so that it can simultaneously focus on the global and local features of all temporal neighbors.

8. The method for filling holes in virtual viewpoint rendering based on multi-level feature fusion according to claim 7, characterized in that: The reconstruction loss L rec , using L1 loss to measure the synthetic video The distance between the original video Y is as follows: The space-time loss L adv , first by using hinge loss as the objective function: Among them, E(·) represents the expectation function, D(Y) and Denote the output of the original video sequence features and the synthetic video sequence features of the spatiotemporal loss discriminator respectively; then the loss function of the generator of the multi-head spatiotemporal attention module is defined as: Finally, construct the total loss function: L total =λ rec L rec +λ adv L adv +λ flow L flow Among them, λ rec ,λ adv and λ flow Represents reconstruction loss, spatiotemporal loss and optical flow loss L respectively flow The weight parameter of .

Images