A virtual viewpoint video generation method based on spatiotemporal joint Transformer

Abstract

The present invention discloses a virtual viewpoint video generation method based on a spatiotemporal joint Transformer. The generation method uses a spatiotemporal joint Transformer model, which includes an encoder, a multi-head spatiotemporal self-attention module, a background restoration module, a spatiotemporal consistency constraint module and a decoder. The overall loss function L is used to generate a virtual viewpoint video. total :L total =L hole +0.5L valid +0.1(L Fore +L Back ) to train the model; the input of the spatiotemporal joint Transformer model is the virtual viewpoint video to be repaired. If it is a multi-viewpoint generated scene, the input also includes the real viewpoint videos on the left and right sides of the virtual viewpoint. This method can specifically solve the error problem of 3D-warping results of different frames and the inconsistency problem of hole filling content, and effectively reduce the pixel flickering problem when playing virtual viewpoint videos.

Description

Technical Field

[0001] The invention belongs to the field of virtual view generation, and relates to a virtual viewpoint video generation method based on a spatiotemporal joint Transformer. Background Art

[0002] 3DTV, a three-dimensional display technology, allows viewers to feel the depth of a scene and enhance the sense of three-dimensionality by showing multiple similar angles of a scene at the same time. Free viewpoint video (FTV) refers to the generation of views from any viewpoint of a scene, providing viewers with comfortable stereoscopic parallax and smooth motion parallax, giving users a better sense of immersion and freedom. As an important branch of 3DTV, FTV allows users to see stereoscopic scenes from any viewpoint without the help of any auxiliary tools, but this also places higher requirements on the image quality of the generated virtual view.

[0003] In the field of computer vision, the synthesis of images from virtual synthetic perspectives by taking real viewpoint images is called virtual view synthesis technology. The virtual view generation method can make full use of computing resources, thereby reducing the demand for hardware equipment, and can display free viewpoint videos without static and dynamic cameras around the scene. Through the viewpoint generation algorithm, more viewpoints can be generated from a small number of viewpoints. At present, the research on virtual view generation method technology has become the hottest research direction in the field of three-dimensional display.

[0004] Depth Image Based Rendering (DIBR) does not require the establishment of complex models. It calculates the positional relationship of the same pixels between a small number of viewpoints and uses these positional relationships to generate new viewpoint images. This technology mainly has two processes, namely 3D-warping and hole filling. 3D-warping maps the pixel content of the color image of the reference viewpoint into three-dimensional space according to the camera imaging principle, the depth map and the internal and external parameters of the camera, and then maps it to the color view of the virtual viewpoint; due to the different parallax of the foreground and background in the reference image and the virtual image, part of the background content in the virtual view is blocked by the foreground in the reference view, so a hole is formed in the virtual view. The hole filling work is to complete this missing content and obtain a complete virtual viewpoint view. In practical applications, free viewpoint generation generates continuous video content, and current research based on the DIBR algorithm is aimed at the generation process of a single-frame virtual view. Although a good single virtual view is generated, the lack of consideration of the temporal and spatial consistency of the video content leads to the phenomenon of pixel flickering when playing the video, which greatly affects the viewer's perception. There are two main reasons for pixel flickering. The first reason is that in the 3D-warping process, the depth maps of different frames are different, and there are errors in the calculation process of coordinate mapping. The second reason is that the positions of holes in different video frames are different, and the contents filled in the holes are also different.

[0005] At present, some researchers have proposed targeted non-reference image evaluation indicators to address the above-mentioned problems, but they have not solved the problem from the perspective of the viewpoint generation process. The pixel flickering problem in the generation of virtual viewpoint videos based on DIBR requires consideration of both the spatial information of a single-frame image and the temporal consistency information of the entire video sequence. However, the method based on two-dimensional convolution is difficult to take into account both the global and local image features of a single video frame, and the method based on three-dimensional convolution is only for a few continuous video frames and cannot take into account the information consistency of the entire video. In response to this, the present invention proposes a virtual viewpoint video generation method based on a spatiotemporal joint Transformer, in which the attention mechanism is not affected by time and space, and can ensure both the image integrity of a single frame of the video and the temporal consistency of all frames of the video, thereby reducing pixel flickering. Summary of the invention

[0006] Aiming at the pixel flickering problem in virtual video generation, the present invention provides a virtual view video generation method based on spatiotemporal joint Transformer. The present invention is applied to the video generation task based on DIBR algorithm, which can solve the error problem of 3D-warping results of different frames and the inconsistency problem of hole filling content, and effectively reduce the pixel flickering problem when playing virtual viewpoint video.

[0007] The technical solution of the present invention is as follows:

[0008] A virtual viewpoint video generation method based on a spatiotemporal joint Transformer, the generation method uses a spatiotemporal joint Transformer model, the spatiotemporal joint Transformer model includes an encoder, a multi-head spatiotemporal self-attention module, a background restoration module, a spatiotemporal consistency constraint module and a decoder;

[0009] Using the overall loss function L total :L total =L hole +0.5L valid +0.1(L Fore +L Back ), to train the model,

[0010] Among them, L hole is the hole area loss, L valid is the loss function of the non-hole area, L Fore is the consistency loss function of the foreground area, L Back is the consistency loss function of the background area;

[0011] The input of the spatiotemporal joint Transformer model is the virtual viewpoint video to be restored. If it is a multi-viewpoint generated scene, the input also includes the real viewpoint videos on the left and right sides of the virtual viewpoint.

[0012] The specific implementation steps of the generation method are:

[0013] The first step is to generate masks of the foreground area, background area, and hole area of ​​the virtual viewpoint video.

[0014] The K-means algorithm is used to perform binary classification on the depth map of the reference video to obtain the masks of the foreground and background. According to the internal and external parameters of the camera and the depth map of the original viewpoint, the spatial coordinate transformation is performed to obtain the depth map of the new viewpoint, and then the original viewpoint color map of all frames is mapped to the new viewpoint to form a virtual viewpoint video: when the original viewpoint color map is mapped to the new viewpoint, the foreground and background are mapped separately to obtain the mask of the foreground area of ​​a single frame, the mask of the background area, and the mask of the hole area. The foreground and background are mapped separately on the color maps of all video frames to obtain the masks of all foreground, background and hole areas of the virtual viewpoint video;

[0015] The second step is to input the virtual viewpoint video and the corresponding mask from the first step into the encoder for encoding and trimming and arranging them into a sequence form.

[0016] The virtual viewpoint video is encoded by the encoder, and the encoding of each frame is cut into N blocks of the same size, where N = W / wp ×H / h p , w p and h p The width and height of each patch code, W is the width of the video frame, H is the height of the video frame; the length of the virtual viewpoint video, that is, the number of frames required to be repaired is f, and after the required f video frames are cropped, the patches of each frame are encoded from left to right and from top to bottom to form a sequence of length N, and then all the sequences of length N of the video frames are arranged in chronological order to obtain a total sequence of length fN, which is then input into the multi-head spatiotemporal self-attention module;

[0017] The third step is to input the arranged total sequence into the multi-head spatiotemporal self-attention module to calculate the attention matrix

[0018] According to the corresponding mask of the virtual viewpoint video, the patch codes in the arranged total sequence are divided into three categories: foreground patch, background patch and hole patch; the total sequence is subjected to three different 1×1 convolutions to obtain three different sequences Q, K, and V, and each patch code of the three sequences Q, K, and V is converted into a one-dimensional vector of length L, and the similarity is calculated by multiplying the transposed matrices of the two vectors. All similarities constitute the attention matrix S of fN×fN;

[0019] The fourth step is to fill the holes in the background area of ​​the virtual viewpoint video encoding according to the attention matrix

[0020] For the column corresponding to the hole patch in the total sequence in the attention matrix, use the softmax operation to calculate the attention value according to the vector corresponding to the hole patch in the Q sequence and the background patch in the K sequence, and get the attention matrix corresponding to the hole patch;

[0021] After calculating the attention matrix corresponding to the hole patch, the background patch code of the V sequence is multiplied by the corresponding attention value and then summed to obtain the output vector code H corresponding to the hole patch. H, the background patch and the foreground patch are reassembled and spliced ​​into the coding arrangement form of the video frame in the order of cropping to obtain the reassembled and spliced ​​video code, so as to fill the holes in the background area of ​​the virtual viewpoint video code;

[0022] Step 5: Perform spatiotemporal consistency constraints on the video encoding obtained in step 4.

[0023] For each scale of attention head, the patches at the same position in the reassembled spliced ​​video code are arranged into a sequence with the same length of the video f, and the attention values ​​corresponding to the patches in these sequences are found from the attention values ​​of the corresponding positions in S calculated in the third step to obtain the f×f attention matrix R; according to R, each patch in the sequence is constrained to be consistent according to the foreground and background areas.

[0024] according to The consistency update calculation for the background area is as follows:

[0025]

[0026] Where i = an integer from 1 to f, B represents the background area, F represents the foreground area, and r i,j is the element in the attention matrix R of f×f, p represents the block, represents the i-th background block, represents the updated block of the i-th background block;

[0027] The consistency loss function L of the background area Back The formula is:

[0028]

[0029] The consistency update calculation formula for the foreground area is:

[0030]

[0031] in,

[0032]

[0033] Consistency loss function L for the foreground area Fore The formula is:

[0034]

[0035] in, represents the updated block of the i-th foreground block, represents the i-th foreground block;

[0036] Step 6: Input the reassembled video code into the decoder to obtain the repaired virtual viewpoint video

[0037] Each frame of the virtual viewpoint with consistency constraints obtained by attention heads of different scales is superimposed together by channel, and the superimposed result is input into the decoder after passing through a 3×3 convolution block to output the repaired virtual viewpoint video.

[0038] A computer-readable storage medium stores a computer program, wherein the program can implement the steps of the generation method when executed by a processor.

[0039] The present invention is a virtual view video generation method based on a spatiotemporal joint Transformer. The K-means clustering algorithm is used to cluster the depth map to obtain the foreground and background masks of the reference video, and then the foreground and background are separately mapped through the 3D-warping process to obtain the foreground, background and hole area masks of the preliminary virtual viewpoint video. Through the multi-head spatiotemporal self-attention module, a complete video background is generated, and the foreground and background videos are constrained in spatiotemporal consistency to generate the final virtual viewpoint video, which effectively reduces the pixel flickering problem when the virtual viewpoint video is played. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Schematic diagram of the process of mask acquisition.

[0041] Figure 2 Schematic diagram of the overall structure of the spatiotemporal joint Transformer model.

[0042] Figure 3 Schematic diagram of the process of spatiotemporal consistency constraints.

[0043] Figure 4 Schematic diagram of the decoder and encoder structure. DETAILED DESCRIPTION

[0044] In order to make the technical solution of the present invention clearer, the present invention is further described below in conjunction with the accompanying drawings.

[0045] The virtual viewpoint video generation method based on the spatiotemporal joint Transformer of the present invention is specifically implemented according to the following steps:

[0046] The first step is to generate masks of the foreground area, background area, and hole area of ​​the virtual viewpoint video.

[0047] The K-means algorithm is used to perform binary classification on the depth map of the reference video (original viewpoint) to obtain the foreground and background masks. Based on the internal and external parameters of the camera and the depth map of the original viewpoint, the spatial coordinate transformation is performed to obtain the depth map of the new viewpoint, and then the original viewpoint color map of all frames is mapped to the new viewpoint to form a virtual viewpoint video: when the original viewpoint color map is mapped to the new viewpoint, the foreground and background are mapped separately to obtain the mask of the foreground area of ​​a single frame, the mask of the background area, and the mask of the hole area. The foreground and background are mapped separately on the color map of all video frames to obtain the masks of all foreground, background, and hole areas of the virtual viewpoint video, such as Figure 1 shown.

[0048] In the second step, the virtual viewpoint video and the corresponding mask are input into the encoder for encoding and trimmed and arranged into a sequence.

[0049] The first step is to use the virtual viewpoint video as the video to be restored. The spatiotemporal joint Transformer model includes a decoder and encoder, a multi-head spatiotemporal self-attention module, a background restoration module, and a spatiotemporal consistency constraint module.

[0050] like Figure 4 As shown, the decoder and encoder are both convolutional neural networks, both including module 1, module 2, and module 3. The encoder is composed of one module 1, two modules 2, one module 3, and module 1 connected in sequence; the decoder is composed of one module 1, module 3, two modules 2, and one module 1 connected in sequence; the module 1 is composed of a 3x3 convolution and a Relu activation function, and module 2 is composed of a 3x3 convolution, a BN layer, and a Relu activation function connected in sequence; the module 3 is a residual network module, including a 3x3 convolution, a BN layer, a Relu activation function, a 3x3 convolution, a BN layer, and an Add layer connected in sequence. The input of module 3 and the output of the last BN layer are added in the Add layer, and the output of the Add layer is the output of module 3. The convolution of the decoder and encoder are both 3x3 convolutions, with padding (filling) of 1 and stride (step length) of 1. The size of the extracted feature map is consistent with the original image.

[0051] The input of the spatiotemporal joint Transformer model is the virtual viewpoint video to be restored. If it is a multi-viewpoint generation scene, the input also includes the real viewpoint videos on the left and right sides of the virtual viewpoint.

[0052] The virtual viewpoint video is encoded by the encoder, and each frame of the code is cut into N patches of the same size, where N = W / w p ×H / h p , w p and h p The width and height of each patch code, W is the width of the video frame, and H is the height of the video frame. The length of the virtual viewpoint video, that is, the number of frames required to be repaired is f. After the required f video frames are cropped, the patches of each frame are arranged from left to right and from top to bottom into a sequence of length N. Then, all the sequences of length N of the video frames are arranged in chronological order to obtain a total sequence of length fN, which is then input into the multi-head spatiotemporal self-attention module. For different attention heads, the scale of the input patch code is also different. This representation based on multi-scale patches coding can make full use of local and global information.

[0053] In the third step, the arranged total sequence is input into the multi-head spatiotemporal self-attention module to calculate the attention matrix.

[0054] According to the corresponding masks of the virtual viewpoint video, the patch encodings in the arranged total sequence are divided into three categories: foreground patches, background patches and hole patches.

[0055] Taking one scale as an example, the total sequence is input into the multi-head spatiotemporal self-attention module and three different 1×1 convolutions are performed to obtain three different sequences Q, K, and V. Each patch encoding of the Q sequence and the K sequence is converted into a one-dimensional vector of length L. The transposed matrix multiplication of the Q sequence vector and the K sequence vector is used to calculate the similarity, as shown in the following formula:

[0056]

[0057] in and They represent the i-th and j-th vectors of the Q sequence and K sequence respectively (1≤i,j≤fN), and the length of the one-dimensional vector is L=h p × p ×c where c is the number of channels, s i,j That is, it represents the similarity of the two vectors. By calculation, we can get the attention matrix S of fN×fN.

[0058] Step 4: Background restoration module: fill the holes in the background area of ​​the virtual viewpoint video encoding according to the attention matrix.

[0059] For the column corresponding to the hole patch in the total sequence in the attention matrix, perform the following softmax operation to calculate the attention matrix corresponding to the hole patch:

[0060]

[0061] Among them, Ω is the visible area in the background, that is, the background patch, is the hole area in the background, α i,j is the element in the attention matrix corresponding to the hole patch, i is the i-th hole patch, I is the number of hole patches, i=1~I; j is the j-th patch, j=1~fN, all patches (including foreground, background and holes), k represents the element in the K sequence. This formula calculates the attention value based on the vector corresponding to the hole patch in the Q sequence and the background patch in the K sequence, ensuring that the filled information is obtained from the visible background area.

[0062]

[0063] After calculating the attention matrix corresponding to the hole patch, the background patch code of the V sequence is multiplied by the corresponding attention value and then summed to obtain the output vector code H corresponding to the hole patch. H, the background patch and the foreground patch are recombined and spliced ​​together in the order of cropping into the coding arrangement form of the video frame to obtain the recombined and spliced ​​video code, thereby filling the holes in the background area of ​​the virtual viewpoint video code.

[0064] The fifth step is the spatiotemporal consistency constraint module: the spatiotemporal consistency constraints are performed on the reassembled and spliced ​​video encoding obtained in the fourth step.

[0065] like Figure 3 As shown in the figure, for each scale of attention head, the patches at the same position in the reassembled spliced ​​video code are arranged into a sequence with the same length as the video, and the attention values ​​corresponding to the patches in these sequences are found from the attention values ​​of the corresponding positions in S calculated in the third step to obtain the f×f attention matrix R. According to R, each patch in the sequence is constrained to be consistent according to the foreground and background areas.

[0066] The consistency update calculation for the background area is as follows:

[0067]

[0068] Where i = 1~f, B represents the background area, F represents the foreground area, r i,j is the element in the attention matrix R of f×f, p represents the block, represents the i-th background block, Represents the updated block of the i-th background block.

[0069] The meaning of the above formula is to consider the differences between the backgrounds at the same position in different video frames when performing consistency update. The consistency loss function L of the background area Back The formula is as follows:

[0070] At this time, i and j can be equal;

[0071] In a video, the foreground changes greatly and its position is not fixed, while the background changes little and is relatively fixed. Therefore, the spatiotemporal consistency constraints on the foreground area are different from those on the background. The size of the video interval should be considered. The closer the video interval, the higher the weight, and the farther the interval, the smaller the weight. The consistency update calculation formula for the foreground area is as follows:

[0072]

[0073] in,

[0074]

[0075] Consistency loss function L for the foreground area Fore The formula is as follows:

[0076]

[0077] in, represents the updated block of the i-th foreground block, Represents the i-th foreground block, i = 1 ~ f, r i,j is the element in the f×f attention matrix R.

[0078] In the sixth step, the reassembled and spliced ​​video code is input into the decoder to obtain the repaired virtual viewpoint video.

[0079] Each frame of the virtual viewpoint with consistency constraints obtained by attention heads of different scales is superimposed together by channel, and the superimposed result is input into the decoder after passing through a 3×3 convolution block to output the repaired virtual viewpoint video.

[0080] Step 7: Calculate the loss function and train the model.

[0081] The loss of the entire spatiotemporal joint Transformer model is divided into three parts: hole region loss, non-hole region loss and consistency loss. hole as follows:

[0082]

[0083] Where M represents the mask of the hole area, G represents the ground-truth of the virtual viewpoint video, and O represents the virtual viewpoint video output by the model. The loss function L for the non-hole area is valid as follows:

[0084]

[0085] In summary, the overall loss function L total for:

[0086] L total =L hole +0.5L valid +0.1(L Fore +L Back )

[0087] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A virtual viewpoint video generation method based on spatiotemporal joint Transformer, characterized in that: The generation method uses a spatiotemporal joint Transformer model, which includes an encoder, a multi-head spatiotemporal self-attention module, a background restoration module, a spatiotemporal consistency constraint module and a decoder; Using the overall loss function L total :L total =L hole +0.5L valid +0.1(L Fore +L Back ), to train the model, Among them, L hole is the hole area loss, L valid is the loss function of the non-hole area, L Fore is the consistency loss function of the foreground area, L Back is the consistency loss function of the background area; The input of the spatiotemporal joint Transformer model is the virtual viewpoint video to be restored. If it is a multi-viewpoint generated scene, the input also includes the real viewpoint videos on the left and right sides of the virtual viewpoint. The specific implementation steps of the generation method are: The first step is to generate masks of the foreground area, background area, and hole area of ​​the virtual viewpoint video. The K-means algorithm is used to perform binary classification on the depth map of the reference video to obtain the masks of the foreground and background. According to the internal and external parameters of the camera and the depth map of the original viewpoint, the spatial coordinate transformation is performed to obtain the depth map of the new viewpoint, and then the original viewpoint color map of all frames is mapped to the new viewpoint to form a virtual viewpoint video: when the original viewpoint color map is mapped to the new viewpoint, the foreground and background are mapped separately to obtain the mask of the foreground area of ​​a single frame, the mask of the background area, and the mask of the hole area. The foreground and background are mapped separately on the color maps of all video frames to obtain the masks of all foreground, background and hole areas of the virtual viewpoint video; The second step is to input the virtual viewpoint video and the corresponding mask from the first step into the encoder for encoding and trimming and arranging them into a sequence form. The virtual viewpoint video is encoded by the encoder, and the encoding of each frame is cut into N blocks of the same size, where N = W / w p ×H / h p , w p and h p The width and height of each patch code, W is the width of the video frame, H is the height of the video frame; the length of the virtual viewpoint video, that is, the number of frames required to be repaired is f, and after the required f video frames are cropped, the patches of each frame are encoded from left to right and from top to bottom to form a sequence of length N, and then all the sequences of length N of the video frames are arranged in chronological order to obtain a total sequence of length fN, which is then input into the multi-head spatiotemporal self-attention module; The third step is to input the arranged total sequence into the multi-head spatiotemporal self-attention module to calculate the attention matrix According to the corresponding mask of the virtual viewpoint video, the patch codes in the arranged total sequence are divided into three categories: foreground patch, background patch and hole patch; the total sequence is subjected to three different 1×1 convolutions to obtain three different sequences Q, K, and V, and each patch code of the three sequences Q, K, and V is converted into a one-dimensional vector of length L, and the similarity is calculated by multiplying the transposed matrices of the two vectors. All similarities constitute the attention matrix S of fN×fN; The fourth step is to fill the holes in the background area of ​​the virtual viewpoint video encoding according to the attention matrix For the column corresponding to the hole patch in the total sequence in the attention matrix, use the softmax operation to calculate the attention value according to the vector corresponding to the hole patch in the Q sequence and the background patch in the K sequence, and get the attention matrix corresponding to the hole patch; After calculating the attention matrix corresponding to the hole patch, the background patch code of the V sequence is multiplied by the corresponding attention value and then summed to obtain the output vector code H corresponding to the hole patch. H, the background patch and the foreground patch are reassembled and spliced ​​into the coding arrangement form of the video frame in the order of cropping to obtain the reassembled and spliced ​​video code, so as to fill the holes in the background area of ​​the virtual viewpoint video code; Step 5: Perform spatiotemporal consistency constraints on the video encoding obtained in step 4. For each scale of attention head, the patches at the same position in the reassembled spliced ​​video code are arranged into a sequence with the same length of the video f, and the attention values ​​corresponding to the patches in these sequences are found from the attention values ​​of the corresponding positions in S calculated in the third step to obtain the f×f attention matrix R; according to R, each patch in the sequence is constrained to be consistent according to the foreground and background areas. according to The consistency update calculation for the background area is as follows: Where i = an integer from 1 to f, B represents the background area, F represents the foreground area, and r i，j is the element in the f×f attention matrix R, p represents the block, represents the i-th background block, represents the updated block of the i-th background block; The consistency loss function L of the background area Back The formula is: The consistency update calculation formula for the foreground area is: in, Consistency loss function L for the foreground area Fore The formula is: in, represents the updated block of the i-th foreground block, represents the i-th foreground block; Step 6: Input the reassembled video code into the decoder to obtain the repaired virtual viewpoint video Each frame of the virtual viewpoint with consistency constraints obtained by attention heads of different scales is superimposed together by channel, and the superimposed result is input into the decoder after passing through a 3×3 convolution block to output the repaired virtual viewpoint video.

2. The virtual viewpoint video generation method based on spatiotemporal joint Transformer according to claim 1 is characterized in that: The decoder and encoder both include module 1, module 2, and module 3. The encoder is composed of one module 1, two modules 2, one module 3, and one module 1 connected in sequence; the decoder is composed of one module 1, one module 3, two modules 2, and one module 1 connected in sequence; the module 1 is composed of a 3x3 convolution and a Relu activation function, and module 2 is composed of a 3x3 convolution, a BN layer, and a Relu activation function connected in sequence; the module 3 is a residual network module, including a 3x3 convolution, a BN layer, a Relu activation function, a 3x3 convolution, a BN layer, and an Add layer connected in sequence, the input of module 3 and the output of the last BN layer are added in the Add layer, and the output of the Add layer is the output of module 3; the convolutions are all 3x3 two-dimensional convolutions, with padding of 1 and stride of 1.

3. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the generation method described in claim 1 or 2 can be implemented.

Images