Single input monocular dynamic scene novel view synthesis method based on self-attention mechanism

Abstract

The application discloses a monocular input dynamic scene new view synthesis method based on a self-attention mechanism, and proposes a new paradigm for dynamic scene new view synthesis, namely, using the self-attention mechanism to express the internal correlation of the dynamic scene in the time dimension, so as to use the time global information to complete the new view synthesis of the dynamic scene, and the real and reliable new view synthesis result can be obtained, the constraints of the optical flow and the scene flow are removed, the training of the model is simplified, and the memory overhead and the training time of the model are reduced; the prediction of the scene flow and the prior of the optical flow information are removed, the optimization of the model is simplified while the reliable new view synthesis result is obtained, and the memory overhead is reduced.

Description

Technical Field

[0001] This invention belongs to the field of novel view synthesis, and more specifically, relates to a method for novel view synthesis of monocular input dynamic scenes based on a self-attention mechanism. Background Technology

[0002] With the continuous development of computer vision, novel view synthesis has become one of the fundamental problems in the field. It has attracted much attention because of its wide range of applications. For example, popular VR technology and virtual 3D maps utilize this technology, and it even shows great promise in industries such as sports and film. Therefore, in recent years, many studies on novel view synthesis have emerged in academia.

[0003] Novel viewpoint synthesis of dynamic scenes is a research hotspot in the field of novel view synthesis. The goal of this task is to reconstruct the four-dimensional information of a scene in time and space from input video or images, ultimately achieving a novel view synthesis effect in both time and perspective dimensions. Novel viewpoint synthesis of dynamic scenes can be applied to film production, sports replay, and other applications. It can break through the limitations of observation in a two-dimensional plane, providing a more comprehensive and intuitive scene display.

[0004] Currently, novel viewpoint synthesis algorithms using neural radiation field models to complete dynamic scenes can be broadly categorized into two types: algorithms based on baseline and deformation models, and algorithms based on dynamic and static models. The baseline and deformation model-based method primarily generates a baseline average scene representation of the dynamic scene using a baseline model. For each time point, the deformation model generates the changes from the average scene deformation to the specific scene at each moment, thus achieving the representation of the dynamic scene. The dynamic and static scene-based method uses a static model to reconstruct the static parts of the scene and a dynamic model to reconstruct the dynamic parts. Finally, the results of the two models are fused to obtain the final dynamic scene representation. To achieve good results, both methods require predicting scene flow information to represent scene changes and using prior optical flow to constrain the scene flow. This complicates model optimization and requires excessive memory consumption. Summary of the Invention

[0005] To address the aforementioned shortcomings or improvement needs of existing technologies, this invention provides a method for synthesizing novel views of monocular input dynamic scenes based on a self-attention mechanism. Its aim is to leverage the high performance of the self-attention mechanism to solve the problem of dynamic scene reconstruction, while simultaneously eliminating the prediction of scene flow parameters, simplifying model optimization, and reducing memory overhead.

[0006] To achieve the above objectives, according to a first aspect of the present invention, a method for synthesizing a new view of a monocular input dynamic scene based on a self-attention mechanism is provided, comprising:

[0007] Training phase:

[0008] The synthetic model is trained using a monocular video sequence of a dynamic scene as the training set; the loss function is the mean squared error between the RGB and depth images generated by the synthetic model and the real RGB and depth images; the synthetic model includes a feature extraction network, a position encoding module, a self-attention encoder, an attention decoder, and a view generation module.

[0009] The feature extraction network is used to extract features from a monocular video sequence to obtain a feature map sequence; the position encoding module is used to add each feature map to a 3D tensor of the same size and containing position information to obtain a feature map sequence incorporating position information.

[0010] The self-attention encoder is used to perform a self-attention operation on the feature map sequence with introduced position information to obtain a feature map sequence with introduced global information; the attention decoder is used to perform an attention operation on the result of performing a self-attention operation on the feature map sequence with introduced position information and the interpolated feature map sequence as a query, and the result of performing convolution processing on the feature map sequence with introduced global information as a key and value to obtain an output feature map sequence.

[0011] The view generation module is used to input time information, coordinates of scene spatial points, and view orientation into a dynamic model, input the coordinates of scene spatial points and view orientation into a static model, and fuse and render the output results of the dynamic and static models to obtain an RGB image and a depth image; wherein, the feature vector obtained by global pooling of the output feature map sequence is used as time information; the spatial points are selected from the camera rays of each view orientation of the monocular video sequence; the view orientation is obtained from the camera parameters corresponding to the monocular video sequence;

[0012] Application phase:

[0013] A specific view orientation, scene space point coordinates, and specific time information are input into the dynamic model, and the specific view orientation and scene space point coordinates are input into the static model. The outputs of the dynamic model and the static model are fused and rendered to obtain an RGB image under the specific view orientation and specific time. The specific time information is a feature vector obtained by globally pooling the result of inputting the interpolated feature map under the specific time into the attention decoder for attention operation.

[0014] According to a second aspect of the present invention, a novel view synthesis system for monocular input dynamic scenes based on a self-attention mechanism is provided, comprising: a computer-readable storage medium and a processor;

[0015] The computer-readable storage medium is used to store executable instructions;

[0016] The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in the first aspect.

[0017] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0018] 1. The monocular input dynamic scene new view synthesis method based on self-attention mechanism provided by this invention proposes a new paradigm for dynamic scene new view synthesis, that is, to use self-attention mechanism to express the intrinsic correlation of dynamic scene in the time dimension, thereby using global time information to complete the synthesis of new viewpoints of dynamic scene, which can obtain real and reliable new view synthesis results. At the same time, it removes the constraints of optical flow and scene flow, simplifies model training, and reduces model memory overhead and training time.

[0019] 2. The monocular input dynamic scene new view synthesis method based on self-attention mechanism provided by the present invention removes the prediction of scene flow and the prior of optical flow information, which simplifies the optimization of the model while obtaining reliable new view synthesis results and reduces memory overhead.

[0020] 3. The monocular input dynamic scene new view synthesis method based on self-attention mechanism provided by the present invention uses a neural radiation field model to synthesize new views, which can obtain new viewpoint images with real and reasonable new time and new perspective. Attached Figure Description

[0021] Figure 1 This is a flowchart of a method for synthesizing a new view of a monocular input dynamic scene based on a self-attention mechanism, provided in an embodiment of the present invention.

[0022] Figure 2 This is a feature extraction network structure diagram provided in an embodiment of the present invention;

[0023] Figure 3 This is a diagram of the convolutional self-attention layer network structure provided in an embodiment of the present invention;

[0024] Figure 4 This is a technical flowchart of the self-attention encoder and attention decoder provided in the embodiments of the present invention;

[0025] Figure 5 (a) and (b) in the figure are respectively the neural radiation field network structure diagrams provided in the embodiments of the present invention;

[0026] Figure 6 These are schematic diagrams illustrating three different forms of model output results provided in embodiments of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0028] This invention provides a method for synthesizing new views of a monocular input dynamic scene based on a self-attention mechanism, such as... Figure 1 As shown, it includes:

[0029] Training phase:

[0030] The synthetic model is trained using a monocular video sequence of a dynamic scene as the training set. The loss function is the mean squared error between the RGB and depth images generated by the synthetic model and the real RGB and depth images (the real RGB images are the input monocular video sequence, and the real depth images are obtained by inputting the monocular video sequence into depth prediction models such as Midas and DPT). The synthetic model includes a feature extraction network, a location encoding module, a self-attention encoder, an attention decoder, and a view generation module.

[0031] The feature extraction network is used to extract features from a monocular video sequence to obtain a feature map sequence; the position encoding module is used to add each feature map to a 3D tensor of the same size and containing position information to obtain a feature map sequence incorporating position information.

[0032] Specifically, the feature extraction network is used for monocular video sequence I i Feature map sequence J is obtained by extracting features from each frame of the image. i i = 1, 2, ..., N.

[0033] The position encoding module is used to generate a position encoding module related to J. i 3D tensors of the same size Pos_Map (i) To J i Perform position encoding and J i With Pos_Map (i) The summation yields the feature map sequence Z incorporating location information. i .

[0034] For each feature map J iEach generates a constant 3D tensor Pos_Map of the same size. (i) To represent feature map J i Location information, feature map sequence J i With position encoding Pos_Map (i) The summation yields a feature map incorporating location information, which is then input into the self-attention encoder.

[0035] The self-attention encoder is used to perform a self-attention operation on the feature map sequence with introduced positional information to obtain a feature map sequence with introduced global information; the attention decoder is used to perform an attention operation on the feature map sequence with introduced positional information and the interpolated frame feature map sequence (the interpolated frames are integer frames, i.e., the interpolated frames in the training phase are integer frames) as a query, and the result of convolution processing on the feature map sequence with introduced global information as the key and value, to obtain an output feature map sequence.

[0036] Preferably, the feature map sequence is interpolated to obtain the interpolated feature map sequence.

[0037] Specifically, feature map J is obtained based on the proportional interpolation method. i Interpolated feature maps between frames.

[0038] The self-attention encoder includes multiple convolutional modules, each of which includes a convolutional self-attention layer for focusing Z-axis signals. i Perform self-attention operations to obtain feature maps that incorporate global information.

[0039] The attention decoder includes multiple convolutional modules, each including a convolutional self-attention layer and a convolutional attention layer; the convolutional attention layer is used to apply the convolutional self-attention layer to J. i and J i Interpolated feature map sequences The output after performing the self-attention operation is used as the query, which will then be used to... The results obtained after convolutional processing through convolutional layers Conv_K and Conv_V are used as keys and values, respectively. An attention operation is then performed to obtain the output feature map F. i ;

[0040] The view generation module is used to input time information, spatial point coordinates, and view orientation into a dynamic model, input the spatial point coordinates and view orientation into a static model, and fuse and render the output results of the dynamic and static models to obtain an RGB image and a depth image. The time information is used as the feature vector obtained by global pooling of the output feature map sequence. The spatial points are selected from the camera rays in each view orientation of the monocular video sequence. The view orientation is obtained from the camera parameters corresponding to the input monocular video sequence.

[0041] Specifically, camera light sampling is performed on each view direction of the monocular video sequence, and spatial points are selected on each light ray. The feature vector obtained by average pooling the final feature map is used as temporal information. Two multilayer perceptrons are used to represent the dynamic scene. The spatial point coordinates, view direction, and feature vector representing temporal information are input into the dynamic model, while the spatial point coordinates and view direction are input into the static model. This completes the separate representation of the dynamic and static information of the scene. Finally, the results of the two are fused, and the final rendered image is obtained using volume rendering technology.

[0042] Application phase:

[0043] A specific view orientation, scene space point coordinates, and specific time information are input into the dynamic model, and the specific view orientation and scene space point coordinates are input into the static model. The outputs of the dynamic model and the static model are fused and rendered to obtain an RGB image under the specific view orientation and specific time. The specific time information is a feature vector obtained by globally pooling the result of inputting the interpolated frame feature map under the specific time into the attention decoder for attention operation (i.e., using the interpolated frame feature map under the specific time as the query, and using the result of convolution processing of the feature map sequence that introduces global information as the key and value).

[0044] In practice, the feature map sequence of the feature extraction network and the output of the self-attention encoder can be saved. During application, the feature map sequence of the feature extraction network can be used to obtain interpolated feature maps at a specific time, where the interpolated feature maps at that time are non-integer frames; the output of the self-attention encoder can be used to obtain information at that specific time.

[0045] Understandably, in the application phase, only the attention decoder, static model, and dynamic model were used.

[0046] Preferably, the self-attention encoder includes multiple convolutional modules, each of which includes a convolutional self-attention layer and a two-dimensional convolutional layer; the convolutional self-attention layer is used to process the feature map sequence Z = {Z1, Z2, ..., Z...}N The results obtained after convolution are used as query graph, key graph, and value graph, and feature map Z is calculated. i query graph Q i key map with each feature map similarity by Weights are applied to the value graphs of each feature map. Weighted calculation is performed to obtain The two-dimensional convolutional layer pairs After performing the convolution operation, a sequence of feature maps incorporating global information is obtained.

[0047] Preferably, the feature map sequence Z = {Z1, Z2, ..., Z...} is... N The results obtained after convolution processing through convolutional layers Conv_Q, Conv_K, and Conv_V are used as the query graph, key graph, and value graph, respectively.

[0048] Preferably, the attention decoder includes multiple convolutional modules, each of which includes a convolutional self-attention layer and a convolutional attention layer;

[0049] The convolutional self-attention layer is used to perform self-attention operations on the feature map sequence and the interpolated feature map sequence;

[0050] The convolutional attention layer is used to take the output of the convolution from the attention layer as the query map, and the result of convolving the feature map sequence that introduces global information as the key map and value map, and then perform attention operations to obtain the output feature map sequence.

[0051] Preferably, the results of convolving the feature map sequence with global information through convolutional layers Conv_K and Conv_V are used as the key map and value map, respectively.

[0052] Preferably, both the dynamic model and the static module are based on a multilayer perceptron network.

[0053] In summary, the monocular input dynamic scene new view synthesis method based on self-attention mechanism provided by this invention achieves the synthesis of new views through the establishment and training of a synthesis model. The synthesis model includes a feature extraction network, a self-attention encoder, an attention decoder, and a view generation network. A deep convolutional network is used as the skeleton network to extract features from the input image sequence. For the self-attention encoder, the feature sequence obtained by the feature extraction network is subjected to self-attention operation based on the self-attention mechanism to obtain intermediate features with global information. For the attention decoder, the feature map obtained by the feature extraction network is interpolated to obtain an intermediate time feature map, which is then used as the query and subjected to attention operation with the intermediate features obtained by the self-attention encoder to obtain the feature representation of the target intermediate time. For the view generation network, referring to the neural radiation field model, a multilayer perceptron network is used with the features obtained by the attention decoder as the driving force to obtain the implicit scene representation of the target time point. Finally, a new viewpoint image is obtained through volume rendering technology. The mean squared error loss function of RGB image and depth map is used to optimize the entire model network.

[0054] The method provided by this invention will be further illustrated below with a specific example.

[0055] To achieve the above objectives, this invention provides a method for synthesizing novel viewpoints in monocular input dynamic scenes based on a self-attention mechanism, comprising:

[0056] (1) Input a monocular video sequence of a dynamic scene, and use ResNet-50 as the skeleton network shared by each frame of the input image to extract features from the input image sequence, obtaining a feature map sequence downsampled by 32 times, including:

[0057] Simultaneously, features from the input image sequence are extracted. The model takes as input a sequence of monocular video images of a dynamic scene, with N frames typically around 20-30. These images are simultaneously input into a feature extraction network, where each original image yields a corresponding feature map. The feature extraction network uses a pre-trained ResNet-50 for the input image sequence. Feature map sequences are obtained through a feature extraction network. Where i∈[1,N].

[0058] A CNN network is used to extract features from the input video image sequence. The data volume and features, as well as the structure of the CNN, are described below:

[0059] In this example, the Nvidia Dynamic Scenes Dataset is used to train the synthetic model. This dataset consists of eight dynamic scenes, each captured by a camera array of 12 cameras, resulting in 24 frames. During training, each frame is taken from only one of the 12 viewpoints to simulate a monocular video effect. The final input to the network model is a 24-frame video sequence. The scenes selected in the Nvidia Dynamic Scenes Dataset are all dynamic, including human actions, scene changes, and the capture of moving objects. During training, the network generates all predictions for the input images and calculates the loss by comparing them with the ground truth values ​​from the RGB and depth images. During actual training, the images are compressed to a size of 512×288.

[0060] A ResNet-50 backbone network is used to extract features from the input image. It consists of Conv1, MaxPool, Conv2-1, Conv2-2, Conv2-3, Conv3-1, Conv3-2, Conv3-3, Conv3-4, Conv4-1, Conv4-2, Conv4-3, Conv4-4, Conv4-5, Conv4-6, Conv5-1, Conv5-2, and Conv5-3. Each Conv layer is followed by a BatchNormalization (BN) layer. This network yields feature maps with 2048 channels and a downsampling factor of 32 (2048-d, 32-r). The CNN network is initialized using weights pre-trained on ImageNet. Its network structure is as follows: Figure 2 As shown.

[0061] (2) For the self-attention encoder, the feature map sequence obtained from the feature extraction network is numerically summed with the positional encoding and then passed through 7 identical convolutional modules. Each module includes a convolutional self-attention layer and a 2D convolutional layer. The final result is a feature map sequence with global information, the same size as the feature map sequence obtained from the feature extraction network, including:

[0062] (2-1) Generate positional encoding. For each feature map J i Each generates a constant 3D tensor Pos_Map of the same size. (i) This tensor is used to represent the positional information of each feature map within the sequence. It is represented using the sin and cosine functions:

[0063] Pos_Map (p，(i，j，2k)) =sin(p / 10000) 2k / D )

[0064] Pos_Map (p，(i，j，2k+1)) =cos(p / 10000) 2k / D )

[0065] Where p∈[1,N] represents the time point position, (i,j) represents the spatial position of the feature and the feature dimension is represented by 2k, and D represents the feature dimension obtained by the feature extraction network (which is 2048). Then, the feature map sequence J... i With Position Encoding Pos-Map (i) The input results of the self-attention encoder are obtained by summing them:

[0066]

[0067] in Z represents the sum of the element dimensions. i The size is

[0068] (2-2) Convolutional Self-Attention Layer. After generating a feature sequence containing positional encodings, a self-attention mechanism can be used to incorporate global information. Specifically, given an input feature map sequence... First, a convolutional network is used to generate the key, query, and value graph for each frame:

[0069]

[0070] Taking the i-th frame as an example, after obtaining Z i Frame corresponding Then, the query graph of the i-th frame can be used. key graph for each frame Calculate similarity In obtaining All corresponding similarity graphs In the future The feature dimension is concatenated to obtain And after passing through the softmax activation function As weights and Perform a weighted summation together: Among them here It means The j-th element in the feature dimension.

[0071] The specific network structure of the convolutional self-attention layer is as follows: Figure 3 As shown, given an input feature map sequence The key, query, and value are obtained by passing the data through convolutional layers Conv_K, Conv_Q, and Conv_V, respectively. The convolution kernel size is 1×1, and the output size is the same as the input size. Taking the i-th frame as an example, the i-th frame... With each frame The similarity is calculated by first... and The tensor is concatenated along the feature dimension, and then a convolutional layer Conv_attn is used to transform the tensor into a one-dimensional scalar:

[0072]

[0073] In obtaining With each frame of image similarity value Then, the softmax function is used to transform these values ​​into weights that sum to 1, thereby guiding... Weighted summation:

[0074]

[0075]

[0076] (2-3) 2D Convolutional Layer. After the feature map passes through the convolutional self-attention layer, it goes through a 2D convolutional layer. This layer is a simple 1×1 2D convolution and does not change the size of the input feature map. This layer consists of three layers: first, a 1×1 convolutional layer conv1 doubles the feature dimension; then, a LeakyReLU layer is used for activation; finally, a 1×1 convolutional layer conv2 restores the dimension to match the input. The final output does not change the size of the input feature map. The specific network structure is as follows: Figure 4 As shown.

[0077] (2-4) Encoder composed of multiple modules. The self-attention encoder consists of 7 identical modules, each containing a convolutional self-attention layer and a 2D convolutional layer. After each layer, a LayerNorm layer is used for normalization and residual connections are made. The size of the feature map does not change after the encoder, and the output feature map... A schematic diagram of the encoder structure is shown below. Figure 4 As shown in the left figure.

[0078] (3) The attention decoder consists of two identical modules, each including a convolutional self-attention layer, a convolutional attention layer, and a 2D convolutional layer. The feature map obtained from the feature extraction network is interpolated to obtain an intermediate time feature map, which is then used as the query and subjected to attention operations with the intermediate features obtained from the self-attention encoder to obtain the feature representation of the target intermediate time, including:

[0079] (3-1) Generate a new time series. For the input sequence of the attention decoder... Given a feature map at a new time point, to obtain the feature map representation at intermediate times, a proportional interpolation method is directly used. Specifically, to obtain the image of the i-th frame at a known time point, the corresponding... Take the J obtained from the corresponding feature extraction network i If you want to obtain I i and I i+1 Interpolated image I in the middle i+0.5 Then the corresponding Take J i With J i+0.5 The average value.

[0080] (3-2) Convolutional self-attention layer. This generates a new time series. The result is added to the positional encoding and then fed into the convolutional self-attention layer. This convolutional self-attention layer has the same structure as the convolutional self-attention layer in the self-attention encoder.

[0081] (3-3) Convolutional attention layer. The output of the convolutional self-attention layer is used as the query, and the output of the self-attention encoder is used as the query. The results obtained from the 1×1 convolutional layers Conv_K and Conv_V are used as keys and values ​​for attention operations. The specific calculation process is the same as that of the convolutional self-attention layer.

[0082] (3-4) Decoder composed of multiple modules. The attention decoder consists of two identical modules, each containing a convolutional self-attention layer, a convolutional attention layer, and a 2D convolutional layer. After each layer, a LayerNorm layer is used for normalization and residual connections are made. After the decoder, the feature map of the target time point is obtained, with the same size as the input. The decoder output can be represented as... A schematic diagram of the specific decoder structure is shown below. Figure 4 As shown in the figure on the right.

[0083] (4) For the neural radiation field network, using the neural radiation field model and volume rendering technology, and driven by the features obtained from the attention decoder, the implicit scene representation of the target time point is obtained and a new viewpoint image is synthesized, including:

[0084] (4-1) Feature compression. The feature map output by the attention decoder is compressed... After being transformed into feature vectors by the global pooling layer.

[0085] (4-2) Multilayer Perceptron Model. Referring to the neural radiation field model, two multilayer perceptron networks are used to model the dynamic and static parts of the scene separately. The dynamic model is constrained not only by the spatial point coordinates x and the view direction d, but also by the feature vector z. i The final output will contain the corresponding color value and volume density.

[0086]

[0087] The static model depends only on the spatial point coordinates and the view orientation, and outputs color values, volume density, and blending weights v.

[0088]

[0089] To enable the model to better represent high-frequency information of the scene, a position encoding function is used to increase the dimensionality of the input, given the spatial point coordinates x and the view direction d:

[0090] γ(p)=(sin(2 0 πp), cos(2 0 πp), ..., sin(2 L-1 πp), cos(2 L-1 πp))

[0091] Where the spatial point coordinate x is set to L = 10; and the view direction d is set to L = 4. The specific MLP network structure is as follows: Figure 5 As shown in (a) and (b) in the figure, Figure 5 (a) and (b) in the diagram are the structure diagrams of the static model and the dynamic model, respectively.

[0092] (4-3) Volume rendering generates images. The results of static and dynamic models are used to render images from the target viewpoint or at a specific time. Traditional volume rendering techniques are used:

[0093]

[0094]

[0095] in This represents the cumulative transmittance. In actual calculations, the integral is discretized into a summation operation, and the final RGB image and depth image are obtained using this summation:

[0096] RGB image:

[0097]

[0098] Depth image:

[0099]

[0100] (5) During training, the mean squared error loss function of RGB images and depth maps is used to optimize the entire model network, including:

[0101] (5-1) RGB Image Optimization. Minimize the mean squared error loss between the synthesized RGB image and the real RGB image:

[0102]

[0103] (5-2) Depth Image Optimization. Minimize the mean squared error loss between the synthesized depth image and the real depth image:

[0104]

[0105] (5-3) Multi-task joint optimization. The model is an end-to-end network that simultaneously optimizes multiple loss functions to update the overall network parameters:

[0106]

[0107] Where λ is a balance factor, which is set to 0.04 in this example.

[0108] The training details of the entire network are as follows: The entire network is trained end-to-end and optimized using the Adam optimizer. The initial learning rate is set to 0.0005, and every 100,000 iterations the learning rate is reduced to one-tenth of the original rate, finally converging after 360,000 iterations.

[0109] (6) In the application phase, the feature maps at the corresponding time points and the new viewpoint direction are input into the neural radiation field model (i.e., the static and dynamic models based on multilayer perceptrons) to obtain the results of the new viewpoint image; for the new time image, the feature maps obtained by the feature extraction network are interpolated and passed through the attention decoder and the neural radiation field model to obtain the results. The final visualization results of the testing process are as follows: Figure 6 As shown, the first row represents the output result with fixed time and interpolated viewpoint; the second row represents the output result with fixed viewpoint and interpolated time; and the third row represents the output result with simultaneous interpolation of time and viewpoint.

[0110] This invention provides a monocular input dynamic scene novel view synthesis system based on a self-attention mechanism, comprising: a computer-readable storage medium and a processor;

[0111] The computer-readable storage medium is used to store executable instructions;

[0112] The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in any of the above embodiments.

[0113] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A monocular input dynamic scene novel view synthesis method based on a self-attention mechanism, characterized in that, The application relates to a method for generating RGB images and depth images from monocular video sequences. The training phase comprises the following steps: The monocular video sequence is used as a training set to train a synthesis model; the mean square deviation of the RGB image and the depth image generated by the synthesis model and the real RGB image and the depth image is used as a loss function; the synthesis model comprises a feature extraction network, a position coding module, a self-attention encoder, an attention decoder and a view generation module; The feature extraction network is used to extract features of the monocular video sequence to obtain a feature map sequence; the position coding module is used to add each feature map to a 3D tensor with the same size and with position information to obtain a feature map sequence with introduced position information; The self-attention encoder is used to perform self-attention operation on the feature map sequence with introduced position information to obtain a feature map sequence with introduced global information; the attention decoder is used to perform attention operation on the feature map sequence with introduced global information by taking the result of self-attention operation on the feature map sequence with introduced position information and the inserted feature map sequence as query and taking the result of convolution processing on the feature map sequence with introduced global information as key and value, to obtain an output feature map sequence; The view generation module is used to input time information, coordinates of a scene space point and a view direction into a dynamic model, input the coordinates of the scene space point and the view direction into a static model, and fuse and render the output results of the dynamic model and the static model to obtain an RGB image and a depth image; the feature vector obtained by performing global pooling on the output feature map sequence is used as the time information; the space point is selected from the camera light of each view direction of the monocular video sequence; the view direction is obtained from the camera parameters corresponding to the monocular video sequence; The application phase comprises the following steps: The specific view direction, the coordinates of the scene space point and the specific time information are input into the dynamic model, the specific view direction and the coordinates of the scene space point are input into the static model, and the output results of the dynamic model and the static model are fused and rendered to obtain an RGB image under the specific view direction and the specific time; the feature vector obtained by performing global pooling on the result of attention operation on the inserted feature map under the specific time by using the attention decoder is used as the specific time information.

2. The method of claim 1, wherein, Interpolation operation is performed on the feature map sequence to obtain an inserted feature map sequence.

3. The method of claim 1, wherein, The self-attention encoder includes multiple convolutional modules, each of which includes a convolutional self-attention layer and a two-dimensional convolutional layer; the convolutional self-attention layer is used to process the feature map sequence Z = {Z1, Z2, ..., Z...} N The results obtained after convolution are used as query graph, key graph, and value graph, and feature map Z is calculated. i query graph key map with each feature map similarity by Weights are applied to the value map v of each feature map. j Weighted calculation is performed to obtain The two-dimensional convolutional layer pairs After performing the convolution operation, a sequence of feature maps incorporating global information is obtained.

4. The method of claim 3, wherein, The sequence of feature maps Z = {Z1, Z2, …, Z N The results obtained after the convolution processing of the sequence of feature maps Z = {Z1, Z2, …, Z 5. The method of claim 1 or 3, wherein, The attention decoder comprises a plurality of convolution modules, each convolution module comprises a convolution self-attention layer and a convolution attention layer; The convolution self-attention layer is used to perform self-attention operation on the feature map sequence and the inserted feature map sequence; The convolution attention layer is used to perform attention operation on the result of the convolution self-attention layer as a query map, the result of convolution processing on the feature map sequence with introduced global information as a key map and a value map, to obtain an output feature map sequence.

6. The method of claim 5, wherein, The results of convolution processing on the feature map sequence with introduced global information by using convolution layers Conv_K and Conv_V are used as a key map and a value map.

7. The method of claim 1, wherein, The dynamic model and the static model are both established based on a multi-layer perceptron network.

8. A system for synthesizing new views of dynamic scenes based on monocular input using self-attention mechanisms, the system comprising: a neural network configured to receive a plurality of input images of a dynamic scene, wherein the neural network is configured to generate a plurality of output images of the dynamic scene from a plurality of different viewpoints. Comprise: A computer readable storage medium and a processor; The computer readable storage medium is used to store executable instructions; The processor is used to read the executable instructions stored in the computer readable storage medium, and execute the method as claimed in any one of claims 1-7.

Images