A new view synthesis method, system and device

Abstract

The application relates to a new view synthesis method, system and device, obtains images and camera poses under multiple viewing angles, encodes to generate input labels to be processed; encodes to generate target labels based on target viewing angle poses; inputs the input labels of each viewing angle into an encoder composed of multiple layers of Transformer blocks, and the output of each layer of the Transformer blocks is saved in a key-value cache; inputs the target labels into a decoder for processing, and outputs label representations of target viewing angle images, the decoder has the same number of layers of Transformer blocks as the encoder, performs self-attention processing on input data as a query, uses the key-value cache of the corresponding layer in the encoder as a key and a value, performs cross-attention query operation, and the result is transmitted into a feedforward neural network to obtain a final output; and the label representations of the target viewing angle images are converted into RGB images under the target viewing angle. Compared with the prior art, the application can more accurately learn scene semantics of input views and rendering rules of target views, and generate high-quality new views.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of 3D scene data processing, in particular to a new view synthesis method. BACKGROUND

[0002] Existing new view synthesis methods can be roughly divided into three categories: one is to rely on three-dimensional specific prior or explicit three-dimensional representation, such as PixelNeRF, MVSNeRF, etc. predict volumetric three-dimensional representation; two is 3D large reconstruction model (LRM) relying on three-plane NeRF, 3DGS, etc. existing three-dimensional representation and corresponding rendering equation; three is to try to directly learn a geometry-free rendering function, such as SRT, etc. learn implicit representation through Transformer, but there are problems such as insufficient model capacity and scalability, difficulty in capturing high-frequency details, etc.

[0003] The technical scheme proposed in the existing paper "LVSM: A LARGE VIEW SYNTHESIS MODEL WITH MINIMAL 3D INDUCTIVE BIAS" adopts a full Transformer architecture, and introduces a more scalable decoder-only architecture, directly learning the rendering function and optional scene representation from data, realizing new view synthesis without predefining three-dimensional structure and rendering equation. But this scheme still has the following problems: 1. The efficiency problem of full self-attention mechanism, the complexity increases quadratically with the number of input views. 2. The performance limitation problem of self-attention mechanism, which does not learn specialized representation for different tasks. 3. The flexibility problem of relying on three-dimensional specific prior or explicit three-dimensional representation method. SUMMARY

[0004] The purpose of the present application is to provide a new view synthesis method, system and device to solve the problems of low efficiency, high complexity and insufficient flexibility in the prior art.

[0005] The purpose of the present application can be achieved by the following technical solutions:

[0006] As a first aspect of the present application, a new view synthesis method is provided, comprising the steps of:

[0007] Obtaining images and camera poses under multiple viewing angles, and encoding to generate input labels to be processed; based on the target viewing angle pose, encoding to generate a target label;

[0008] The input label of each viewing angle is input into an encoder composed of multiple Transformer blocks, and the output of each Transformer block of the encoder is saved as a key-value cache as the keyK and value V ;

[0009] The target marker is input into a decoder to output a marker representation of the target view image; the decoder has the same number of Transformer blocks as the encoder, and the Transformer block of the encoder performs self-attention processing on the input data as a query Q , uses the key-value cache of the corresponding Transformer block layer in the corresponding view encoder as a key K and value V , performs cross-attention query operation, and the cross-attention result is transmitted into a feedforward neural network with residual connection to obtain the final output of the Transformer block;

[0010] The marker representation of the target view image is converted into an RGB image under the target view.

[0011] As a preferred technical solution, the encoding generation process of the input marker and the target marker is specifically as follows:

[0012] The camera poses and the target view poses are encoded using the Plucker ray embedding to obtain embedded representations of the camera poses and the target view poses, respectively;

[0013] The image is processed in blocks to obtain a marker representation of the image;

[0014] For the input marker: the marker representation of the image and the Plucker ray embedded representation of the camera pose are spliced in the hidden dimension, and then processed by a multilayer perceptron to obtain the input marker to be processed;

[0015] For the target marker: the embedded representation of the target view pose is input into a multilayer perceptron to obtain the target marker.

[0016] As a preferred technical solution, the Transformer block of the decoder includes a self-attention layer with residual connection and a feedforward neural network layer with residual connection connected in turn;

[0017] In the first layer Transformer block, the input marker is input into the self-attention layer for processing, and then the output result of the self-attention layer is added to the input in a residual connection manner;

[0018] The result of the self-attention layer after residual connection is transmitted into a feedforward neural network with residual connection to obtain the final output of the layer Transformer block;

[0019] The output of the previous Transformer block is taken as the input of the next Transformer block, and the inference operation is performed in this way;

[0020] The output of each Transformer block is saved in the form of key-value cache, which is used as the key K and value V of the corresponding cross-attention layer of the decoder.

[0021] As a preferred technical solution, the Transformer block of the encoder sequentially includes a self-attention layer with a residual connection, a cross-attention layer, and a feedforward neural network layer with a residual connection.

[0022] The target token is taken as the query Q , key K and value V of the first layer Transformer block, and is input to the self-attention layer for processing, and then the output of the self-attention layer is added to the input target token in a residual connection manner.

[0023] The cross-attention layer takes the output of the self-attention layer after the residual connection as the query Q , and uses the key-value cache of the corresponding Transformer block layer in the corresponding view encoder as the key K and value V to perform cross-attention query operation.

[0024] The output of the cross-attention is input to the feedforward neural network with a residual connection to obtain the output of the first layer Transformer block.

[0025] The output of the previous layer Transformer block is taken as the input of the next layer Transformer block, which is processed by the self-attention layer of the next layer Transformer block, and so on, and finally the token representation of the target view image is obtained in the decoding stage.

[0026] As a preferred technical solution, the token representation of the target view image is converted into an RGB image under the target view, which is specifically as follows:

[0027] The token of the target view image is sequentially processed by a linear layer and a sigmoid activation function to obtain a pixel value representation of the image.

[0028] The pixel value representation is rearranged according to the original position of the corresponding image block to restore it to a three-channel RGB image.

[0029] As a second aspect of the present application, a new view synthesis system is provided, which performs the new view synthesis method as described above, and specifically includes:

[0030] The label generation module: obtains images and camera poses under multiple perspectives, encodes to generate input labels to be processed; based on the target perspective pose, encodes to generate target labels;

[0031] The three-dimensional scene understanding and guiding module: the input labels of each perspective are respectively input into the encoder composed of multiple layers of Transformer blocks, and the output of each layer of Transformer blocks of the encoder is saved as the key K and value V of the corresponding Transformer block key

[0032] The target view encoding module: input the target token into the decoder for processing, and output the label representation of the target perspective image; the decoder has the same number of Transformer blocks as the encoder, and the Transformer blocks of the encoder perform self-attention processing on the input data as queries Q , use the key K and value V of the corresponding Transformer block layer in the corresponding perspective encoder as the key

[0033] The target perspective image generation module: converts the label representation of the target perspective image into an RGB image under the target perspective.

[0034] As a preferred technical solution, the Transformer block of the decoder in the three-dimensional scene understanding and guiding module includes a self-attention layer with residual connection and a feedforward neural network layer with residual connection connected in turn;

[0035] In the first layer of the Transformer block, the input label is sent to the self-attention layer for processing, and then the output result of the self-attention layer is added to the input in a residual connection manner;

[0036] The result of the residual connection self-attention layer is transmitted to the feedforward neural network with residual connection to obtain the final output of the layer of the Transformer block;

[0037] The output of the previous Transformer block is taken as the input of the next Transformer block, and inference operation is performed accordingly;

[0038] The output of each layer of the Transformer block is saved in the form of key K and value V.

[0039] As a preferred technical solution, the Transformer block of the encoder in the target view encoding module sequentially includes a self-attention layer with residual connections, a cross-attention layer, and a feedforward neural network layer with residual connections.

[0040] Use the target token as a query for the first-level Transformer block. Q ,key K Sum V The input is fed into the self-attention layer for processing, and then the output of the self-attention layer is added to the input target label using a residual connection method.

[0041] The cross-attention layer uses the output of the self-attention layer after the residual connection as the query. Q The key-value cache of the corresponding Transformer block layer in the corresponding view encoder is used as the key. K Sum V Perform cross-attention query operation;

[0042] The output of cross attention is fed into a feedforward neural network with residual connections to obtain the output of the first Transformer block.

[0043] The output of the previous Transformer block is used as the input of the next Transformer block, which is then processed by the self-attention layer of the next Transformer block. This process continues until the labeled representation of the target view image is obtained in the decoding stage.

[0044] As a third aspect of the present invention, an electronic device is provided, comprising:

[0045] One or more processors;

[0046] Memory, used to store one or more programs;

[0047] When the one or more programs are executed by the one or more processors, the one or more processors implement the new view composition method as described above.

[0048] As a fourth aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the new view synthesis method described above.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] 1) The present application proposes an efficient new view generation technology, constructs a new view generation model of a double-flow architecture, and input encoders and target decoders process input labels and target labels respectively. In the target decoder, the target label is used as a query, and the input label is used as a key and a value for cross-attention calculation, which avoids heterogeneous information sharing parameters, so that the model can more accurately learn the scene semantics of the input view and the rendering rules of the target view, improve the efficiency of the self-attention mechanism, and reduce the complexity to linear growth with the number of input views.

[0051] 2) The method proposed in the present application is suitable for variable view number and has strong generalization: the input self-attention processes each view independently, and the transformation of each view does not interfere with each other, so that the model has better generalization ability to the change of the number of input views during testing, and can flexibly cope with the new view synthesis task of different number of input views.

[0052] 3) The method proposed in the present application can efficiently calculate and store, and support incremental inference: the input encoder processes each input view independently, and the target decoder uses cross-attention to reduce the computational complexity related to the number of input views from quadratic to linear; and the decoupled structure supports KV caching of input view features, so that when a new input view is added, only the new view needs to be processed, and when a new target view is added, the KV cache can be directly reused, the cost of adding a new view is close to a constant, incremental inference is realized, and the inference efficiency of multi-view synthesis is significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 The overall flowchart of the efficient new view synthesis method of the present application.

[0054] Figure 2 The structure diagram of the new view synthesis overall model in the present application.

[0055] Figure 3 The structure diagram of the encoder Transformer block in the present application.

[0056] Figure 4 The structure diagram of the cross-attention layer of the decoder Transformer block in the present application. DETAILED DESCRIPTION

[0057] The present application will be described in detail below in combination with the drawings and specific embodiments. The present embodiment is implemented on the basis of the technical solution of the present application, and detailed implementation and specific operation process are given, but the protection scope of the present application is not limited to the following examples.

[0058] Example 1

[0059] The application provides a new view synthesis method with high efficiency, which is mainly composed of an encoder and a decoder, and the specific solution is as follows: the encoder is responsible for processing the splicing token of the input image and the pose information obtained through preprocessing. Through the self-attention mechanism of the Transformer block, the model learns and has the understanding ability of the three-dimensional scene; in the encoding process, the input tokens of different angles are processed separately to reduce the computational complexity, and the output of each layer is used as the key K and value V , which provides guidance for the rendering synthesis of the decoding stage. The decoder is used for processing the target image pose information label obtained through preprocessing. Each Transformer block of the decoder aggregates scene information through the self-attention mechanism, and learns the rendering synthesis rule of the new view by means of the guidance information of the output of the corresponding layer of the encoder through the cross-attention mechanism.

[0060] Further, the REPA distillation loss function is used in the model training process to normalize and constrain the output of the encoding layer and the decoding layer; in the inference stage of multi-view synthesis, the inference speed can be significantly improved by effectively reusing the calculated key-value cache.

[0061] As shown in Figure 1 , the specific steps of the new view synthesis method proposed by the application are as follows:

[0062] Step 1: as shown in Figure 2 , the input image, camera pose and other information and target view pose and other information are encoded and preprocessed to prepare for subsequent token generation, and the specific steps are as follows:

[0063] Step 1.1: use Plücker ray embedding to encode all camera poses to obtain the embedded representation of the pose.

[0064] Step 1.2: perform patchify processing on the input image to obtain the token representation of the image.

[0065] Step 1.3: for the input token: splice the token representation of the image and the Plücker ray block in the hidden dimension, and then process it through a multilayer perceptron to obtain the form of the input token to be processed. For the target token: input the Plücker ray block into another multilayer perceptron MLP to obtain the form of the target token.

[0066] Step 2: process each view input token through an encoder composed of 12 Transformer blocks, and the Transformer block structure of the encoder is as follows:Figure 3 The output of each layer is also retained, in preparation for the decoding process for the subsequent target label, as follows:

[0067] Step 2.1: In the first layer of the Transformer block, the input label is first sent to the self-attention layer for processing, and then the output of this layer is added to the input using a residual connection.

[0068] Step 2.2: Next, the result processed in step 2.1 is transmitted to the feedforward neural network with a residual connection to obtain the final output of the Transformer block.

[0069] Step 2.3: The output of the previous Transformer block is taken as the input of the next Transformer block, and the inference operation is performed in this way. The output of each Transformer block is saved in the form of key-value cache (kv cache) for the decoding stage as the key (K) and value (V).

[0070] Step 3: The target token is processed by the decoder composed of 12 layers of Transformer blocks to obtain the token representation of the output target perspective image. Each Transformer block contains a self-attention layer, a cross-attention layer, and a feedforward neural network layer. The decoder has only one more cross-attention layer compared to the Transformer block of the encoder, as shown in Figure 4 The specific process is as follows:

[0071] Step 3.1: The target token is taken as the query (Q), key (K), and value (V) of the first layer of the Transformer block, and is input to the self-attention layer for processing. Then, the output of this layer is added to the input target label token using a residual connection.

[0072] Step 3.2: The result obtained in step 3.1 is taken as the query (Q), and the output of the corresponding Transformer block layer in the encoder corresponding to the perspective is taken as the key (K) and value (V) to perform cross-attention query operation.

[0073] Step 3.3: The output result of step 3.2 is transmitted to the feedforward neural network with a residual connection to obtain the final output of the Transformer block.

[0074] Step 3.4: The output of the previous Transformer block is taken as the input of the next Transformer block, and so on, and finally the labeled representation of the target view image is obtained in the decoding stage.

[0075] Step 4: Convert the labeled token representation of the target view image into an RGB image, and the specific process is as follows:

[0076] Step 4.1: Linear layer processing and sigmoid activation function operation are performed on the labeled tokens of the target view image in turn to obtain the pixel value representation of the image.

[0077] Step 4.2: Through the unpatchify operation, the pixel value representation is restored to a three-channel RGB image.

[0078] Embodiment 2

[0079] As another embodiment of the present application, the embodiment also provides a new view synthesis system, which performs the new view synthesis method as described in Embodiment 1, and specifically includes:

[0080] The label generation module: obtains images and camera poses under multiple views, encodes to generate input labels to be processed; and encodes to generate target labels based on the target view pose;

[0081] The three-dimensional scene understanding and guiding module: the input labels of each view are respectively input into an encoder composed of multiple Transformer blocks, and the output of each Transformer block of the encoder is saved as the key K and value V of the corresponding Transformer block key and value

[0082] of the decoder; The target view encoding module: inputs the target token into the decoder for processing, and outputs the labeled representation of the target view image; the decoder has the same number of Transformer blocks as the encoder, and the Transformer blocks of the encoder perform self-attention processing on the input data as the query Q , and the key K and value V of the corresponding Transformer block layer in the corresponding view encoder are used as the key and value

[0083] for cross-attention query operation, and the cross-attention result is further input into a feedforward neural network with residual connection to obtain the final output of the Transformer block; The target view image generation module: converts the labeled representation of the target view image into an RGB image under the target view.

[0084] Further, the Transformer block of the decoder in the three-dimensional scene understanding and guiding module comprises a self-attention layer with a residual connection and a feedforward neural network layer with a residual connection connected in sequence;

[0085] In the first layer Transformer block, the input token is sent into the self-attention layer for processing, and then the output result of the self-attention layer is added to the input by using the residual connection mode;

[0086] The result of the self-attention layer after the residual connection is transmitted into the feedforward neural network with a residual connection to obtain the final output of the layer Transformer block;

[0087] The output of the previous Transformer block is taken as the input of the next Transformer block for inference operation;

[0088] The output of each layer Transformer block is saved in the form of key-value cache, which is used as the key K and value V of the corresponding cross-attention layer of the decoder.

[0089] Further, the Transformer block of the encoder in the target view encoding module comprises a self-attention layer with a residual connection, a cross-attention layer and a feedforward neural network layer with a residual connection connected in sequence;

[0090] The target token is taken as the query Q , key K and value V of the first layer Transformer block, which is input into the self-attention layer for processing, and then the output result of the self-attention layer is added to the input target token by using the residual connection mode;

[0091] The cross-attention layer takes the output result of the self-attention layer after the residual connection as the query Q , and uses the key-value cache of the corresponding Transformer block layer in the corresponding view encoder as the key K and value V to perform cross-attention query operation;

[0092] The output result of the cross-attention is transmitted into the feedforward neural network with a residual connection to obtain the output of the first layer Transformer block;

[0093] The output of the previous layer Transformer block is taken as the input of the next layer Transformer block, which is processed by the self-attention layer of the next layer Transformer block, and the process is repeated until the token representation of the target view image is obtained in the decoding stage.

[0094] Embodiment 3

[0095] As a third aspect of the present application, an electronic device is provided, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above new view synthesis method. In addition to the above processor, memory and interface, any data processing device in which the device is located in the embodiment can also include other hardware according to the actual function of the data processing device, which will not be described here.

[0096] Embodiment 4

[0097] As a fourth aspect of the present application, a computer readable storage medium is also provided, which stores computer instructions, and the instructions are executed by a processor to implement the above new view synthesis method. The computer readable storage medium can be an internal storage unit of any data processing device, such as a hard disk or a memory. The computer readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. Further, the computer readable storage medium can include both the internal storage unit and the external storage device of any data processing device. The computer readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0098] The above functions, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0099] The preferred embodiments of the application have been described above in detail. It should be understood that modifications and variations to the preferred embodiments could be made by those skilled in the art in light of the teachings above. It is therefore contemplated that the application can encompass other variations and modifications that fall within the scope of the claims.

Claims

1. A novel view synthesis method, characterized in that, The method steps include: Acquire images and camera poses from multiple viewpoints, and encode them to generate input labels to be processed; based on the target viewpoint pose, encode and generate target labels; The input markers for each viewpoint are input into an encoder consisting of multiple Transformer blocks. The output of each Transformer block in the encoder is stored as a key-value buffer and used as the key for the corresponding Transformer block in the decoder. K Sum V ; The target label is input into the decoder for processing, and the output is a labeled representation of the target viewpoint image; the decoder has Transformer blocks with the same number of layers as the encoder, and the encoder's Transformer blocks perform self-attention processing on the input data and then use it as a query. Q The key-value cache of the corresponding Transformer block layer in the corresponding view encoder is used as the key. K Sum V The cross-attention query operation is performed, and the cross-attention result is then fed into a feedforward neural network with residual connections to obtain the final output of the Transformer block. Convert the labeled representation of the target view image into an RGB image under the target view.

2. The novel view synthesis method according to claim 1, characterized in that, The specific process for generating the input and target tags is as follows: All camera poses and target view poses are encoded using Plück ray embedding to obtain the embedding representations of camera poses and target view poses, respectively. The image is divided into blocks to obtain a labeled representation of the image; For the input label: the label representation of the image is concatenated with the Plück ray embedding representation of the camera pose on the hidden dimension, and then processed by a multilayer perceptron to obtain the input label to be processed; For target labeling: The embedded representation of the target view pose is input into a multilayer perceptron to obtain the target label.

3. The novel view synthesis method according to claim 1, characterized in that, The Transformer block of the decoder includes a self-attention layer with residual connections and a feedforward neural network layer with residual connections connected in sequence. In the first Transformer block, the input labels are fed into the self-attention layer for processing, and then the output of the self-attention layer is added to the input using a residual connection method. The result of the residual connection and self-attention layer is fed into the feedforward neural network with residual connection to obtain the final output of the Transformer block of that layer; The output of the previous Transformer block is used as the input of the next Transformer block, and inference operations are performed accordingly. The output of each Transformer block is saved as a key-value buffer, which is used by the decoder to store the key K and value V of the corresponding cross-attention layer.

4. The novel view synthesis method according to claim 1, characterized in that, The encoder's Transformer block sequentially includes a self-attention layer with residual connections, a cross-attention layer, and a feedforward neural network layer with residual connections. Use the target token as a query for the first-level Transformer block. Q ,key K Sum V The input is fed into the self-attention layer for processing, and then the output of the self-attention layer is added to the input target label using a residual connection method. The cross-attention layer uses the output of the self-attention layer after the residual connection as the query. Q The key-value cache of the corresponding Transformer block layer in the corresponding view encoder is used as the key. K Sum V Perform cross-attention query operation; The output of cross attention is fed into a feedforward neural network with residual connections to obtain the output of the first Transformer block. The output of the previous Transformer block is used as the input of the next Transformer block, which is then processed by the self-attention layer of the next Transformer block. This process continues until the labeled representation of the target view image is obtained in the decoding stage.

5. The novel view synthesis method according to claim 1, characterized in that, The process of converting the labeled representation of the target view image into an RGB image under the target view is as follows: The tokens of the target view image are sequentially processed by linear layers and activated by the sigmoid function to obtain the pixel value representation of the image; The pixel values ​​are rearranged to their original positions relative to the corresponding image blocks, restoring the image to a three-channel RGB image.

6. A novel view synthesis system, characterized in that, The system executes the new view synthesis method as described in any one of claims 1-5, specifically including: The tag generation module acquires images and camera poses from multiple viewpoints, encodes them to generate input tags to be processed, and generates target tags based on the target viewpoint pose. The 3D scene understanding and guidance module: Input tags for each viewpoint are input into an encoder composed of multiple Transformer blocks. The output of each Transformer block in the encoder is stored as a key-value buffer, serving as the key for the corresponding Transformer block in the decoder. K Sum V ; Target view encoding module: Inputs the target token into the decoder for processing, and outputs a labeled representation of the target view image; the decoder has Transformer blocks with the same number of layers as the encoder, and the encoder's Transformer blocks perform self-attention processing on the input data as the query. Q The key-value cache of the corresponding Transformer block layer in the corresponding view encoder is used as the key. K Sum V The cross-attention query operation is performed, and the cross-attention result is then fed into a feedforward neural network with residual connections to obtain the final output of the Transformer block. Target View Image Generation Module: Converts the labeled representation of the target view image into an RGB image under the target view.

7. A novel view synthesis system according to claim 6, characterized in that, The Transformer block of the decoder in the 3D scene understanding and guidance module includes a self-attention layer with residual connections and a feedforward neural network layer with residual connections connected in sequence. In the first Transformer block, the input labels are fed into the self-attention layer for processing, and then the output of the self-attention layer is added to the input using a residual connection method. The result of the residual connection and self-attention layer is fed into the feedforward neural network with residual connection to obtain the final output of the Transformer block of that layer; The output of the previous Transformer block is used as the input of the next Transformer block, and inference operations are performed accordingly. The output of each Transformer block is saved as a key-value buffer, which is used as the key for the corresponding cross-attention layer in the decoder. K Sum V .

8. A novel view synthesis system according to claim 6, characterized in that, The Transformer block of the encoder in the target view encoding module includes, in sequence, a self-attention layer with residual connections, a cross-attention layer, and a feedforward neural network layer with residual connections. Use the target token as a query for the first-level Transformer block. Q ,key K Sum V The input is fed into the self-attention layer for processing, and then the output of the self-attention layer is added to the input target label using a residual connection method. The cross-attention layer uses the output of the self-attention layer after the residual connection as the query. Q The key-value cache of the corresponding Transformer block layer in the corresponding view encoder is used as the key. K Sum V Perform cross-attention query operation; The output of cross attention is fed into a feedforward neural network with residual connections to obtain the output of the first Transformer block. The output of the previous Transformer block is used as the input of the next Transformer block, which is then processed by the self-attention layer of the next Transformer block. This process continues until the labeled representation of the target view image is obtained in the decoding stage.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the new view composition method as described in any one of claims 1-5.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the new view synthesis method as described in any one of claims 1-5.

Images