A new viewpoint image synthesis method based on camera pose light field coding

Abstract

The present invention discloses a new viewpoint image synthesis method based on camera pose light field coding, including: designing a two-stage training scene new viewpoint synthesis model; determining a pose light field coding scheme to encode the viewpoint pose into a pose light field of target resolution; providing an effective training strategy; the beneficial effect is: based on the advantages of the Transformer module, the present invention trains a scene potential representation coding model with a large amount of priors and a scene new viewpoint synthesis module. Then, by introducing a pose light field coding, the problem of inaccurate viewpoint control is solved, and the generation speed of traditional pixel-based light rendering is optimized. The present invention introduces a random noise addition process of the target pose in the training, which effectively avoids the overfitting phenomenon of the specific target pose in the second stage training, so that the generalization and generation quality of the generated model are greatly improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a new viewpoint image synthesis method based on camera pose light field coding. Background Art

[0002] New viewpoint synthesis is an important task in the field of computer vision, which aims to generate images from new viewpoints based on single or multiple known viewpoint images through algorithmic means. New viewpoint synthesis has broad application prospects and research significance in many fields, such as scene roaming, virtual reality, 3D object and scene reconstruction, etc.

[0003] Traditional viewpoint synthesis methods are usually based on geometric reconstruction or image interpolation, but they face certain limitations in practical applications. Geometric reconstruction relies on accurate depth estimation or point cloud data, which is difficult to handle for occluded areas and irregular surfaces. Image interpolation methods are prone to blur or distortion, especially in the case of large parallax. These traditional methods are difficult to achieve high-quality and realistic viewpoint synthesis in real scenes. In recent years, viewpoint synthesis methods based on deep learning have made significant progress. They use the powerful representation ability of deep neural networks to directly learn the mapping relationship of viewpoint synthesis from data, making up for the shortcomings of traditional methods.

[0004] The current new viewpoint synthesis algorithms can be mainly divided into three directions: methods based on three-dimensional geometric fields, such as neural radiance field (NeRF) and three-dimensional Gaussian sputtering; methods based on generative models, such as combined diffusion models; methods based on feature extraction and feature fusion, such as using CNN and Transformer.

[0005] Among them, the method based on 3D geometric field has the problem of poor generalization and requires separate training for each scene; while the method based on generative model, especially diffusion model, has the problem of slow generation speed and requires more denoising steps to generate reasonable new viewpoint images. In contrast, the method based on feature extraction and feature fusion has strong generalization and fast generation speed, but it also has the following defects:

[0006] 1. The existing new viewpoint synthesis method based on feature extraction and feature fusion will still introduce additional geometric information, which increases the complexity of the algorithm and also restricts three-dimensional elements such as the parallax of the input viewpoint. The algorithm that does not introduce additional geometric information cannot fully realize the effective representation of scene features.

[0007] 2. The existing new viewpoint synthesis method based on feature extraction and feature fusion still uses a pixel-by-pixel light rendering algorithm to obtain the final target pixel value, which also leads to an increase in the complexity of the algorithm. Summary of the invention

[0008] To solve the above problems, the present invention discloses a new viewpoint image synthesis method based on camera pose light field coding, which utilizes the rich priors of the pre-trained Transformer model and introduces pose light field coding, which greatly improves the generalization ability of the model and greatly enhances the generation speed and controllability of new viewpoint synthesis of the scene.

[0009] To achieve the above object, the technical solution of the present invention is as follows:

[0010] A new viewpoint image synthesis method based on camera pose light field coding comprises the following steps:

[0011] Step 1: Build a multi-scene viewpoint dataset containing images and poses from multiple viewpoints in multiple scenes;

[0012] Step 2: Preprocess the viewpoint data in the data set in step 1, divide it into input viewpoints, training target viewpoints, and test target viewpoints, resize and linearly normalize the viewpoint images, and apply pose light field encoding to the viewpoint poses;

[0013] Step 3: Determine the first-stage learning framework, input the input viewpoint image obtained in step 2 into the encoder part of the VAE autoencoder, and generate compressed image features of the input viewpoint;

[0014] Step 4: Concatenate the compressed image features of the input viewpoint obtained in step 3 with the pose light field of the input viewpoint obtained in step 2, and send them to the scene potential representation encoding module to generate the scene potential representation;

[0015] Step 5: Send the scene potential representation generated in step 4 to the scene potential representation decoding module for decoding to obtain the reconstructed image features of the input viewpoint;

[0016] Step 6: Compare the reconstructed image features of the input viewpoint obtained in step 5 with the compressed image features of the input viewpoint obtained in step 3, calculate the loss value according to the first loss function, and then optimize the loss to the global minimum value, thereby obtaining a trained first-stage scene potential representation encoding module network model;

[0017] Step 7: Determine the second-stage learning framework, freeze the network parameters of the scene potential representation encoding module obtained in step 6, and obtain the scene potential representation according to step 4;

[0018] Step 8: Send the scene potential representation obtained in step 7 and the pose light field of the training target viewpoint obtained in step 2 to the scene new viewpoint synthesis module, and perform random noise processing on the training target viewpoint pose in step 2 only during the training process to generate the synthetic image features of the training target viewpoint;

[0019] Step 9: Input the training target viewpoint image obtained in step 2 into the encoder part of the VAE autoencoder to generate compressed image features of the training target viewpoint;

[0020] Step 10: Compare the synthetic image features of the training target viewpoint obtained in step 8 with the compressed image features of the training target viewpoint obtained in step 9, calculate the loss value according to the second loss function, and then optimize the loss to the global minimum value, and obtain the trained second-stage scene new viewpoint synthesis module network model;

[0021] Step 11: Freeze the network parameters of the scene new viewpoint synthesis module obtained in step 10, obtain a set of scene test target viewpoint data, pre-process the input viewpoint image, input viewpoint pose, and test target viewpoint pose according to step 2, and perform steps 3, 7, and 8 in sequence to obtain the synthetic image features of the test target viewpoint;

[0022] Step 12: Input the synthetic image features of the test target viewpoint obtained in step 11 into the decoding part of the VAE autoencoder to obtain the final synthetic image of the test target viewpoint.

[0023] Furthermore, the preprocessing in step 2 uses a variable-resolution pose light field encoding;

[0024] Variable-resolution pose light field encoding encodes a given camera pose into a feature light field of a target resolution.

[0025] Its expression is as follows:

[0026] ;

[0027] Among them, K is the camera pose intrinsic parameter matrix, R is the camera pose extrinsic parameter rotation matrix, T is the camera pose extrinsic parameter translation matrix; LF is the target feature light field; (o, d) is the channel structure of the target feature light field, where o is the origin of the light field and d is the direction of the light field; H is the length of the target feature light field; W is the width of the target feature light field; C is the number of channels of the target feature light field;

[0028] More specifically,

[0029] ;

[0030] ;

[0031] ;

[0032] ;

[0033] Among them, u and v are the pixel coordinates of the image, L is the concatenation of o and d in the channel dimension, and N is the length of the cosine position code used, and the default value is N=15.

[0034] Furthermore, the first-stage learning framework used in steps 3-6 adopts the STEncoder-Dncoder model architecture, which consists of an image feature compression module and a scene potential representation encoding and decoding module; the image feature compression module uses a pre-trained VAE autoencoder; the scene potential representation encoding and decoding module consists of 8 self-attention STBlocks, and the self-attention STBlock consists of a residual self-attention Transformer block and an MLP; the input image feature Z S The image blocks are divided into multiple 1x1 blocks by the block module, and are input into the first STBlock after image position encoding and viewpoint position encoding. They are passed through the linear layer as Q, K, V, and then through the residual self-attention Transformer block. After passing through the MLP output, they are used as the input of the next layer of self-attention STBlock. After passing through a total of 8 STBlocks, the scene potential representation S is output. r .

[0035] Furthermore, the first loss function used in step 6 is expressed as:

[0036] ;

[0037] ;

[0038] ;

[0039] ;

[0040] in is the input viewpoint image, and Represent the input viewpoint image features and the reconstructed image features respectively, is the camera pose light field of the input viewpoint, It is the potential representation of the scene generated by the encoding module, which will be used as the input of the second-stage scene new viewpoint image generation model.

[0041] Furthermore, the second-stage learning framework used in steps 7-10 adopts the NVSDncoder model architecture, which consists of a scene new view synthesis module and an image feature decoding module; the image feature decoding module uses a VAE autoencoder; the scene new view synthesis module consists of two cross-attention STBlock blocks, and the cross-attention STBlock consists of a residual cross-attention Transformer block and an MLP; the input scene potential representation S rAfter passing through the linear layer, it is converted to K and V, and the pose light field of the target viewpoint is divided into multiple 1x1 image blocks after an image position encoding, and converted to Q after passing through the linear layer. Q, K, and V pass through the residual cross-attention Transformer block and then pass through the MLP output as the scene potential representation S of the next cross-self-attention STBlock r Input, after passing through the second STBlock, directly outputs the image feature block group of the target viewpoint through the final MLP layer, and is restored to the complete image feature through normalization and linear layers.

[0042] Furthermore, the random noise processing used in step 8 is specifically implemented by adding an offset noise to the camera pose matrices K, R, and T of the training target viewpoint. The noise is a randomly sampled Gaussian noise, and its disturbance size is controlled by a coefficient factor to be no more than 0.1% of the standard scale of the scene.

[0043] Furthermore, the second loss function used in step 10 is expressed as:

[0044] ;

[0045] ;

[0046] ;

[0047] in and 'represent the training target viewpoint image features and the generated target viewpoint image features, respectively, is the training target viewpoint image, is the camera pose light field of the target viewpoint, is the scene potential representation generated by the scene potential representation encoding module in the first stage, is the second loss function.

[0048] Furthermore, steps 6 and 10 respectively perform two stages of model training, first training the scene latent representation encoding module and then training the scene new viewpoint synthesis module, rather than training the two models simultaneously for joint optimization.

[0049] The beneficial effects of the present invention are:

[0050] 1. This invention not only utilizes the rich priors of the pre-trained Transformer model, greatly improving the generalization ability of the model, but also introduces a pose light field encoding, which greatly enhances the generation speed and controllability of the synthesis of new viewpoints in the scene. Compared with three-dimensional geometric methods such as neural radiation field, it does not require separate training for a single scene; compared with diffusion model methods such as zero123, it achieves precise control of viewpoint pose; compared with the recent similar methods based on image features, the generation speed is increased by more than 3 times.

[0051] 2. The present invention uses a two-stage Transformer codec architecture to train the scene potential representation encoding module and the scene new viewpoint image generation module in sequence, thereby avoiding the overall training failure caused by the unsatisfactory first stage during joint training.

[0052] 3. The present invention introduces a random noise-adding process for the target posture during training. Through extremely small posture noise disturbance, the overfitting phenomenon of the specific target posture in the second stage training is effectively avoided without affecting the posture control, so that the generalization and generation quality of the generated model are greatly improved, and high-quality and three-dimensional consistent viewpoint images can still be generated at completely unknown viewpoint positions. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 The figure is the overall flow chart of the algorithm of the present invention.

[0054] Figure 2 This is a diagram of the overall architecture of the training algorithm described in the present invention.

[0055] Figure 3 This is a schematic diagram of the structure of the self-attention STBlock module described in the present invention.

[0056] Figure 4 This is a schematic diagram of the cross-attention STBlock module structure described in the present invention.

[0057] Figure 5 This is a diagram of the overall architecture of the inference algorithm described in the present invention. DETAILED DESCRIPTION

[0058] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0059] like Figure 1 As shown, the new viewpoint image synthesis method based on camera pose light field coding described in the present invention comprises the following steps:

[0060] Step 1: Build a multi-scene viewpoint dataset containing images and poses from multiple viewpoints in multiple scenes;

[0061] Step 2: Preprocess the viewpoint data in the data set in step 1, divide it into input viewpoints, training target viewpoints, and test target viewpoints, resize and linearly normalize the viewpoint images, and apply pose light field encoding to the viewpoint poses;

[0062] A variable-resolution pose light field encoding is used for preprocessing;

[0063] Variable-resolution pose light field encoding encodes a given camera pose into a feature light field of a target resolution.

[0064] Its expression is as follows:

[0065] ;

[0066] Among them, K is the camera pose intrinsic parameter matrix, R is the camera pose extrinsic parameter rotation matrix, T is the camera pose extrinsic parameter translation matrix; LF is the target feature light field; (o, d) is the channel structure of the target feature light field, where o is the origin of the light field and d is the direction of the light field; H is the length of the target feature light field; W is the width of the target feature light field; C is the number of channels of the target feature light field;

[0067] More specifically,

[0068] ;

[0069] ;

[0070] ;

[0071] ;

[0072] Among them, u and v are the pixel coordinates of the image, L is the concatenation of o and d in the channel dimension, and N is the length of the cosine position code used, and the default value is N=15.

[0073] Step 3: According to Figure 2 The training algorithm framework diagram shown in the figure determines the first stage learning framework, inputs the input viewpoint image obtained in step 2 into the encoder part of the VAE autoencoder, and generates the compressed image features of the input viewpoint;

[0074] The first stage learning framework adopted adopts the STEncoder-Dncoder model architecture, which consists of an image feature compression module and a scene potential representation encoder-decoder module. The image feature compression module uses a pre-trained VAE autoencoder. The scene potential representation encoder-decoder module consists of 8 self-attention STBlocks, which are composed of residual self-attention Transformer blocks (such as Swin Transformer, Twins Transformer, etc.) and MLP. The detailed model structure is as follows Figure 3 As shown. Input image feature Z S The image blocks are divided into multiple 1x1 blocks by the block module, and are input into the first STBlock after image position encoding and viewpoint position encoding. They are passed through the linear layer as Q, K, V, and then through the residual self-attention Transformer block. After passing through the MLP output, they are used as the input of the next layer of self-attention STBlock. After passing through a total of 8 STBlocks, the scene potential representation S is output. r .

[0075] Step 4: Concatenate the compressed image features of the input viewpoint obtained in step 3 with the pose light field of the input viewpoint obtained in step 2, and send them to the scene potential representation encoding module to generate the scene potential representation;

[0076] Step 5: Send the scene potential representation generated in step 4 to the scene potential representation decoding module for decoding to obtain the reconstructed image features of the input viewpoint;

[0077] Step 6: Compare the reconstructed image features of the input viewpoint obtained in step 5 with the compressed image features of the input viewpoint obtained in step 3, calculate the loss value according to the first loss function, and then optimize the loss to the global minimum value, thereby obtaining a trained first-stage scene potential representation encoding module network model;

[0078] The first loss function adopted is expressed as:

[0079] ;

[0080] ;

[0081] ;

[0082] ;

[0083] in is the input viewpoint image, and Represent the input viewpoint image features and the reconstructed image features respectively, is the camera pose light field of the input viewpoint, It is the potential representation of the scene generated by the encoding module, which will be used as the input of the second-stage scene new viewpoint image generation model.

[0084] Step 7: According to Figure 2 The training algorithm framework diagram shown in the figure determines the second stage learning framework, freezes the network parameters of the scene potential representation encoding module obtained in step 6, and obtains the scene potential representation according to step 4;

[0085] The second-stage learning framework adopted adopts the NVSDncoder model architecture, which consists of a scene new view synthesis module and an image feature decoding module. The image feature decoding module uses a VAE autoencoder. The scene new view synthesis module consists of two cross-attention STBlock blocks, and the cross-attention STBlock consists of a residual cross-attention Transformer block and an MLP. The detailed model structure is as follows Figure 4 As shown. The input scene potential representation S r After passing through the linear layer, it is converted to K and V, and the pose light field of the target viewpoint is divided into multiple 1x1 image blocks after an image position encoding, and converted to Q after passing through the linear layer. Q, K, and V pass through the residual cross-attention Transformer block and then pass through the MLP output as the scene potential representation S of the next cross-self-attention STBlock r Input, after passing through the second STBlock, directly outputs the image feature block group of the target viewpoint through the final MLP layer, and is restored to the complete image feature through normalization and linear layers.

[0086] Step 8: Send the scene potential representation obtained in step 7 and the pose light field of the training target viewpoint obtained in step 2 to the scene new viewpoint synthesis module, and perform random noise processing on the training target viewpoint pose in step 2 only during the training process to generate the synthetic image features of the training target viewpoint;

[0087] The random noise processing used is specifically implemented by adding an offset noise to the camera pose matrices K, R, and T of the training target viewpoint. The noise is a randomly sampled Gaussian noise, and its disturbance size is controlled by a coefficient factor to be no higher than 0.1% of the standard scale of the scene.

[0088] Step 9: Input the training target viewpoint image obtained in step 2 into the encoder part of the VAE autoencoder to generate compressed image features of the training target viewpoint;

[0089] Step 10: Compare the synthetic image features of the training target viewpoint obtained in step 8 with the compressed image features of the training target viewpoint obtained in step 9, calculate the loss value according to the second loss function, and then optimize the loss to the global minimum value, and obtain the trained second-stage scene new viewpoint synthesis module network model;

[0090] The second loss function adopted is expressed as:

[0091] ;

[0092] ;

[0093] ;

[0094] in and 'represent the training target viewpoint image features and the generated target viewpoint image features, respectively, is the training target viewpoint image, is the camera pose light field of the target viewpoint, is the scene potential representation generated by the scene potential representation encoding module in the first stage, is the second loss function.

[0095] Among them, steps 6 and 10 respectively perform two stages of model training, first training the scene latent representation encoding module, and then training the scene new viewpoint synthesis module, rather than jointly optimizing the two models and training them simultaneously.

[0096] Step 11: According to Figure 5 The inference algorithm framework diagram shown in the figure determines the model inference framework, freezes the network parameters of the scene new viewpoint synthesis module obtained in step 10, obtains a set of scene test target viewpoint data, pre-processes the input viewpoint image, input viewpoint pose, and test target viewpoint pose according to step 2, and performs steps 3, 7, and 8 in sequence to obtain the synthetic image features of the test target viewpoint;

[0097] Step 12: Input the synthetic image features of the test target viewpoint obtained in step 11 into the decoding part of the VAE autoencoder to obtain the final synthetic image of the test target viewpoint.

[0098] Based on the advantages of the Transformer module, the present invention designs a two-stage scene new viewpoint synthesis model, which includes a scene potential representation coding model with a large amount of priors and a scene new viewpoint synthesis module.

[0099] By introducing a pose light field encoding, the problem of inaccurate viewpoint control is solved and the generation speed of traditional pixel-ray rendering is optimized.

[0100] The present invention introduces a random noise adding process of the target posture in the training, which effectively avoids the overfitting phenomenon of the specific target posture in the second stage training, so that the generalization and generation quality of the generated model are greatly improved.

[0101] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A new viewpoint image synthesis method based on camera pose light field coding, characterized by: The steps include: Step 1: Build a multi-scene viewpoint dataset containing images and poses from multiple viewpoints in multiple scenes; Step 2: Preprocess the viewpoint data in the data set in step 1, divide it into input viewpoints, training target viewpoints, and test target viewpoints, resize and linearly normalize the viewpoint images, and apply pose light field encoding to the viewpoint poses; The preprocessing uses a variable resolution pose light field encoding; Variable-resolution pose light field encoding encodes a given camera pose into a feature light field of target resolution; Its expression is as follows: (K,R,T)→(o,d)∈R H×W×6 →LF∈R H×W×C Among them, K is the camera pose intrinsic parameter matrix, R is the camera pose extrinsic parameter rotation matrix, T is the camera pose extrinsic parameter translation matrix; LF is the target feature light field; (o, d) is the channel structure of the target feature light field, where o is the origin of the light field and d is the direction of the light field; H is the length of the target feature light field; W is the width of the target feature light field; C is the number of channels of the target feature light field; More specifically, o i ∈R H×W×3 =T d i ∈R H×W×3 =R·K -1 ·(u,v,1) L=(o,d) LF=[sin(2 0 πL),cos(2 0 πL),...sin(2 N-1 πL),cos(2 N-1 πL)] Where u, v are the pixel coordinates of the image, L is the concatenation of o, d in the channel dimension, and N is the length of the cosine position code used, with N = 15 by default; Step 3: Determine the first-stage learning framework, input the input viewpoint image obtained in step 2 into the encoder part of the VAE autoencoder, and generate compressed image features of the input viewpoint; Step 4: Concatenate the compressed image features of the input viewpoint obtained in step 3 with the pose light field of the input viewpoint obtained in step 2, and send them to the scene potential representation encoding module to generate the scene potential representation; Step 5: Send the scene potential representation generated in step 4 to the scene potential representation decoding module for decoding to obtain the reconstructed image features of the input viewpoint; Step 6: Compare the reconstructed image features of the input viewpoint obtained in step 5 with the compressed image features of the input viewpoint obtained in step 3, calculate the loss value according to the first loss function, and then optimize the loss to the global minimum value, thereby obtaining a trained first-stage scene potential representation encoding module network model; The first stage learning framework adopts the STEncoder-Dncoder model architecture, which consists of an image feature compression module and a scene potential representation encoding and decoding module; the image feature compression module uses a pre-trained VAE autoencoder; the scene potential representation encoding and decoding module consists of 8 self-attention STBlocks, which are composed of residual self-attention Transformer blocks and MLP; the input image feature Z S The image blocks are divided into multiple 1x1 blocks by the block module, and are input into the first STBlock after image position encoding and viewpoint position encoding. They are passed through the linear layer as Q, K, V, and then through the residual self-attention Transformer block. After passing through the MLP output, they are used as the input of the next layer of self-attention STBlock. After passing through a total of 8 STBlocks, the scene potential representation S is output. r ; Step 7: Determine the second-stage learning framework, freeze the network parameters of the scene potential representation encoding module obtained in step 6, and obtain the scene potential representation according to step 4; Step 8: Send the scene potential representation obtained in step 7 and the pose light field of the training target viewpoint obtained in step 2 to the scene new viewpoint synthesis module, and perform random noise processing on the training target viewpoint pose in step 2 only during the training process to generate the synthetic image features of the training target viewpoint; Step 9: Input the training target viewpoint image obtained in step 2 into the encoder part of the VAE autoencoder to generate compressed image features of the training target viewpoint; Step 10: Compare the synthetic image features of the training target viewpoint obtained in step 8 with the compressed image features of the training target viewpoint obtained in step 9, calculate the loss value according to the second loss function, and then optimize the loss to the global minimum value, and obtain the trained second-stage scene new viewpoint synthesis module network model; The second stage learning framework adopts the NVSDncoder model architecture, which consists of a scene new view synthesis module and an image feature decoding module; the image feature decoding module uses a VAE autoencoder; the scene new view synthesis module consists of two cross-attention STBlock blocks, and the cross-attention STBlock consists of a residual cross-attention Transformer block and an MLP; the input scene potential representation S r After passing through the linear layer, it is converted to K and V, and the pose light field of the target viewpoint is divided into multiple 1x1 image blocks after an image position encoding, and converted to Q after passing through the linear layer. Q, K, and V pass through the residual cross-attention Transformer block and then pass through the MLP output as the scene potential representation S of the next cross-self-attention STBlock r Input, after passing through the second STBlock, the image feature block group of the target viewpoint is directly output through the final MLP layer, and restored to the complete image feature through normalization and linear layers; Step 11: Freeze the network parameters of the scene new viewpoint synthesis module obtained in step 10, obtain a set of scene test target viewpoint data, pre-process the input viewpoint image, input viewpoint pose, and test target viewpoint pose according to step 2, and perform steps 3, 7, and 8 in sequence to obtain the synthetic image features of the test target viewpoint; Step 12: Input the synthetic image features of the test target viewpoint obtained in step 11 into the decoding part of the VAE autoencoder to obtain the final synthetic image of the test target viewpoint.

2. A new viewpoint image synthesis method based on camera pose light field coding according to claim 1, characterized in that: The first loss function used in step 6 is expressed as: Z S =VAEEncoder(I S ) S r =STEncoder(Z S ,LF S ) Z ref =STDncoder(S r ) Among them I S is the input viewpoint image, Z S and Z ref Represent the input viewpoint image features and the reconstructed image features, LF S is the camera pose light field of the input viewpoint, S r It is the potential representation of the scene generated by the encoding module, which will be used as the input of the second-stage scene new viewpoint image generation model.

3. The new viewpoint image synthesis method based on camera pose light field coding according to claim 1, characterized in that: The random noise processing used in step 8 is specifically implemented by adding an offset noise to the camera pose matrix K, R, T of the training target viewpoint. The noise is a randomly sampled Gaussian noise, and its disturbance size is controlled by a coefficient factor to be no more than 0.1% of the standard scale of the scene.

4. The new viewpoint image synthesis method based on camera pose light field coding according to claim 1, characterized in that: The second loss function used in step 10 is expressed as: Z T '=NVSDncoder(S r ,LF T ) Z T =VAEEncoder(I T ) Where Z T and Z T 'represent the training target viewpoint image features and the generated target viewpoint image features, I T is the training target viewpoint image, LF T is the camera pose light field of the target viewpoint, S r is the scene potential representation generated by the scene potential representation encoding module in the first stage, is the second loss function.

5. The new viewpoint image synthesis method based on camera pose light field coding according to claim 1, characterized in that: Step 6 and step 10 respectively perform two stages of model training, first training the scene latent representation encoding module, and then training the scene new viewpoint synthesis module, rather than training the two models simultaneously for joint optimization.

Images