Rendering method and device, equipment and storage medium

Abstract

The invention discloses a rendering method, device and equipment and a storage medium, and relates to the technical field of image processing, and the method comprises the steps: obtaining a three-plane vector and a to-be-input image; the three-plane vector and the to-be-input image are input into a pre-constructed three-plane generator, viewpoint rendering is carried out on the three-plane vector and the to-be-input image through a two-stage preheating training strategy, a final rendering result is obtained, the three-plane generator is obtained based on deep neural network training, and the three-plane generator is obtained based on deep neural network training. The training time can be shortened.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a rendering method, apparatus, device and storage medium. Background Art

[0002] Recently, the neural network-based Neural Radiance Fields (NeRF) has brought great improvements in the quality of high-fidelity synthesis for new view synthesis tasks.

[0003] NeRF represents 3D radiation fields through multi-layer perceptrons and obtains new views through differentiable volume rendering.

[0004] However, current methods take too long to train.

[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of the present application is to provide a rendering method, apparatus, device and storage medium, aiming to solve the technical problem that the current method has a long training time.

[0007] To achieve the above object, the present application proposes a rendering method, which comprises:

[0008] Get the three-plane vectors and the image to be input;

[0009] The three-plane vectors and the image to be input are input into a pre-built three-plane generator, and the three-plane vectors and the image to be input are viewpoint rendered through a two-stage warm-up training strategy to obtain a final rendering result. The three-plane generator is obtained based on deep neural network training.

[0010] In one embodiment, the step of inputting the three-plane vector and the image to be input into a pre-built three-plane generator, performing viewpoint rendering on the three-plane vector and the image to be input through a two-stage warm-up training strategy, and obtaining a final rendering result includes:

[0011] Inputting the three-plane vector and the image to be input into the three-plane generator to generate a first characteristic three-plane;

[0012] Visualizing the first three characteristic planes to obtain a first rendering result;

[0013] Optimizing the three-plane generator based on the first rendering result;

[0014] Freeze the model parameters of the optimized three-plane generator;

[0015] The second characteristic three-plane is obtained by forward reasoning and adjusting the shape through the three-plane generator after the model parameters are frozen;

[0016] Setting parameters of the second characteristic triplane as learnable variables to obtain a learnable second characteristic triplane;

[0017] The learnable second characteristic three-plane is visualized to obtain the final rendering result.

[0018] In one embodiment, before the step of inputting the three-plane vector and the image to be input into the three-plane generator to generate the first characteristic three-plane, the step further includes:

[0019] Inputting the three-plane vector into the three-plane generator to generate a feature grid;

[0020] Optimizing the position and posture of a camera that shoots the image to be input based on the feature grid to obtain an optimized camera position and posture;

[0021] Inputting the image to be input into a feature extractor to extract a feature vector;

[0022] Aggregating the feature vector with the three-plane vector through a cross-attention mechanism to obtain a cross-aggregation feature;

[0023] The step of inputting the three-plane vector and the image to be input into the three-plane generator to generate the first characteristic three-plane comprises:

[0024] The cross-convergence feature is input into the three-plane generator to generate the first feature three-plane.

[0025] In one embodiment, the step of visualizing the first characteristic three planes to obtain a first rendering result includes:

[0026] Aggregating the optimized camera pose and the first feature three planes through a decoupled three-plane aggregation algorithm to generate a first aggregated feature;

[0027] Decoding the first aggregated feature by a decoder to obtain corresponding radiation color and volume density;

[0028] Performing volume rendering on the corresponding radiation color and volume density to obtain the first rendering result;

[0029] The step of visualizing the learnable second characteristic three-plane to obtain the final rendering result comprises:

[0030] Aggregating the optimized camera pose and the learnable second feature three-plane through the decoupled three-plane aggregation algorithm to generate a second aggregated feature;

[0031] Decoding the second aggregate feature by the decoder to obtain corresponding radiation color and volume density;

[0032] Volume rendering is performed on the corresponding radiation color and volume density to obtain the final rendering result.

[0033] In one embodiment, the step of inputting the image to be input into a feature extractor to extract a feature vector comprises:

[0034] The feature vector is obtained by extracting features of the image to be input through a pre-built dual interlaced network optimization model.

[0035] In one embodiment, the step of aggregating the feature vector and the three-plane vector through a cross attention mechanism to obtain a cross-aggregated feature includes:

[0036] The three-plane vector is used as a query vector, and the feature vector is used as a key and a value to perform the cross-attention mechanism to obtain the cross-aggregated feature.

[0037] In one embodiment, the step of inputting the three-plane vector and the image to be input into a pre-built three-plane generator includes:

[0038] Construct the three-plane generator, wherein each of the plane generators is composed of an intermediate block and an upsampling block, the intermediate block is composed of a convolutional layer, a structure using group normalization and an activation function, and an attention layer, the upsampling block is composed of a convolutional layer, a structure using group normalization and an activation function, and except for the last upsampling block, the upsampling block also includes a bilinear upsampler.

[0039] In addition, to achieve the above purpose, the present application also proposes a rendering device, the rendering device comprising:

[0040] A data acquisition module, used for acquiring three-plane vectors and images to be input;

[0041] A rendering module is used to input the three-plane vectors and the image to be input into a pre-built three-plane generator, and perform viewpoint rendering on the three-plane vectors and the image to be input through a two-stage warm-up training strategy to obtain a final rendering result. The three-plane generator is obtained based on deep neural network training.

[0042] In addition, to achieve the above objectives, the present application also proposes a rendering device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the rendering method described above.

[0043] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the rendering method described above are implemented.

[0044] One or more technical solutions proposed in this application have at least the following technical effects:

[0045] This application provides fast reasoning, efficient data compression performance, and better convergence under fuzzy poses by inputting the three-plane vectors and the input image into a pre-built three-plane generator. Therefore, the training time of this application is significantly shortened, while improving the performance of new viewpoint rendering and camera pose estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0048] Figure 1 A schematic diagram of a process flow provided for Embodiment 1 of the rendering method of the present application;

[0049] Figure 2 A flowchart of the warm-up training strategy in the rendering method of this application;

[0050] Figure 3a A schematic diagram of the camera and feature grid for this application;

[0051] Figure 3b This is a schematic diagram of the characteristic plane visualization results of the Horn angle of the scene;

[0052] Figure 4 This is a schematic diagram of the plane generator effect of the rendering method of this application;

[0053] Figure 5 A complete flowchart of the rendering method for this application;

[0054] Figure 6a A schematic diagram comparing the rendering results of this application with other methods;

[0055] Figure 6b This is a schematic diagram comparing the training efficiency of this application with other methods;

[0056] Figure 7 A schematic diagram of the new viewpoint rendering effect between the NeRF-Synthetic neural radiation field synthesis dataset and other methods of this application;

[0057] Figure 8 A schematic diagram of the new viewpoint rendering effect of this application between the real-shot LLFF (Local Light Field Fusion) dataset and other methods;

[0058] Fig. 9 This is a schematic diagram of the module structure of the rendering device according to an embodiment of the present application;

[0059] Fig.10 Schematic diagram of the device structure of the hardware operating environment involved in the rendering method in the embodiment of the present application.

[0060] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0061] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0062] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0063] Technical terms involved in the embodiments of the present invention:

[0064] NeRF: NeRF (Neural Radiance Field) is a method of 3D reconstruction. By training a neural network to fit a 3D scene, the corresponding image can be obtained from a new viewpoint that has never been seen before.

[0065] PSNR: PSNR (Peak Signal Noise Ratio) is usually used to evaluate the similarity between two images. The unit is decibel dB. The higher the peak signal-to-noise ratio, the higher the image similarity.

[0066] iNeRF: iNeRF (Inverting neural radiance fields for pose estimation) demonstrates the ability to perform pose estimation using a reconstructed NeRF model.

[0067] NeRFmm: NeRFmm (Neural radiance fields without known camera parameters) jointly estimates the camera's intrinsic parameters, extrinsic parameters, and NeRF.

[0068] BARF: BARF (Bundle-adjusting neural radiance fields) proposes a coarse-to-fine position encoding method.

[0069] SC-NeRF: SC-NeRF (Self-calibrating neural radiance fields) estimates camera distortion and proposes geometric regularization.

[0070] GNeRF: GNeRF (Gan-based neural radiance field without posed camera) uses a generative adversarial network to handle the joint optimization problem.

[0071] VMRF: VMRF (View Matching Neural Radiance Fields) uses a generative adversarial network to handle joint optimization problems.

[0072] SiNeRF: SiNeRF (Sinusoidal neural radiance fields for joint poseestimation and scene reconstruction) modifies different activation functions to obtain more appropriate pose estimation.

[0073] GARF: GARF (Gaussian activated neural radiance fields for high fidelity reconstruction and pose estimation) modifies different activation functions to obtain more appropriate pose estimation.

[0074] NoPe-NeRF: NoPe-NeRF (Learning neural radiance field with no poseprior) uses a single depth estimate to geometrically guide the joint optimization, which is designed for long sequence images.

[0075] FDC-NeRF: FDC-NeRF (Learning pose-free neural radiance fields with flow-depth consistency) uses a single depth estimate to geometrically guide the joint optimization, which is designed for long sequences of images.

[0076] L2G-NeRF: L2G-NeRF (Local-to-Global Neural Radiance Fields) proposes a local-to-global registration method to estimate pose from multiple learned local poses.

[0077] SPARF: SPARF (Neural radiance fields from sparse and noisy poses) uses a pre-trained image matching model as a prior in the sparse view setting.

[0078] LU-NeRF: LU-NeRF (Scene and pose estimation by synchronizing local unposed nerfs) optimizes NeRF without any pose initialization.

[0079] LocalRF: LocalRF (Progressively optimized local radiance fields for robust view synthesis) is proposed for long sequence static videos.

[0080] Robust-DynRF: Robust-DynRF (Robust Dynamic Neural Radiance Fields) is proposed for dynamic videos.

[0081] Hash-BARF: Hash-BARF (Instant neural graphics primitives with a multiresolution hash encoding) uses multiresolution hash encoding to accelerate the joint optimization of pose and neural radiation field in previous methods.

[0082] The main solution of the embodiment of the present application is: obtaining three-plane vectors and the image to be input; inputting the three-plane vectors and the image to be input into a pre-built three-plane generator, performing viewpoint rendering on the three-plane vectors and the image to be input through a two-stage warm-up training strategy to obtain a final rendering result, wherein the three-plane generator is obtained based on deep neural network training.

[0083] In this embodiment, for the convenience of description, the following description is made with the rendering system as the execution subject.

[0084] The current method takes too long to train.

[0085] Therefore, this application provides a solution, which brings fast reasoning, efficient data compression performance, and better convergence under fuzzy poses by inputting the three-plane vectors and the input image into a pre-built three-plane generator. Therefore, the training time of this application is significantly shortened, and the performance of new viewpoint rendering and camera pose estimation is improved.

[0086] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, a rendering device, etc. capable of realizing the above functions, or an electronic system, a rendering system, etc. capable of realizing the above functions. The following takes the rendering system as an example to illustrate this embodiment and the following embodiments.

[0087] Recently, Neural Radiance Fields (NeRF) based on neural networks has brought great improvements in the quality of high-fidelity synthesis for novel view synthesis tasks. NeRF represents 3D radiance fields through multi-layer perceptrons and obtains novel views through differentiable volume rendering. The key prerequisite for achieving ideal rendering results using NeRF is accurately calibrated camera parameters.

[0088] However, the accurate camera pose is not easy to obtain, which depends heavily on the accuracy and robustness of the Structure from Motion (SfM) algorithm such as the COLMAP algorithm.

[0089] To this end, previous researchers have increasingly focused on reducing the dependence on camera parameters by optimizing NeRF and camera parameters simultaneously.

[0090] In recent years, the joint optimization problem of camera parameters and neural radiance fields has been widely studied. iNeRF first demonstrated the ability to perform pose estimation using a reconstructed NeRF model. NeRFmm first jointly estimates the camera's intrinsic parameters, extrinsic parameters, and NeRF. BARF proposed a coarse-to-fine position encoding method. SC-NeRF estimates camera distortion and proposes geometric regularization. GNeRF and VMRF adopt adversarial generative networks to deal with the joint optimization problem, and SiNeRF and GARF modify different activation functions to obtain more appropriate pose estimates. Recently, NoPe-NeRF and FDC-NeRF use a single depth estimate to geometrically guide the joint optimization, which is designed for long sequence images. L2G-NeRF proposes a local-to-global registration method to estimate pose from multiple learned local poses. SPARF uses a pre-trained image matching model as a prior in sparse view settings. LU-NeRF optimizes NeRF without any pose initialization. LocalRF and Robust-DynRF are proposed for long sequence static videos and dynamic videos, respectively. The recent Hash-BARF uses multi-resolution hash coding to accelerate the joint optimization of pose and neural radiance field in previous methods.

[0091] However, previous joint optimization methods are usually based on the original multi-layer perceptron representation, and most existing methods take hours or even days to train using modern powerful GPUs, which severely limits their use in practical application scenarios. Therefore, it is of great significance to improve the training efficiency of such joint optimization.

[0092] Based on this, the present application embodiment provides a rendering method, referring to Figure 1 , Figure 1 A flowchart diagram of the first embodiment of the rendering method of the present application is provided.

[0093] In this embodiment, the rendering method includes steps S20 to S30:

[0094] Step S20, obtaining three-plane vectors and an image to be input;

[0095] Among them, three-plane vector is a method of using three planes in three-dimensional space to represent or reconstruct a scene. These three planes are usually used to capture and represent different perspectives and details of a three-dimensional scene. In the rendering process, three-plane vector helps the model more accurately represent and process the three-dimensional information of the scene, making the rendering result more refined and accurate.

[0096] The image to be input refers to the original image that needs to be rendered or processed. This image can be a photo actually taken or a synthesized image. In this embodiment, a picture actually taken by a camera is taken as an example.

[0097] It should be noted that the three-plane vectors may be three-plane noise vectors following Gaussian distribution.

[0098] Step S30, input the three-plane vector and the image to be input into a pre-built three-plane generator, perform viewpoint rendering on the three-plane vector and the image to be input through a two-stage warm-up training strategy to obtain a final rendering result, and the three-plane generator is obtained based on deep neural network training.

[0099] It should be noted that the three-plane generator is a model based on a convolutional deep neural network, which aims to generate three independent planes with spatial features from a noise vector. The generator consists of three neural networks with the same structure but no shared parameters, each of which is responsible for generating a different plane to cope with the anisotropy and decoupling characteristics of the feature plane at different viewpoints. This design ensures that the plane features extracted from the input noise can be better adapted to different viewpoints and feature requirements in the three-dimensional scene.

[0100] It should be noted that, in the first stage of warm-up training, the three-plane vector and the image to be input are input into the three-plane generator to generate a first characteristic three-plane; the first characteristic three-plane is visualized to obtain a first rendering result; and the three-plane generator is optimized based on the first rendering result.

[0101] In the second stage of warm-up training, the model parameters of the optimized three-plane generator are frozen; forward reasoning and shape adjustment are performed on the three-plane generator after the model parameters are frozen to obtain a second characteristic three-plane; the parameters of the second characteristic three-plane are set as learnable variables to obtain a learnable second characteristic three-plane; the learnable second characteristic three-plane is visualized to obtain the final rendering result.

[0102] Reference Figure 2 , Figure 2 A flowchart of the warm-up training strategy in the rendering method of this application;

[0103] It should be noted that given a three-plane noise vector following a Gaussian distribution as input, a convolution-based neural network is introduced as a three-plane generator for extracting plane features, with the goal of parameterizing the spatial feature grid into a deep neural network.

[0104] Since the feature planes of different perspectives of a 3D scene are highly decoupled, the features between different planes are anisotropic. Therefore, three independent neural networks with the same structure but no shared parameters can be used as disentangled generators to generate three different planes respectively.

[0105] The generator is randomly initialized in each scene instead of being pre-trained, so similar to previous work, it still needs to be optimized scene by scene.

[0106] Reference Figure 4 , Figure 4 This is a schematic diagram of the plane generator effect of the rendering method of this application;

[0107] Figure 4 The following lists (a) a feature visualization method that directly combines three planes and camera poses and (b) a feature visualization method of the present application using a three-plane generator. Figure 4 It can be seen that compared with the feature visualization method that directly combines the three planes and the camera pose, the three-plane generator of the present application reduces the error of local updates and shows less noise in the visualization of the planar feature map.

[0108] Furthermore, the triplanar generator will mine the geometric and textural features of the scene and embed them into the triplanar representation instead of simply overfitting to new viewpoints.

[0109] It is worth noting that in order to ensure that the generated planes conform to the spatial characteristics and structure of the real world, the input of the three-plane generator is a randomly initialized three-plane noise vector. During the generation process, the core features are maintained in the three-plane structure from input to output to adapt to the decoupled scene representation.

[0110] After training with the three-plane generator, we can get the following insights: once the camera pose is optimized to a good position, the effect of directly optimizing the three planes will be better than using the three-plane generator to generate the three planes from beginning to end, especially in the rendering effect of high-frequency areas.

[0111] Therefore, although the three-plane generator alleviates local update errors, it is implicitly constrained by different feature blocks on the plane grid caused by the generator, which introduces over-smoothing on the feature grid. Therefore, a two-stage warm-up training strategy can be proposed, which switches to direct plane optimization after the faster and better joint optimization in the first stage, thereby obtaining better rendering quality for new viewpoints overall.

[0112] In the second stage, the parameters generated by the three-plane generator in the first stage can be used (i.e., the second characteristic three-plane is obtained by forward reasoning and adjusting the shape through the three-plane generator after the model parameters are frozen) instead of initializing the parameters from scratch.

[0113] The present embodiment provides a rendering method, which brings fast reasoning, efficient data compression performance and better convergence under fuzzy pose by inputting three-plane vectors and the image to be input into a pre-built three-plane generator. Therefore, the training time of the present application is significantly shortened, while improving the performance of new viewpoint rendering and camera pose estimation. In addition, the present application proposes to use a three-plane neural radiation field with fast training and rendering speed, highest compression efficiency, best performance and compact data structure to perform joint estimation of camera pose-neural radiation field, thereby solving the problems of long training time and slow reasoning speed of previous methods.

[0114] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can refer to the above description, and will not be described in detail later. On this basis, step S30, inputting the three-plane vector and the image to be input into the pre-built three-plane generator, performing viewpoint rendering on the three-plane vector and the image to be input through a two-stage warm-up training strategy, and obtaining the final rendering result also includes steps S35 to S311:

[0115] Step S35, inputting the three-plane vector and the image to be input into the three-plane generator to generate a first characteristic three-plane;

[0116] Among them, for the two-stage warm-up training strategy, in the first stage, the embodiment of the present application inputs the random three-plane noise vector and the image to be input into the proposed three-plane generator to generate a feature three-plane.

[0117] Specifically, the three-plane generator extracts features of different levels or resolutions from the input image and maps these features to three independent planes. Each plane represents specific features or information. The combination of these planes forms a rich feature representation, which is the first feature three-plane.

[0118] Step S36, visualizing the first characteristic three planes to obtain a first rendering result;

[0119] The optimized camera pose and the first feature three planes may be aggregated by a decoupled three-plane aggregation algorithm to generate a first aggregate feature; the first aggregate feature is decoded by a decoder to obtain a corresponding radiation color and volume density; the corresponding radiation color and volume density are volume rendered to obtain the first rendering result;

[0120] Step S37, optimizing the three-plane generator based on the first rendering result;

[0121] Wherein, in the second stage, the three-plane generator may be optimized based on the first rendering result;

[0122] Furthermore, the optimization of the three-plane generator can be obtained by minimizing the photometric consistency loss, the specific formula is:

[0123]

[0124] Among them, L stands for Loss, which is the loss function; Σ stands for sum.

[0125] If N points are collected on a picture, there will be N corresponding rays. The color of each ray output can be obtained by the above method, making it consistent with the real image color in the data set. By constraining the optimization, the loss function is MSE.

[0126] Among them, I i The color output for each ray, is the actual image color.

[0127] Step S38, freezing the model parameters of the optimized three-plane generator;

[0128] Step S39, performing forward reasoning and adjusting the shape by the three-plane generator after the model parameters are frozen to obtain a second characteristic three-plane;

[0129] Among them, forward reasoning refers to passing the input data (3D shape or other feature description) through the model to generate output results (feature three-planes). These feature three-planes are generated by the model parameters of the optimized three-plane generator, but they also need to be adjusted or post-processed according to specific needs to further optimize their representation.

[0130] At the beginning of the second stage three-plane optimization process, the generator can be abandoned and a single forward inference can be performed to obtain the second feature three-plane.

[0131] As an implementation method, based on the first characteristic three-plane, a multi-scale second characteristic three-plane can be obtained through bilinear interpolation to perform direct three-plane optimization.

[0132] In addition, when performing a single forward inference to obtain the second feature three-plane, multiple feature grids with different resolutions are used to enhance the representation of the structure and details of the second feature three-plane in a coarse-to-fine manner.

[0133] Step S310, setting the parameters of the second characteristic three-plane as learnable variables to obtain a learnable second characteristic three-plane;

[0134] The parameters of the three feature planes are set as learnable variables. This means that the parameters will be fine-tuned in the following training to better suit the specific task or dataset. By setting the parameters of the three feature planes as learnable variables, the model can optimize these parameters through back propagation during the training process to further improve the final rendering or estimation performance.

[0135] Step S311, visualizing the learnable second characteristic three-plane to obtain the final rendering result.

[0136] Among them, the optimized camera pose and the learnable second feature three-plane can be aggregated through the decoupled three-plane aggregation algorithm to generate a second aggregate feature; the second aggregate feature is decoded by the decoder to obtain the corresponding radiation color and volume density; the corresponding radiation color and volume density are volume rendered to obtain the final rendering result.

[0137] The two stages in the two-stage warm-up training strategy refer to the training process of the model. The two stages are switched according to the training iterations. In the first stage optimization process, the network parameters of the three-plane generator and the camera pose are jointly optimized. When a specific number of iterations is reached, the model training switches to the second stage.

[0138] In the second stage, the model parameters of the three-plane generator are first frozen, and the generator network parameters are no longer updated. At this time, the three-plane generator will forward infer a three-plane vector, reshape the three-plane vector into a feature three-plane, and set the parameters in this three-plane as learnable variables. Therefore, in the second stage, the camera pose and the learnable feature three-plane are jointly optimized, and the generator of the first stage is abandoned. The feature three-plane is a decoupled representation of three-dimensional space. After the camera optical center emits a ray, the point on the ray is projected to the corresponding position on the three-plane to obtain the corresponding feature. Aggregating the features of different planes and decoding them through the decoder can obtain the 3D features of the corresponding points on the ray. Repeating this operation, several points on the ray can get the corresponding 3D features, and then through volume rendering in traditional graphics, the color of the final ray corresponding to the 2D image can be output. During the training process, the parameters of the three planes are directly optimized.

[0139] This embodiment proposes a new training strategy, a two-stage warm-up training strategy, in which the three-plane generator is discarded in the second stage and the three-plane is directly trained to reduce the problem of over-smoothing of the feature plane caused by using the three-plane generator.

[0140] Based on the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the second embodiment can be referred to the above description, and will not be described in detail later. On this basis, step S35, inputting the three-plane vector and the image to be input into the three-plane generator, and before generating the first characteristic three-plane, steps S31 to S34 are also included:

[0141] Step S31, inputting the three-plane vector into the three-plane generator to generate a feature grid;

[0142] Step S32, optimizing the position and posture of the camera that shoots the image to be input based on the feature grid to obtain an optimized camera position and posture;

[0143] Among them, in the first stage, the embodiment of the present application inputs a random three-plane noise vector into the proposed three-plane generator to generate different feature grids for scene representation while optimizing the camera pose.

[0144] Step S33, inputting the image to be input into a feature extractor to extract a feature vector;

[0145] Step S34, aggregating the feature vector and the three-plane vector through a cross attention mechanism to obtain a cross-aggregation feature;

[0146] Among them, the feature vector extracted from the input image after passing through the feature extractor is aggregated into a three-plane vector through a cross-attention mechanism to obtain a cross-aggregated feature.

[0147] The step S35, inputting the three-plane vector and the image to be input into the three-plane generator, and generating the first characteristic three-plane further comprises the step S351:

[0148] Step S351, inputting the cross-aggregation feature into the three-plane generator to generate the first feature three-plane.

[0149] In addition, this embodiment improves the explicit representation of the three planes into an implicit network-explicit three-plane hybrid representation based on deep network parameterization. Since the neural network brings enough smoothness and global information to the three planes, the training is easier to converge without falling into the local optimum, so the result of camera pose estimation is more accurate.

[0150] Previous aggregation methods did not take into account the differences in training difficulty between different planes, so the different quality of supervision signals resulted in suboptimal camera pose optimization.

[0151] Therefore, embodiment 4 of the present application is proposed.

[0152] Based on the third embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as those in the third embodiment can refer to the above description, and will not be repeated in the following. On this basis, step S36, visualizing the first feature three planes, and obtaining the first rendering result also includes steps S361 to S363:

[0153] Step S361, aggregating the optimized camera pose and the first feature three planes through a decoupled three-plane aggregation algorithm to generate a first aggregated feature;

[0154] Before decoding and rendering, it is necessary to aggregate the features of the sampling points queried from different planes. The embodiment of the present application proposes a decoupled plane aggregation method.

[0155] Since the three-plane representation is an interpretable explicit radiation field that decomposes the scene into three orthogonal planes aligned with the X, Y, and Z axes, each plane contains 3D scene features from an orthogonal perspective. Images taken from different angles provide more comprehensive scene information for the three-plane, while images from limited or unevenly distributed viewpoints provide different quality supervision information for each plane.

[0156] Reference Figure 3a , Figure 3a A schematic diagram of the camera and feature grid for this application;

[0157] Depend on Figure 3a As shown, the angles of images or videos are usually consistent, all objects are mainly facing the camera, and the YZ plane obtains richer scene information and is easier to learn.

[0158] Reference Figure 3b , Figure 3b This is a schematic diagram of the characteristic plane visualization results of the Horn angle of the scene;

[0159] During the optimization process, a plane that is easier to learn brings richer supervision information, thereby providing better supervision for camera parameter optimization. In contrast, learning a more challenging plane requires additional training iterations to accurately represent the decoupled scene information, thereby increasing the probability of providing fuzzy supervision for camera pose optimization. The difference in learning difficulty between different planes increases the difficulty of pose optimization. In this regard, the embodiment of the present application proposes an aggregation algorithm for decoupling three planes, and the implementation formula is as follows:

[0160]

[0161] Where D(x) represents the gradient separation operation, P represents the feature plane, ψ represents the interpolation operation on the feature plane, and π represents the projection operation of 3D points to each plane. The result of the decoupled plane feature aggregation algorithm is equivalent to Π k (Fk +1), is a form of Hadamard product, so this guarantees the expressiveness of the three planes. In practice, the addition term 1 can be changed to a hyperparameter λ. This decoupled three-plane aggregation method realizes the back propagation of the gradient of the three planes from the product and the back propagation of the gradient of the camera pose from the sum.

[0162] Step S362, decoding the first aggregated feature through a decoder to obtain corresponding radiation color and volume density;

[0163] As an implementation, the decoder may be an MLP multi-layer perceptron decoder.

[0164] Step S363, performing volume rendering on the corresponding radiation color and volume density to obtain the first rendering result;

[0165] The first aggregated feature is decoded by a decoder to obtain the radiation color and volume density, and the image to be rendered can be synthesized and rendered along the ray r(t)=o+td of the camera that took the input image between the near plane and the far plane to obtain a first rendering result. The volume rendering formula can be expressed as:

[0166]

[0167] Among them, o is the optical center of the camera, d represents the direction of the light emitted from the optical center of the camera, t represents the distance along the sampling, o+td can be analogous to a vector, and t represents the actual vector length. r(t) as a whole represents the point at position t where the light is emitted from o along the direction d.

[0168] I represents the RGB value of the ray on the 2D image. tn, tf represent the nearest (tnear) and farthest intervals (tfar) of the ray emission. W(t) represents the cumulative transmittance at the position, σ represents the volume density, σ(r(t)) represents the volume density value of a certain point at the index ray (from the previous text, a certain point is projected onto three planes to obtain three features, and the aggregated 3D features are decoded to obtain the color c and volume density sigma of the 3D point), c represents the color (also from the previous text, the 3D features after the three planes are aggregated are decoded). c(r(t), d) represents the color of the 3D point in the observation direction d. The size of t increases from tn to tf on the ray, so the final color of the ray can be calculated by integration. W represents the cumulative transmittance, and the depth can be rendered by the cumulative transmittance.

[0169] Step S311, visualizing the learnable second characteristic three-plane, and obtaining the final rendering result includes steps S3111 to S3113:

[0170] Step S3111, aggregating the optimized camera pose and the learnable second feature three-plane through the decoupled three-plane aggregation algorithm to generate a second aggregated feature;

[0171] Among them, the implementation formula of the decoupled three-plane aggregation algorithm is as described above.

[0172] Step S3112, decoding the second aggregate feature by the decoder to obtain corresponding radiation color and volume density;

[0173] Furthermore, the decoder and camera poses of the second stage are directly inherited from the first stage since they apply to the same three planes.

[0174] Step S3113, performing volume rendering on the corresponding radiation color and volume density to obtain the final rendering result.

[0175] Among them, the implementation formula of volume rendering is as described above.

[0176] This embodiment improves the aggregation method and proposes a decoupled three-plane aggregation algorithm. Since the gradient of the camera pose estimation and the gradient of the features of different three planes are decoupled after the improvement, the joint optimization is more robust, so the performance of new viewpoint rendering and camera pose estimation is improved. In addition, this application proposes a new plane feature aggregation method. After the three-plane features are obtained, they need to be aggregated in some way and sent to the subsequent modules. This application proposes a decoupled aggregation method to aggregate different plane features and improve the quality of joint optimization.

[0177] Based on the third embodiment of the present application, in the fifth embodiment of the present application, the same or similar contents as those in the third embodiment can be referred to the above description, and will not be described in detail later. On this basis, step S33, inputting the image to be input into the feature extractor, extracting the feature vector also includes step S331:

[0178] Step S331, extracting features of the image to be input through a pre-built dual interlaced network optimization model to obtain the feature vector.

[0179] After parameterizing the three planes using the three-plane generator, the three-plane texture representation of the scene can also be enhanced through the scene texture embedding module, thereby alleviating the problem of camera pose-neural radiance field optimization gradient ambiguity.

[0180] As an implementation method, a pre-built DINOv2 dual interleaved network optimization model may be applied to encode the input image into a feature vector.

[0181] The steps of constructing a dual interlaced network optimization model are as follows:

[0182] 1. Build two intertwined neural network branches, which can be convolutional neural networks (CNNs) or other suitable structures. One branch is responsible for extracting low-level features, and the other branch processes higher-level features.

[0183] 2. Implement feature interleaving between certain layers of the network, that is, combining the output of one branch with the input of another branch. This interleaving can help the network capture features at different levels.

[0184] 3. Use optimization algorithms (such as gradient descent) to adjust network parameters. Through the training process, the model will learn how to better extract and integrate features.

[0185] 4. Usually, the two branches are trained together to optimize the entire network through a shared loss function. This helps the information flow between the two branches and improves the performance of the model.

[0186] This embodiment extracts features from the image to be input through a pre-built dual interlaced network optimization model to obtain the feature vector, wherein the dual interlaced network optimization model is a well-designed and trained deep learning model that can effectively extract high-level abstract features from the image to be input. These features are not just pixel-level information, but can capture semantic and structural information in the image, which helps to accurately execute subsequent tasks.

[0187] Based on the third embodiment of the present application, in the sixth embodiment of the present application, the same or similar contents as those in the third embodiment can be referred to the above description, and will not be described in detail later. On this basis, step S34, aggregating the feature vector with the three-plane vector through a cross attention mechanism to obtain a cross-aggregation feature also includes step S341:

[0188] Step S341, using the three-plane vector as a query vector and the feature vector as a key and a value to execute the cross-attention mechanism to obtain the cross-aggregated feature.

[0189] Among them, the feature vector extracted by the feature extractor is a 2D feature vector, and the cross attention mechanism can be applied to merge the 2D feature vector into three planes.

[0190] During the optimization process, the 2D feature vectors and the three-plane vectors will automatically learn the alignment between the patterns, thus better integrating the image features into the three-plane.

[0191] Specifically, a fixed three-plane noise vector can be used as the query vector, and the extracted feature vectors can be used as keys and values ​​to perform a cross-attention mechanism. Finally, the aggregated cross-aggregated features are input into the three-plane generator for subsequent operations.

[0192] This embodiment uses the three-plane vector as the query vector and the feature vector as the key and value to perform the cross-attention mechanism to obtain the cross-aggregated feature, wherein the cross-attention mechanism allows the aggregation of feature information at different levels to form a more comprehensive and comprehensive feature representation. This multi-level aggregation can improve the system's ability to understand and process complex information.

[0193] Based on the first embodiment of the present application, in the seventh embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description, and will not be described in detail later. On this basis, step S20, before inputting the three-plane vector and the image to be input into the pre-built three-plane generator, also includes step S10:

[0194] Step S10, constructing the three-plane generator, wherein each of the plane generators is composed of an intermediate block and an upsampling block, the intermediate block is composed of a convolutional layer, a structure using group normalization and an activation function, and an attention layer, the upsampling block is composed of a convolutional layer, a structure using group normalization and an activation function, and except for the last upsampling block, the upsampling block also includes a bilinear upsampler.

[0195] As an implementation mode, the specific implementation of the three-plane generator is as follows: the shape of the three-plane noise vector is set to (3×8×20×20), where 3 represents the number of feature planes, 8 represents the hidden dimension, and 20×20 is the spatial shape.

[0196] The three-plane vector is reshaped into (3×400×8), and each plane (1×400×8) is set as the query vector and cross-attention operations are performed with the extracted feature vector (1×3889×384) to obtain the cross-aggregated features.

[0197] Then, the cross-aggregated features are re-converted back to (3×8×20×20) for the input of the three-plane generator.

[0198] It should be noted that the generator of each plane consists of an intermediate block and L upsampling blocks (L=5). The intermediate block consists of two 2D convolutional layers (res-conv-layers) with residual connections and an attention layer. Group normalization and SiLU activation function are after each residual convolution layer.

[0199] For the upsampling block, two residual convolutional layers can be used, followed by group normalization and SiLU activation function.

[0200] After that, each block is appended with a bilinear upsampler, except the last block.

[0201] The upsampling layer expands the spatial size of the feature vector (3×8×20×20) to the shape of the final three-plane grid (3×64×320×320).

[0202] This application proposes a new three-plane generator based on deep neural networks, constructs a new explicit-implicit hybrid scene representation based on three planes, and uses it to jointly estimate camera poses and synthesize new viewpoints. This method of combining deep network parameterized three planes with pose optimization is the first application in the field of joint optimization.

[0203] For example, to help understand the implementation process of the rendering method of the above embodiment, please refer to Figure 5 , Figure 5 A complete flowchart of the rendering method for this application;

[0204] Figure 5 The overall description is as follows: in the training stage, the vector of the three-plane is input to the three-plane generator of the first stage to obtain the corresponding three-plane. At the same time, the input image also promotes the generation of the three-plane under the action of the cross-attention mechanism. At this time, it is judged whether the current training iteration round is the second stage. If not, the feature vector generated by the three-plane generator is directly used, and the shape is re-adjusted to become the feature three-plane. The light emitted by the camera posture is used to sample the points on the three-plane. The structure is aggregated and input into the decoder, and the final result is obtained by volume rendering. If the current round is judged to be the second stage, the parameters of the three-plane generator are frozen, the output three-plane vector is set as a learnable parameter, and the shape is re-adjusted to become the feature three-plane, and the above operations are repeated.

[0205] The difference between the two stages is that during back propagation, the three planes in the first stage are generated by the three plane generator, so the gradient acts on the network parameters of the generator. The update of the parameters in the three planes in the first stage is reflected in the change of the generator network parameters. In the second stage, the three plane generator is frozen, the three planes become learnable parameters, and the gradient acts directly on the parameters of the three planes without going through the three plane generator.

[0206] Specifically, the process includes the following steps:

[0207] Step S100, inputting the three-plane vector into a three-plane generator to generate a feature grid;

[0208] Step S101, optimizing the camera pose based on the feature grid to obtain an optimized camera pose;

[0209] Step S103, inputting the image to be input into a feature extractor to extract a feature vector;

[0210] Step S104, aggregating the feature vector and the three-plane vector through a cross attention mechanism to obtain a cross-aggregation feature;

[0211] Step S105, inputting the cross-aggregation features into a three-plane generator to generate feature three-planes.

[0212] Step S106, determining whether to switch to the second stage.

[0213] Step S107: if not, aggregating the optimized camera pose and the feature three planes through a decoupled three-plane aggregation algorithm to generate a first aggregated feature;

[0214] Step S108, decoding the first aggregated feature through a decoder to obtain corresponding radiation color and volume density;

[0215] Step S109, performing volume rendering on the corresponding radiation color and volume density to obtain the first rendering result;

[0216] Step S110: If yes, freeze the model parameters of the optimized three-plane generator;

[0217] Step S111, performing forward reasoning and adjusting the shape by the three-plane generator after the model parameters are frozen to obtain a second characteristic three-plane;

[0218] Step S112, setting the parameters of the second characteristic three-plane as learnable variables to obtain a learnable second characteristic three-plane;

[0219] Step S113, aggregating the optimized camera pose and the learnable feature three planes through the decoupled three-plane aggregation algorithm to generate a second aggregated feature;

[0220] Step S114, decoding the second aggregate feature by the decoder to obtain corresponding radiation color and volume density;

[0221] Step S115, performing volume rendering on the corresponding radiation color and volume density to obtain the final rendering result.

[0222] This application is compared with other methods as follows:

[0223] like Figure 6a As shown, Figure 6a A schematic diagram comparing the rendering results of this application with other methods;

[0224] Depend on Figure 6aIt can be seen that the PSNR of the neural radiation field of the BARF bundle adjustment method is 22.55; the PSNR of the local to global neural radiation field of L2G-NeRF is 23.49; the PSNR of the baseline method is 21.86; the PSNR of the rendering method of this application is 25.35, which is higher than other methods.

[0225] like Figure 6b As shown, Figure 6b This is a schematic diagram comparing the training efficiency of this application with other methods;

[0226] Depend on Figure 6b It can be seen that the training time of this method is shorter than that of the neural radiation field of the BARF bundle adjustment method and the local to global neural radiation field of L2G-NeRF; the displacement error of this method is smaller than that of other methods.

[0227] Reference Figure 7 , Figure 7 A schematic diagram of the new viewpoint rendering effect between the NeRF-Synthetic neural radiation field synthesis dataset and other methods of this application;

[0228] like Figure 7 As shown, the rendering method of the present application has better rendering effect than other methods and is closer to the real image.

[0229] Reference Figure 8 , Figure 8 A schematic diagram of the new viewpoint rendering effect of this application between the real-shot LLFF (Layered Light Field Flow) dataset and other methods;

[0230] like Figure 8 As shown, the rendering method of the present application has better rendering effect than other methods and is closer to the real image.

[0231] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the rendering method of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0232] This application also provides a rendering device, please refer to Fig. 9 , the rendering device comprises:

[0233] A data acquisition module 10 is used to acquire three-plane vectors and an image to be input;

[0234] The rendering module 20 is used to input the three-plane vectors and the image to be input into a pre-built three-plane generator, and perform viewpoint rendering on the three-plane vectors and the image to be input through a two-stage warm-up training strategy to obtain a final rendering result. The three-plane generator is obtained based on deep neural network training.

[0235] The rendering device provided by the present application adopts the rendering method in the above embodiment, which can solve the technical problem that the current method takes too long to train. Compared with the prior art, the beneficial effects of the rendering device provided by the present application are the same as the beneficial effects of the rendering method provided by the above embodiment, and the other technical features in the rendering device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0236] The present application provides a rendering device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the rendering method in the above-mentioned embodiment 1.

[0237] Reference below Fig.10 , which shows a schematic diagram of the structure of a rendering device suitable for implementing an embodiment of the present application. The rendering device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.10 The rendering device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0238] like Fig.10As shown, the rendering device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the rendering device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the rendering device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a rendering device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0239] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0240] The rendering device provided by the present application adopts the rendering method in the above embodiment, which can solve the technical problem that the current method takes too long to train. Compared with the prior art, the beneficial effects of the rendering device provided by the present application are the same as the beneficial effects of the rendering method provided by the above embodiment, and the other technical features in the rendering device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0241] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0242] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0243] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the rendering method in the above-mentioned embodiment.

[0244] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0245] The computer-readable storage medium may be included in the rendering device; or may exist independently without being assembled into the rendering device.

[0246] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a rendering device, the rendering device: obtains three-plane vectors and an image to be input; inputs the three-plane vectors and the image to be input into a pre-built three-plane generator, and performs viewpoint rendering on the three-plane vectors and the image to be input through a two-stage warm-up training strategy to obtain a final rendering result, wherein the three-plane generator is obtained based on deep neural network training.

[0247] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0248] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0249] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0250] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned rendering method, and can solve the technical problem that the current method takes too long to train. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the rendering method provided in the above-mentioned embodiment, and will not be repeated here.

[0251] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A rendering method, characterized in that: The method comprises: Get the three-plane vectors and the image to be input; The three-plane vectors and the image to be input are input into a pre-built three-plane generator, and the three-plane vectors and the image to be input are viewpoint rendered through a two-stage warm-up training strategy to obtain a final rendering result. The three-plane generator is obtained based on deep neural network training.

2. The method according to claim 1, characterized in that The step of inputting the three-plane vector and the image to be input into a pre-built three-plane generator, performing viewpoint rendering on the three-plane vector and the image to be input through a two-stage warm-up training strategy, and obtaining a final rendering result comprises: Inputting the three-plane vector and the image to be input into the three-plane generator to generate a first characteristic three-plane; Visualizing the first three characteristic planes to obtain a first rendering result; Optimizing the three-plane generator based on the first rendering result; Freeze the model parameters of the optimized three-plane generator; The second characteristic three-plane is obtained by forward reasoning and adjusting the shape through the three-plane generator after the model parameters are frozen; Setting parameters of the second characteristic triplane as learnable variables to obtain a learnable second characteristic triplane; The learnable second characteristic three-plane is visualized to obtain the final rendering result.

3. The method according to claim 2, characterized in that Before the step of inputting the three-plane vector and the image to be input into the three-plane generator to generate the first characteristic three-plane, the step further includes: Inputting the three-plane vector into the three-plane generator to generate a feature grid; Optimizing the position and posture of a camera that shoots the image to be input based on the feature grid to obtain an optimized camera position and posture; Inputting the image to be input into a feature extractor to extract a feature vector; Aggregating the feature vector with the three-plane vector through a cross-attention mechanism to obtain a cross-aggregation feature; The step of inputting the three-plane vector and the image to be input into the three-plane generator to generate the first characteristic three-plane comprises: The cross-convergence feature is input into the three-plane generator to generate the first feature three-plane.

4. The method according to claim 3, characterized in that The step of visualizing the first characteristic three planes to obtain a first rendering result includes: Aggregating the optimized camera pose and the first feature three planes through a decoupled three-plane aggregation algorithm to generate a first aggregated feature; Decoding the first aggregated feature by a decoder to obtain corresponding radiation color and volume density; Performing volume rendering on the corresponding radiation color and volume density to obtain the first rendering result; The step of visualizing the learnable second characteristic three-plane to obtain the final rendering result comprises: Aggregating the optimized camera pose and the learnable second feature three-plane through the decoupled three-plane aggregation algorithm to generate a second aggregated feature; Decoding the second aggregate feature by the decoder to obtain corresponding radiation color and volume density; Volume rendering is performed on the corresponding radiation color and volume density to obtain the final rendering result.

5. The method according to claim 3, characterized in that The step of inputting the image to be input into a feature extractor to extract a feature vector comprises: The feature vector is obtained by extracting features of the image to be input through a pre-built dual interlaced network optimization model.

6. The method according to claim 3, characterized in that The step of aggregating the feature vector and the three-plane vector through a cross attention mechanism to obtain a cross-aggregation feature comprises: The three-plane vector is used as a query vector, and the feature vector is used as a key and a value to perform the cross-attention mechanism to obtain the cross-aggregated feature.

7. The method according to claim 1, characterized in that The step of inputting the three-plane vector and the image to be input into a pre-built three-plane generator includes: Construct the three-plane generator, wherein each of the plane generators is composed of an intermediate block and an upsampling block, the intermediate block is composed of a convolutional layer, a structure using group normalization and an activation function, and an attention layer, the upsampling block is composed of a convolutional layer, a structure using group normalization and an activation function, and except for the last upsampling block, the upsampling block also includes a bilinear upsampler.

8. A rendering device, characterized in that: The device comprises: A data acquisition module, used for acquiring three-plane vectors and images to be input; A rendering module is used to input the three-plane vectors and the image to be input into a pre-built three-plane generator, and perform viewpoint rendering on the three-plane vectors and the image to be input through a two-stage warm-up training strategy to obtain a final rendering result. The three-plane generator is obtained based on deep neural network training.

9. A rendering device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the rendering method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the rendering method according to any one of claims 1 to 7 are implemented.

Images