Rendering method, device, equipment and storage medium

Abstract

The present application discloses a rendering method, apparatus, device and storage medium, relating to the field of image processing technology. The method comprises: obtaining a three-plane vector and an image to be input; inputting the three-plane vector and the image to be input into a pre-built three-plane generator, performing viewpoint rendering on the three-plane vector and the image to be input through a two-stage warm-up training strategy to obtain a final rendering result. The three-plane generator is obtained based on deep neural network training, and the present application can shorten the training time.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a rendering method, apparatus, device, and storage medium. Background Art

[0002] Recently, the neural network-based Neural Radiance Field (NeRF) has brought significant improvements in high-fidelity synthesis quality to the new view synthesis task.

[0003] NeRF represents 3D radiance fields through multi-layer perceptrons and obtains new views through differentiable volume rendering.

[0004] However, current methods take too long to train.

[0005] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0006] The main purpose of this application is to provide a rendering method, apparatus, device and storage medium, aiming to solve the technical problem that the current method has a long training time.

[0007] To achieve the above objectives, the present application proposes a rendering method, which includes:

[0008] Get the three-plane vector and the image to be input;

[0009] The three-plane vectors and the image to be input are input into a pre-built three-plane generator, and the three-plane vectors and the image to be input are viewpoint rendered through a two-stage warm-up training strategy to obtain a final rendering result. The three-plane generator is obtained based on deep neural network training.

[0010] In one embodiment, the step of inputting the three-plane vectors and the image to be input into a pre-built three-plane generator, performing viewpoint rendering on the three-plane vectors and the image to be input using a two-stage warm-up training strategy, and obtaining a final rendering result includes:

[0011] Inputting the three-plane vector and the image to be input into the three-plane generator to generate a first characteristic three-plane;

[0012] Visualizing the first characteristic three planes to obtain a first rendering result;

[0013] Optimizing the three-plane generator based on the first rendering result;

[0014] Freeze the model parameters of the optimized three-plane generator;

[0015] The second feature three-plane is obtained by forward reasoning and shape adjustment of the three-plane generator after the model parameters are frozen;

[0016] Setting parameters of the second characteristic triplane as learnable variables to obtain a learnable second characteristic triplane;

[0017] The learnable second characteristic three-plane is visualized to obtain the final rendering result.

[0018] In one embodiment, before the step of inputting the three-plane vector and the image to be input into the three-plane generator to generate the first characteristic three-plane, the step further includes:

[0019] Inputting the three-plane vector into the three-plane generator to generate a feature grid;

[0020] Optimizing the position and posture of a camera that captures the image to be input based on the feature grid to obtain an optimized camera position and posture;

[0021] Inputting the image to be input into a feature extractor to extract a feature vector;

[0022] Aggregating the feature vector and the three-plane vector through a cross-attention mechanism to obtain a cross-aggregated feature;

[0023] The step of inputting the three-plane vector and the image to be input into the three-plane generator to generate the first characteristic three-plane includes:

[0024] The cross-convergence feature is input into the three-plane generator to generate the first feature three-plane.

[0025] In one embodiment, the step of visualizing the first characteristic three planes to obtain a first rendering result includes:

[0026] Aggregating the optimized camera pose and the first feature three-plane using a decoupled three-plane aggregation algorithm to generate a first aggregated feature;

[0027] Decoding the first aggregated feature by a decoder to obtain corresponding radiation color and volume density;

[0028] Performing volume rendering on the corresponding radiation color and volume density to obtain the first rendering result;

[0029] The step of visualizing the learnable second characteristic three-plane to obtain the final rendering result includes:

[0030] Aggregating the optimized camera pose and the learnable second feature three-plane through the decoupled three-plane aggregation algorithm to generate a second aggregated feature;

[0031] decoding the second aggregate feature by the decoder to obtain corresponding radiation color and volume density;

[0032] Volume rendering is performed on the corresponding radiation color and volume density to obtain the final rendering result.

[0033] In one embodiment, the step of inputting the image to be input into a feature extractor to extract a feature vector includes:

[0034] The feature vector is obtained by extracting features from the image to be input using a pre-built dual-interlaced network optimization model.

[0035] In one embodiment, the step of aggregating the feature vector and the three-plane vector through a cross-attention mechanism to obtain a cross-aggregated feature includes:

[0036] The three-plane vector is used as a query vector and the feature vector is used as a key and a value to perform the cross-attention mechanism to obtain the cross-aggregated feature.

[0037] In one embodiment, the step of inputting the three-plane vector and the image to be input into a pre-built three-plane generator includes:

[0038] Construct the three-plane generator, wherein each plane generator is composed of an intermediate block and an upsampling block, the intermediate block is composed of a convolutional layer, a structure using group normalization and an activation function, and an attention layer, and the upsampling block is composed of a convolutional layer, a structure using group normalization and an activation function, and except for the last upsampling block, the upsampling block also includes a bilinear upsampler.

[0039] In addition, to achieve the above-mentioned purpose, the present application also proposes a rendering device, which includes:

[0040] A data acquisition module, used to acquire three-plane vectors and images to be input;

[0041] A rendering module is used to input the three-plane vectors and the image to be input into a pre-built three-plane generator, and perform viewpoint rendering on the three-plane vectors and the image to be input through a two-stage warm-up training strategy to obtain a final rendering result. The three-plane generator is obtained based on deep neural network training.

[0042] In addition, to achieve the above objectives, the present application also proposes a rendering device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the rendering method described above.

[0043] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is executed by a processor, the steps of the rendering method described above are implemented.

[0044] One or more technical solutions proposed in this application have at least the following technical effects:

[0045] By feeding the three-plane vectors and the input image into a pre-built three-plane generator, this application achieves fast inference, efficient data compression, and better convergence under ambiguous poses. As a result, this application significantly reduces training time and improves the performance of rendering from new viewpoints and camera pose estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0048] Figure 1 A flowchart of the first embodiment of the rendering method of this application is provided;

[0049] Figure 2 This is a flowchart of the warm-up training strategy in the rendering method of this application;

[0050] Figure 3a Schematic diagram of the camera and feature grid for this application;

[0051] Figure 3b This is a diagram showing the visualization results of the characteristic plane of the Horn angle of the scene;

[0052] Figure 4 This is a schematic diagram of the plane generator effect of the rendering method of this application;

[0053] Figure 5 A complete flowchart of the rendering method for this application;

[0054] Figure 6a A schematic diagram comparing the rendering results of this application with other methods;

[0055] Figure 6b A diagram comparing the training efficiency of this application and other methods;

[0056] Figure 7 A schematic diagram of the new viewpoint rendering effect of this application compared with the NeRF-Synthetic neural radiation field synthesis dataset and other methods;

[0057] Figure 8 This is a schematic diagram of the new viewpoint rendering effect of this application compared with other methods in the real-life LLFF (Local Light Field Fusion) dataset;

[0058] Figure 9 This is a schematic diagram of the module structure of the rendering device according to an embodiment of the present application;

[0059] Figure 10 Schematic diagram of the device structure of the hardware operating environment involved in the rendering method in the embodiment of the present application.

[0060] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0061] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0062] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0063] Technical terms involved in the embodiments of the present invention:

[0064] NeRF: NeRF (Neural Radiance Field) is a method of 3D reconstruction that trains a neural network to fit a 3D scene, so that corresponding images can be obtained from new viewpoints that have never been seen before.

[0065] PSNR: PSNR (Peak Signal Noise Ratio) is usually used to evaluate the similarity between two images. The unit is decibel dB. The higher the peak signal-to-noise ratio, the higher the image similarity.

[0066] iNeRF: iNeRF (Inverting neural radiance fields for pose estimation) demonstrates the ability to perform pose estimation using a reconstructed NeRF model.

[0067] NeRFmm: NeRFmm (Neural radiance fields without known camera parameters) jointly estimates the camera's intrinsic parameters, extrinsic parameters, and NeRF.

[0068] BARF: BARF (Bundle-adjusting neural radiance fields) proposes a coarse-to-fine position encoding method.

[0069] SC-NeRF: SC-NeRF (Self-calibrating neural radiance fields) estimates camera distortion and proposes geometric regularization.

[0070] GNeRF: GNeRF (Gan-based neural radiance field without posed camera) uses a generative adversarial network to handle the joint optimization problem.

[0071] VMRF: VMRF (View Matching Neural Radiance Fields) uses a generative adversarial network to handle the joint optimization problem.

[0072] SiNeRF: SiNeRF (Sinusoidal neural radiance fields for joint poseestimation and scene reconstruction) modifies different activation functions to obtain more appropriate pose estimates.

[0073] GARF: GARF (Gaussian activated neural radiance fields for high fidelity reconstruction and pose estimation) modifies different activation functions to obtain more appropriate pose estimation.

[0074] NoPe-NeRF: NoPe-NeRF (Learning neural radiance field with no pose prior) uses a single depth estimate to geometrically guide the joint optimization, which is designed for long sequences of images.

[0075] FDC-NeRF: FDC-NeRF (Learning pose-free neural radiance fields with flow-depth consistency) uses a single depth estimate to geometrically guide the joint optimization, which is designed for long sequences of images.

[0076] L2G-NeRF: L2G-NeRF (Local-to-Global Neural Radiance Fields) proposes a local-to-global registration method to estimate pose from multiple learned local poses.

[0077] SPARF: SPARF (Neural radiance fields from sparse and noisy poses) uses a pre-trained image matching model as a prior in the sparse view setting.

[0078] LU-NeRF: LU-NeRF (Scene and pose estimation by synchronizing local unposed nerfs) optimizes NeRF without any pose initialization.

[0079] LocalRF: LocalRF (Progressively optimized local radiance fields for robust view synthesis) is proposed for long sequences of static videos.

[0080] Robust-DynRF: Robust-DynRF (Robust Dynamic Neural Radiance Fields) is proposed for dynamic videos.

[0081] Hash-BARF: Hash-BARF (Instant neural graphics primitives with a multiresolution hash encoding) uses multiresolution hash encoding to accelerate the joint optimization of pose and neural radiation field in previous methods.

[0082] The main solution of the embodiment of the present application is: obtaining three-plane vectors and the image to be input; inputting the three-plane vectors and the image to be input into a pre-built three-plane generator, and performing viewpoint rendering on the three-plane vectors and the image to be input through a two-stage warm-up training strategy to obtain the final rendering result. The three-plane generator is obtained based on deep neural network training.

[0083] In this embodiment, for ease of description, the following description is made with the rendering system as the execution entity.

[0084] Because the current method takes too long to train.

[0085] Therefore, this application provides a solution. By feeding the three-plane vectors and the input image into a pre-built three-plane generator, this application achieves fast inference, efficient data compression, and better convergence under ambiguous poses. As a result, this application significantly reduces training time and improves the performance of new viewpoint rendering and camera pose estimation.

[0086] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication, and program execution capabilities, such as a tablet computer, personal computer, or mobile phone, or an electronic device or rendering device capable of implementing the aforementioned functions, or an electronic system or rendering system capable of implementing the aforementioned functions. This embodiment and the following embodiments will be described below using a rendering system as an example.

[0087] Recently, the neural network-based Neural Radiance Field (NeRF) has significantly improved the quality of high-fidelity synthesis for novel view synthesis tasks. NeRF represents 3D radiance fields using a multi-layer perceptron and extracts novel views through differentiable volume rendering. A key prerequisite for achieving ideal rendering results with NeRF is accurately calibrated camera parameters.

[0088] However, accurate camera pose is not easy to obtain, which depends heavily on the accuracy and robustness of structure-from-motion (SfM) algorithms such as the COLMAP algorithm.

[0089] To this end, previous researchers have increasingly focused on reducing the dependence on camera parameters by simultaneously optimizing NeRF and camera parameters.

[0090] In recent years, the joint optimization problem of camera parameters and neural radiance fields has been widely studied. iNeRF first demonstrated the ability to utilize a reconstructed NeRF model for pose estimation. NeRFmm first jointly estimated camera intrinsics, extrinsic parameters, and NeRF. BARF proposed a coarse-to-fine position encoding method. SC-NeRF estimated camera distortion and proposed geometric regularization. GNeRF and VMRF adopted generative adversarial networks to address the joint optimization problem, while SiNeRF and GARF modified different activation functions to obtain more appropriate pose estimates. Recently, NoPe-NeRF and FDC-NeRF utilized a single depth estimate for geometric guidance of the joint optimization, which is specifically designed for long image sequences. L2G-NeRF proposed a local-to-global registration method to estimate pose from multiple learned local poses. SPARF used a pretrained image matching model as a prior in the sparse view setting. LU-NeRF optimized NeRF without any pose initialization. LocalRF and Robust-DynRF were proposed for long sequences of static and dynamic videos, respectively. The recent Hash-BARF uses multi-resolution hash coding to accelerate the joint optimization of pose and neural radiance field in previous methods.

[0091] However, previous joint optimization methods are typically based on primitive multi-layer perceptron representations, and most existing methods require hours or even days to train using modern powerful GPUs, which severely limits their use in practical applications. Therefore, improving the training efficiency of such joint optimization is of great significance.

[0092] Based on this, the present application embodiment provides a rendering method, referring to Figure 1 , Figure 1 A flowchart illustrating the first embodiment of the rendering method of this application is provided.

[0093] In this embodiment, the rendering method includes steps S20 to S30:

[0094] Step S20, obtaining three-plane vectors and the image to be input;

[0095] Triplane vectorization is a method of representing or reconstructing a scene using three planes in 3D space. These three planes are often used to capture and represent different perspectives and details of a 3D scene. During the rendering process, triplane vectorization helps the model more accurately represent and process the 3D information of the scene, resulting in more detailed and accurate rendering results.

[0096] The image to be input refers to the original image that needs to be rendered or processed. This image can be a photo actually taken or a synthesized image. In this embodiment, a picture actually taken by a camera is used as an example.

[0097] It should be noted that the three-plane vectors may be three-plane noise vectors following Gaussian distribution.

[0098] In step S30, the three-plane vectors and the image to be input are input into a pre-built three-plane generator, and viewpoint rendering is performed on the three-plane vectors and the image to be input through a two-stage warm-up training strategy to obtain a final rendering result. The three-plane generator is obtained based on deep neural network training.

[0099] It's important to note that the three-plane generator is a convolutional deep neural network-based model designed to generate three independent planes with spatial features from a noise vector. The generator consists of three neural networks with identical structures but shared parameters, each responsible for generating a different plane to account for the anisotropy and decoupling of feature planes at different viewpoints. This design ensures that the plane features extracted from the input noise are better adapted to different viewpoints and feature requirements in 3D scenes.

[0100] It should be noted that in the first stage of warm-up training, the three-plane vector and the image to be input are input into the three-plane generator to generate a first feature three-plane; the first feature three-plane is visualized to obtain a first rendering result; and the three-plane generator is optimized based on the first rendering result.

[0101] In the second stage of warm-up training, the model parameters of the optimized three-plane generator are frozen; forward reasoning and shape adjustment are performed on the three-plane generator after the model parameters are frozen to obtain a second characteristic three-plane; the parameters of the second characteristic three-plane are set as learnable variables to obtain a learnable second characteristic three-plane; the learnable second characteristic three-plane is visualized to obtain the final rendering result.

[0102] Reference Figure 2 , Figure 2 This is a flowchart of the warm-up training strategy in the rendering method of this application;

[0103] It should be noted that given a three-plane noise vector following a Gaussian distribution as input, a convolution-based neural network is introduced as a three-plane generator to extract planar features, with the aim of parameterizing the spatial feature grid into a deep neural network.

[0104] Since the feature planes of different viewpoints in a 3D scene are highly decoupled and the features between different planes are anisotropic, three independent neural networks with the same structure but no shared parameters can be used as disentangled generators to generate three different planes respectively.

[0105] The generator is randomly initialized in each scene rather than being pre-trained, so similar to previous work, it still needs to be optimized scene by scene.

[0106] Reference Figure 4 , Figure 4 This is a schematic diagram of the plane generator effect of the rendering method of this application;

[0107] Figure 4 The following are (a) a feature visualization method that directly combines three planes and camera poses and (b) a feature visualization method of the present application using a three-plane generator. Figure 4 It can be seen that compared with the feature visualization method that directly combines the three planes and camera pose, the three-plane generator of this application reduces the error of local updates and shows less noise in the visualization of the planar feature map.

[0108] Furthermore, the triplanar generator mines the geometric and textural features of the scene and embeds them into the triplanar representation instead of simply overfitting to new viewpoints.

[0109] It is worth noting that to ensure that the generated planes conform to the spatial characteristics and structure of the real world, the input of the three-plane generator is a randomly initialized three-plane noise vector. During the generation process, the core features are maintained in the three-plane structure from input to output to adapt to the decoupled scene representation.

[0110] After training with the three-plane generator, we can get the following insights: once the camera pose is optimized to a good position, the effect of directly optimizing the three planes will be better than using the three-plane generator to generate the three planes from beginning to end, especially the rendering effect in high-frequency areas.

[0111] Therefore, although the three-plane generator alleviates local update errors, it is implicitly constrained by the different feature blocks on the plane grid caused by the generator, which introduces over-smoothing on the feature grid. Therefore, a two-stage warm-up training strategy can be proposed. After the faster and better joint optimization in the first stage, it switches to direct plane optimization, thereby achieving better rendering quality for new viewpoints overall.

[0112] In the second stage, the parameters generated by the three-plane generator in the first stage can be utilized (i.e., the second feature three-plane is obtained by forward reasoning and adjusting the shape through the three-plane generator after the model parameters are frozen) instead of initializing the parameters from scratch.

[0113] This embodiment provides a rendering method, which brings fast reasoning, efficient data compression performance and better convergence under fuzzy poses by inputting three-plane vectors and the image to be input into a pre-built three-plane generator. Therefore, the training time of this application is significantly shortened, while the performance of new viewpoint rendering and camera pose estimation is improved. In addition, this application proposes to use a three-plane neural radiation field with fast training and rendering speed, highest compression efficiency, best performance and compact data structure to perform joint estimation of camera pose-neural radiation field, thereby solving the problems of long training time and slow reasoning speed of previous methods.

[0114] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description and will not be described in detail. On this basis, step S30, inputting the three-plane vector and the image to be input into the pre-built three-plane generator, performing viewpoint rendering on the three-plane vector and the image to be input through a two-stage warm-up training strategy, and obtaining the final rendering result also includes steps S35 to S311:

[0115] Step S35, inputting the three-plane vector and the image to be input into the three-plane generator to generate a first characteristic three-plane;

[0116] Among them, for the two-stage warm-up training strategy, in the first stage, the embodiment of the present application inputs the random three-plane noise vector and the image to be input into the proposed three-plane generator to generate a feature three-plane.

[0117] Specifically, the three-plane generator extracts features of different levels or resolutions from the input image and maps these features onto three independent planes. Each plane represents specific features or information. The combination of these planes forms a rich feature representation, which is the first feature three-plane.

[0118] Step S36, visualizing the first characteristic three planes to obtain a first rendering result;

[0119] The optimized camera pose and the first feature three-plane can be aggregated using a decoupled three-plane aggregation algorithm to generate a first aggregate feature; the first aggregate feature can be decoded using a decoder to obtain corresponding radiation color and volume density; and volume rendering can be performed on the corresponding radiation color and volume density to obtain the first rendering result;

[0120] Step S37: optimizing the three-plane generator based on the first rendering result;

[0121] Wherein, in the second stage, the three-plane generator may be optimized based on the first rendering result;

[0122] Furthermore, the optimization of the three-plane generator can be obtained by minimizing the photometric consistency loss, and the specific formula is:

[0123]

[0124] Among them, L stands for Loss, which is the loss function; Σ stands for sum.

[0125] If N points are collected on a picture, there will be N rays corresponding to them. The color of each ray output can be obtained by the above method, making it consistent with the real image color in the data set. By constraining the optimization, the loss function is MSE.

[0126] Among them, I i The color output for each ray, is the actual image color.

[0127] Step S38, freezing the model parameters of the optimized three-plane generator;

[0128] Step S39, performing forward reasoning and adjusting the shape by the three-plane generator after the model parameters are frozen to obtain a second characteristic three-plane;

[0129] Forward reasoning involves passing input data (3D shapes or other feature descriptions) through the model to generate output results (feature triplanes). These feature triplanes are generated using the optimized model parameters of the triplane generator, but they may also require shape adjustment or post-processing to further optimize their representation based on specific needs.

[0130] At the beginning of the second stage three-plane optimization process, the generator can be abandoned and a single forward inference can be performed to obtain the second feature three-plane.

[0131] As an implementation method, based on the first characteristic three-plane, a multi-scale second characteristic three-plane can be obtained by bilinear interpolation, and direct three-plane optimization can be performed.

[0132] In addition, when performing a single forward inference to obtain the second feature triplane, multiple feature grids with different resolutions are used to enhance the representation of the structure and details of the second feature triplane in a coarse-to-fine manner.

[0133] Step S310, setting the parameters of the second characteristic triplane as learnable variables to obtain a learnable second characteristic triplane;

[0134] The parameters of the three feature planes are set as learnable variables. This means that the parameters will be fine-tuned during subsequent training to better suit the specific task or dataset. By setting the parameters of the three feature planes as learnable variables, the model can optimize these parameters through backpropagation during training to further improve the final rendering or estimation performance.

[0135] Step S311 : Visualize the learnable second characteristic three-plane to obtain the final rendering result.

[0136] Among them, the optimized camera pose and the learnable second feature three-plane can be aggregated through the decoupled three-plane aggregation algorithm to generate a second aggregate feature; the second aggregate feature is decoded by the decoder to obtain the corresponding radiation color and volume density; the corresponding radiation color and volume density are volume rendered to obtain the final rendering result.

[0137] The two phases in the two-phase warmup training strategy refer to the model training process. The switching between the two phases is determined by the number of training iterations. During the first phase, the network parameters of the three-plane generator and the camera pose are jointly optimized. After reaching a certain number of iterations, the model training switches to the second phase.

[0138] In the second phase, the model parameters of the three-plane generator are first frozen, and the generator network parameters are no longer updated. The three-plane generator then forward infers a three-plane vector, reshapes the three-plane vector into a feature three-plane, and sets the parameters of this three-plane as learnable variables. Therefore, in the second phase, the camera pose and the learnable feature three-plane are jointly optimized, and the generator from the first phase is discarded. The feature three-plane is a decoupled representation of three-dimensional space. After emitting a ray from the camera's optical center, the points on the ray are projected onto corresponding positions on the three-plane to obtain the corresponding features. Aggregating features from different planes and decoding them through a decoder yields the 3D features corresponding to the points on the ray. This process is repeated until corresponding 3D features are obtained for multiple points on the ray. Then, through volume rendering in traditional graphics, the colors of the 2D image corresponding to the ray are output. During training, the parameters of the three-plane are directly optimized.

[0139] This embodiment proposes a new training strategy—a two-stage warm-up training strategy. In the second stage, the three-plane generator is discarded and the three planes are trained directly to reduce the problem of over-smoothing of the feature plane caused by using the three-plane generator.

[0140] Based on the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the second embodiment can be referred to the above introduction and will not be repeated hereafter. On this basis, step S35 inputs the three-plane vector and the image to be input into the three-plane generator, and before generating the first characteristic three-plane, steps S31 to S34 are also included:

[0141] Step S31, inputting the three-plane vector into the three-plane generator to generate a feature grid;

[0142] Step S32, optimizing the posture of the camera that captures the input image based on the feature grid to obtain an optimized camera posture;

[0143] In the first stage, the embodiment of the present application inputs a random three-plane noise vector into the proposed three-plane generator to generate different feature grids for scene representation while optimizing the camera pose.

[0144] Step S33, inputting the image to be input into a feature extractor to extract a feature vector;

[0145] Step S34, aggregating the feature vector and the three-plane vector through a cross attention mechanism to obtain a cross-aggregated feature;

[0146] Among them, the feature vector extracted from the input image after passing through the feature extractor is aggregated into a three-plane vector through a cross-attention mechanism to obtain a cross-aggregated feature.

[0147] The step S35 of inputting the three-plane vector and the image to be input into the three-plane generator to generate the first characteristic three-plane further includes step S351:

[0148] Step S351: input the cross-aggregation feature into the three-plane generator to generate the first feature three-plane.

[0149] In addition, this embodiment improves the explicit representation of the three planes into an implicit network-explicit three-plane hybrid representation based on deep network parameterization. Since the neural network brings sufficient smoothness and global information to the three planes, the training is easier to converge without falling into local optimality, so the camera pose estimation result is more accurate.

[0150] Previous aggregation methods did not take into account the differences in training difficulty between different planes, so the different quality of supervision signals resulted in suboptimal camera pose optimization.

[0151] Therefore, embodiment 4 of the present application is proposed.

[0152] Based on the third embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as those in the third embodiment can be referred to the above introduction and will not be described in detail. On this basis, step S36, visualizing the first feature three planes to obtain the first rendering result also includes steps S361 to S363:

[0153] Step S361, aggregating the optimized camera pose and the first feature three-plane using a decoupled three-plane aggregation algorithm to generate a first aggregated feature;

[0154] Before decoding and rendering, it is necessary to aggregate the features of the sampling points queried from different planes. The embodiment of the present application proposes a decoupled plane aggregation method.

[0155] Since the three-plane representation is an interpretable explicit radiance field that decomposes the scene into three orthogonal planes aligned with the X, Y, and Z axes, each plane contains 3D scene features from an orthogonal perspective. Images taken from different angles provide more comprehensive scene information for the three-plane, while images from limited or unevenly distributed viewpoints provide different quality supervision information for each plane.

[0156] Reference Figure 3a , Figure 3a Schematic diagram of the camera and feature grid for this application;

[0157] Depend on Figure 3a As shown, the angles of images or videos are usually consistent, all objects are mainly facing the camera, and the YZ plane obtains richer scene information and is easier to learn.

[0158] Reference Figure 3b , Figure 3b This is a diagram showing the visualization results of the characteristic plane of the Horn angle of the scene;

[0159] During the optimization process, a plane that is easier to learn brings richer supervision information, thereby providing better supervision for camera parameter optimization. On the contrary, learning a more challenging plane requires additional training iterations to accurately represent the decoupled scene information, thereby increasing the probability of providing fuzzy supervision for camera pose optimization. The difference in learning difficulty between different planes increases the difficulty of pose optimization. To this end, the embodiment of the present application proposes an aggregation algorithm for decoupling three planes, and the implementation formula is as follows:

[0160]

[0161] Where D(x) represents the gradient separation operation, P represents the feature plane, ψ represents the interpolation operation on the feature plane, and π represents the projection operation of 3D points to each plane. The result of the decoupled plane feature aggregation algorithm is equivalent to π k (Fk +1), is a Hadamard product, thus preserving the expressive power of the three planes. In practice, the additive term 1 can be replaced with the hyperparameter λ. This decoupled three-plane aggregation approach allows backpropagation of the three-plane gradients from the product and backpropagation of the camera pose gradient from the sum.

[0162] Step S362: decoding the first aggregated feature through a decoder to obtain corresponding radiation color and volume density;

[0163] As an implementation, the decoder may be an MLP multi-layer perceptron decoder.

[0164] Step S363: performing volume rendering on the corresponding radiation color and volume density to obtain the first rendering result;

[0165] The first aggregated feature is decoded by a decoder to obtain the radiation color and volume density. The final image to be rendered can be rendered by synthesizing the ray r(t) = o + td between the near plane and the far plane along the camera that captured the input image to obtain a first rendering result. The volume rendering formula can be expressed as:

[0166]

[0167] Where o is the camera's optical center, d represents the direction of the light emitted from the camera's optical center, and t represents the distance along the sampling line. o + td can be analogized to a vector, with t representing the actual vector length. r(t) represents the point at position t where the light is emitted from o along direction d.

[0168] I represents the RGB value corresponding to the ray on the 2D image. tn and tf represent the nearest (tnear) and farthest (tfar) intervals of the ray's emission. W(t) represents the cumulative transmittance at that location, σ represents the volume density, and σ(r(t)) represents the volume density value at a point on the index ray (as previously mentioned, a point is projected onto three planes to obtain three features. The aggregated 3D features are decoded to obtain the color c and volume density sigma of the 3D point). c represents the color (also as previously mentioned, the 3D features after the three-plane aggregation are decoded). c(r(t), d) represents the color of the 3D point in the observation direction d. The size of t increases from tn to tf along the ray, so the final overall color of the ray can be calculated by integration. W represents the cumulative transmittance, which can be used to render depth.

[0169] Step S311, visualizing the learnable second characteristic three-plane to obtain the final rendering result includes steps S3111 to S3113:

[0170] Step S3111, aggregating the optimized camera pose and the learnable second feature three-plane through the decoupled three-plane aggregation algorithm to generate a second aggregated feature;

[0171] The implementation formula of the decoupled three-plane aggregation algorithm is as described above.

[0172] Step S3112: decoding the second aggregated feature by the decoder to obtain corresponding radiation color and volume density;

[0173] Furthermore, the decoder and camera poses of the second stage are directly inherited from the first stage since they apply to the same three planes.

[0174] Step S3113: Perform volume rendering on the corresponding radiation color and volume density to obtain the final rendering result.

[0175] The implementation formula of volume rendering is as described above.

[0176] This embodiment improves the aggregation method and proposes a decoupled three-plane aggregation algorithm. Since the gradient of the camera pose estimation and the gradient of the features on different three planes are decoupled after the improvement, the joint optimization is more robust, so the performance of new viewpoint rendering and camera pose estimation is improved. In addition, this application proposes a new plane feature aggregation method. After obtaining the three-plane features, they need to be aggregated in some way and sent to the subsequent module. This application proposes a decoupled aggregation method to aggregate different plane features and improve the quality of joint optimization.

[0177] Based on the third embodiment of the present application, in the fifth embodiment of the present application, the same or similar contents as those in the third embodiment can be referred to the above introduction and will not be described in detail. On this basis, step S33, inputting the image to be input into the feature extractor, extracting the feature vector also includes step S331:

[0178] Step S331 : extracting features from the image to be input by using a pre-built dual interlaced network optimization model to obtain the feature vector.

[0179] After parameterizing the three planes using the three-plane generator, the three-plane texture representation of the scene can also be enhanced through the scene texture embedding module, thereby alleviating the problem of camera pose-neural radiance field optimization gradient ambiguity.

[0180] As an implementation method, a pre-built DINOv2 dual interleaved network optimization model may be applied to encode the input image into a feature vector.

[0181] The steps for constructing a dual-interleaved network optimization model are as follows:

[0182] 1. Build two intertwined neural network branches, which can be convolutional neural networks (CNNs) or other suitable structures. One branch is responsible for extracting low-level features, while the other branch processes higher-level features.

[0183] 2. Implement feature interleaving between certain layers of the network, that is, combining the output of one branch with the input of another branch. This interleaving can help the network capture features at different levels.

[0184] 3. Use optimization algorithms (such as gradient descent) to adjust network parameters. Through the training process, the model will learn how to better extract and integrate features.

[0185] 4. Typically, the two branches are trained together, using a shared loss function to optimize the entire network. This facilitates information flow between the two branches and improves model performance.

[0186] This embodiment uses a pre-built dual-interleaved network optimization model to extract features from the input image to obtain the feature vector. The dual-interleaved network optimization model is a carefully designed and trained deep learning model that effectively extracts high-level, abstract features from the input image. These features are not just pixel-level information; they capture semantic and structural information within the image, facilitating the precise execution of subsequent tasks.

[0187] Based on the third embodiment of the present application, in the sixth embodiment of the present application, the same or similar contents as those in the third embodiment can be referred to the above introduction and will not be repeated hereafter. On this basis, step S34, aggregating the feature vector and the three-plane vector through the cross attention mechanism to obtain the cross-aggregated feature also includes step S341:

[0188] Step S341: Use the three-plane vector as a query vector and the feature vector as a key and a value to execute the cross-attention mechanism to obtain the cross-aggregated feature.

[0189] Among them, the feature vector extracted by the feature extractor is a 2D feature vector, and the cross attention mechanism can be applied to merge the 2D feature vector into three planes.

[0190] During the optimization process, the 2D feature vectors and the three-plane vectors will automatically learn the alignment between the patterns, thereby better integrating the image features into the three-plane.

[0191] Specifically, a fixed three-plane noise vector can be used as the query vector, and the extracted feature vectors can be used as keys and values ​​to perform a cross-attention mechanism. Finally, the aggregated cross-aggregated features are input into the three-plane generator for subsequent operations.

[0192] This embodiment uses the three-plane vector as the query vector and the feature vector as the key and value to execute the cross-attention mechanism to obtain the cross-aggregated features. The cross-attention mechanism allows for the aggregation of feature information at different levels, thereby forming a more comprehensive and comprehensive feature representation. This multi-level aggregation can improve the system's ability to understand and process complex information.

[0193] Based on the first embodiment of the present application, in the seventh embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description and will not be described in detail. On this basis, step S20, before inputting the three-plane vector and the image to be input into the pre-built three-plane generator, also includes step S10:

[0194] Step S10, constructing the three-plane generator, wherein each of the plane generators is composed of an intermediate block and an upsampling block, the intermediate block is composed of a convolutional layer, a structure using group normalization and an activation function, and an attention layer, and the upsampling block is composed of a convolutional layer, a structure using group normalization and an activation function, and except for the last upsampling block, the upsampling block also includes a bilinear upsampler.

[0195] As an implementation, the specific implementation of the three-plane generator is as follows: the shape of the three-plane noise vector is set to (3×8×20×20), where 3 represents the number of feature planes, 8 represents the hidden dimension, and 20×20 is the spatial shape.

[0196] The three-plane vector is reshaped into (3×400×8), and each plane (1×400×8) is set as the query vector, and a cross-attention operation is performed with the extracted feature vector (1×3889×384) to obtain the cross-aggregated features.

[0197] The cross-aggregated features are then re-transformed back to (3×8×20×20) for input to the three-plane generator.

[0198] It should be noted that the generator for each plane consists of an intermediate block and L upsampling blocks (L = 5). The intermediate block consists of two 2D convolutional layers with residual connections (res-conv-layers) and an attention layer. Group normalization and SiLU activation functions are applied after each residual convolution layer.

[0199] For the upsampling block, two residual convolution layers can be used, followed by group normalization and SiLU activation function.

[0200] After that, each block is appended with a bilinear upsampler, except the last block.

[0201] The upsampling layer expands the spatial size of the feature vector (3×8×20×20) to the shape of the final three-plane grid (3×64×320×320).

[0202] This application proposes a novel three-plane generator based on a deep neural network, constructing a new explicit-implicit hybrid scene representation based on three planes for joint estimation of camera poses and synthesis of new viewpoints. This approach, combining deep network parameterized three-planes with pose optimization, is the first application of this joint optimization approach.

[0203] For example, to help understand the implementation process of the rendering method of the above embodiment, please refer to Figure 5 , Figure 5 A complete flowchart of the rendering method for this application;

[0204] Figure 5 The overall description is as follows: During the training phase, the vector of the three-plane is input to the three-plane generator of the first stage to obtain the corresponding three-plane. At the same time, the input image also promotes the generation of the three-plane under the action of the cross-attention mechanism. At this time, it is determined whether the current training iteration round is the second stage. If not, the feature vector generated by the three-plane generator is directly used, and the shape is re-adjusted to become the feature three-plane. The light emitted by the camera posture is used to sample the points on the three-plane. The structure is aggregated and input to the decoder for volume rendering to obtain the final result. If the current round is judged to be the second stage, the parameters of the three-plane generator are frozen, the output three-plane vector is set to a learnable parameter, and the shape is re-adjusted to become the feature three-plane, and the above operations are repeated.

[0205] The difference between the two phases is that during backpropagation, the three-plane generator in the first phase generates the three-planes, so the gradients act on the generator's network parameters. Updates to the parameters within the three-planes in the first phase are reflected in changes to the generator's network parameters. In the second phase, the three-plane generator is frozen, and the three-planes become learnable parameters. Gradients act directly on the three-plane parameters without passing through the generator.

[0206] Specifically, the process includes the following steps:

[0207] Step S100, inputting the three-plane vector into a three-plane generator to generate a feature grid;

[0208] Step S101, optimizing the camera pose based on the feature grid to obtain an optimized camera pose;

[0209] Step S103, inputting the image to be input into a feature extractor to extract a feature vector;

[0210] Step S104, aggregating the feature vector and the three-plane vector through a cross attention mechanism to obtain a cross-aggregated feature;

[0211] Step S105: input the cross-aggregation features into a three-plane generator to generate a feature three-plane.

[0212] Step S106: Determine whether to switch to the second stage.

[0213] Step S107: If not, the optimized camera pose and the feature three planes are aggregated using a decoupled three-plane aggregation algorithm to generate a first aggregated feature;

[0214] Step S108: decoding the first aggregated feature through a decoder to obtain corresponding radiation color and volume density;

[0215] Step S109, performing volume rendering on the corresponding radiation color and volume density to obtain the first rendering result;

[0216] Step S110: If yes, freeze the model parameters of the optimized three-plane generator;

[0217] Step S111, performing forward reasoning and adjusting the shape by the three-plane generator after the model parameters are frozen to obtain a second characteristic three-plane;

[0218] Step S112, setting the parameters of the second characteristic triplane as learnable variables to obtain a learnable second characteristic triplane;

[0219] Step S113, aggregating the optimized camera pose and the learnable feature three-plane through the decoupled three-plane aggregation algorithm to generate a second aggregated feature;

[0220] Step S114, decoding the second aggregated feature by the decoder to obtain corresponding radiation color and volume density;

[0221] Step S115 , performing volume rendering on the corresponding radiation color and volume density to obtain the final rendering result.

[0222] This application is compared with other methods as follows:

[0223] like Figure 6a As shown, Figure 6a A schematic diagram comparing the rendering results of this application with other methods;

[0224] Depend on Figure 6aIt can be seen that the PSNR of the neural radiation field of the BARF bundle adjustment method is 22.55; the PSNR of the local to global neural radiation field of L2G-NeRF is 23.49; the PSNR of the baseline method is 21.86; the PSNR of the rendering method of this application is 25.35, which is higher than other methods.

[0225] like Figure 6b As shown, Figure 6b A diagram comparing the training efficiency of this application and other methods;

[0226] Depend on Figure 6b It can be seen that the training time of this method is shorter than that of the neural radiation field of the BARF bundle adjustment method and the local to global neural radiation field of L2G-NeRF; the displacement error of this method is smaller than that of other methods.

[0227] Reference Figure 7 , Figure 7 A schematic diagram of the new viewpoint rendering effect of this application compared with the NeRF-Synthetic neural radiation field synthesis dataset and other methods;

[0228] like Figure 7 As shown in the figure, the rendering method of this application has better rendering effect than other methods and is closer to the real image.

[0229] Reference Figure 8 , Figure 8 A schematic diagram of the new viewpoint rendering effect of this application in the real-life LLFF (Layered Light Field Flow) dataset compared with other methods;

[0230] like Figure 8 As shown in the figure, the rendering method of this application has better rendering effect than other methods and is closer to the real image.

[0231] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the rendering method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0232] This application also provides a rendering device, please refer to Figure 9 , the rendering device includes:

[0233] A data acquisition module 10 is used to acquire three-plane vectors and an image to be input;

[0234] The rendering module 20 is used to input the three-plane vectors and the image to be input into a pre-built three-plane generator, and perform viewpoint rendering on the three-plane vectors and the image to be input through a two-stage warm-up training strategy to obtain a final rendering result. The three-plane generator is obtained based on deep neural network training.

[0235] The rendering device provided in this application, utilizing the rendering method described in the aforementioned embodiments, can address the technical issue of excessively long training times associated with current methods. Compared to the prior art, the rendering device provided in this application achieves the same beneficial effects as the rendering method described in the aforementioned embodiments. Other technical features of the rendering device are the same as those disclosed in the aforementioned embodiments and are not further elaborated upon here.

[0236] The present application provides a rendering device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the rendering method in the above-mentioned embodiment 1.

[0237] Reference below Figure 10 , which shows a schematic diagram of the structure of a rendering device suitable for implementing the embodiments of the present application. The rendering device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 10 The rendering device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0238] like Figure 10As shown, the rendering device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the rendering device. Processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 1003 including, for example, a magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 can allow the rendering device to communicate with other devices wirelessly or wired to exchange data. Although the figure shows a rendering device with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems can be implemented or have alternatively.

[0239] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0240] The rendering device provided in this application, employing the rendering method described in the aforementioned embodiment, can address the technical issue of excessively long training times associated with current methods. Compared to the prior art, the rendering device provided in this application achieves the same beneficial effects as the rendering method described in the aforementioned embodiment. Other technical features of the rendering device are the same as those disclosed in the aforementioned embodiment and are not further elaborated upon here.

[0241] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0242] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0243] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer program) stored thereon, and the computer-readable program instructions are used to execute the rendering method in the above embodiment.

[0244] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0245] The computer-readable storage medium may be included in the rendering device, or may exist independently without being incorporated into the rendering device.

[0246] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the rendering device, the rendering device: obtains a three-plane vector and an image to be input; inputs the three-plane vector and the image to be input into a pre-built three-plane generator, and performs viewpoint rendering on the three-plane vector and the image to be input through a two-stage preheating training strategy to obtain a final rendering result. The three-plane generator is obtained based on deep neural network training.

[0247] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0248] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0249] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0250] The computer-readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned rendering method. This computer-readable storage medium can address the technical issue of excessive training time associated with current methods. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are similar to those of the rendering method provided in the aforementioned embodiments and are not further elaborated here.

[0251] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A rendering method, characterized in that: The method comprises: Get the three-plane vector and the image to be input; Inputting the three-plane vectors and the image to be input into a pre-built three-plane generator, performing viewpoint rendering on the three-plane vectors and the image to be input through a two-stage warm-up training strategy to obtain a final rendering result, wherein the three-plane generator is obtained based on deep neural network training; The step of inputting the three-plane vector and the image to be input into a pre-built three-plane generator, performing viewpoint rendering on the three-plane vector and the image to be input through a two-stage warm-up training strategy, and obtaining a final rendering result includes: Inputting the three-plane vector and the image to be input into the three-plane generator to generate a first characteristic three-plane; Visualizing the first characteristic three planes to obtain a first rendering result; Optimizing the three-plane generator based on the first rendering result; Freeze the model parameters of the optimized three-plane generator; The second feature three-plane is obtained by forward reasoning and shape adjustment of the three-plane generator after the model parameters are frozen; Setting parameters of the second characteristic triplane as learnable variables to obtain a learnable second characteristic triplane; The learnable second characteristic three-plane is visualized to obtain the final rendering result.

2. The method according to claim 1, wherein Before the step of inputting the three-plane vector and the image to be input into the three-plane generator to generate the first characteristic three-plane, the method further includes: Inputting the three-plane vector into the three-plane generator to generate a feature grid; Optimizing the position and posture of a camera that captures the image to be input based on the feature grid to obtain an optimized camera position and posture; Inputting the image to be input into a feature extractor to extract a feature vector; Aggregating the feature vector and the three-plane vector through a cross-attention mechanism to obtain a cross-aggregated feature; The step of inputting the three-plane vector and the image to be input into the three-plane generator to generate the first characteristic three-plane includes: The cross-convergence feature is input into the three-plane generator to generate the first feature three-plane.

3. The method according to claim 2, wherein The step of visualizing the first characteristic three planes to obtain a first rendering result includes: Aggregating the optimized camera pose and the first feature three-plane using a decoupled three-plane aggregation algorithm to generate a first aggregated feature; Decoding the first aggregated feature by a decoder to obtain corresponding radiation color and volume density; Performing volume rendering on the corresponding radiation color and volume density to obtain the first rendering result; The step of visualizing the learnable second characteristic three-plane to obtain the final rendering result includes: Aggregating the optimized camera pose and the learnable second feature three-plane through the decoupled three-plane aggregation algorithm to generate a second aggregated feature; decoding the second aggregate feature by the decoder to obtain corresponding radiation color and volume density; Volume rendering is performed on the corresponding radiation color and volume density to obtain the final rendering result.

4. The method according to claim 2, wherein The step of inputting the image to be input into the feature extractor to extract a feature vector comprises: The feature vector is obtained by performing feature extraction on the image to be input through a pre-built dual interlaced network optimization model.

5. The method according to claim 2, wherein The step of aggregating the feature vector and the three-plane vector through a cross attention mechanism to obtain a cross-aggregated feature includes: The three-plane vector is used as a query vector and the feature vector is used as a key and a value to perform the cross-attention mechanism to obtain the cross-aggregated feature.

6. The method according to claim 1, wherein The step of inputting the three-plane vector and the image to be input into a pre-built three-plane generator includes: Construct the three-plane generator, wherein each plane generator is composed of an intermediate block and an upsampling block, the intermediate block is composed of a convolutional layer, a structure using group normalization and an activation function, and an attention layer, and the upsampling block is composed of a convolutional layer, a structure using group normalization and an activation function, and except for the last upsampling block, the upsampling block also includes a bilinear upsampler.

7. A rendering device, characterized in that: The device comprises: A data acquisition module, used to acquire three-plane vectors and images to be input; a rendering module, configured to input the three-plane vectors and the image to be input into a pre-built three-plane generator, and perform viewpoint rendering on the three-plane vectors and the image to be input using a two-stage warm-up training strategy to obtain a final rendering result, wherein the three-plane generator is trained based on a deep neural network; The rendering module is also used to input the three-plane vector and the image to be input into the three-plane generator to generate a first characteristic three-plane; visualize the first characteristic three-plane to obtain a first rendering result; optimize the three-plane generator based on the first rendering result; freeze the model parameters of the optimized three-plane generator; perform forward reasoning and shape adjustment on the three-plane generator after the model parameters are frozen to obtain a second characteristic three-plane; set the parameters of the second characteristic three-plane to learnable variables to obtain a learnable second characteristic three-plane; visualize the learnable second characteristic three-plane to obtain the final rendering result.

8. A rendering device, characterized in that The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the rendering method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the rendering method according to any one of claims 1 to 6 are implemented.

Images