Fourier grid based three-dimensional neural rendering method and apparatus
By introducing the FourierGrid operator, the data acquisition and computation process of the NeRF model is simplified, the computational efficiency and accuracy of the model are improved, the computational complexity and robustness of the NeRF model in large-scale datasets and complex scenes are solved, and efficient and accurate new perspective image generation is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- COWA TECHNOLOGY CO LTD
- Filing Date
- 2023-08-30
- Publication Date
- 2026-07-24
AI Technical Summary
Existing NeRF models suffer from computational complexity and inefficiency when dealing with large-scale datasets. They require large amounts of data, are highly dependent on data quality and diversity, lack robustness and generalization ability, and face challenges in terms of geometric and lighting consistency.
A 3D neural rendering method based on Fourier grids is adopted. The FourierGrid operator expands the input signal into multiple frequency components, and the grid is used to model the components of different frequencies, which simplifies the data acquisition process, improves the computational efficiency, and enhances the generalization ability and scene scalability of the model.
It achieves efficient computation and accurate rendering of NeRF models, enabling faster generation of high-quality new perspective images, improving the robustness and generalization ability of the models, adapting to a wider range of scenes, and enhancing the consistency of geometry and lighting.
Smart Images
Figure CN117132697B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of novel perspective synthesis technology, and in particular to a three-dimensional neural rendering method and apparatus based on Fourier meshes. Background Technology
[0002] Viewpoint synthesis refers to the technique of generating new perspectives outside the original viewpoint using existing image or scene information. This field is of great significance in computer vision and computer graphics, and involves related technologies such as model building, image processing, and signal processing.
[0003] Neural Radiance Fields (NeRF): NeRF is a method that uses multilayer perceptrons (MLPs) to represent color and density and learns implicit 3D representations through differentiable volumetric rendering. NeRF has become a leading method for novel perspective synthesis and has been widely applied in multiple fields. Previous work has also made some improvements to the efficiency of NeRF, such as FastNeRF and Instant-NGP using advanced caching techniques, and KiloNeRF, which decomposes a high-capacity MLP into multiple smaller MLPs.
[0004] Unbounded Scene Reconstruction: This field primarily focuses on learning to reconstruct unbounded or large-scale radiative fields. Due to the significant success of NeRF in rendering novel perspectives, recent research has turned its attention to learning unbounded or large-scale radiative fields. For example, NeRF++ extends NeRF to 360-degree capture and has conducted experiments on the Unbounded Tanks & Temples dataset. Mip-NeRF-360 improves upon Mip-NeRF in terms of parameterization and efficiency.
[0005] The existing technical approach mainly includes the following steps:
[0006] (1) Image Acquisition: The NeRF workflow typically begins with acquiring multiple images of the scene. These images can be taken from different perspectives to capture various aspects and details of the scene.
[0007] (2) Camera localization device regresses camera pose: In order to accurately reconstruct the scene, NeRF needs to know the camera pose (position and orientation) of each image. To obtain this information, it is usually necessary to use a camera localization device (such as IMU, GPS, etc.) to record the camera pose, and then predict the camera pose parameters through regression methods.
[0008] (3) Solving the Neural Render Field: NeRF uses neural networks to model the radiance field of a scene, that is, the color and density of each point in the scene. The neural network maps the three-dimensional coordinates of the input point to the color and density values of the corresponding point, thereby realizing the modeling of the scene.
[0009] (4) Obtaining the Input Viewpoint: Before rendering a new viewpoint, NeRF needs to obtain the required input viewpoint. This can be achieved by sampling input points in the scene at known viewpoints. Typically, a set of input points is sampled evenly in the scene, and the viewpoints of these points will be used to render the new viewpoint.
[0010] (5) Sampling from the input viewpoint and rendering the new viewpoint image: Once the input viewpoint and corresponding scene points are obtained, NeRF can use a neural network model to render the new viewpoint. By sampling light rays at the input viewpoint and integrating the radiation field of the scene along the light rays, NeRF can generate realistic images of the new viewpoint, showing how the scene looks as viewed from that viewpoint.
[0011] However, existing technologies have the following drawbacks:
[0012] (1) Computational complexity and inefficiency: Some techniques face problems of computational complexity and inefficiency when dealing with large-scale datasets or complex models. This can lead to high time and resource consumption in the training and inference processes, limiting their application in real-time applications and resource-constrained environments.
[0013] (2) High data requirements: Many technologies require a large amount of labeled data. In some tasks, a large amount of labeled data is needed to train accurate models. The cost of data collection and labeling is often high, which may limit the application scope of the technology.
[0014] (3) Dependence on data quality and diversity: Some techniques rely heavily on high-quality and diverse datasets. If the training dataset is noisy, biased, or imbalanced, the model's performance and generalization ability may be affected.
[0015] (4) Lack of robustness and generalization ability: Some techniques may exhibit poor robustness and generalization ability when faced with unseen data or domain transfers. This may limit the application of the model in the real world, especially in complex, dynamic or uncertain environments. Summary of the Invention
[0016] The present invention aims to solve the following technical problems:
[0017] (1) Data Requirements and Acquisition Complexity: The NeRF model requires a large number of input images and corresponding camera pose information to build the neural rendering field of the scene. Acquiring this data requires high-precision camera positioning devices and a complex data acquisition process. Therefore, reducing data requirements and simplifying the data acquisition process is an important technical issue.
[0018] (2) Computational efficiency and real-time performance: Due to the high computational complexity of the NeRF model, rendering images from new perspectives requires a significant amount of computational resources and time. Improving the computational efficiency of the NeRF model to enable it to generate high-quality images in real time is a challenge.
[0019] (3) Generalization ability and scene scalability: Existing NeRF models have limitations in terms of scene generalization ability and scalability. This means that the model may not be able to generate accurate rendering results when dealing with unseen or complex scenes. Improving the generalization ability and scene scalability of NeRF models so that they can adapt to a wider range of scenes is a key issue.
[0020] (4) Geometric and lighting consistency: NeRF models face challenges in terms of scene geometry and lighting consistency. The generated images may suffer from geometric distortion, inconsistent lighting, and other issues. Solving these problems to make the generated images more realistic and consistent in terms of geometry and lighting is an important research direction.
[0021] To achieve the above objectives, the technical solution of this invention provides a three-dimensional neural rendering method based on Fourier grids, which includes the following steps: S1: Image acquisition, acquiring multiple viewpoint images of the scene through a photographic device; S2: Camera positioning device regresses camera pose, using the camera positioning device to determine the camera pose information of each image; S3: Solving the neural rendering field, modeling the unbounded radiation field through the FourierGrid operator, wherein the FourierGrid uses Fourier feature mapping to expand the input signal into multiple frequency components and uses a grid to model the components of different frequencies; S4: Obtaining the input viewpoint, using the acquired camera pose information to convert the camera position and orientation into the corresponding input viewpoint; S5: Sampling based on the input viewpoint and rendering an image from the new viewpoint.
[0022] Furthermore, for the $i$-th frequency component of the input signal, the Fourier feature map $\gamma(\bx,i)\in\mathbb{R}^{3}$ is calculated as follows:
[0023] If the remainder when $i$ is divided by 2 is 0, then $\gamma(\bx,i)=\sin(2^{\lfloor i / 2\rfloor}\pi\bx)$;
[0024] Otherwise, $\gamma(\bx,i)=\cos(2^{\lfloor i / 2\rfloor}\pi\bx)$.
[0025] Furthermore, each frequency component is modeled using the corresponding learnable grid tensor $\mathbf{V}^{(i)}$.
[0026] Furthermore, modeling each frequency component using the corresponding learnable mesh tensor $\mathbf{V}^{(i)}$ specifically includes:
[0027] For the $i$-th frequency component, the feature map is applied to the grid using the interpolation operator $\texttt{Interpolate}$ to obtain the feature vector $\boldsymbol{h_i}$ representing that frequency component:
[0028] $\boldsymbol{h_i}=\texttt{Interpolate}(\gamma(\bx,i),\mathbf{V}^{(i)})$, where $0\leq i\leq 2l-1$, and $l$ is a hyperparameter that controls the number of Fourier features used;
[0029] Finally, FourierGrid outputs the summarized signal through the summarization function $\Phi$, obtaining the final feature vector $\bu$:
[0030] $\bu=\texttt{FourierGrid}(\bx)=\Phi({\boldsymbol{h_0},\boldsymbol{h_1},...,\boldsymbol{h_{2l-1}}})$;
[0031] In summary functions, using the average value is the best way to summarize:
[0032] $\Phi({\boldsymbol{h_0},\boldsymbol{h_1},...,\boldsymbol{h_{n-1}}})=\frac{1}{n}\sum_{i=0}^{n-1}{\boldsymbol{h_i}}$.
[0033] Furthermore, in step S5, by performing interpolation on the grid in FourierGrid, the radiation field properties of the new viewpoint are inferred from the existing viewpoint image, thereby generating the new viewpoint image.
[0034] The technical solution of this invention also provides a three-dimensional neural rendering device based on Fourier grids, which includes the following modules: an image acquisition module for acquiring multiple viewpoint images of a scene using a photographic device; a camera pose regression module for determining the camera pose information of each image using a camera positioning device; a neural rendering field solving module for modeling an unbounded radiation field using the FourierGrid operator, wherein the FourierGrid uses Fourier feature mapping to expand the input signal into multiple frequency components and uses a grid to model the components of different frequencies; an input viewpoint acquisition module for converting the camera position and orientation into corresponding input viewpoints using the acquired camera pose information; and a new viewpoint generation module for sampling and rendering images of new viewpoints based on the input viewpoints.
[0035] Furthermore, for the $i$-th frequency component of the input signal, the Fourier feature map $\gamma(\bx,i)\in\mathbb{R}^{3}$ is calculated as follows:
[0036] If the remainder when $i$ is divided by 2 is 0, then $\gamma(\bx,i)=\sin(2^{\lfloor i / 2\rfloor}\pi\bx)$;
[0037] Otherwise, $\gamma(\bx,i)=\cos(2^{\lfloor i / 2\rfloor}\pi\bx)$.
[0038] Furthermore, each frequency component is modeled using the corresponding learnable grid tensor $\mathbf{V}^{(i)}$.
[0039] Furthermore, modeling each frequency component using the corresponding learnable mesh tensor $\mathbf{V}^{(i)}$ specifically includes:
[0040] For the $i$-th frequency component, the feature map is applied to the grid using the interpolation operator $\texttt{Interpolate}$ to obtain the feature vector $\boldsymbol{h_i}$ representing that frequency component:
[0041] $\boldsymbol{h_i}=\texttt{Interpolate}(\gamma(\bx,i),\mathbf{V}^{(i)})$, where $0\leq i\leq 2l-1$, and $l$ is a hyperparameter that controls the number of Fourier features used;
[0042] Finally, FourierGrid outputs the summarized signal through the summarization function $\Phi$, obtaining the final feature vector $\bu$:
[0043] $\bu=\texttt{FourierGrid}(\bx)=\Phi({\boldsymbol{h_0},\boldsymbol{h_1},...,\boldsymbol{h_{2l-1}}})$;
[0044] In summary functions, using the average value is the best way to summarize:
[0045] $\Phi({\boldsymbol{h_0},\boldsymbol{h_1},...,\boldsymbol{h_{n-1}}})=\frac{1}{n}\sum_{i=0}^{n-1}{\boldsymbol{h_i}}$.
[0046] Furthermore, the new perspective generation module infers the radiation field properties of the new perspective from the existing perspective image by interpolating the grid in the FourierGrid, and then generates the new perspective image.
[0047] This invention improves the efficiency and accuracy of processing unbounded radiation fields by introducing the FourierGrid operator. FourierGrid can better model frequency components and perform interpolation operations through the grid for faster processing of input data. Compared to the existing VoxelGrid operator, FourierGrid exhibits better performance and efficiency in scene modeling and view composition tasks. Attached Figure Description
[0048] Figure 1 This is a schematic diagram of the FourierGrid operator of the present invention;
[0049] Figure 2 This is an overall architecture diagram of the application of the FourierGrid operator in this invention;
[0050] Figure 3 This is the overall flowchart of the algorithm of this invention. Detailed Implementation
[0051] The technical solution of the present invention will be further described below with reference to specific embodiments, but the present invention is not limited to these embodiments.
[0052] VoxelGrid is a grid-based operator widely used in 3D scene modeling and view composition tasks. It has been used to replace multilayer perceptrons (MLPs) to achieve higher computational speed.
[0053] The input to VoxelGrid is a field point $\bx$ in three-dimensional space, and the output is a feature vector $\bu$, where the dimension $d$ of the feature vector can be set according to the needs of the task (for example, $d=3$ for color information and $d=1$ for density information).
[0054] VoxelGrid is implemented by interpolating the features of nearby grid cells to map field points $\bx$ to the corresponding feature vectors $\bu$. The specific operation is as follows:
[0055] \begin{equation}
[0056] \bu=\texttt{VoxelGrid}(\bx)=\texttt{Interpolate}(\bx,\boldsymbol{V})
[0057] \label{eq:dvgo-operator}
[0058] \end{equation}
[0059] Where $\boldsymbol{V}\in\mathbb{R}^{d\times N_x\times N_y\times N_z}$ is a voxel mesh with a resolution of $N_x\times N_y\times N_z$. VoxelGrid operates in the spatial domain and runs faster than MLP.
[0060] The advantage of VoxelGrid lies in its ability to quickly obtain feature representations of specific field points through interpolation of grid features. It is widely used in view synthesis tasks, generating new views based on given scene and camera parameters. Furthermore, VoxelGrid is also used in applications such as density field reconstruction and point cloud generation, exhibiting high computational efficiency and good performance. However, for tasks involving complex scenes such as unbounded radiation fields, VoxelGrid may have some limitations, prompting this invention to propose a more advanced operator—FourierGrid.
[0061] This invention proposes a grid-based operator called FourierGrid to replace the existing VoxelGrid operator for handling unbounded radiation fields. Figure 1 and Figure 2As shown, FourierGrid expands the input signal into multiple frequency components using Fourier feature maps and uses a grid to model the different frequency components. Specifically, the Fourier feature map is defined as follows:
[0062] For the $i$-th component, the Fourier feature map $\gamma(\bx,i)\in\mathbb{R}^{3}$ is calculated as follows:
[0063] If the remainder when $i$ is divided by 2 is 0, then $\gamma(\bx,i)=\sin(2^{\lfloor i / 2\rfloor}\pi\bx)$;
[0064] Otherwise, $\gamma(\bx,i)=\cos(2^{\lfloor i / 2\rfloor}\pi\bx)$.
[0065] Then, each frequency component is modeled using the corresponding learnable grid tensor $\mathbf{V}^{(i)}$. Specifically, for the $i$-th frequency component, we apply the feature map to the grid using the interpolation operator $\texttt{Interpolate}$ to obtain the feature vector $\boldsymbol{h_i}$ representing that frequency component:
[0066] $\boldsymbol{h_i}=\texttt{Interpolate}(\gamma(\bx,i),\mathbf{V}^{(i)})$, where $0\leq i\leq 2l-1$, and $l$ is a hyperparameter controlling the number of Fourier features used.
[0067] Finally, FourierGrid outputs the summarized signal through the summarization function $\Phi$, obtaining the final feature vector $\bu$:
[0068] $\bu=\texttt{FourierGrid}(\bx)=\Phi({\boldsymbol{h_0},\boldsymbol{h_1},...,\boldsymbol{h_{2l-1}}})$.
[0069] In summary functions, using the average value is the best way to summarize:
[0070] $\Phi({\boldsymbol{h_0}, \boldsymbol{h_1}, ...,\boldsymbol{h_{n-1}}})=\frac{1}{n}\sum_{i=0}^{n-1}{\boldsymbol{h_i}}$.
[0071] By introducing the FourierGrid operator, this invention aims to improve the efficiency and accuracy of processing unbounded radiative fields. FourierGrid can better model frequency components and perform interpolation operations through the grid to process input data more quickly. Compared to the existing VoxelGrid, FourierGrid exhibits better performance and efficiency in scene modeling and view compositing tasks.
[0072] like Figure 3 As shown, the overall algorithm flow of this invention is as follows:
[0073] Image acquisition: Capture images of the scene from multiple perspectives using photographic equipment.
[0074] Camera positioning device regresses camera pose: The camera positioning device is used to determine the camera pose information for each image, such as camera position and orientation.
[0075] Solving the neural rendering field: A novel grid operator, FourierGrid, as described above, is introduced to model the unbounded radiation field. FourierGrid uses Fourier feature maps to expand the input signal into multiple frequency components and uses a grid to model the components at different frequencies.
[0076] Obtaining the input viewpoint: Using known camera pose information, the camera position and orientation are converted into the corresponding input viewpoint.
[0077] Based on the input viewpoint sampling, a new viewpoint image is rendered: The input viewpoint is sampled by manipulating the FourierGrid to generate an image from the new viewpoint. This process involves interpolating the grid within the FourierGrid to obtain the radiation field properties of the new viewpoint, such as color and density.
[0078] In an embodiment of the present invention, a three-dimensional neural rendering method based on Fourier grids is provided, comprising the following steps: S1: acquiring images, acquiring multiple viewpoint images of the scene through a photographic device; S2: regressing the camera pose using a camera positioning device, determining the camera pose information for each image; S3: solving the neural rendering field, modeling the unbounded radiation field using the FourierGrid operator, wherein the FourierGrid uses Fourier feature mapping to expand the input signal into multiple frequency components and uses a grid to model the components of different frequencies; S4: acquiring the input viewpoint, using the acquired camera pose information to convert the camera position and orientation into the corresponding input viewpoint; S5: sampling based on the input viewpoint and rendering an image from the new viewpoint.
[0079] Furthermore, for the $i$-th frequency component of the input signal, the Fourier feature map $\gamma(\bx,i)\in\mathbb{R}^{3}$ is calculated as follows:
[0080] If the remainder when $i$ is divided by 2 is 0, then $\gamma(\bx,i)=\sin(2^{\lfloor i / 2\rfloor}\pi\bx)$;
[0081] Otherwise, $\gamma(\bx,i)=\cos(2^{\lfloor i / 2\rfloor}\pi\bx)$.
[0082] Furthermore, each frequency component is modeled using the corresponding learnable grid tensor $\mathbf{V}^{(i)}$.
[0083] Furthermore, modeling each frequency component using the corresponding learnable mesh tensor $\mathbf{V}^{(i)}$ specifically includes:
[0084] For the $i$-th frequency component, the feature map is applied to the grid using the interpolation operator $\texttt{Interpolate}$ to obtain the feature vector $\boldsymbol{h_i}$ representing that frequency component:
[0085] $\boldsymbol{h_i}=\texttt{Interpolate}(\gamma(\bx,i),\mathbf{V}^{(i)})$, where $0\leq i\leq 2l-1$, and $l$ is a hyperparameter that controls the number of Fourier features used;
[0086] Finally, FourierGrid outputs the summarized signal through the summarization function $\Phi$, obtaining the final feature vector $\bu$:
[0087] $\bu=\texttt{FourierGrid}(\bx)=\Phi({\boldsymbol{h_0},\boldsymbol{h_1},...,\boldsymbol{h_{2l-1}}})$;
[0088] In summary functions, using the average value is the best way to summarize:
[0089] $\Phi({\boldsymbol{h_0},\boldsymbol{h_1},...,\boldsymbol{h_{n-1}}})=\frac{1}{n}\sum_{i=0}^{n-1}{\boldsymbol{h_i}}$.
[0090] Furthermore, in step S5, the radiation field properties of the new perspective are inferred from the existing perspective image by interpolating the grid in the FourierGrid, thereby generating the new perspective image.
[0091] In other embodiments of the present invention, a three-dimensional neural rendering device based on Fourier grids is also provided, comprising the following modules: an image acquisition module for acquiring multiple viewpoint images of a scene using a photographic device; a camera pose regression module for determining the camera pose information of each image using a camera positioning device; a neural rendering field solving module for modeling an unbounded radiation field using the FourierGrid operator, wherein the FourierGrid uses Fourier feature mapping to expand the input signal into multiple frequency components and uses a grid to model the components of different frequencies; an input viewpoint acquisition module for converting the camera position and orientation into corresponding input viewpoints using the acquired camera pose information; and a new viewpoint generation module for sampling and rendering images of new viewpoints based on the input viewpoints.
[0092] Furthermore, for the $i$-th frequency component of the input signal, the Fourier feature map $\gamma(\bx,i)\in\mathbb{R}^{3}$ is calculated as follows:
[0093] If the remainder when $i$ is divided by 2 is 0, then $\gamma(\bx,i)=\sin(2^{\lfloor i / 2\rfloor}\pi\bx)$;
[0094] Otherwise, $\gamma(\bx,i)=\cos(2^{\lfloor i / 2\rfloor}\pi\bx)$.
[0095] Furthermore, each frequency component is modeled using the corresponding learnable grid tensor $\mathbf{V}^{(i)}$.
[0096] Furthermore, modeling each frequency component using the corresponding learnable mesh tensor $\mathbf{V}^{(i)}$ specifically includes:
[0097] For the $i$-th frequency component, the feature map is applied to the grid using the interpolation operator $\texttt{Interpolate}$ to obtain the feature vector $\boldsymbol{h_i}$ representing that frequency component:
[0098] $\boldsymbol{h_i}=\texttt{Interpolate}(\gamma(\bx,i),\mathbf{V}^{(i)})$, where $0\leq i\leq 2l-1$, and $l$ is a hyperparameter that controls the number of Fourier features used;
[0099] Finally, FourierGrid outputs the summarized signal through the summarization function $\Phi$, obtaining the final feature vector $\bu$:
[0100] $\bu=\texttt{FourierGrid}(\bx)=\Phi({\boldsymbol{h_0},\boldsymbol{h_1},...,\boldsymbol{h_{2l-1}}})$;
[0101] In summary functions, using the average value is the best way to summarize:
[0102] $\Phi({\boldsymbol{h_0},\boldsymbol{h_1},...,\boldsymbol{h_{n-1}}})=\frac{1}{n}\sum_{i=0}^{n-1}{\boldsymbol{h_i}}$.
[0103] Furthermore, the new perspective generation module performs interpolation operations on the grid in FourierGrid to infer the radiation field properties of the new perspective from the existing perspective image, thereby generating a new perspective image.
[0104] The beneficial technical effects obtained by this invention are as follows:
[0105] (1) Efficiency: The FourierGrid operator, based on a grid, is introduced, which enables more efficient computation compared to the traditional MLP model. Experimental results show that using FourierGrid to replace MLP can achieve a speedup of more than 100 times, making the radiation field modeling and rendering process more efficient.
[0106] (2) Accuracy: By introducing the FourierGrid operator, unbounded radiation fields can be modeled better. FourierGrid utilizes Fourier feature maps and grids to model components of different frequencies, enabling it to capture the characteristics of the radiation field more accurately. Compared to the traditional VoxelGrid operator, FourierGrid has better fitting and learning capabilities, and can generate more accurate representations of the radiation field.
[0107] (3) Multi-view image generation: The technical solution of the present invention can generate images from new perspectives based on known camera pose information. By manipulating and interpolating the FourierGrid, the radiation field properties of other perspectives can be inferred from existing perspective images, and high-quality new perspective images can be generated, which provides an important foundation for applications such as virtual reality and augmented reality.
[0108] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the protection scope of the present invention.
Claims
1. A three-dimensional neural rendering method based on Fourier meshes, characterized in that, Includes the following steps: S1: Image acquisition, capturing multiple perspective images of the scene using photographic equipment; S2: The camera positioning device returns to the camera pose, and the camera pose information for each image is determined using the camera positioning device; S3: Solve the neural rendering field. Model the unbounded radiation field using the FourierGrid operator. FourierGrid uses Fourier feature maps to expand the input signal into multiple frequency components and uses a grid to model the components at different frequencies. S4: Obtain the input viewpoint and use the acquired camera pose information to convert the camera position and orientation into the corresponding input viewpoint; S5: Sample based on the input viewpoint and render an image from a new viewpoint.
2. The method according to claim 1, characterized in that, For the $i$-th frequency component of the input signal, the Fourier feature map $\gamma(\bx,i)\in\mathbb{R}^{3}$ is calculated as follows: If the remainder when $i$ is divided by 2 is 0, then $\gamma(\bx,i)=\sin(2^{\lfloor i / 2\rfloor}\pi\bx)$; Otherwise, $\gamma(\bx,i)=\cos(2^{\lfloor i / 2\rfloor}\pi\bx)$.
3. The method according to claim 2, characterized in that, Each frequency component is modeled using the corresponding learnable mesh tensor $\mathbf{V}^{(i)}$.
4. The method according to claim 3, characterized in that, Modeling each frequency component using the corresponding learnable mesh tensor $\mathbf{V}^{(i)}$ specifically includes: For the $i$-th frequency component, the feature map is applied to the grid using the interpolation operator $\texttt{Interpolate}$ to obtain the feature vector $\boldsymbol{h_i}$ representing that frequency component: $\boldsymbol{h_i}=\texttt{Interpolate}(\gamma(\bx,i),\mathbf{V}^{(i)})$, where $0\leq i\leq 2l-1$, and $l$ is a hyperparameter that controls the number of Fourier features used; Finally, FourierGrid outputs the summarized signal through the summarization function $\Phi$, obtaining the final feature vector $\bu$: $\bu=\texttt{FourierGrid}(\bx)=\Phi({\boldsymbol{h_0},\boldsymbol{h_1},...,\boldsymbol{h_{2l-1}}})$; In summary functions, using the average value is the best way to summarize: $\Phi({\boldsymbol{h_0},\boldsymbol{h_1},...,\boldsymbol{h_{n-1}}})=\frac{1}{n}\sum_{i=0}^{n-1}{\boldsymbol{h_i}}$.
5. The method according to claim 4, characterized in that, In step S5, the radiation field properties of the new perspective are inferred from the existing perspective image by interpolating the grid in the FourierGrid, thereby generating the new perspective image.
6. A three-dimensional neural rendering device based on Fourier meshes, characterized in that, Includes the following modules: The image acquisition module is used to acquire images of a scene from multiple perspectives using photographic equipment; The camera pose regression module is used to determine the camera pose information for each image by using a camera positioning device. The Neural Render Field Solver module is used to model unbounded radiation fields using the FourierGrid grid operator. FourierGrid uses Fourier feature maps to expand the input signal into multiple frequency components and uses a grid to model the components at different frequencies. The input view acquisition module is used to convert the camera position and orientation into the corresponding input view using the acquired camera pose information; The new perspective generation module is used to sample and render images from new perspectives based on the input perspective.
7. The apparatus according to claim 6, characterized in that, For the $i$-th frequency component of the input signal, the Fourier feature map $\gamma(\bx,i)\in\mathbb{R}^{3}$ is calculated as follows: If the remainder when $i$ is divided by 2 is 0, then $\gamma(\bx,i)=\sin(2^{\lfloor i / 2\rfloor}\pi\bx)$; Otherwise, $\gamma(\bx,i)=\cos(2^{\lfloor i / 2\rfloor}\pi\bx)$.
8. The apparatus according to claim 7, characterized in that, Each frequency component is modeled using the corresponding learnable mesh tensor $\mathbf{V}^{(i)}$.
9. The apparatus according to claim 8, characterized in that, Modeling each frequency component using the corresponding learnable mesh tensor $\mathbf{V}^{(i)}$ specifically includes: For the $i$-th frequency component, the feature map is applied to the grid using the interpolation operator $\texttt{Interpolate}$ to obtain the feature vector $\boldsymbol{h_i}$ representing that frequency component: $\boldsymbol{h_i}=\texttt{Interpolate}(\gamma(\bx,i),\mathbf{V}^{(i)})$, where $0\leq i\leq 2l-1$, and $l$ is a hyperparameter that controls the number of Fourier features used; Finally, FourierGrid outputs the summarized signal through the summarization function $\Phi$, obtaining the final feature vector $\bu$: $\bu=\texttt{FourierGrid}(\bx)=\Phi({\boldsymbol{h_0},\boldsymbol{h_1},...,\boldsymbol{h_{2l-1}}})$; In summary functions, using the average value is the best way to summarize: $\Phi({\boldsymbol{h_0},\boldsymbol{h_1},...,\boldsymbol{h_{n-1}}})=\frac{1}{n}\sum_{i=0}^{n-1}{\boldsymbol{h_i}}$.
10. The apparatus according to claim 9, characterized in that, The new perspective generation module infers the radiation field properties of the new perspective from the existing perspective image by interpolating the grid in the FourierGrid, and then generates the new perspective image.