Meta-light field cross-scale intelligent reconstruction method and device

Abstract

The present invention discloses a meta-light field cross-scale intelligent reconstruction method and device, the method comprising constructing a multi-resolution hash feature grid based on the rough three-dimensional surface structure information of the scene to be reconstructed; determining the index of the multi-resolution hash feature grid by rasterizing the rough three-dimensional surface, and performing an interpolation operation on the multi-resolution hash feature grid according to the index to obtain a surface multi-resolution hash feature; performing a warp supersampling process on the surface multi-resolution hash feature to obtain a pre-filtered hash feature; and decoding the pre-filtered hash feature to obtain a meta-light field color rendering result. The present invention can effectively and realistically depict fine textures at a local microscale, thereby achieving highly realistic free-viewing angle light field rendering at several different scales spanning distant and near views.

Description

Technical Field

[0001] The present invention relates to the technical field of light field reconstruction, and in particular to a method and device for cross-scale intelligent reconstruction of a meta-light field. Background Art

[0002] Existing light field representations can achieve realistic free-view light field rendering for small-scale simple scenes based on highly structured multi-view images. However, for unstructured meta-light fields of large-scale and complex outdoor scenes, the scalability of existing representations is still significantly lacking.

[0003] Existing light field representations based on explicit geometric structures usually define feature descriptors discretely on parameterized UV maps, 3D meshes, or 3D point clouds, and then use convolutional neural networks or fully connected multi-layer perceptrons to decode local features to obtain high-quality free-view light field rendering. These methods have a strong perception of the global structure of the scene, but it is difficult to perfectly depict the complex and rich local details in real large scenes. Specifically, due to the distortion problem of parameterized mapping, the UV map-based representation is difficult to maintain a regular feature distribution on the actual 3D surface, which easily leads to distortion and blurring of the rendering of details. In addition, in existing representations based on 3D meshes or 3D point clouds, the spatial resolution of feature descriptors often depends on the resolution of discrete geometric primitives (such as points, patches, voxels, etc.), and the texture content above this resolution is described by the weighted sum of adjacent feature descriptors. Therefore, these representations cannot efficiently depict extremely fine texture details that are far higher than the resolution of geometric primitives based on limited storage.

[0004] On the other hand, in the implicit light field representation based on three-dimensional volume space, multi-resolution hash coding has gradually been widely used, which has significantly improved the quality and efficiency of light field reconstruction. However, when describing real large-scale scenes, the volume-based implicit representation will inevitably lead to the dispersion of volume rendering weights and cause surface ambiguity due to the lack of constraints on the complex geometric structure of the scene. That is, several multi-view inconsistent, non-scene surface but highly weighted sampling points weaken the pixel color supervision of the actual surface points, and hinder the precise optimization of surface features with highly inconsistent color gradients. Summary of the invention

[0005] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0006] To this end, the present invention proposes a meta-light field cross-scale intelligent reconstruction method, which effectively solves the problems of limited resolution and parametric mapping distortion of existing light field representation based on explicit geometric structures by constructing an efficient multi-resolution hash feature grid in three-dimensional volume space.

[0007] Another object of the present invention is to provide a meta-light field cross-scale intelligent reconstruction device.

[0008] To achieve the above objectives, the present invention proposes, on one hand, a meta-light field cross-scale intelligent reconstruction method, comprising:

[0009] Construct a multi-resolution hash feature grid based on the rough three-dimensional surface structure information of the scene to be reconstructed;

[0010] Determine the index of the multi-resolution hash feature grid by rasterizing the rough three-dimensional surface, and perform an interpolation operation on the multi-resolution hash feature grid according to the index to obtain a surface multi-resolution hash feature;

[0011] Performing a warp supersampling process on the surface multi-resolution hash feature to obtain a pre-filtered hash feature;

[0012] The pre-filtered hash feature is decoded to obtain a meta-light field color rendering result.

[0013] The meta-light field cross-scale intelligent reconstruction method of the embodiment of the present invention may also have the following additional technical features:

[0014] In one embodiment of the present invention, a multi-resolution hash feature grid is constructed based on rough three-dimensional surface structure information of a scene to be reconstructed, including:

[0015] Extracting a cross-scale input image of the scene to be reconstructed;

[0016] Solving the camera pose corresponding to each of the cross-scale input images and constructing a rough surface model of the scene to be reconstructed;

[0017] A bounding box of a scene to be reconstructed is determined according to the rough surface model, and a multi-resolution hash feature grid is constructed in a three-dimensional space volume contained in the bounding box.

[0018] In one embodiment of the present invention, determining the index of the multi-resolution hash feature grid by rasterizing the rough three-dimensional surface, and performing an interpolation operation on the multi-resolution hash feature grid according to the index to obtain the surface multi-resolution hash feature includes:

[0019] Projecting the reconstructed rough three-dimensional surface to the view angle to be rendered, and obtaining the coordinates of the intersection of each pixel ray of the view angle to be rendered and the rough three-dimensional surface;

[0020] Obtain hash features of eight mesh vertices of the intersection coordinates and calculate the intersection features by trilinear interpolation;

[0021] The multi-resolution hash feature grid is interpolated based on the intersection feature to obtain a feature cascade, and a surface multi-resolution hash feature of the surface intersection is obtained according to the feature cascade result.

[0022] In one embodiment of the present invention, the surface multi-resolution hash feature is subjected to a warp supersampling process to obtain a pre-filtered hash feature, including:

[0023] Supersample each pixel of the rendering perspective to obtain multiple pixel rays;

[0024] Input the surface multi-resolution hash feature of each pixel ray index and the corresponding pixel ray direction vector into the first fully connected multi-layer perceptron neural network prediction feature residual, and perform residual connection with the surface multi-resolution hash feature of each ray index to obtain a feature connection result;

[0025] Feature aggregation is performed based on the feature connection result to obtain a pre-filtered hash feature of each pixel.

[0026] In one embodiment of the present invention, decoding the pre-filtered hash feature to obtain a meta-light field color rendering result includes:

[0027] Obtaining the pre-filtered hash feature of each pixel of the perspective to be rendered;

[0028] The pre-filtered hash features of each pixel of the to-be-rendered viewing angle and the corresponding pixel ray direction vector are input into the second fully connected multi-layer perceptron neural network for decoding to obtain the meta-light field color rendering result.

[0029] To achieve the above-mentioned object, the present invention further provides a meta-light field cross-scale intelligent reconstruction device, comprising:

[0030] A feature grid construction module is used to construct a multi-resolution hash feature grid based on the rough three-dimensional surface structure information of the scene to be reconstructed;

[0031] A surface rasterization module, used to determine the index of the multi-resolution hash feature grid by rasterizing a rough three-dimensional surface, and to perform an interpolation operation on the multi-resolution hash feature grid according to the index to obtain a surface multi-resolution hash feature;

[0032] A warp supersampling module, used for performing warp supersampling processing on the surface multi-resolution hash feature to obtain a pre-filtered hash feature;

[0033] The feature decoding module is used to decode the pre-filtered hash feature to obtain a meta-light field color rendering result.

[0034] The meta-light field cross-scale intelligent reconstruction method and device of the embodiment of the present invention can robustly describe the macroscopic global structure of the scene while efficiently and realistically depicting the fine texture of the local microscale, thereby achieving highly realistic free-viewing-angle light field rendering at several different scales across distant and near views.

[0035] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0037] Figure 1 is a flow chart of a method for cross-scale intelligent reconstruction of a meta-light field according to an embodiment of the present invention;

[0038] Figure 2 is a framework diagram of a meta-light field cross-scale intelligent reconstruction method according to an embodiment of the present invention;

[0039] Figure 3 is a comparison diagram of the light field rendering effects of the present invention according to an embodiment of the present invention and other existing methods;

[0040] Figure 4 4 is a structural diagram of a meta-light field cross-scale intelligent reconstruction device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0042] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0043] The following describes a method and device for cross-scale intelligent reconstruction of a meta-light field according to an embodiment of the present invention with reference to the accompanying drawings.

[0044] Figure 1 It is a flow chart of the meta-light field cross-scale intelligent reconstruction method according to an embodiment of the present invention.

[0045] like Figure 1 As shown, the method includes but is not limited to the following steps:

[0046] S1, constructing a multi-resolution hash feature grid based on the rough three-dimensional surface structure information of the scene to be reconstructed;

[0047] S2, determining the index of the multi-resolution hash feature grid by rasterizing the rough three-dimensional surface, and performing an interpolation operation on the multi-resolution hash feature grid according to the index to obtain the surface multi-resolution hash feature;

[0048] S3, performing warp supersampling processing on the surface multi-resolution hash feature to obtain a pre-filtered hash feature;

[0049] S4, decoding the pre-filtered hash feature to obtain the meta-light field color rendering result.

[0050] Figure 2 The framework of the meta-light field cross-scale intelligent reconstruction method of the present invention is as follows: Figure 2 As shown, a hash feature manifold meta-light field representation is introduced, rasterization-based surface rendering is used to guide the optimization of multi-resolution hash features, and a meta-light field rendering framework based on hash feature manifold is further proposed.

[0051] In one embodiment of the present invention, the existing multi-view 3D reconstruction technology can be used to obtain the rough 3D surface structure of the scene to be reconstructed, and a multi-resolution hash feature grid can be constructed based on it.

[0052] It is understandable that multi-view 3D reconstruction technology can include data acquisition, feature extraction and matching, camera pose estimation, 3D point cloud generation, point cloud alignment and optimization, meshing and simplification, post-processing and texture mapping, etc. In order to increase the realism of the model, the texture information of the original image can be mapped to the surface of the 3D model to complete the entire reconstruction process.

[0053] Furthermore, based on the scene to be reconstructed, a number of multi-view images are collected. The embodiment of the present invention is a cross-scale input image, and then the existing multi-view 3D reconstruction technology is used to solve the camera parameters and camera pose corresponding to each image, and the rough surface model of the scene is obtained at the same time. Then, the bounding box of the scene to be reconstructed is determined according to the rough surface model, and a feature grid with L levels of increasing spatial resolution is constructed in the three-dimensional space volume contained in the bounding box. It can be known that the spatial resolution of the first-level feature grid is (16,16,16), and the spatial resolution of the L-th level feature grid is (M,M,M), and the spatial resolution of the feature grids at all levels increases exponentially according to the power of 2. The features of the vertices of each level of the grid are respectively stored in a hash table of length T, and the hash query method is fixed in advance. The features stored in the hash table will be optimized together with the neural network through end-to-end gradient descent.

[0054] It is understandable that the use of existing multi-view 3D reconstruction technology to solve the camera parameters corresponding to each image generally includes camera model definition, feature extraction and matching, motion recovery structure, camera parameter solution, verification and post-processing, etc. Through the above steps, the camera parameters corresponding to each image can be solved. These parameters are not only crucial for 3D reconstruction, but also provide a basis for subsequent image correction, scene analysis and other applications. Modern 3D reconstruction software and libraries (such as OpenCV, COLMAP, VisualSFM, etc.) provide functions for automating these steps.

[0055] Through the above process, the embodiment of the present invention can efficiently reconstruct the three-dimensional structure of the scene from multi-view images and build a multi-resolution hash feature grid system that is easy to query and process.

[0056] In one embodiment of the present invention, sparse indexing of a multi-resolution hash feature grid is achieved by rasterizing a rough three-dimensional surface. According to the camera pose of the view to be rendered, the reconstructed rough three-dimensional surface is efficiently rasterized using an existing surface rasterization tool, and the intersection coordinates of each pixel ray of the view to be rendered and the rough three-dimensional surface are obtained by projecting the rough three-dimensional surface to the view to be rendered.

[0057] Furthermore, for each level of hash feature grid, the hash features of the eight grid vertices containing the intersection coordinates are indexed, and the intersection features are calculated by trilinear interpolation. The features interpolated from the hash feature grids at each level are cascaded to express the feature cascade results as multi-resolution hash features of the surface intersection.

[0058] It is understandable that when the perspective to be rendered is trained / optimized, the input is a perspective image in the cross-scale multi-perspective image. After the optimization is completed, the algorithm can render the image of the scene to be reconstructed at any specified perspective.

[0059] It can be understood that each pixel ray refers to a ray formed by back-projecting each pixel on the image into the three-dimensional space using the camera posture of the viewing angle to be rendered.

[0060] It can be understood that the hash feature grid is essentially a regular three-dimensional grid. For example, the spatial resolution of the L-th level feature grid is (M, M, M), which means a three-dimensional cube containing MxMxM grid vertices, each grid vertex stores an optimizable feature vector. Therefore, for any three-dimensional point, the present invention finds the eight closest grid vertices to it, and performs trilinear interpolation (i.e., weighted summation based on the distance of the three-dimensional point from each grid vertex) on the features stored in these vertices as the features of the three-dimensional point.

[0061] It can be understood that the present invention utilizes existing surface rasterization tools to perform rasterization processing, including selecting appropriate tools or libraries, optimizing three-dimensional models before rasterization, setting rasterization parameters, performing rasterization, multi-resolution rasterization, and finally outputting a rasterized image.

[0062] Through the above steps, the embodiment of the present invention can effectively implement sparse indexing of multi-resolution hash feature grids based on rasterization, thereby providing efficient data access and processing capabilities for various three-dimensional applications.

[0063] In one embodiment of the present invention, the present invention adaptively adjusts and aggregates the extracted surface multi-resolution hash features through a warp supersampling mechanism to form a pre-filtering feature representation that has multi-view consistency perception capabilities and is more robust to cross-scale input images.

[0064] Specifically, first, during the rasterization process, each pixel of the rendering perspective is supersampled, and a total of S rays are sampled. Then, the multi-resolution hash features indexed by each ray and the ray direction are fed into a lightweight fully connected multi-layer perceptron neural network to predict the feature residual, and residually connected with the multi-resolution hash features directly indexed by each ray. Finally, the features of the S rays are aggregated through mathematical expectation calculation to obtain the pre-filtered hash features of each pixel.

[0065] It can be known that each pixel of the viewing angle to be rendered is determined by the image resolution. For example, a 1080x1920 image has a total of 1080x1920 pixels. Based on the camera pose of the viewing angle to be rendered, each pixel on the image can be back-projected into three-dimensional space to form a ray, i.e., a pixel ray. The supersampling of the present invention refers to evenly dividing each pixel into S sub-pixels, each sub-pixel generates a camera ray, so each pixel can generate S pixel rays. Thus, the present invention determines each pixel ray, i.e., the color of each pixel of the viewing angle to be rendered, thereby obtaining a rendered image of the viewing angle to be rendered.

[0066] Through the above processing steps, the present invention can obtain a pre-filtering feature representation that is more robust to cross-scale input images and has multi-view consistency perception capabilities, providing a more powerful and accurate foundation for subsequent three-dimensional reconstruction, object recognition, scene analysis and other tasks.

[0067] In one embodiment of the present invention, the pre-filtered hash feature is decoded to obtain a meta-light field color rendering result.

[0068] Specifically, the pre-filtered hash features and ray directions of each pixel of the to-be-rendered perspective are sent to a lightweight fully-connected multi-layer perceptron neural network for decoding to obtain high-quality meta-light field color rendering results.

[0069] Specifically, the decoding process can be: extracting pre-filtered hash features from each pixel position of the perspective to be rendered. These features should contain information such as the geometry, texture, and illumination of the scene, and be preprocessed to adapt to the input format of the neural network. Encode the direction of the ray emitted by each pixel into a vector form that can be processed by the neural network. This usually involves converting the three-dimensional direction vector into two angles in a spherical coordinate system or using other types of encoding methods. At the same time, a lightweight multi-layer perceptron network is designed to efficiently decode the color information of the light field from the input features. The network usually contains multiple fully connected layers, each followed by an activation function, and the last layer outputs the predicted RGB color value or spectral information. The network is trained using a large number of known light field samples, which include images of different perspectives and their corresponding light field information. Through optimization algorithms such as backpropagation and gradient descent, the network weights are adjusted so that the network can accurately predict the color under given features and ray directions.

[0070] It can be understood that for each pixel to be rendered, its pre-filtered hash feature is extracted and the ray direction is calculated, and then both are sent as input to the neural network. The neural network outputs the color value of the ray in the scene. The predicted color values ​​of all pixels are combined together to obtain the final rendered image.

[0071] The details and clarity of the rendering results can also be enhanced through network design techniques (such as skip connections, residual learning) or post-processing steps (such as super-resolution technology). The network architecture, training strategy or input features can also be continuously adjusted based on evaluation feedback to further improve the rendering effect.

[0072] Furthermore, the present invention also conducted relevant experiments. Figure 3 This is a comparison diagram of the meta-light field rendering effects of the present invention and other existing methods, from which it can be seen that the effect of the meta-light field cross-scale intelligent reconstruction method applied in the embodiment of the present invention.

[0073] The present invention can effectively utilize the pre-filtered hash features and the concept of ray tracing by decoding the pre-filtered hash features, and combine the power of machine learning to achieve high-quality and efficient meta-light field rendering, which has important application value in the fields of virtual reality, augmented reality, and film production.

[0074] According to the meta-light field cross-scale intelligent reconstruction method of the embodiment of the present invention, multi-resolution hash space features are extracted only on the surface of the scene by means of rasterization, and sparse two-dimensional manifolds are used to guide the multi-resolution hash features to represent the three-dimensional space, so that local features are focused on sparse areas with strong multi-view consistency, avoiding the dense expression of redundant three-dimensional volume space. At the same time, the proposed hash feature manifold representation enables widely related feature grid nodes to obtain color gradients only from sparse surfaces that are consistent with multiple perspectives, avoiding the interference introduced by optimizing in areas with inconsistent multi-views, significantly improving the feature representation capabilities of multi-resolution hash codes, and being able to describe extremely fine texture information that is far higher than the geometric resolution, enabling compact, efficient and high-quality meta-light field free-view rendering tasks.

[0075] In order to implement the above embodiment, Figure 4 As shown, this embodiment also provides a meta-light field cross-scale intelligent reconstruction device 10, which includes a feature grid construction module 100, a surface rasterization module 200, a warp supersampling module 300 and a feature decoding module 400.

[0076] A feature grid construction module 100 is used to construct a multi-resolution hash feature grid based on the rough three-dimensional surface structure information of the scene to be reconstructed;

[0077] The surface rasterization module 200 is used to determine the index of the multi-resolution hash feature grid by rasterizing the rough three-dimensional surface, and to obtain the surface multi-resolution hash feature by performing an interpolation operation on the multi-resolution hash feature grid according to the index;

[0078] The warp super sampling module 300 is used to perform warp super sampling processing on the surface multi-resolution hash feature to obtain a pre-filtered hash feature;

[0079] The feature decoding module 400 is used to decode the pre-filtered hash feature to obtain a meta-light field color rendering result.

[0080] Furthermore, the above-mentioned feature grid construction module 100 is also used for:

[0081] Extracting a cross-scale input image of the scene to be reconstructed;

[0082] Solving the camera pose corresponding to each of the cross-scale input images and constructing a rough surface model of the scene to be reconstructed;

[0083] A bounding box of a scene to be reconstructed is determined according to the rough surface model, and a multi-resolution hash feature grid is constructed in a three-dimensional space volume contained in the bounding box.

[0084] Furthermore, the surface rasterization module 200 is also used for:

[0085] Projecting the reconstructed rough three-dimensional surface to the view angle to be rendered, and obtaining the coordinates of the intersection of each pixel ray of the view angle to be rendered and the rough three-dimensional surface;

[0086] Obtain hash features of eight mesh vertices of the intersection coordinates and calculate the intersection features by trilinear interpolation;

[0087] The multi-resolution hash feature grid is interpolated based on the intersection feature to obtain a feature cascade, and a surface multi-resolution hash feature of the surface intersection is obtained according to the feature cascade result.

[0088] Furthermore, the above-mentioned warp super sampling module 300 is also used for:

[0089] Supersample each pixel of the rendering perspective to obtain multiple pixel rays;

[0090] Input the surface multi-resolution hash feature of each pixel ray index and the corresponding pixel ray direction vector into the first fully connected multi-layer perceptron neural network prediction feature residual, and perform residual connection with the surface multi-resolution hash feature of each ray index to obtain a feature connection result;

[0091] Feature aggregation is performed based on the feature connection result to obtain a pre-filtered hash feature of each pixel.

[0092] Furthermore, the feature decoding module 400 includes:

[0093] Obtain the pre-filtered hash features of each pixel of the perspective to be rendered;

[0094] The pre-filtered hash features of each pixel of the to-be-rendered viewing angle and the corresponding pixel ray direction vector are input into the second fully connected multi-layer perceptron neural network for decoding to obtain the meta-light field color rendering result.

[0095] According to the meta-light field cross-scale intelligent reconstruction device of the embodiment of the present invention, multi-resolution hash space features are extracted only on the surface of the scene by means of rasterization, and sparse two-dimensional manifolds are used to guide the representation of the three-dimensional space by the multi-resolution hash features, so that local features are focused on sparse areas with strong multi-view consistency, avoiding the dense expression of redundant three-dimensional volume space. At the same time, the proposed hash feature manifold representation enables widely related feature grid nodes to obtain color gradients only from sparse surfaces that are consistent with multiple perspectives, avoiding the interference introduced by optimizing in inconsistent areas of multiple perspectives, significantly improving the feature representation capability of multi-resolution hash coding, and being able to describe extremely fine texture information that is far higher than the geometric resolution, enabling compact, efficient and high-quality meta-light field free-view rendering tasks.

[0096] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0097] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

Claims

1. A meta-light field cross-scale intelligent reconstruction method, characterized in that: include: Construct a multi-resolution hash feature grid based on the rough three-dimensional surface structure information of the scene to be reconstructed; Determine the index of the multi-resolution hash feature grid by rasterizing the rough three-dimensional surface, and perform an interpolation operation on the multi-resolution hash feature grid according to the index to obtain a surface multi-resolution hash feature; Performing a warp supersampling process on the surface multi-resolution hash feature to obtain a pre-filtered hash feature; Decoding the pre-filtered hash feature to obtain a meta-light field color rendering result; Determining the index of the multi-resolution hash feature grid by rasterizing the rough three-dimensional surface, and performing an interpolation operation on the multi-resolution hash feature grid according to the index to obtain a surface multi-resolution hash feature, including: Projecting the reconstructed rough three-dimensional surface to the view angle to be rendered, and obtaining the coordinates of the intersection of each pixel ray of the view angle to be rendered and the rough three-dimensional surface; Obtain hash features of eight mesh vertices of the intersection coordinates and calculate the intersection features by trilinear interpolation; The multi-resolution hash feature grid is interpolated based on the intersection feature to obtain a feature cascade, and a surface multi-resolution hash feature of the surface intersection is obtained according to the feature cascade result.

2. The method according to claim 1, characterized in that A multi-resolution hash feature grid is constructed based on the rough three-dimensional surface structure information of the scene to be reconstructed, including: Extracting a cross-scale input image of the scene to be reconstructed; Solving the camera pose corresponding to each of the cross-scale input images and constructing a rough surface model of the scene to be reconstructed; A bounding box of a scene to be reconstructed is determined according to the rough surface model, and a multi-resolution hash feature grid is constructed in a three-dimensional space volume contained in the bounding box.

3. The method according to claim 1, characterized in that The surface multi-resolution hash feature is subjected to a warp supersampling process to obtain a pre-filtered hash feature, including: Supersample each pixel of the rendering perspective to obtain multiple pixel rays; Input the surface multi-resolution hash feature of each pixel ray index and the corresponding pixel ray direction vector into the first fully connected multi-layer perceptron neural network prediction feature residual, and perform residual connection with the surface multi-resolution hash feature of each ray index to obtain a feature connection result; Feature aggregation is performed based on the feature connection result to obtain a pre-filtered hash feature of each pixel.

4. The method according to claim 1, characterized in that: The pre-filtered hash feature is decoded to obtain a meta-light field color rendering result, including: Obtaining the pre-filtered hash feature of each pixel of the perspective to be rendered; The pre-filtered hash features of each pixel of the to-be-rendered viewing angle and the corresponding pixel ray direction vector are input into the second fully connected multi-layer perceptron neural network for decoding to obtain the meta-light field color rendering result.

5. A meta-light field cross-scale intelligent reconstruction device, characterized in that: include: A feature grid construction module is used to construct a multi-resolution hash feature grid based on the rough three-dimensional surface structure information of the scene to be reconstructed; A surface rasterization module, used to determine the index of the multi-resolution hash feature grid by rasterizing a rough three-dimensional surface, and to perform an interpolation operation on the multi-resolution hash feature grid according to the index to obtain a surface multi-resolution hash feature; A warp supersampling module, used for performing warp supersampling processing on the surface multi-resolution hash feature to obtain a pre-filtered hash feature; A feature decoding module, used for decoding the pre-filtered hash feature to obtain a meta-light field color rendering result; The surface rasterization module is further used for: Projecting the reconstructed rough three-dimensional surface to the view angle to be rendered, and obtaining the coordinates of the intersection of each pixel ray of the view angle to be rendered and the rough three-dimensional surface; Obtain hash features of eight mesh vertices of the intersection coordinates and calculate the intersection features by trilinear interpolation; The multi-resolution hash feature grid is interpolated based on the intersection feature to obtain a feature cascade, and a surface multi-resolution hash feature of the surface intersection is obtained according to the feature cascade result.

6. The device according to claim 5, characterized in that The feature grid construction module is also used to: Extracting a cross-scale input image of the scene to be reconstructed; Solving the camera pose corresponding to each of the cross-scale input images and constructing a rough surface model of the scene to be reconstructed; A bounding box of a scene to be reconstructed is determined according to the rough surface model, and a multi-resolution hash feature grid is constructed in a three-dimensional space volume contained in the bounding box.

7. The device according to claim 5, characterized in that The warp supersampling module is also used for: Supersample each pixel of the rendering perspective to obtain multiple pixel rays; Input the surface multi-resolution hash feature of each pixel ray index and the corresponding pixel ray direction vector into the first fully connected multi-layer perceptron neural network prediction feature residual, and perform residual connection with the surface multi-resolution hash feature of each ray index to obtain a feature connection result; Feature aggregation is performed based on the feature connection result to obtain a pre-filtered hash feature of each pixel.

8. The device according to claim 5, characterized in that The feature decoding module comprises: Obtaining the pre-filtered hash feature of each pixel of the perspective to be rendered; The pre-filtered hash features of each pixel of the to-be-rendered viewing angle and the corresponding pixel ray direction vector are input into the second fully connected multi-layer perceptron neural network for decoding to obtain the meta-light field color rendering result.

Images