A semi-implicit neural map construction method based on grid-like cell spatial coding

Abstract

The application discloses a semi-implicit neural map construction method based on grid-like cell space coding. The method uses grid-like cell space coding to abstractly encode a three-dimensional space, inputs the coding result into a neural network for decoding, and generates a new visual view through rendering. The grid-like cell space coding on the three-dimensional space can improve the correlation between data, eliminate redundant information in the neural map, realize compression sensing of the environment, maximize the use of information, and thus improve the map reconstruction quality. The semi-implicit neural map construction method can be applied to technical fields such as medical imaging, automatic driving, game development or indoor design, and can automatically generate a three-dimensional scene model according to an input two-dimensional image.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology and relates to methods for three-dimensional reconstruction or new perspective synthesis, specifically to a semi-implicit neural map construction method based on grid-like cell spatial coding. Background Technology

[0002] 3D reconstruction or new perspective generation technology refers to the use of computer algorithms and artificial intelligence to generate new perspectives or angles from existing image or video data. This technology has wide applications in virtual reality, augmented reality, and 3D modeling. Traditional 3D modeling requires significant time and manpower, while new perspective generation technology can automatically extract 3D scenes from existing 2D images or videos and generate new perspectives or angles through deep learning model training, thus achieving rapid 3D modeling. In virtual reality and augmented reality, new perspective generation technology allows users to observe virtual scenes or augmented reality objects from different angles or perspectives, enhancing the user's immersion and experience. It also has wide applications in medicine, architectural design, and game development.

[0003] Currently, existing 3D reconstruction or new view synthesis methods can be divided into explicit 3D reconstruction and implicit 3D reconstruction. Based on existing multi-view visual image information, explicit 3D reconstruction methods extract relevant feature information, complete data association between images, and construct explicit models such as point clouds and meshes, thereby reconstructing the 3D scene. Implicit 3D reconstruction methods, on the other hand, utilize neural network models to store and fit the spatial information of the scene to be reconstructed, predict the observation results of visual images from different viewpoints, calculate the difference between the predicted observation results and the actual measurement data, and backpropagate to optimize the neural network model, thus achieving implicit 3D scene reconstruction.

[0004] Implicit neural mapping representations have proven effective in representing shape and scene information, typically utilizing multilayer perceptrons (MLPs) to encode signed distance fields, occupancy, or volume density. Together with differentiable rendering, these methods can reconstruct the geometry and appearance of objects and scenes. Among them, Neural Radiation Fields (NeRF) have achieved remarkable results in synthesizing new views of static scenes given a set of pose input images. The key idea of ​​NeRF is to use a neural network to fit the radiation information of the scene, thereby constructing a continuous radiation field, and obtaining the image through differentiable voxel rendering. Therefore, the optimization process can be achieved by minimizing photometric loss. However, implicit neural mapping techniques still have several problems: First, due to the limited information storage capacity of the network, the forgetting phenomenon during the learning process makes it difficult for NeRF methods to encode large-scale scenes. Simultaneously, neural mapping networks are large, computationally complex, and have a large number of parameters to be trained, requiring a large amount of training data. Furthermore, the training process of neural maps relies on relatively accurate camera pose data.

[0005] To further improve the performance of neural maps, some methods attempt to construct auxiliary codes in space to replace part of the network structure and parameters, implicitly storing scene information in the coding parameters of the explicit 3D spatial structure, i.e., constructing semi-implicit neural maps. Combining the coding of auxiliary spatial structures with these methods can improve the coding spatial range and coding performance of neural maps. This series of studies demonstrates that designing semi-implicit neural map models is key to improving the efficiency and accuracy of neural map representation. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a semi-implicit neural map construction method based on grid-like cell spatial encoding. This method proposes grid-like cell spatial encoding, which explicitly abstracts and encodes three-dimensional space, improving the correlation between data, eliminating redundant information in the neural map, and achieving compressed perception of the environment. It can construct high-quality semi-implicit neural maps based on new perspective images provided by sensors, thereby improving the neural map's ability to express scenes. A semi-implicit neural map construction method based on grid-like cell spatial encoding includes the following steps:

[0007] Step 1. Selection of Grid-like Cell Spatial Encoding Model

[0008] One or more combinations of spatial encoding methods for grid cells are selected as the spatial encoding methods for grid-like cells. Multi-scale grid-like cells are constructed according to the selected encoding methods, and the area of ​​the grid-like cells increases with the scale.

[0009] Preferably, the spatial encoding method of the grid cells includes:

[0010] ① Multiple grid cells are arranged in a longitudinal, periodic, and staggered manner through a hexagonal close-packed (HCP) arrangement;

[0011] ② Multiple grid cells are arranged in a longitudinally periodic, staggered, and overlapping manner through face-centered cubic (FCC) stacking;

[0012] ③ Hexagonal patterns aligned in the z-direction between every two layers in the two grid cell spaces;

[0013] ④ A hexagonal pattern in which all individual grid cells are aligned in the z-direction of space;

[0014] ⑤ Individual grid cells are randomly distributed in the z-direction of space and have a hexagonal pattern in the horizontal direction.

[0015] Step 2. Sampling Search

[0016] From one perspective, for each two-dimensional imaging coordinate to be observed, the ray from the sensor's optical center to the imaging plane is extended in reverse by ray tracing, and the three-dimensional spatial coordinate x is sampled on the ray. Then, based on the coordinate x, the adjacent grid cells are found to obtain the multi-scale grid cell set around the coordinate x.

[0017] Step 3. Encoding Calculation

[0018] Based on the spatial relationship between coordinate x and the relative positions of each grid cell at a single scale, the weights of the interpolation combination are calculated to combine the grid cell encoding outputs, obtaining the encoding result of coordinate x at a single scale. Subsequently, the encoding results of grid cells at different scales are combined to obtain the final encoding information of the grid cells.

[0019] Step 4. Decoding and Rendering

[0020] The neural network model combines the ray tracing vector in step 2 as the viewpoint observation direction information to decode the final encoded information of the grid cells in step 3, estimate the sensor observable information corresponding to the coordinate point in three-dimensional space, and integrate the information of multiple sampling points on the extended ray in step 2 through a rendering method related to the sensor type, and output the sensor's observation results from that viewpoint.

[0021] Preferably, the rendering method includes a voxel rendering method for visible light cameras and a spherical ray tracing search rendering method for depth sensors.

[0022] Step 5. Backpropagation Training

[0023] By comparing the observation results obtained in step 4 with the original measurement results from the sensor, a corresponding loss function is constructed. Since each grid cell has trainable encoding parameters, both the implicit decoding model and the explicit grid cell encoding parameters are updated during backpropagation training of the map model. Ultimately, through continuous training, the loss function is minimized, thereby completing the construction of a semi-implicit neural map.

[0024] The present invention has the following beneficial effects:

[0025] This method primarily focuses on explicitly encoding spatially distributed information. By using grid-like spatial encoding in 3D space, it maximizes information utilization and improves map reconstruction quality. The grid-like spatial encoding model used in this method is a special type of 3D spatial hash encoding. Each grid-like cell appears in multiple locations in space; that is, adjacent grid cell sets at different 3D coordinates have a certain degree of overlap. Therefore, different sampling points will have similar encoded information, similar to hash collisions. Hash collisions can increase the correlation between data and eliminate redundant information in the hash table, thus achieving data compression of the environment. Simultaneously, this encoding method exhibits a certain periodicity. During compression, each grid-like cell pays more attention to the periodic patterns in the scene, summarizing the information patterns of the scene. Based on existing 3D spatial structure experience, it can quickly complete the cognition of 3D structures in new scenes, improving the performance of semi-implicit neural radiation fields in structured scenes. Therefore, this method has significant scientific and practical value. Attached Figure Description

[0026] Figure 1 This is a flowchart of the overall process of neural radiation field based on mesh-like cells;

[0027] Figure 2 A schematic diagram of a brain-like neural coding pattern;

[0028] Figure 3 This is a schematic diagram illustrating the construction of brain-like neural coding in the embodiment. Detailed Implementation

[0029] The present invention will be further explained below with reference to the accompanying drawings;

[0030] like Figure 1 As shown, the specific implementation steps of a semi-implicit neural map construction method based on grid-like cell spatial encoding are as follows:

[0031] Step 1: Select image data and brain-like neural coding method

[0032] The training data chosen was open-sourced by the authors of the neural radiation field dataset on Google Drive. This training data includes Lego data and scene data stored separately in two folders. Taking the "lego" category in the scene data as an example, it contains a "train / val / test" folder and three JSON files containing the camera pose for each image. Specifically, during the loading process, the "load_blender_data" function reads image, transform_matrix, and other information from the aforementioned JSON files to form the training dataset.

[0033] Different encoding methods of grid cells in three-dimensional space all present a hexagonal two-dimensional encoding in their top view. Five common grid cell spatial encoding methods are as follows: Figure 2 As shown, this embodiment selects the fourth type, namely the hexagonal pattern where all individual grid cells are aligned in the z-direction of space. According to neuroscience research, grid cells exist at different scales to correspond to objects of different sizes within the map. Therefore, this method designs multi-scale grid-like cells based on this phenomenon, with the area of ​​each scale-like grid cell being larger than that of the previous scale.

[0034] Step 2: Constructing brain-like neural coding

[0035] By combining the LEGO image data and camera pose data from step 1, a 3D spatial scene of the scene at the time it was captured is constructed. For example... Figure 3 As shown, firstly, hexagonal mesh-like cells are densely packed across the xy-plane of 3D space, dividing the hexagons into six congruent equilateral triangles. Then, for the 3D spatial coordinate x of the sampling point, its x-axis and y-axis coordinates are linearly transformed while the z-axis coordinate remains unchanged. This transforms the original equilateral triangle mesh-like cells into a space divided by right-angled triangles along the diagonals of a square. Subsequently, each dimension of the transformed 3D spatial coordinate x is rounded up or down, and the rounded results are arranged and combined to obtain the coordinates of the four vertices of the square surrounding the sampling point. Each square is composed of two right-angled triangles, so the location of the sampling point determines the right-angled triangle it occupies. Based on the vertex coordinates, the index value of each coordinate is calculated using a modulo function. Then, the index value is used to determine the number of the adjacent mesh cells at that coordinate point, extracting the parameters for the mesh cell encoding.

[0036] Next, based on the relative positions of the x-coordinate and each grid cell in the original xyz space at a single scale, the weights of the linear interpolation combination are calculated by relating the input x-coordinate to the coordinates of the three vertices of the equilateral triangle surrounding the sampling point in step two. Figure 1As shown in the upper right section, the combination of grid cell encoding outputs is implemented to obtain the encoding result of coordinate x at a single scale. Subsequently, the combined results of grid cell encoding at different scales are combined to obtain the final grid cell encoding information.

[0037] Step 3. Construct a neural radiation field based on brain-like neural coding.

[0038] 3-1 Decoding and Sampling

[0039] The neural network will combine the viewpoint information of the camera's observation direction in the training dataset. The final encoded information of the grid cells obtained in step 2 is decoded to estimate the observable information of the corresponding sensor in three-dimensional space, thus completing the construction of the NeRF model. The output of the model is the predicted image information (R, G, B, σ), where R, G, and B represent RGB color values, and σ represents voxel density. In this embodiment, an MLP is selected as the neural network.

[0040] 3-2 Voxel Rendering

[0041] 3-2-1. From a given perspective, for each two-dimensional imaging coordinate to be observed, a ray from the sensor's optical center to the imaging plane is extended backward using ray tracing. First, 64 points are sampled along this ray. Then, the spatial position and viewing angle information of each sampled point are analyzed. Encode using the method described in step 2, and obtain the corresponding scene representation (R, G, B, σ) through the decoding process described in step 3-1;

[0042] 3-2-2. Render the ray using the classic rendering formula to obtain the pixel color value C(r) corresponding to the ray:

[0043]

[0044] Where r(t) is the expression for the ray, tσ(r(t)) and c(r(t),d) are the voxel density and RGB color value of the ray at point t, respectively, both predicted by a neural network, where d is the direction vector in the ray o+td, and o is the origin of the coordinate system; T(t) is the transmittance of the ray at point t, obtained by integrating the voxel density σ(r(t)).

[0045]

[0046] Using the above ray tracing formula, the RGB color value of a pixel in the image can be calculated based on the prediction results output by the neural network.

[0047] 3-2-3. Choose L1 loss, i.e., the mean squared error loss function, for model training. The formula for calculating L1 loss is:

[0048] L1 loss = (RGB) NeRF -RGB GT ) 2

[0049] Among them, RGB NeRF It refers to the color information of the image obtained in step 3-2-2, RGB. GT The training dataset contains real image color information. The NeRF model is continuously optimized using gradient calculation and gradient descent to improve the quality of NeRF-rendered images. Since each grid cell has trainable encoding parameters, both the implicit decoding model and the explicit grid cell encoding parameters are updated during backpropagation training. Through training, the weights of important regions increase while the weights of blank regions decrease. By continuously training, the L1 loss value is minimized, thus completing the construction of the semi-implicit neural radiation field.

[0050] After the semi-implicit neural radiation field is constructed, scene information is stored in the neural network. Based on any input perspective information, the constructed semi-implicit neural radiation field model based on grid-like cell spatial encoding can provide color information and spatial geometric structure information under arbitrary spatial coordinates. Therefore, in medical imaging, the proposed technology can convert two-dimensional medical images into three-dimensional scenes, thereby helping doctors better understand the patient's condition and make more accurate diagnoses and treatment plans. In autonomous driving, based on real-time scene information of autonomous vehicles, the proposed technology can help autonomous vehicles obtain more comprehensive and accurate scene information, thereby improving driving safety and efficiency. In game development, the proposed technology can automatically generate three-dimensional models of scenes and characters in games, thereby accelerating game development speed and reducing development costs. In interior design, the proposed technology can convert two-dimensional interior design drawings into three-dimensional scenes, allowing users to more intuitively understand the interior design effect, improving designers' design efficiency and customer satisfaction. In architectural design, the proposed technology can convert architectural drawings into three-dimensional scenes, allowing designers to more intuitively understand the building's appearance and internal layout, reducing errors and repetitive work in architectural design.

Claims

1. A method for constructing a semi-implicit neural map based on grid-like cell spatial encoding, characterized in that: Specifically, the following steps are included: Step 1. Selection of Grid-like Cell Spatial Encoding Model One or more combinations of spatial coding schemes for grid cells are selected as the spatial coding schemes for grid-like cells. Multi-scale grid-like cells are constructed according to the selected coding schemes, and the area of ​​the grid-like cells increases with the scale. Step 2. Sampling Search From one perspective, for each two-dimensional imaging coordinate to be observed, the ray from the sensor optical center to the imaging plane is extended in reverse by ray tracing, and the three-dimensional spatial coordinate x is sampled on the ray. Then, based on the coordinate x, the adjacent grid cells are found to obtain the multi-scale grid cell set around the coordinate x. Step 3. Encoding Calculation Based on the spatial relationship between coordinate x and the relative positions of each grid cell at a single scale, the weight of the interpolation combination is calculated to realize the combination of grid cell encoding outputs and obtain the encoding result of coordinate x at a single scale; then, the encoding results of grid cells at different scales are combined to obtain the final encoding information of the grid cells. Step 4. Decoding and Rendering The neural network model combines the ray-traced vectors from step 2 as the viewpoint observation direction information to decode the final encoded information of the mesh cells in step 3, estimate the sensor observable information corresponding to the coordinate points in three-dimensional space, and, through a rendering method related to the sensor type, integrates the information of multiple sampling points on the extended ray from step 2 to output the sensor's observation results from that viewpoint: Step 4.1: The neural network decodes the final encoded information of the grid cells obtained in Step 2 based on the orientation information (θ, φ) of the viewpoint, estimates the observable information of the corresponding sensor in three-dimensional space, and completes the construction of the NeRF model; the output of the model is the predicted image information (R, G, B, σ), where R, G, B represent RGB color values, and σ represents voxel density; Step 4.2: Select 64 sampling points for the extended ray in Step 2, and then encode the spatial position and view information (x, y, z, θ, φ) of each sampling point using the methods described in Steps 2 and 3. Then, obtain the corresponding scene representation (R, G, B, σ) through the decoding process described in Step 4.

1. Step 4.3: Render the ray using the classic rendering formula to obtain the pixel color value C(r) corresponding to the ray: Where r(t) is the expression for the ray, tσ(r(t)) and c(r(t),d) are the voxel density and RGB color value of the ray at point t, respectively, both predicted by a neural network, where d is the direction vector in the ray o+td, and o is the origin of the coordinate system; T(t) is the transmittance of the ray at point t, obtained by integrating the voxel density σ(r(t)). Using the above ray tracing formula, the RGB color value of a pixel in the image can be calculated based on the prediction results output by the neural network. Step 5. Backpropagation Training By comparing the observation results obtained in step 4 with the original measurement results of the sensor, a corresponding loss function is constructed. Through continuous training, the loss function is minimized, thereby completing the construction of the semi-implicit neural map.

2. The semi-implicit neural map construction method based on grid-like cell spatial encoding as described in claim 1, characterized in that: The spatial encoding method of the grid cells includes: ① Multiple grid cells are arranged in a longitudinal, periodic, and staggered manner through hexagonal close-packing of HCPs; ② Multiple grid cells are arranged in a face-centered cubic stacking pattern, forming a longitudinally periodic, staggered combination of FCCs; ③ Hexagonal patterns aligned in the z-direction between every two layers in the two grid cell spaces; ④ A hexagonal pattern in which all individual grid cells are aligned in the z-direction of space; ⑤ Individual grid cells are randomly distributed in the z-direction of space and have a hexagonal pattern in the horizontal direction.

3. The semi-implicit neural map construction method based on grid-like cell spatial encoding as described in claim 1, characterized in that: In step 2, hexagonal mesh-like cells are first densely packed across the xy-plane of three-dimensional space, and the hexagons are divided into six congruent equilateral triangles. Then, for the three-dimensional spatial coordinate x of the sampling point, the x-axis and y-axis coordinates are linearly transformed, while the z-axis coordinate remains unchanged. This transforms the mesh-like cells, which were originally distributed in equilateral triangles, into a right-angled triangle space divided by the diagonals of a square. Subsequently, each dimension of the transformed three-dimensional spatial coordinate x is rounded up or down, and the rounded results are arranged and combined to obtain the coordinates of the four vertices of the square surrounding the sampling point. Since each square is composed of two right-angled triangles, the right-angled triangle in which the sampling point is located is determined based on the position of the sampling point. Based on the vertex coordinates, the index value of each coordinate is calculated using the modulo function. Then, the index value is used to determine the number of the adjacent mesh cells of that coordinate point, and the parameters of the mesh cell encoding are extracted to obtain the multi-scale mesh cell set around coordinate x.

4. The semi-implicit neural map construction method based on grid-like cell spatial encoding as described in claim 3, characterized in that: Based on the relative positional relationship between coordinate x and each grid cell in the original xyz space at a single scale, the weight of the linear interpolation combination is calculated by inputting the relationship between coordinate x and the coordinates of the three vertices of the equilateral triangle around the sampling point in step two, thereby realizing the combination of grid cell encoding outputs and obtaining the encoding result of coordinate x at a single scale; subsequently, by combining the combination results of grid cell encoding at different scales, the final grid cell encoding information is obtained.

5. The semi-implicit neural map construction method based on grid-like cell spatial encoding as described in claim 1, characterized in that: The rendering methods include voxel rendering for visible light cameras and spherical ray tracing search rendering for depth sensors.

6. The semi-implicit neural map construction method based on grid-like cell spatial encoding as described in claim 1, characterized in that: In step 5, select the L1 loss model for training. The formula for calculating L1 loss is: in, It is the color information of the image obtained after rendering. It contains real image color information; the model is continuously optimized through gradient calculation and gradient descent.

Images