A scalable incremental visual mapping method based on neural radiance fields

Abstract

The application provides an extensible incremental visual mapping method based on neural radiance fields. The method dynamically divides the scene into multiple partially overlapping cubic spaces according to the current frame camera pose and image depth information during the mapping process, and each cubic space is implicitly represented by a neural network to represent the environment geometry and texture attributes. During the optimization process of each related cubic space, the key frames related to the current space block are selected, the image frame information is sparsely sampled, and the distributed optimization method is used to optimize the related implicit neural space block and the current frame pose based on the information. The method provided by the application can realize continuous reconstruction and lightweight representation based on neural radiance fields in a large unknown environment, and solves the poor scalability problem of the existing related method in large scene mapping.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of robot vision mapping, in particular to an extensible incremental vision mapping method based on neural radiance field. BACKGROUND

[0002] Mapping, as an important part of the field of robot simultaneous localization and mapping (SLAM), is an important prerequisite for robot autonomous path planning and task execution. Existing methods discretely represent the scene using point clouds, voxel grids, and symbolic distance fields. Although these methods can achieve real-time scene mapping, they are difficult to balance representation quality and query efficiency in large-scale scene mapping, which greatly limits the efficiency of dense scene mapping.

[0003] Scene reconstruction methods based on neural implicit representation have achieved great success in the field of three-dimensional reconstruction. By sampling the pixel values of the perceived visual RGB or RGBD images and using voxel rendering to implicitly train a continuous neural network, the continuity of the environment representation is guaranteed, and the representation quality is improved. In addition, compared with the discrete storage methods of point cloud, voxel grid, and symbolic distance field, the network parameter storage of neural network occupies less space, and the spatial position query and reading speed is faster, and the data access efficiency is higher.

[0004] Chinese patent publication No. CN112613609A discloses a neural radiance field enhancement method based on joint pose optimization, which generates a new perspective of neural radiance field by using the rendering principle, optimizes the radiance field and pose at the same time, but essentially focuses more on the new perspective generation field, and cannot guarantee the geometric integrity of the neural radiance field representation; Chinese patent publication No. CN114004941A discloses an indoor scene three-dimensional reconstruction system and method based on neural radiance field, but this method uses a cloud-edge-end collaborative method to implicitly train the entire indoor scene into a single neural radiance field, which reduces the computational demand of the end device, but the cloud computing complexity and spatial complexity are very high, and the training speed is slow, but it still cannot meet the real-time needs of robots in unknown environments; Chinese patent publication No. CN113706714A discloses a new perspective synthesis method based on depth images and neural radiance field, which generates a three-dimensional point cloud of the target object using a depth map, converts the point cloud into a grid, generates multiple images, and finally trains the neural radiance field, which is a redundant process with high computational overhead, and cannot meet the real-time mapping requirements. When applied to large scene generation, the calculation time will increase geometrically. In summary, the current neural radiance field-based method cannot achieve real-time scene mapping requirements, and focuses on new perspective generation while ignoring the geometric consistency of neural radiance field representation. In addition, this method uses a single radiance field to represent the entire scene, which is limited by the capacity of a single network and cannot be expanded to large scenes. SUMMARY

[0005] The purpose of the present application is to provide an extensible incremental visual mapping method based on neural radiance field, which solves the problem that the current neural radiance field-based mapping method cannot be expanded to large scenes.

[0006] The first aspect of the present application provides an extensible incremental visual mapping method based on neural radiance field, and the technical solution is:

[0007] S1, dynamic scene division and implicit neural space block search:

[0008] In the mapping process, the scene is dynamically divided into multiple square spaces of the same size and partially overlapping each other according to the current frame camera pose and image information obtained from the visual odometry, and each square space is implicitly represented by a neural network based on a multi-layer perception to represent the voxel density and color attribute of the space, i.e. an implicit neural space block.

[0009] According to the current frame camera pose and image depth information, a view plane of the current frame is generated, and the implicit neural space block related to the current view plane is searched according to its effective range;

[0010] S2, optimizing the searched space block one by one:

[0011] Obtain all key frames related to the current implicit neural space block, and dynamically select M key frames according to the average depth value error size of the key frames in the current implicit neural space block;

[0012] Sparse sampling N pixel points in the selected key frames and the current frame, and projecting these pixel points into the world coordinate to generate corresponding space rays;

[0013] Using the voxel rendering method to generate the depth value and color of the corresponding rays, and obtaining the voxel density and color value of these rays at multiple position points in the overlapping region with the previous implicit neural space block, and using a step-by-step optimization method to iteratively optimize the camera pose, geometry and texture features represented by the implicit neural block;

[0014] If it is not the last space block, go back to S2 and start the next space block optimization;

[0015] S3, update the key frame relationship table:

[0016] According to the information entropy and the pose change threshold, determine whether the current frame is created as a key frame and recorded in the related relationship table;

[0017] Obtain the next frame information and go back to S1.

[0018] Optionally, the size consistent and partially overlapping cubic space includes:

[0019] The side length of the cubic space is between 4-10 meters, and there is a 20%-50% overlapping area between any two adjacent cubic spaces with consistent two-dimensional coordinates.

[0020] Optionally, the current frame camera pose of S1 includes:

[0021] The initial current frame camera pose T raw Multiply by a cumulative drift error correction term T before calculation drift , obtain the optimized camera pose T old , T old =T drift T raw ;

[0022] When S2 current frame optimization is completed, the correction term will be updated as , where T new is the optimized current frame pose.

[0023] Optionally, the S1 generates a current frame view cone according to the current frame camera pose and image depth information, and searches for the passed implicit neural space block according to its effective range, including:

[0024] According to the current frame pose, N pixels in the current frame image are uniformly sampled and the ray direction r is calculated i = T wc K -1 [u i , v i ], i ∈ {1,..., N}, wherein T wc is the camera pose, K is the camera image parameter, and the pixel point selection quantity N is less than 100;

[0025] For each ray, M position points x near = o wc + t j r, j ∈ {1,..., M} are uniformly sampled from t i to min{D far , t j}, wherein D i is the termination point of the ray according to the depth value, t near is the lower limit of the ray, t far is the upper limit of the ray, and the value is determined by the effective range of the current depth camera, and M is selected between 16 and 32;

[0026] According to the center position x cur of the current spatial block, the spatial block half side length b cur , and the center position interval between any two adjacent spatial blocks, all spatial blocks related to the current frame are determined according to If there is none, it is created;

[0027] According to the number of spatial points in the spatial block in descending order, the optimization order is sorted.

[0028] Optionally, the dynamic selection of M key frames and sparse sampling of N pixel points in S2 includes:

[0029] The key frame quantity M is selected between 2 and 5, and the pixel point quantity N is determined by N = (M + 1)N pix , wherein N pix is selected between 200 and 400.

[0030] Optionally, the distributed optimization method of S2 includes:

[0031] Determine whether the current spatial block is the first item in the optimization sequence;

[0032] 1) If it is the first item, the predicted depth value of the effective ray with all termination distances in the spatial block is output according to the voxel rendering method Color vector Optimize the following objective function within a limited number of iterations e all ​

[0033] 2) If not the first item, get the current spatial block, and optimize the current spatial block as a knowledge distillation teacher, the knowledge distillation error is represented as L dt , λ dt is the corresponding hyperparameter, and the optimization objective function is:

[0034]

[0035] In the formula, e cur is the current iteration optimization times, and the iteration times is set not to exceed 20, when e cur <e all / 3, the neural network parameters θ of the current spatial block are fixed, and only the current frame pose is optimized, if the number of depth points in the current spatial block range exceeds the threshold t d , the pose of the current frame is optimized as T new , L g is the average L1 norm error of the predicted depth value and the real depth value, L p is the average L1 norm error of the predicted color and the real color, λ g and λ p are manually set hyperparameters; when e all / 3≤e cur , the camera pose is fixed, and only the neural network parameters θ of the current spatial block are optimized, L fs is the average voxel density of all points before the termination point on all rays, λ fs is the corresponding hyperparameter.

[0036] Optionally, the knowledge distillation method in the distributed optimization method comprises:

[0037] From the currently selected rays, the rays passing through the overlapping area of the current spatial block and the previous spatial block are screened, and 16-32 spatial points are uniformly sampled from the near boundary max{0, t onear} to the far boundary min{D, t ofar} of the overlapping area, the voxel density values of these points are predicted from the two spatial blocks respectively, and the error is calculated by the average L1 norm.

[0038] Optionally, the method according to the voxel rendering outputs the predicted depth value of the effective ray of the space block with all termination distances Color vector includes the following formula:

[0039] In the formula, N sample is the number of sampling points on the ray, which is set to be between 16-32, is the termination probability of the ray, oi = 1 - exp(-σ i δ i ) is a voxel occupancy probability, δ i = ||x i+1 -x i || is the spatial distance between two adjacent points on the ray.

[0040] The second aspect of the present application provides an electronic device, comprising: at least one processor and a memory;

[0041] The memory stores computer-executable instructions;

[0042] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the method described in the above first aspect and various possible designs of the first aspect.

[0043] The third aspect of the present application provides a computer-readable storage medium, the computer-readable storage medium stores computer-executable instructions, when the processor executes the computer-executable instructions, the method described in the above first aspect and various possible designs of the first aspect is realized.

[0044] Compared with the prior art, the technical scheme of the present application has the following advantages:

[0045] (1) The present application provides an extensible incremental visual mapping method based on neural radiance field, which dynamically divides the scene into multiple implicit neural spatial blocks according to the current frame pose and RGBD image information during the mapping process. While ensuring the continuity and compactness of the data structure of the neural radiance field representing the environment, it solves the problem that the existing method cannot completely represent the entire scene due to the insufficient capacity of a single neural radiance field in a large scene.

[0046] (2) The present application adopts a sparse sampling method and a distributed optimization strategy, which can not only realize real-time mapping refresh efficiency compared with other methods based on neural radiance field, but also can optimize the initial camera pose at the same time, reducing the cumulative drift error caused by visual odometry.

[0047] (3) The present application adopts a knowledge distillation method to optimize the relevance between spatial blocks, avoiding the geometric discontinuity caused by the divided spatial blocks in the overlapping area, and fundamentally ensuring the geometric consistency of the entire scene representation. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 A flowchart of an extensible incremental visual mapping method based on neural radiance field provided by an embodiment of the present application.

[0049] Figure 2This is a schematic diagram of ray sampling under multiple hidden neural spatial blocks. Detailed Implementation

[0050] See Figure 1 This application provides a scalable incremental visual mapping method based on neural radiation fields. During the implicit mapping process, the scene is incrementally divided into multiple overlapping cubic spaces of uniform size based on the pose obtained from visual odometry. Each cubic space is implicitly represented by a neural network to represent its multidimensional data. This application refers to the cubic space represented by the neural network based on a multilayer perceptron as an implicit neural space block. During the current frame optimization process, keyframes related to the current space block are selected, and each related implicit neural space block is optimized one by one. This ultimately avoids the parameter forgetting problem caused by using a single neural network with limited capacity to represent the entire scene. This includes steps 101-112.

[0051] After the program starts executing, it first executes step 101, waiting to acquire the current frame's RGBD image data and the corresponding image pose data. Each pose data is multiplied by a cumulative drift error correction term T. drift .

[0052] Step 102, Implicit Neural Spatial Block Query and Creation. The program uniformly samples N pixels within the current frame image based on the current frame pose and calculates the ray direction r. i =T wc K -1 [u i v i ], i∈{1,...,N}, where T wc Let be the camera pose, and K be the camera image intrinsic parameter. Then, for each ray from t... near to min{D i , t far Uniformly sample M location points x j =o wc +t j r, j∈{1, ..., M}, where D i This is the endpoint of the ray obtained based on the depth value. Then, based on the center position x of the current space block... cur Half-side length b cur And the distance between the centers of any two spatial blocks, we based on Identify all spatial blocks related to the current frame; create them if they do not exist. Finally, sort the spatial points in descending order of the number of spatial blocks they belong to, and proceed to the next step.

[0053] Step 103, Space Block Selection. Select the space blocks to be optimized according to the order in which they are to be optimized.

[0054] Step 104, keyframe retrieval. Query the keyframe library 112 for keyframes related to the current control block and randomly select N frame frames of keyframes according to their normalized error loss probability.

[0055] Step 105, ray sampling. As shown in FIG. 5, randomly sample N Figure 2 pixels in the keyframes and back-project them into rays in the world coordinate system according to their poses. pix sp strat imp strat surf j j sp

[0056] Step 106, space block optimization. Determine whether the current space block is the first item in the optimization sequence:

[0057] (1) If it is the first item, output the predicted depth value of the effective ray in the space block whose termination distance is in the current space block according to the voxel rendering method color vector where N sample is the number of sampled points on the ray, p is the termination probability of the ray, o i = 1-exp(-σ i δ i ) is the voxel occupancy probability, δ i = ||x i+1 -x i || is the spatial distance between two adjacent points on the ray, and e all is the number of iterations. Optimize the following objective function within a limited number of iterations e

[0058] (2) If it is not the first item, obtain the previous space block 109 of the current frame and perform step 107 knowledge distillation optimization on the current space block as the knowledge imparting party. Select N ol rays from the current selected rays that pass through the overlapping area of the current space block and the previous space block 109, and uniformly sample M onear points on each ray from the near boundary max{0, t ofar} to the far boundary min{D, t sp} of the overlapping area.​​​​​​​​Spatial points, and predict the voxel density values of two spatial blocks at these points, and calculate the error of the two by the average L1 norm, which is denoted as L dt , λ dt are corresponding hyperparameters, and the optimization objective function is:

[0059]

[0060] where e cur is the current iteration optimization number, when e cur < e all / 3, fix the neural network parameters θ of the current spatial block, and only optimize the current frame pose, if the number of depth points within the current spatial block exceeds the threshold t d , the pose of the current frame is optimized to T new . L g is the average L1 norm error of the predicted depth value and the real depth value, L p is the average L1 norm error of the predicted color and the real color, λ g and λ p are manually set hyperparameters; when e all / 3≤e cur , fix the camera pose, and only optimize the neural network parameters θ of the current spatial block. L fs is the average voxel density value of all points before the termination point on all rays, λ g is the corresponding hyperparameter value.

[0061] Step 108, key frame detection. After optimization, 110 is executed to judge whether the information entropy of the current frame and the angle and distance of the pose transformation exceed the set threshold, if they exceed, step 111 is executed, and the key frame is added to the key frame library 112.

[0062] After step 108 is executed, the previous spatial block 109 is saved as the current optimization spatial block, and whether the current optimization spatial block is the last spatial block of the current frame is judged:

[0063] (1) is the last spatial block, the embodiment jumps back to step 101 (RGBD image and pose input) to continue to execute the optimization of the next frame;

[0064] (2) is not the last spatial block, the embodiment jumps back to step 103 (spatial block selection) to continue to execute the operation on the next spatial block to be optimized of the current frame.

[0065] The embodiment of the application ends running when it is manually terminated or there is no subsequent input value in step 101 (RGBD image and pose input).

[0066] The embodiment of the present application further provides a terminal device, comprising: at least one processor and a memory; the memory is used for storing program instructions; the processor is used for calling and executing the program instructions stored in the memory, so that the terminal device executes the neural radiance field based scalable incremental visual mapping method provided in the foregoing embodiments.

[0067] The processor is used for calling and executing the program instructions stored in the memory, so that the terminal device executes the neural radiance field based scalable incremental visual mapping method provided in the foregoing embodiments.

[0068] The embodiment of the present application further provides a computer readable storage medium, wherein instructions are stored in the computer readable storage medium, when the instructions are executed on a computer, the computer executes the neural radiance field based scalable incremental visual mapping method provided in the foregoing embodiments.

[0069] The foregoing embodiments are only used to illustrate the technical solutions of the present application, rather than limiting the same, and those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some or all of the technical features can be replaced by equivalents, and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A scalable incremental visual mapping method based on neural radiation fields, characterized in that, include: S1, Scene Dynamic Partitioning and Implicit Neural Spatial Block Search: During the mapping process, the scene is dynamically divided into multiple cubic spaces of the same size that partially overlap each other, based on the current frame camera pose and image information obtained from visual odometry. Each cubic space is implicitly represented by a neural network based on a multilayer perceptron, which is called an implicit neural space block. The current frame view frustum is generated based on the current frame camera pose and image depth information, and implicit neural spatial blocks related to the current view frustum are searched according to its effective range; S2, optimize the searched space blocks one by one: Obtain all keyframes related to the current implicit neural spatial block, and dynamically select M keyframes based on the average depth error of these keyframes within the current implicit neural spatial block. Sparsely sample N pixels in the selected keyframe and the current frame, and back-project these pixels onto world coordinates to generate the corresponding spatial ray; The depth and color of the corresponding rays are generated using voxel rendering. At the same time, the voxel density and color values ​​of multiple points in the overlapping area of ​​these rays with the previous implicit neural block are obtained. The obtained prediction data are then iteratively optimized using a step-by-step optimization method to sequentially optimize the camera pose, the geometric and texture features represented by the implicit neural block. If it is not the last space block, roll back to the beginning of S2 and perform optimization on the next space block; S3, Update the keyframe relationship table: The decision to create a keyframe for the current frame and record it in the relevant relationship table is based on the information entropy and pose change threshold. Obtain the information for the next frame and revert to S1; the process ends or is manually terminated when there are no subsequent input values ​​for the RGBD image and pose input.

2. The scalable incremental visual mapping method based on neural radiation fields according to claim 1, characterized in that, The cubic spaces of uniform size that partially overlap each other include: The cubes have a side length between 4 and 10 meters, and there is a 20% to 50% overlap between any two adjacent cubes with the same two-dimensional coordinates.

3. The scalable incremental visual mapping method based on neural radiation fields according to claim 1, characterized in that, The current frame camera pose of S1 includes: Initial current frame camera pose T raw Before calculation, multiply by a cumulative drift error correction term T. drift The camera pose T before optimization is obtained. old T old =T drift T raw ; Once the current frame of S2 is optimized, this correction will be applied. To update, in the formula, T new This is for the optimized pose of the current frame.

4. The scalable incremental visual mapping method based on neural radiation fields according to claim 1, characterized in that, The step S1, which involves generating the current frame view frustum based on the current frame camera pose and image depth information, and searching for the implicit neural spatial blocks it passes through based on its effective range, includes: Based on the current frame pose, uniformly sample N pixels within the current frame image and calculate the ray direction r. i =T wc K -1 [u i v i ], i∈{1,...,N}, where T wc Where K is the camera pose, K is the camera image intrinsic parameter, and N is selected with a pixel count of less than 100. For each ray from t near to min{D i , t far Uniformly sample M location points x j =o wc +t j r, j∈{1, ..., M}, where D i The endpoint of the ray is determined by the depth value, t. near The lower limit of radiation, t far The upper limit of the ray is determined by the effective range of the current depth camera, and M is selected between 16 and 32. Based on the current center position x of the space block cur The half-side length of the space block is b cur And the distance between the centers of any two adjacent spatial blocks, according to Determine all spatial blocks related to the current frame; create them if they do not exist. The order of optimization is sorted in descending order based on the number of spatial points in the spatial block.

5. The scalable incremental visual mapping method based on neural radiation fields according to claim 1, characterized in that, The dynamic selection of M keyframes and sparse sampling of N pixels in S2 includes: The number of keyframes M ranges from 2 to 5, and the number of pixels is determined by N = (M+1)N. pix Determine, in the formula, N pix The selection quantity is between 200 and 400.

6. The scalable incremental visual mapping method based on neural radiation fields according to claim 1, characterized in that, The distributed optimization method of S2 includes: Determine if the current spatial block is the first item in the sequence to be optimized: 1) If it is the first item, output the predicted depth values ​​of all effective rays with termination distances in this space block according to the voxel rendering method. Color vector Within a finite number of iterations e all Internal optimization of the following objective function: 2) If it is not the first option, obtain the spatial block before the current frame and use it as the knowledge transmitter to perform knowledge distillation to optimize the current spatial block. The knowledge distillation error is represented as L. dt , λ dt To correspond to the hyperparameters, the objective function for optimization is: In the formula e cur For the current iteration optimization number, set the iteration number to no more than 20, when e cur <e all At / 3, the neural network parameters θ of the current spatial block are fixed, and only the pose of the current frame is optimized. If the number of depth points within the current spatial block exceeds the threshold t... d Then the pose of the current frame is optimized to T. new L g To calculate the L1 norm average error between the predicted depth value and the true depth value, L p To calculate the average L1 norm error between the predicted color and the true color, λ g and λ p For manually set hyperparameters; when e all / 3≤e cur At that time, with the camera pose fixed, only the neural network parameters θ and L of the current spatial block are optimized. fs λ is the average voxel density at all points along all rays up to the termination point. fs These are the corresponding hyperparameters.

7. The scalable incremental visual mapping method based on neural radiation fields according to claim 5, characterized in that, The knowledge distillation method includes: Select rays from the currently selected rays that pass through the overlapping region between the current spatial block and the previous spatial block, and select rays from the near boundary of the overlapping region max{0, t onear } to the far boundary min{D, t ofar In the process, 16-32 spatial points are uniformly sampled for each ray, and the voxel density values ​​of these points are predicted from two spatial blocks respectively. The error between the two is calculated using the average L1 norm.

8. The scalable incremental visual mapping method based on neural radiation fields according to claim 6, characterized in that, The method based on voxel rendering outputs the predicted depth values ​​of all effective rays with termination distances within the spatial block. Color vector This includes using the following formulas: In the formula N sample The number of sampling points on the ray should be set between 16 and 32. Let o be the termination probability of the ray. i =1-exp(-σ i δ i ) represents the voxel occupancy probability, δ i =||x i+1 -x i || represents the spatial distance between two adjacent points on the ray.

9. An electronic device, comprising: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the method as described in any one of claims 1-7.

Images