Fast indoor reconstruction method based on visual-inertial navigation
By synchronously acquiring data from depth cameras and IMUs, and combining dynamic octree models and ORB feature matching, real-time 3D reconstruction of indoor environments is achieved on ordinary onboard computers. This solves the problems of high cost and heavy load in existing technologies, and improves the convenience and lightweight nature of the system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-22
- Publication Date
- 2026-04-10
AI Technical Summary
Existing technologies for rapid real-time 3D reconstruction using handheld cameras in unknown indoor environments are costly and computationally intensive, making it impossible to run on a lightweight CPU.
Data is collected synchronously using a depth camera and an IMU. Local maps are generated through feature extraction and keyframe judgment. Dense 3D reconstruction is performed using a dynamic octree model, combined with IMU pre-integration and backend optimization. The pose is optimized by ORB feature matching and Gauss-Newton method. Real-time reconstruction is performed using a regular onboard computer.
It enables real-time 3D reconstruction of indoor environments on ordinary onboard computers, reducing equipment load and cost, and improving the convenience and lightweight nature of the system.
Smart Images
Figure CN116592892B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the field of three-dimensional reconstruction, and particularly relates to a fast indoor reconstruction method based on visual inertial navigation. BACKGROUND
[0002] SLAM is a technology for acquiring self-position information by a computing device through the input of self-sensor, and is widely applied to the fields of automatic driving and VR / AR. At present, the algorithm of vSLAM based on vision is relatively mature, two main research directions are respectively to improve the positioning and the effect in a dynamic environment by using deep learning or other machine learning methods, and the other is to fuse with multiple sensors to improve the robustness of positioning, and the research on the application value after positioning is also one of the current popular directions.
[0003] The indoor three-dimensional reconstruction applying the SLAM technology has a broad development prospect. Before the proposal of Kinect Fusion, the three-dimensional reconstruction in an indoor environment is mainly achieved by using the SfM (Struct from Motion) method, the position relationship of multiple images is restored through a large amount of calculation to reconstruct the indoor environment, and the main disadvantage of the method is that it cannot be performed in real time. Kinect Fusion first uses GPU to complete real-time reconstruction, but the lightweight scheme on CPU is relatively few, and the robustness needs to be further verified. SUMMARY
[0004] Therefore, the application aims to provide a fast indoor reconstruction method based on visual inertial navigation to solve the problems of high cost and heavy load of fast real-time three-dimensional reconstruction in an indoor environment under a handheld camera.
[0005] To achieve the above object, the technical scheme of the application is as follows:
[0006] The fast indoor reconstruction method based on visual inertial navigation comprises the following steps:
[0007] Synchronously collecting data information by using a depth camera and an IMU to obtain images and IMU data corresponding to the images;
[0008] Selecting two images to obtain the relative pose transformation between the two images;
[0009] Extracting features from the images to obtain the pose of the current frame image;
[0010] Setting a judgment criterion, and defining the current frame meeting the judgment criterion as a key frame;
[0011] The pose of the key frame and the feature points in the key frame are generated into a three-dimensional landmark point and added to the local map, and the IMU pre-integration between the continuous key frames is also put into the local map, and the continuous key frames and the landmark points are further optimized in the local map;
[0012] The depth information and the pose information obtained by the depth camera are used for patch-based dense three-dimensional reconstruction in an indoor environment;
[0013] A global octree model is established, and a TSDF value is established for each voxel, and the TSDF value is updated each time new data arrives;
[0014] Surfaces are extracted from all meshes at the minimum resolution to generate the final reconstruction surface;
[0015] The generated triangular patches are rendered and displayed, and the files are saved.
[0016] Further, the IMU data between the images is pre-integrated in the Lie group and Lie algebra form to obtain the relative pose transformation between the two images.
[0017] Further, the ORB feature extraction is performed on the images, and the feature matching between the two images is performed to recover the three-dimensional landmark points tracked by the current frame image, and the visual observation model is established between the current frame pose and the tracking points, and the inertial observation residual is constructed by the two frame poses and the IMU measurement, and the current frame pose is optimized by the Gauss-Newton method.
[0018] Further, the criteria include:
[0019] The area of the feature points observed by the previous frame in the current frame does not exceed X%;
[0020] And
[0021] The distance from the previous frame has been Y consecutive frames.
[0022] Further, when the continuous key frames and the landmark points in the local map are further optimized, the common view frames that do not belong to the most adjacent key frames are fixed.
[0023] Further, when the patch-based dense three-dimensional reconstruction is performed in the indoor environment, the pose comes from the back-end optimization thread, and when the back-end optimization thread is not executed, the estimated result of the front-end pose tracking is selected.
[0024] Further, a global octree model is established, and the child nodes of the octree represent a specific space cube, in order to use parallel acceleration calculation, the child nodes of the octree are octuple into a unit grid, and the unit grid contains 512 voxels.
[0025] For the place where there is no object, the octree does not expand, and for the place where there is a space object, the octree expands the grid at the minimum resolution. The minimum resolution is set to 6mm to ensure the accuracy of the reconstruction. The TSDF value is established for each voxel under the grid, and is updated every time new data arrives.
[0026] Further, the scheme discloses an electronic device, comprising a processor and a memory connected with the processor and used for storing executable instructions of the processor, and the processor is used for executing the fast indoor reconstruction method based on visual inertial navigation.
[0027] Further, the scheme discloses a server, comprising at least one processor and a memory connected with the processor, wherein the memory stores executable instructions of the at least one processor, and the instructions are executed by the processor to make the at least one processor execute the fast indoor reconstruction method based on visual inertial navigation.
[0028] Further, the scheme discloses a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the fast indoor reconstruction method based on visual inertial navigation.
[0029] Compared with the prior art, the fast indoor reconstruction method based on visual inertial navigation has the following beneficial effects:
[0030] (1) The fast indoor reconstruction method based on visual inertial navigation can position and track the terminal moving autonomously in an indoor environment, and can perform real-time three-dimensional reconstruction on the surrounding environment, and output a model that can support editing of various three-dimensional modeling software.
[0031] (2) The fast indoor reconstruction method based on visual inertial navigation uses a dynamic octree to utilize the minimum memory space for representing a three-dimensional model, and uses an octant division strategy to accelerate the calculation speed, so that the system can run on an ordinary onboard computer without GPU acceleration, and the whole system is more convenient and lightweight. BRIEF DESCRIPTION OF DRAWINGS
[0032] The accompanying drawings, which form a part of the present application, are intended to provide further understanding of the present application, and are used to interpret the illustrative embodiments of the present application and their descriptions, and do not constitute improper limitations on the present application. In the drawings:
[0033] Figure 1 A multi-thread schematic diagram of real-time reconstruction according to the embodiments of the present application;
[0034] Figure 2 A schematic diagram of different frequency data acquisition of the IMU and camera described in the embodiments of the present application;
[0035] Figure 3 A schematic diagram of graph reconstruction for a certain activity room described in the embodiments of the present application;
[0036] Figure 4 A schematic diagram of qualitative viewing of the reconstruction result of a certain activity room described in the embodiments of the present application. DETAILED DESCRIPTION
[0037] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.
[0038] The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0039] The existing indoor technology mostly focuses on indoor positioning, and there are many indoor visual and IMU coupled positioning schemes, but they rarely pay attention to real-time reconstruction of the surrounding environment or the generated map is a sparse three-dimensional point cloud, and the three-dimensional model of the present application has great application value. At the same time, some existing dense mapping schemes mostly use point cloud stacking, which cannot support long-term operation, and the present application uses a dynamic octree method to represent the three-dimensional model with minimum memory space, and uses an octant strategy to speed up the calculation, so that it can run on a normal onboard computer without GPU acceleration, making the entire system more convenient and lightweight.
[0040] The present application is a software and hardware combined system, which needs to be deployed on a corresponding ubuntu18.04 computer when used, and the deployment is completed by a script for automatic deployment. At the same time, the hardware device supports Microsoft's kinect dk depth camera. The software is opened through the desktop program, and the start key is clicked, so that the visual interface can be seen to reconstruct the image shot by the camera. In an indoor environment, the camera can be held to reconstruct, or the onboard computer and the camera can be placed on a robot car to reconstruct the environment while moving with the car. After the reconstruction is completed, the result is saved in the output path of the current folder, and the result can be copied or the remote sending command of the software is sent to the local server.
[0041] In the implementation process, the onboard computer (without high-performance graphics card) is used as the terminal, the terminal with depth camera and IMU is used for indoor free movement, the motion estimation of the depth camera is used to determine the deviation of the IMU and the gravity direction, the information of the depth camera is used to select the key points of the image to generate the initial local map, the kinematics of the IMU and the characteristics of the high-frequency output are used to determine the motion relationship between the two frames of images, then the local map points are projected onto the current frame to form the re-projection error, the measurement of the IMU and the pose residual formed between the poses are considered comprehensively, and the random walk of the deviation is considered, the pose is continuously tracked, the key frames are selected to continue to generate the local map, and the map points that are not observed between the key frames are removed, the key frames, the pre-integration of the IMU and the local map are input into the backend optimization thread, the optimized pose and the depth image information are used to update the global map based on the octree symbol distance function SDF, and the batch object face is reconstructed, finally the results are saved to the obj and ply format three-dimensional files through Qt rendering and visualization, and the current indoor three-dimensional reconstruction is solved. The situation of heavy load equipment and high price product monopoly is solved.
[0042] Specifically, the following steps are included:
[0043] S1: The information of the depth camera and the IMU is synchronously collected by setting multiple threads, and the collection frequencies are 30FPS and 200Hz respectively. The different frequency diagrams are shown as follows.
[0044] S2: The relative pose transformation between the two frames of images is obtained by performing matrix pre-integration on the IMU data between the images in the Lie group and Lie algebra form, the ORB features of the images are extracted, the feature matching between the two frames is performed, the three-dimensional landmark points tracked by the current frame image are recovered, the visual observation model between the current frame pose and the tracking points is established, and the inertial observation residual is constructed by using the two frame poses and the IMU measurement, and the current frame pose is solved by using the Gauss-Newton method.
[0045] S3: The key frame determination is performed on the current frame, and the determination criteria are 1) the area of the feature points observed by the previous frame in the current frame is not more than 60%, and 2) there are 5 consecutive frames away from the previous frame.
[0046] S4: After the pose calculation is completed, the three-dimensional landmark points of the key frame and the feature points are added to the local map, the IMU pre-integration between the continuous key frames is also put into the local map, and the continuous key frames and the landmark points are further optimized in the local map. Considering that different key frames may observe the same landmark points, the common view frames that do not belong to the most adjacent key frames are fixed, that is, the optimization equation is added but not iteratively updated.
[0047] S5: At the same time when the pose estimation thread is started, start the reconstruction thread to use the depth information and pose information obtained by the depth camera to do the dense three-dimensional reconstruction based on the patch for the indoor environment, in order to ensure the accuracy of the image pose information, the pose used by the reconstruction comes from the back-end optimization thread, and when the back-end optimization thread is not executed, the estimated result of the front-end pose tracking is selected.
[0048] S6: Establish a global octree model, the child nodes of the octree represent a specific space cube, in order to use application and parallel acceleration calculation, the child nodes of the octree are octantized into a unit grid, and the unit grid contains 512 voxels in total. For places where no object exists, the octree is not expanded, and for places where space objects exist, the octree is expanded in the minimum resolution grid. The minimum resolution is set to 6mm to ensure the accuracy of the reconstruction, and a TSDF value is established for each voxel in the grid, which is updated each time new data arrives.
[0049] S7: Use the Marching Cube algorithm to extract the surface of all grids under the minimum resolution to generate the final reconstruction surface.
[0050] S8: Use Qt to render and display the generated triangular patches, and save the file according to the obj after the reconstruction tree is built;
[0051] In the specific implementation process, the specific steps of the present application are as follows:
[0052] Step 1, use multiple threads to simultaneously collect depth camera (depth image and color image) and IMU data.
[0053] 1a) The data size of the depth image is 640*576, and the color image is cropped according to the depth image. The camera shooting frequency is required to be not less than 30 frames per second.
[0054] 1b) The frequency of the IMU has no specific requirement, the angular velocity frequency is generally higher than twice the camera frequency, and the acceleration frequency is higher.
[0055] Step 2, perform image pyramid processing on the color image, the scaling factor is 1.2, the number of extracted layers is 6, ORB features are used to extract feature points, and a quadtree is used for non-maximum suppression to make the feature points uniformly distributed on the image.
[0056] Step 3, perform feature matching, since the landmark of the feature points of the previous frame is known, the matching of the 3D landmark points and the 2D feature points is actually completed.
[0057] Step 4. Pre-integrate the IMU data between images to get the relative pose observation between two frames. The next step is synchronized with step 2, 3 in the case of multi-threading, as shown in the description of the accompanying drawings. Figure 1
[0058] Step 5. Establish a non-linear optimization based on Gauss-Newton method for IMU observation residual and visual observation residual, and establish a random walk model between biases to update all variables of the current frame.
[0059] Step 6. Determine the key frame for the image, establish a local map for several consecutive key frames and the map points observed by them, use the sliding window method to insert new frames and remove old frames, for the outer frame which does not belong to the local map but can also observe the local map, also add to this optimization model, but as a fixed term, that is, only provide constraints and do not participate in updating, use marginalization calculation for landmark points to solve the pose.
[0060] Step 7. Establish a global octree model at the beginning of the system, each child node represents a corresponding cube in space, and whenever a new key frame or ordinary frame is added, perform TSDF calculation and update for all unit grids and voxel grids.
[0061] Step 8. Use the Marching Cube algorithm to extract the surface for all minimum resolution voxel grids to get all the results.
[0062] Step 9. Add the reconstruction results of the algorithm to the rendering queue, use the lightweight visualization library QGLviewer to render and visualize the results.
[0063] Those of ordinary skill in the art can realize that the units and method steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description in general terms. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.
[0064] In several embodiments provided in the present application, it should be understood that the disclosed method and system can be implemented in other manners. For example, the division of the above-mentioned units is only a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the above-mentioned units can or can not be physical units, and can or can not be distributed on a network. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments of the present application.
[0065] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application, and they should be covered in the scope of the claims and the specification of the present application.
[0066] The above-mentioned is only the preferred embodiment of the present application, and does not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.
Claims
1. A fast indoor reconstruction method based on visual-inertial navigation, characterized in that, The method comprises the following steps: Synchronously collecting data information by using a depth camera and an IMU, obtaining an image and IMU data corresponding to the image; Selecting two images to obtain a relative pose transformation between the two images; Extracting features from the image to obtain a pose of the current frame image; Setting a judgment criterion, and defining a current frame meeting the judgment criterion as a key frame; Adding a three-dimensional landmark point of the key frame and the feature point in the key frame to a local map, and meanwhile adding IMU pre-integration between continuous key frames to the local map, and further optimizing the continuous key frames and the landmark point in the local map; Using depth information and pose information obtained by the depth camera to perform patch-based dense three-dimensional reconstruction on an indoor environment; Establishing a global octree model, establishing a TSDF value for each voxel, and updating the TSDF value each time new data arrives; Extracting a surface from all voxel grids under a minimum resolution to generate a final reconstruction surface; Rendering and displaying the generated triangular patches, and saving a file; The judgment criterion comprises: An area of feature points observed by a previous frame in the current frame is less than 60%; And There are five continuous frames from the previous frame; When further optimizing the continuous key frames and the landmark point in the local map, fixing a common view frame not belonging to the most adjacent key frame; When performing patch-based dense three-dimensional reconstruction on the indoor environment, the pose is obtained from a back-end optimization thread, and when the back-end optimization thread is not executed, an estimated result obtained by a front-end pose tracking is selected; Establishing a global octree model, and a child node of the octree represents a specific space cube; in order to apply parallel acceleration calculation, the child node of the octree is octuple into a unit grid, and the unit grid contains 512 voxels in total. The octree is not expanded in a place where no object exists, and the octree is expanded in a place where a space object exists; a TSDF value is established for all unit grids and voxel grids, and the TSDF value is updated each time new data arrives.
2. The method of claim 1, wherein: Performing matrix pre-integration on IMU data between images in a Lie group and a Lie algebra form to obtain a relative pose transformation between the two images.
3. The method of claim 1, wherein: Extracting ORB features from the image, and performing feature matching between the two images to restore a three-dimensional landmark point tracked by the current frame image; a visual observation model is established between the pose of the current frame and the tracking point, and an inertial observation residual is constructed by using two poses and IMU measurement; the pose of the current frame is optimized by using a Gauss-Newton method.
4. The fast indoor reconstruction method based on visual-inertial navigation according to claim 1, wherein: The minimum resolution is set to 6 mm to ensure the accuracy of the reconstruction.
5. An electronic device, comprising a processor and a memory connected to the processor in communication, and configured to store executable instructions of the processor, characterized in that: The processor is configured to execute the fast indoor reconstruction method based on visual-inertial navigation according to any one of claims 1-4.
6. A server, characterized by: A computer program product comprising a computer readable medium having stored thereon instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1-4.
7. A computer readable storage medium storing a computer program, characterized in that: The computer program is executed by a processor to implement the method of any one of claims 1-4.