A method for constructing real-time dense maps of unknown environments based on RGB-D cameras

Abstract

This invention discloses a method for constructing real-time dense maps of unknown environments based on an RGB-D camera, belonging to the field of machine vision technology. Specifically, it involves: using a wheeled robot platform equipped with an RGB-D camera, obtaining the robot's pose using visual odometry and wheeled odometry calculation; obtaining a voxel space composed of voxel blocks based on the robot's pose and depth image, and constructing a hash table; synthesizing distance information into each voxel block based on the depth image, determining the accurate position of the corresponding voxel block in the map, and assigning distance field information and depth weights; finding the voxel block closest to the map surface based on the distance field information stored in each voxel block, assigning color information, and forming a 3D dense map; and using a BOW (Browser Overlap) detection matching algorithm for loop closure detection to update the 3D dense map and the wheeled robot's pose in real time. This invention is applicable to practical engineering applications, achieving high-performance, high-real-time, robust, and high-precision construction of large-scale dense maps.

Description

Technical Field

[0001] This invention belongs to the field of machine vision technology, specifically relating to a method for constructing real-time dense maps of unknown environments based on an RGB-D camera. Background Technology

[0002] Map building is an important research topic in the field of machine vision. The problem it addresses is: given RGB (color image) graphics and depth images in the form of a video stream, how to obtain a 3D map of the entire scene? Visual mapping techniques can use sensors such as monocular, binocular, and RGB-D (color + depth image) cameras to perform incremental dense mapping of the location environment. However, the depth information from monocular and binocular cameras needs to be estimated by matching RGB images, which results in accuracy loss. Therefore, RGB-D cameras are typically used for mapping.

[0003] RGB-D mapping technology has experienced rapid growth since 2011. The KinectFusion method, released in 2011, which can build dense 3D maps in real time, ushered in a new era for the field of 3D reconstruction. Based on this algorithm, excellent dense map building algorithms such as Kintinuous, InfiniTAM, ElasticFusion, Fusion4D, and RTAB-MAP have emerged. However, these algorithms suffer from limitations such as the inability to build large-scale maps due to memory constraints, the inability to perform loop closure detection and updates in dense mapping over large areas, and difficulties in map reuse.

[0004] Based on this, the present invention proposes a method for constructing dense maps that can build large-scale maps, has high efficiency in loop closure detection and updating, and has map reuse capabilities. Summary of the Invention

[0005] To address the shortcomings of the aforementioned dense mapping techniques, this invention provides a method for constructing real-time dense maps of unknown environments based on an RGB-D camera. This method can achieve accurate and efficient mapping of large-scale scenes, eliminate accumulated errors through loop closure detection and update strategies, and has the function of map reuse.

[0006] A method for constructing real-time dense maps of unknown environments based on RGB-D cameras, characterized by the following steps:

[0007] Step 1: Based on a wheeled robot platform equipped with an RGB-D camera, RGB-D images of the unknown environment are acquired using the RGB-D camera. Under the visual odometry method, the FAST (Features from accelerated segment test) algorithm is used to extract feature points from the RGB-D images. The motion matrix of the RGB-D camera is solved using the PNP (Perspective-n-Points) algorithm. Then, the bundle adjustment method (BA) is used to minimize reprojection error and photometric error to obtain the pose within the ranging range of the RGB-D camera. Outside the ranging range of the RGB-D camera, the wheeled odometry method is used to locate the robot, thereby obtaining the pose of the wheeled robot.

[0008] Step 2: Based on the pose and depth image of the wheeled robot (obtained by extracting from the RGB-D image), allocate the space that the map may occupy in the three-dimensional space, allocate the voxel space in the corresponding space based on the Voxel Hashing algorithm, obtain the voxel space composed of voxel blocks, and construct a hash table.

[0009] Step 3: Based on the depth image, the distance information is synthesized into each voxel block in the voxel space. Then, based on the wheeled robot's pose and the depth image, the accurate position of the corresponding voxel block of the map is determined. The distance field information (SDF) and depth weights are assigned to the corresponding voxel block. Based on the distance field information stored in each voxel block, the voxel block closest to the map surface is found and assigned color information to form a complete 3D dense map.

[0010] Step 4: Loop closure detection is performed using the BOW (Bag of Words) algorithm to update the 3D dense map and the wheeled robot's pose in real time. The BOW algorithm uses visual words to represent features at various locations in the 3D dense map and creates nodes based on the currently processed RGB-D image. All features of the currently processed RGB-D image are represented by node signatures composed of all corresponding visual words. The loop closure is then compared with the current node L... t The weight of the previous processing node L c The weights are updated for L. t The weights are calculated using the following formula:

[0011]

[0012] Where, N pair For L t With L c The number of matching words; and L respectively t With L c Total word count; s(Lt ,L c The ) represents the minimum ratio of the number of matched words to the total number of words.

[0013] If s(L) t ,L c () greater than the preset threshold T s Then delete the L from the vocabulary of the BOW detection matching algorithm. t The newly added visual words, and L c Merge into L t In the middle, L t After merging, only L's node signature is retained. c The visual word, L t The weight is updated to L t The weights and L c Add 1 to the sum of the weights, and also add L c Neighbors and loopback redirection to L t Middle; otherwise, not L t Perform the operation.

[0014] Furthermore, step 3 employs a GPU (Graphics Processing Unit) processing method with a memory management strategy to synthesize distance information into each voxel block, assign distance field information and depth weights to the voxel blocks, and assign color information to the voxel block closest to the map surface; the specific memory management strategy is as follows:

[0015] Step 3.1: Calculate the hash value of each voxel block in voxel space using world coordinates. Based on the hash value, find the corresponding hash entry in the hash table, store it in the hash table in memory, and obtain the voxel block pointer stored in the value of the hash entry. According to the voxel block pointer, call the corresponding connected voxel block to the external memory, and set the voxel block pointer to the voxel block. Then, store all voxel blocks in voxel space in the external memory.

[0016] Step 3.2: For voxel blocks that require GPU processing, calculate their hash value based on world coordinates, then use the hash value to find the corresponding voxel block pointed to by the voxel block pointer in external memory, and copy it to memory for computation. When processing voxel blocks using GPU, first copy the hash entry corresponding to the voxel block in the hash table from memory to the intermediate buffer of video memory, and at the same time copy the voxel block from memory to the heap space of video memory. Then, perform GPU computation in video memory. After GPU processing is completed, update the hash entry and voxel block in video memory to memory, and then update them from memory to external memory.

[0017] Furthermore, if, during the construction of a 3D dense map, only a portion of the 3D dense map is constructed due to communication interruptions or other reasons, the vocabulary corresponding to the portion of the 3D dense map is stored in external storage, and the references to each visual word in external storage are deleted, thus completing the storage of the portion of the 3D dense map.

[0018] Multiple parts of the 3D dense map stored in external memory are extracted, and loop closure detection is performed using the BOW detection matching algorithm to merge the multiple parts of the 3D dense map and obtain a complete 3D dense map.

[0019] Furthermore, in step 4, the threshold T s The value ranges from [0.5, 1].

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0021] 1. The real-time dense map construction method for unknown environments based on RGB-D cameras proposed in this invention can provide accurate pose information for wheeled robots even outside the ranging range of RGB-D cameras, resulting in more complete positioning. Furthermore, through loop closure detection and update strategies, it solves the problem of cumulative errors caused by long-term positioning. It is suitable for practical engineering applications and achieves high-performance, high-real-time, robust, and high-precision large-scale dense map construction.

[0022] 2. Preferably, by employing a GPU processing method with memory management strategies, the contradiction between computing speed and storage space can be balanced, and a larger map can be built while ensuring computing speed, thus achieving the goal of efficient mapping.

[0023] 3. Preferably, in cases where only a partial 3D dense map is constructed due to communication interruptions or other reasons, a map management strategy is used to realize the functions of storing, retrieving, updating, and merging maps, thereby achieving the purpose of map reuse. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the principle of the method for constructing a real-time dense map of an unknown environment based on an RGB-D camera, as proposed in Embodiment 1 of the present invention.

[0025] Figure 2 This is a schematic diagram of the GPU processing method with memory management strategy in Embodiment 1 of the present invention;

[0026] Figure 3 This is a comparison image of the dataset mapping effect in Embodiment 1 of the present invention and a real scene image;

[0027] Figure 4 This is a comparison image of the real-time mapping effect of the mobile robot in Embodiment 1 of the present invention and a real scene image;

[0028] Figure 5 This refers to the five-part three-dimensional dense map in Embodiment 1 of the present invention;

[0029] Figure 6 In Embodiment 1 of the present invention, the following steps are performed sequentially: Figure 5 The image shows the effect of reusing a 3D dense map consisting of 5 parts. Detailed Implementation

[0030] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0031] Example 1

[0032] The principle block diagram of the real-time dense map construction method for unknown environments based on RGB-D cameras proposed in this embodiment is as follows: Figure 1 As shown, it includes the following steps:

[0033] Step 1: Based on a wheeled robot platform equipped with an RGB-D camera, RGB-D images of the unknown environment are acquired using the RGB-D camera. Under the visual odometry method, the FAST algorithm is used to extract feature points from the RGB-D images. The motion matrix of the RGB-D camera is solved using the PNP algorithm. Then, the bundle adjustment method is used to minimize the reprojection error and photometric error to obtain the pose within the ranging range of the RGB-D camera. Outside the ranging range of the RGB-D camera, the wheeled odometry method is used to locate the robot and obtain its pose.

[0034] Step 2: Based on the pose and depth image of the wheeled robot (obtained by extracting from the RGB-D image), allocate the space that the map may occupy in the three-dimensional space, allocate the voxel space in the corresponding space based on the Voxel Hashing algorithm, obtain the voxel space composed of voxel blocks, and construct a hash table.

[0035] Step 3: Based on the depth image, the distance information is synthesized into each voxel block in the voxel space. Then, based on the wheeled robot's pose and the depth image, the accurate position of the corresponding voxel block of the map is determined, and the distance field information and depth weights are assigned to the corresponding voxel block. Based on the distance field information stored in each voxel block, the voxel block closest to the map surface is found and assigned color information to form a complete 3D dense map.

[0036] The GPU processing method employing a memory management strategy is illustrated in the following block diagram: Figure 2 As shown, distance information is synthesized into each voxel block, distance field information and depth weights are assigned to the voxel blocks, and color information is assigned to the voxel block closest to the map surface; the specific memory management strategy is as follows:

[0037] Step 3.1: Calculate the hash value of each voxel block in voxel space using world coordinates. Based on the hash value, find the corresponding hash entry in the hash table and store it in the hash table in the running memory (corresponding to arrow 1). Obtain the voxel block pointer stored in the value of the hash entry. According to the voxel block pointer, call the corresponding connected voxel block to the external hard disk storage (corresponding to arrow 2) and point the voxel block pointer to the voxel block. Then, store all voxel blocks in voxel space in the external hard disk storage.

[0038] Step 3.2: For voxel blocks that require GPU processing, calculate their hash values ​​based on world coordinates, then use the hash values ​​to locate the corresponding voxel block pointer in the external hard disk storage, and copy it to the runtime memory for computation (corresponding to arrow 3); when processing the voxel block using the GPU, first copy the hash entry corresponding to the voxel block in the hash table from the runtime memory to the intermediate buffer of the GPU memory (corresponding to arrow 4), and simultaneously copy the voxel block from the runtime memory to the heap space of the GPU memory (corresponding to arrow 5), and then perform GPU computation in the GPU memory; after the GPU memory processing is completed, update the hash entry and voxel block in the GPU memory to the runtime memory (corresponding to arrows 6 and 7), and then update the external hard disk storage from the runtime memory;

[0039] Step 4: Loop closure detection is performed using the BOW (Browser-Warp) algorithm to update the 3D dense map and the wheeled robot's pose in real time. The BOW algorithm uses visual words to represent the features of each location in the 3D dense map and creates nodes based on the currently processed RGB-D image. All features of the currently processed RGB-D image are represented by node signatures composed of all corresponding visual words. The loop closure is then compared with the current node L. t The weight of the previous processing node L c The weights are updated for L. t The weights are calculated using the following formula:

[0040]

[0041] Where, N pair For L t With L c The number of matching words; and L respectively t With L c Total word count; s(L t ,L c The ) represents the minimum ratio of the number of matched words to the total number of words.

[0042] If s(L) t ,L c () greater than the preset threshold Ts =0.7, then delete entries from the vocabulary of the BOW detection matching algorithm that are from L. t The newly added visual words, and L c Merge into L t In the middle, L t After merging, only L's node signature is retained. c The visual word, L t The weight is updated to L t The weights and L c Add 1 to the sum of the weights, and also add L c Neighbors and loopback redirection to L t Middle; otherwise, not L t Perform the operation.

[0043] Figure 3 The images shown are of the results of building a map from a known dataset using the real-time dense map construction method provided in this embodiment, and a comparison image with a real-world scene image. Figure 4 The images shown are the results of building a real-time map of a scene using the real-time dense map construction method provided in this embodiment, and a comparison image with a real scene image. Both of these images demonstrate that this embodiment can achieve robust and high-precision dense map construction.

[0044] Furthermore, if, during the construction of a 3D dense map, only a portion of the 3D dense map is constructed due to communication interruptions or other reasons, a map management strategy is adopted to construct the complete 3D dense map, specifically as follows:

[0045] Store the vocabulary corresponding to a portion of the 3D dense map in external storage, and delete the references to each visual word in external storage to complete the storage of a portion of the 3D dense map.

[0046] Multiple parts of the 3D dense map stored in external memory are extracted, and loop closure detection is performed using the BOW detection matching algorithm to merge the multiple parts of the 3D dense map and obtain a complete 3D dense map.

[0047] Example: such as Figure 5 The five parts of the 3D dense map shown are respectively Figure 5 (a) Figure 5 (b) Figure 5 (c) Figure 5 (d) and Figure 5 (e). Figure 6 This is a rendering of the map reuse effect of five parts of a dense 3D map. Figure 6 (a) and Figure 5 (a) Same, will Figure 5 (a) and Figure 5 (b) The result of merging Figure 6 (b) Merge sequentially Figure 5(c) Figure 5 (d) and Figure 5 (e) yields the corresponding merge result. Figure 6 (c) Figure 6 (d) and Figure 6 (e) indicates that large-scale dense map construction can be achieved through map management strategies.

[0048] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A method for constructing real-time dense maps of unknown environments based on RGB-D cameras, characterized in that, Includes the following steps: Step 1: Based on a wheeled robot platform equipped with an RGB-D camera, RGB-D images of the unknown environment are acquired using the RGB-D camera. Under the visual odometry method, the FAST algorithm is used to extract feature points from the RGB-D images. The motion matrix of the RGB-D camera is solved using the PNP algorithm. Then, the bundle adjustment method is used to minimize the reprojection error and photometric error to obtain the pose within the ranging range of the RGB-D camera. Outside the ranging range of the RGB-D camera, the wheeled odometry method is used to locate the robot and obtain its pose. Step 2: Based on the pose and depth image of the wheeled robot, allocate the space that the map may occupy in the 3D space, allocate the voxel space in the corresponding space based on the Voxel Hashing algorithm, obtain the voxel space composed of voxel blocks, and construct a hash table. Step 3: Based on the depth image, the distance information is synthesized into each voxel block in the voxel space. Then, based on the wheeled robot's pose and the depth image, the accurate position of the corresponding voxel block of the map is determined, and the distance field information and depth weights are assigned to the corresponding voxel block. Based on the distance field information stored in each voxel block, the voxel block closest to the map surface is found. Using a GPU processing method with a memory management strategy, the distance information is synthesized into each voxel block, the distance field information and depth weights are assigned to the voxel block, and color information is assigned to the voxel block closest to the map surface to form a complete 3D dense map. Step 4: Loop closure detection is performed using the BOW (Browser-Warp) algorithm to update the 3D dense map and the wheeled robot's pose in real time. The BOW algorithm uses visual words to represent the features of each location in the 3D dense map and creates nodes based on the currently processed RGB-D image. All features are represented by a node signature composed of all corresponding visual words. The current node signature is then compared with the existing visual words. The weight of the previous processing node Weights, update The weights are calculated using the following formula: in, for and The number of matching words; and They are respectively and Total word count; This represents the minimum ratio of the number of matched words to the total number of words. like Greater than the preset threshold Then delete entries from the vocabulary of the BOW detection matching algorithm. The newly added visual words, and will Merge to middle, The merged node signature only retains Visual words, The weights are updated to weights and The sum of the weights plus 1, and in addition... Neighbors and loopback redirection to Middle; otherwise, not correct. Perform the operation.

2. The method for constructing a real-time dense map of an unknown environment based on an RGB-D camera according to claim 1, characterized in that, The memory management strategy described in step 3 is as follows: Step 3.1: Calculate the hash value of each voxel block in voxel space using world coordinates. Based on the hash value, find the corresponding hash entry in the hash table, store it in the hash table in memory, and obtain the voxel block pointer stored in the value of the hash entry. According to the voxel block pointer, call the corresponding connected voxel block to the external memory, and set the voxel block pointer to the voxel block. In this way, all voxel blocks in voxel space are stored in external memory. Step 3.2: For voxel blocks that require GPU processing, calculate their hash value based on world coordinates, then use the hash value to find the corresponding voxel block pointed to by the voxel block pointer in external memory, and copy it to memory for computation. When processing voxel blocks using GPU, first copy the hash entry corresponding to the voxel block in the hash table from memory to the intermediate buffer of video memory, and at the same time copy the voxel block from memory to the heap space of video memory. Then, perform GPU computation in video memory. After GPU processing is completed, update the hash entry and voxel block in video memory to memory, and then update them from memory to external memory.

3. The method for constructing a real-time dense map of an unknown environment based on an RGB-D camera according to claim 1, characterized in that, If only a portion of the 3D dense map is constructed during the process of building the 3D dense map, the vocabulary corresponding to the portion of the 3D dense map is stored in external storage, and the references to each visual word in the external storage are deleted, thus completing the storage of the portion of the 3D dense map. Multiple parts of the 3D dense map stored in external memory are extracted, and loop closure detection is performed using the BOW detection matching algorithm to merge the multiple parts of the 3D dense map and obtain a complete 3D dense map.

4. The method for constructing a real-time dense map of an unknown environment based on an RGB-D camera according to claim 1, characterized in that, Threshold in step 4 The value ranges from [0.5, 1].

Images