A lightweight UAV map construction method based on facets
Through the drone map construction method based on the surface element, the problem of large memory occupancy and poor consistency of the drone map construction algorithm is solved, and efficient and accurate map construction is achieved, which is suitable for drones with limited resources.
Patent Information
- Application Number
- CN202211245143.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-10-12
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-10-12
AI Technical Summary
The existing drone map construction algorithm occupies a lot of memory and has poor consistency in establishing global maps, especially in the absence of GPS scenarios, which affects the mission's flight efficiency and accuracy.
Using a lightweight drone map construction method based on surface elements, the depth image and RGB image are obtained through sensors, superpixel extraction and normal vector calculation are performed, the camera position is estimated using SLAM method, the surface element is extracted and searched in the position image. Locally consistent keyframes, integrating local and global metas to ensure map consistency.
It reduces the computing resource requirements of drone maps, improves map construction accuracy and system efficiency, and reduces cumulative errors. It is suitable for drones with limited computing and storage resources.
Smart Images

Figure CN115507842B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of unmanned aerial vehicles and map construction, and in particular to a facet-based lightweight unmanned aerial vehicle map construction method. Background Art
[0002] When a drone is performing a mission, it needs to build a map of the surrounding environment and plan and generate an executable trajectory. Currently, online maintenance of drone maps requires the onboard computer to have strong processing capabilities. In addition, in scenarios without GPS, such as indoors, drone positioning is poor. In the presence of cumulative errors, it is also difficult to maintain drone maps. The existing methods occupy a large amount of memory, and the global map established is Figure 1 The consistency is poor and there may be large cumulative errors. In summary, the existing map construction algorithms have certain limitations, which affect the efficiency and accuracy of UAV mission flight. Summary of the invention
[0003] In order to solve the above problems, the present invention provides a lightweight UAV map construction method based on facets, which introduces facets to construct maps, reduces the space occupied by maps, improves system efficiency, sets a consistency principle, and improves map construction accuracy.
[0004] In order to achieve the above object, the technical solution adopted by the present invention is:
[0005] A lightweight UAV map construction method based on facets includes the following steps:
[0006] Step 1: obtain a depth image and an RGB image through a sensor, extract superpixels from the depth image, and calculate the normal vector of the superpixel;
[0007] Step 2: Use SLAM method to estimate the camera pose;
[0008] Step 3, using superpixel information and keyframe information to perform facet extraction;
[0009] Step 4: Search for local consistent keyframes in the pose graph, extract local maps, and fuse local facets;
[0010] Step 5: Create a global map, fuse facets, and complete map construction.
[0011] Furthermore, in step 1, superpixel extraction is performed on the depth image, and the specific method of calculating the normal vector of the superpixel is:
[0012] Step 1.1, define the input i-th depth image as D i , using clustering method to divide the image into regular grids;
[0013] Step 1.2: Initialize the cluster center C on each grid. i =[x i ,y i ,d i ,r i ] T , where [x i ,y i ] T is the average position of clustered pixels, defined as the cluster center position; d i is the average depth value, defined as the pixel [x i ,y i ] T The depth value of r i Defined as the radius of the superpixel; for cluster centers initialized at pixels without valid depth estimates, d i Initialized to empty;
[0014] Cluster center [x i ,y i ] T and the average depth d i By minimizing the regression loss function E with radius δ d To update:
[0015]
[0016] Among them, u is the set of pixels within the radius, L δ is the distance from the pixel to the cluster center, u d is the pixel depth, d i is the average depth value of cluster centers;
[0017] Step 1.3, through the camera imaging function u = π (p) to convert the point u in the pixel coordinate system: = [u, v] T Convert to point p=[x,y,z] in the camera coordinate system T ;
[0018] Step 1.4, calculate the normal vector of the superpixel based on the three-dimensional information of the points within the superpixel range.
[0019] Furthermore, the specific method of step 2 is:
[0020] Use SLAM method to estimate the camera pose T i , get the same as the i-th RGB image I i The corresponding rotation matrix R of key frame F i and the translation vector t i .
[0021] Furthermore, the specific method of step 3 is:
[0022] Define a single facet: S = [S p ,S n ,S r ,S t ,S i ] T ,
[0023] Among them, the location is the 3D coordinate corresponding to the center of the superpixel in the corresponding depth image; normal is the normal vector of the corresponding superpixel; the radius is the 3D radius of the superpixel; update time Latest marker time for temporary detection of outliers or dynamic objects; additional keyframe index The last keyframe corresponding to the surface element.
[0024] Furthermore, the specific method of step 4 is:
[0025] When a feature exists in two key frames at the same time, it is defined as a shared feature of the key frames, and the number of shared features of the two key frames is recorded as g;
[0026] When a feature exists in two facets at the same time, it is defined as the shared feature of the facets, and the number of shared features of the two facets is recorded as h;
[0027] If the number of keyframe shared features between the current keyframe and a certain historical keyframe is greater than the threshold G δ , then these keyframes are locally consistent;
[0028] Find the locally consistent keyframes by performing a breadth-first search on the historical keyframes in the pose graph;
[0029] When a locally consistent keyframe is detected, if the number of shared features between the facets is greater than the threshold H δ , the previous face element will continue to be used, thereby reducing the growth of the map and updating the key frame information of the face element;
[0030] Extracting the surfels containing the local consistent keyframes as a local map;
[0031] If the face element S n With the surface element S l There is a similar correspondence between depth and normal, namely:
[0032]
[0033] and
[0034]
[0035] Then the surface element S nWith local surface element S l The fusion is performed in the same way as the method of updating the mesh in step 1.2, that is, the face element is used as the mesh and updated in the manner of step 1.2; the key frame information of the face element is updated;
[0036] in, and They are respectively n With S l The depth value of the keyframe, b is the baseline distance between keyframes, f is the camera focal length, σ is the surface element radius, and They are respectively n With S l The normal vector of .
[0037] Furthermore, the specific method of step 5 is:
[0038] After detecting loops and optimizing the pose graph, the locally consistent keyframes between loops are considered to be globally consistent;
[0039] Use T for all facets i Convert to the global frame, move to the global map, and create an index from the center pixel coordinates of the face to the global coordinates;
[0040] If the initial surface element S n With the surface element S l There is a similar correspondence between depth and normal, namely:
[0041]
[0042] and
[0043]
[0044] Then the surface element S n With local surface element S l The fusion is performed in the same way as the method of updating the mesh in step 1.2, that is, the face element is used as the mesh and updated in the manner of step 1.2; the key frame information of the face element is updated;
[0045] in, and They are respectively n With S l The depth value of the keyframe, b is the baseline distance between keyframes, f is the camera focal length, σ is the surface element radius, and They are respectively n With S l The normal vector of .
[0046] Compared with the existing method, the beneficial effects of the present invention are:
[0047] 1. The present invention reduces the computing resources of drone maps and can be deployed on drones with limited computing and storage resources, with the characteristics of high efficiency.
[0048] 2. The present invention sets a consistency criterion to ensure the accuracy of facet fusion and reduce the cumulative error of the facet map. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.
[0050] Figure 1 It is a flow chart of the present invention. Specific implementation methods
[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0052] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.
[0053] A lightweight UAV map construction method based on surface element, such as Figure 1 As shown, including:
[0054] S1: extract superpixels from the depth image obtained by the sensor and calculate the normal vector of the superpixel;
[0055] S2: Estimate the camera pose using SLAM method;
[0056] S3: Use superpixel information and keyframe information to perform surface element extraction;
[0057] S4: Search for local consistent keyframes in the pose graph, extract local maps, and fuse local facets;
[0058] S5: Build a global map and fuse facets.
[0059] In step S1, superpixel extraction is performed on the depth image obtained by the sensor, and the normal vector of the superpixel is calculated, including:
[0060] S11: Define the i-th input depth image as D i ,For the depth image, a clustering algorithm is used to divide the image into regular grids;
[0061] S12: Initialize the cluster center C on each network i =[x i ,y i ,d i ,r i ] T , where [x i ,y i ] T is the average position of clustered pixels, defined as the cluster center position; d i is the average depth value, defined as the pixel [x i ,y i ] T The depth value of r i Defined as the radius of the superpixel. For cluster centers initialized at pixels without valid depth estimates, d i Initialized to NaN;
[0062] S13: Cluster center [x i ,y i ] T and the average depth d i Update by minimizing the regression loss function with radius δ:
[0063]
[0064] Among them, u is the set of pixels within the radius, L δ is the distance from the pixel to the cluster center, u d is the pixel depth, d i is the average depth value of cluster centers;
[0065] S14: The point u in the pixel coordinate system is converted to [u, v] by the camera imaging function u = π (p). T Convert to point p=[x,y,z] in the camera coordinate system T ;
[0066] S15: Calculate the normal vector of the superpixel according to the three-dimensional information of the points within the superpixel range.
[0067] For step S2, the camera pose is estimated using the SLAM method, including:
[0068] S21: Estimate the camera pose T using SLAM method i , get the same as the i-th RGB image I i The corresponding rotation matrix R of key frame F i and the translation vector t i .
[0069] In step S3, using superpixel information and keyframe information, facet extraction is performed, including:
[0070] Define a single facet: S = [S p ,S n ,S r ,S t ,S i ] T ,
[0071] Where: Location is the 3D coordinate corresponding to the center of the superpixel in the corresponding depth image; normal is the normal vector of the corresponding superpixel; the radius is the 3D radius of the superpixel; update time Latest marker time for temporary detection of outliers or dynamic objects; additional keyframe index The last keyframe corresponding to the surface element.
[0072] For step S4, searching for local consistent keyframes in the pose graph, extracting local maps, and fusing local facets include:
[0073] S41: When a feature exists in two key frames at the same time, it is defined as a key frame shared feature, and the number of shared features of the two key frames is recorded as g;
[0074] S42: When a feature exists in two facets at the same time, it is defined as a facet shared feature, and the number of shared features of the two facets is recorded as h;
[0075] S43: If the number of common features shared between the current key frame and a certain historical key frame is greater than G δ , then these keyframes are locally consistent;
[0076] S44: Find the locally consistent keyframes by performing a breadth-first search on the historical keyframes in the pose graph;
[0077] S45: When a locally consistent keyframe is detected, if the number of shared common features between the facets is greater than H δ , you can continue to use the previous face element, thereby reducing the growth of the map and updating the key frame information of the face element;
[0078] S46: extracting the face elements including the key frames as a local map;
[0079] S47: If the face element S n With the surface element S l There is a similar correspondence between depth and normal: and Then the surface element S n With local surface element S l The fusion is performed in the same way as the grid update method, and the key frame information of the face element is updated.
[0080] in, and They are respectively n With S l The depth value of the keyframe, b is the baseline distance between keyframes, f is the camera focal length, σ is the surface element radius, and They are respectively n With S l The normal vector of .
[0081] For step S5, a global map is established and global facets are fused, including:
[0082] S51: After detecting the loop and optimizing the pose graph, the locally consistent keyframes between the loops are considered to be globally consistent;
[0083] S52: Use T for all facets i Convert to the global frame, move to the global map, and create an index from the center pixel coordinates of the face to the global coordinates;
[0084] S53: If the initial surface element S n With the surface element S l There is a similar correspondence between depth and normal: and Then the surface element S n With local surface element S l The fusion is performed in the same way as the grid update method, and the key frame information of the face element is updated.
[0085] in, and They are respectively n With S l The depth value of the keyframe, b is the baseline distance between keyframes, f is the camera focal length, σ is the surface element radius, and They are respectively n With S l The normal vector of .
[0086] In summary, the present invention reduces the computing resources of UAV maps and can be deployed on UAVs with limited computing and storage resources; it sets a consistency criterion to ensure the accuracy of facet fusion and reduce the cumulative error of facet maps.
Claims
1. A lightweight UAV map construction method based on facets, characterized in that: The following steps are involved: Step 1: obtain a depth image and an RGB image through a sensor, extract superpixels from the depth image, and calculate the normal vector of the superpixel; wherein the specific method of extracting superpixels from the depth image and calculating the normal vector of the superpixel is: Step 1.1, define the input i-th depth image as D i , using clustering method to divide the image into regular grids; Step 1.2: Initialize the cluster center C on each grid. i =[x i ,y i ,d i ,r i ] T , where [x i ,y i ] T is the average position of clustered pixels, defined as the cluster center position; d i is the average depth value, defined as the pixel [x i ,y i ] T The depth value of r i Defined as the radius of the superpixel; for cluster centers initialized at pixels without valid depth estimates, d i Initialized to empty; Cluster center [x i ,y i ] T and the average depth d i By minimizing the regression loss function E with radius δ d To update: Among them, u is the set of pixels within the radius, L δ is the distance from the pixel to the cluster center, u d is the pixel depth, d i is the average depth value of cluster centers; Step 1.3, through the camera imaging function u = π (p) to convert the point u in the pixel coordinate system: = [u, v] T Convert to point p=[x,y,z] in the camera coordinate system T ; Step 1.4, calculating the normal vector of the superpixel based on the three-dimensional information of the points within the superpixel range; Step 2: Use the SLAM method to estimate the camera pose; the specific method is: Use SLAM method to estimate the camera pose T i , get the same as the i-th RGB image I i The corresponding rotation matrix R of key frame F i and the translation vector t i ; Step 3: Use superpixel information and keyframe information to extract surface elements; the specific method is as follows: Define a single facet: S = [S p ,S n ,S r ,S t ,S i ] T , Among them, the location is the 3D coordinate corresponding to the center of the superpixel in the corresponding depth image; normal is the normal vector of the corresponding superpixel; the radius is the 3D radius of the superpixel; update time Latest marker time for temporary detection of outliers or dynamic objects; additional keyframe index The last key frame corresponding to the surface element; Step 4: Search for local consistent keyframes in the pose graph, extract the local map, and fuse the local facets. The specific method is as follows: When a feature exists in two key frames at the same time, it is defined as a shared feature of the key frames, and the number of shared features of the two key frames is recorded as g; When a feature exists in two facets at the same time, it is defined as the shared feature of the facets, and the number of shared features of the two facets is recorded as h; If the number of keyframe shared features between the current keyframe and a certain historical keyframe is greater than the threshold G δ , then these keyframes are locally consistent; Find the locally consistent keyframes by performing a breadth-first search on the historical keyframes in the pose graph; When a locally consistent keyframe is detected, if the number of shared features between the facets is greater than the threshold H δ , the previous face element will continue to be used, thereby reducing the growth of the map and updating the key frame information of the face element; Extracting the surfels containing the local consistent keyframes as a local map; If the face element S n With the surface element S l There is a similar correspondence between depth and normal, namely: and Then the surface element S n With local surface element S l The fusion is performed in the same way as the method of updating the mesh in step 1.2, that is, the face element is used as the mesh and updated in the manner of step 1.2; the key frame information of the face element is updated; in, and They are respectively n With S l The depth value of the keyframe, b is the baseline distance between keyframes, f is the camera focal length, σ is the surface element radius, and They are respectively n With S l The normal vector of Step 5: Establish a global map, fuse the facets, and complete the map construction; the specific method is: After detecting loops and optimizing the pose graph, the locally consistent keyframes between loops are considered to be globally consistent; Use T for all facets i Convert to the global frame, move to the global map, and create an index from the center pixel coordinates of the face to the global coordinates; If the initial surface element S n With the surface element S l There is a similar correspondence between depth and normal, namely: and Then the surface element S n With local surface element S l The fusion is performed in the same way as the method of updating the mesh in step 1.2, that is, the face element is used as the mesh and updated in the manner of step 1.2; the key frame information of the face element is updated; in, and They are respectively n With S l The depth value of the keyframe, b is the baseline distance between keyframes, f is the camera focal length, σ is the surface element radius, and They are respectively n With S l The normal vector of .
Citation Information
Patent Citations
Map merging method of unmanned aerial vehicle visual SLAM under city complex environment
CN106595659A
A multi-UAV (Unmanned Aerial Vehicle) collaborative map construction method oriented to data sharing
CN109559277A
Unmanned aerial vehicle scene dense reconstruction method based on VI-SLAM and depth estimation network
CN112435325A
Three-dimensional dense surface element mapping method and system based on SLAM (Simultaneous Localization and Mapping) and electronic equipment
CN114445549A
Visual slam method applicable to indoor dynamic environment
WO2022041596A1
