A depth map based rich information calibration method and device
By acquiring and mapping depth data in the map coordinate system using a depth camera, a height map is generated, which solves the problem of missing rich information in robot composition and achieves efficient and accurate rich information calibration.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU YUNXIANG COMMERCIAL MASCH CO LTD
- Filing Date
- 2023-04-12
- Publication Date
- 2026-06-02
AI Technical Summary
In the current process of robot mapping, rich information about many objects on the ground cannot be added to the map, resulting in a lack of a lot of information in the planning task.
Depth data is acquired using a depth camera, and the effective information is transformed into a map coordinate system through mapping relationships to generate a height map, which is then output in a visual form to label rich map information.
It improves the accuracy and efficiency of rich information calibration, reduces miscalibration and omissions, is low in cost, and is applicable to various types of rich information calibration.
Smart Images
Figure CN116499451B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of map marking, and more particularly to a rich-information marking method and apparatus based on depth maps. Background Technology
[0002] Rich information refers to the rich information in the scene besides map information during the robot's mapping process.
[0003] Currently, most maps are constructed after LiDAR scanning. During the robot mapping process, many scene objects on the ground cannot be added to the map, such as transparent glass walls, some recessed pits, and gently sloping speed bumps, resulting in a lack of a lot of information in the planning task.
[0004] Existing robot navigation map building methods are designed to address the problem of not being able to identify transparent glass walls during map building. For example, a method for building a navigation map for a mobile robot in a complex indoor environment, disclosed in Chinese patent literature (publication number CN113203409A), includes: acquiring the robot's own pose information and environmental information; performing initial SLAM and building an original map; processing lidar intensity data to filter areas where glass is suspected; selecting RGB images based on areas where glass is suspected; performing glass detection in the complex environment based on the selected RGB images; determining the state of the corresponding area grid based on the glass detection results; updating the map and providing a two-dimensional grid map containing indoor glass.
[0005] This solution detects glass in complex environments using only RGB cameras, but it still cannot obtain rich information about the environment, such as pits and gently convex speed bumps, resulting in a lack of information in the planning task. Summary of the Invention
[0006] This invention primarily addresses the problem in existing technologies where rich information about many scene objects on the ground cannot be added to the map during robot mapping, resulting in a lack of information in planning tasks. It provides a rich information calibration method and apparatus based on depth maps, using a depth camera as an aid to extract effective information from the depth camera data and transfer it to the map coordinate system to generate a height map. The height map is then output in a visual form to calibrate the rich information of the map. This helps mobile robots complete the calibration of rich information in complex scenes, in addition to map information, thereby improving the accuracy and efficiency of rich information calibration.
[0007] The above-mentioned technical problems of the present invention are mainly solved by the following technical solutions:
[0008] A depth map-based rich information labeling method includes the following steps:
[0009] S1: Install the depth camera and perform extrinsic parameter calibration on the depth camera;
[0010] S2: Scan areas with rich information, acquire real-time depth camera data, and extract useful information;
[0011] S3: By mapping the depth camera to the vehicle coordinate system and the vehicle's current location value, effective information is converted into the global map;
[0012] S4: Combine all valid information using probabilistic statistics to construct a height map of the scanned scene;
[0013] S5: Perform rich information calibration based on the height map output by visualization.
[0014] This system extracts valid information from depth camera data and converts it to a map coordinate system to generate a height map. The height map is then output in a visual format to calibrate rich information on the map. This helps mobile robots calibrate rich information in complex scenes, in addition to map information, improving the accuracy and efficiency of rich information calibration. It is cost-effective, requiring only one depth camera. The solution is versatile and applicable to various rich information calibration methods. Deployment is rapid, requiring only depth camera installation and calibration. The solution is efficient, enabling fast and accurate rich information calibration. Human-computer interaction reduces miscalibration and omissions.
[0015] Preferably, the mapping relationship between the depth camera and the vehicle coordinate system is determined through the extrinsic parameter calibration of the depth camera; the calibration process is as follows:
[0016] A1: Fix the feature object to the ground plane;
[0017] A2: Obtain the pose relationship between the depth camera and the feature object by using the feature information of the feature object, and obtain the mapping from the depth camera to the feature object;
[0018] A3: The mapping from the depth camera coordinate system to the vehicle coordinate system is completed through hand-eye calibration.
[0019] Deployment is quick, requiring only the installation and calibration of the depth camera.
[0020] As a preferred method, the process for extracting effective information is as follows:
[0021] After obtaining real-time depth camera data, the ground plane determined by the calibration value is used as the reference plane to fit the ground plane of the current data;
[0022] After fitting the ground plane, the ground plane data is removed, and the point clouds of objects above and below the ground are extracted as valid data.
[0023] Applicable to various information-rich calibration methods, the solution is universal.
[0024] Preferably, the global map is a global probabilistic raster map; all valid information is projected onto the probabilistic raster, and by performing probability statistics on each raster, rasteres with probabilities greater than a set threshold are identified as valid rasteres. This filters out dynamic objects, sensor noise, and other points that are inconsistent with the scene during vehicle operation.
[0025] Preferably, the maximum height of the effective grid is projected onto the ground plane to form a two-dimensional height map.
[0026] As a preferred method, the visualization output process of the height map is as follows:
[0027] The height values below ground level in the elevation map are normalized to a certain filled image series and displayed as an image.
[0028] The height values above ground in the elevation map are normalized to another filled image series and displayed as an image.
[0029] Visualize the height map to improve the accuracy and efficiency of rich information calibration.
[0030] Preferably, positive and negative height values are assigned to different fill image series, and different height grids are assigned to different fill images in the fill image series.
[0031] As a preferred option, the rich information calibration process is as follows:
[0032] Select the rich information type to be calibrated, and automatically filter out the raster that matches the characteristics for highlighting;
[0033] Identify the highlighted areas and mark the locations of the rich information;
[0034] Generate a rich information file based on the selected area.
[0035] Human-computer interaction reduces the occurrence of mislabeling and omissions.
[0036] A depth map-based rich information calibration device includes:
[0037] Depth camera, acquires depth camera data of the probe area in the scanned scene;
[0038] The data transformation and fusion module extracts effective information from the depth camera data, transforms the effective information into the global map through the mapping relationship between the depth camera and the vehicle coordinate system and the current positioning value of the vehicle; and fuses all effective information in a probabilistic statistical manner to finally generate a height map of the scanned scene.
[0039] The visual interactive platform outputs a height map, displays the corresponding area of the height map based on the selected rich information type, and generates a calibration file by selecting and calibrating the area.
[0040] The depth camera is mounted on the top of the vehicle body, with its lens pointing downwards. The angle between the depth camera lens and the horizontal plane is 30° to 60°. Due to the limited field of view of the depth camera, it is mounted on the vehicle body with its viewing angle downwards to ensure that it can see low-lying objects.
[0041] The beneficial effects of this invention are:
[0042] 1. Extract effective information from depth camera data and transfer it to the map coordinate system to generate a height map. Output the height map in a visual form to calibrate rich information on the map. This helps mobile robots complete the calibration of rich information other than map information in complex scenes, thereby improving the accuracy and efficiency of rich information calibration.
[0043] 2. Low cost, requiring only one depth camera.
[0044] 3. The solution is universal and applicable to various types of rich information calibration.
[0045] 4. Rapid deployment, requiring only depth camera installation and calibration.
[0046] 5. The solution is highly efficient and can quickly and accurately complete rich information calibration.
[0047] 6. Human-computer interaction reduces the occurrence of mislabeling and omissions. Attached Figure Description
[0048] Figure 1 This is a schematic diagram of the structure of the information-rich calibration device of the present invention.
[0049] Figure 2 This is a flowchart of the rich information calibration method of the present invention.
[0050] Figure 3 This is a schematic diagram of the height map visualization of the present invention.
[0051] In the image: 1. Vehicle body; 2. Depth camera. Detailed Implementation
[0052] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings.
[0053] Example:
[0054] This embodiment provides a depth map-based rich information calibration device, such as... Figure 1 As shown, it includes vehicle body 1 and depth camera 2.
[0055] Depth camera 2 acquires depth camera data of the detection area in the scanned scene. Depth camera 2 is mounted on the top of vehicle body 1, with its lens pointing downwards, and the angle between the lens and the horizontal plane is 30° to 60°. In this embodiment, the angle between depth camera 2 and the horizontal plane is 45°.
[0056] Due to the limited field of view of depth camera 2, in order to ensure that depth camera 2 can see low-lying objects, depth camera 2 is mounted on vehicle body 1 with its viewing angle facing downwards.
[0057] This embodiment of a depth map-based rich information calibration device also includes a data conversion and fusion module and a visual interactive platform.
[0058] The data transformation and fusion module extracts effective information from the depth camera data, transforms the effective information into the global map through the mapping relationship between the depth camera and the vehicle coordinate system and the current positioning value of the vehicle; and fuses all effective information in a probabilistic statistical manner to finally generate a height map of the scanned scene.
[0059] The visual interactive platform outputs a height map, displays the corresponding area of the height map based on the selected rich information type, and generates a calibration file by selecting and calibrating the area.
[0060] A rich information labeling method based on depth maps, such as Figure 2 As shown, it includes the following steps:
[0061] S1: Install a depth camera and perform extrinsic parameter calibration on the depth camera.
[0062] The mapping relationship between the depth camera and the vehicle coordinate system is determined through extrinsic parameter calibration of the depth camera. The calibration process is as follows:
[0063] A1: Fix the feature to the ground plane (it cannot be moved in subsequent steps).
[0064] A2: Camera-to-feature mapping;
[0065] The vehicle observes the feature from two known positioning points and obtains the pose relationship between the depth camera and the feature by using the feature's feature information.
[0066] A3: Mapping from the depth camera coordinate system to the robot coordinate system;
[0067] Given the relative poses of the vehicle at two observation points, the mapping from the camera to the feature is completed by hand-eye calibration, transforming the depth camera coordinate system into the robot coordinate system.
[0068] Deployment is quick, requiring only the installation and calibration of the depth camera.
[0069] S2: Scan areas with rich information, acquire real-time depth camera data, and extract useful information.
[0070] The process of extracting effective information is as follows:
[0071] B1: After obtaining real-time depth camera data, the ground plane determined by the calibration value is used as the reference plane to fit the ground plane of the current data, thereby obtaining point cloud data after removing the ground plane.
[0072] B2: After fitting the ground plane, remove the ground plane data and extract the point clouds of objects above and below the ground as valid data.
[0073] Applicable to various information-rich calibration methods, the solution is universal.
[0074] S3: By mapping the depth camera to the vehicle coordinate system and the vehicle's current location value, effective information is transformed into the global map.
[0075] In this embodiment, the global map is a global probabilistic raster map.
[0076] First, a global grid map is constructed, projecting all valid information from the robot's operation onto the probabilistic grid. By performing probability statistics on each grid, points that do not conform to the scene, such as dynamic objects and sensor noise during vehicle operation, are filtered out. Grids with probabilities greater than a certain threshold are called valid grids.
[0077] S4: Combine all valid information using probabilistic statistics to construct a height map of the scanned scene.
[0078] Specifically, the maximum height of the effective grid cells is projected onto the ground plane to form a two-dimensional height map. For example... Figure 3 As shown, on the probabilistic grid map constructed in step S3, the maximum height of the effective grid in the vertical direction is projected onto the ground plane to form a two-dimensional height map.
[0079] S5: Perform rich information calibration based on the height map output by visualization.
[0080] The visualization output process of the height map is as follows:
[0081] C1: Normalizes the height values below ground level in the height map to a certain filled image series for display.
[0082] C2: Normalize the height values above ground in the height map to another filled image series for display.
[0083] Visualize the height map to improve the accuracy and efficiency of rich information calibration.
[0084] Positive and negative height values are assigned to different fill image series, and different height gratings are assigned to different fill images in the fill image series.
[0085] In this embodiment, the fill image series uses a color space, and the height map is output as an image visualization. First, the height values below the ground in the height map are normalized to one color space for image display, and the height values above the ground in the height map are normalized to another color space for image display. As shown in the figure, different height grids will be assigned different colors, and positive and negative height values will be assigned to different color spaces.
[0086] The rich information calibration process is as follows:
[0087] D1: Select the rich information type to be calibrated, and automatically filter out the raster cells that match the characteristics for highlighting.
[0088] D2: Identify the highlighted areas and mark the locations of rich information.
[0089] D3: Generate a rich information file based on the selected area.
[0090] Human-computer interaction reduces the occurrence of mislabeling and omissions.
[0091] This embodiment extracts valid information from depth camera data and transfers it to a map coordinate system to generate a height map. The height map is then output in a visual format to calibrate rich information on the map. This helps mobile robots calibrate rich information in complex scenes, in addition to map information, thereby improving the accuracy and efficiency of rich information calibration. It is low-cost, requiring only one depth camera; the solution is versatile and applicable to various types of rich information calibration; and it can quickly and accurately complete rich information calibration, making it highly efficient.
[0092] It should be understood that the embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.
Claims
1. A depth map based rich information calibration method, characterized in that, Includes the following steps: S1: Install a depth camera, perform extrinsic parameter calibration of the depth camera based on features, and determine the mapping relationship between the depth camera coordinate system and the vehicle coordinate system; S2: Scan areas with rich information, acquire real-time depth camera data, and extract point clouds of objects above and below the ground as valid information; S3: By mapping the depth camera coordinate system to the vehicle coordinate system and the vehicle's current positioning value, the effective information is transformed into the global map, which is a global probabilistic grid map. S4: Integrate all valid information in a probabilistic statistical manner, project all valid information onto the probability grid of the global map, perform probability statistics on each grid, determine the grid with a probability greater than a set threshold as a valid grid, project the maximum height of the valid grid onto the ground plane to form a two-dimensional height map, and construct the height map of the scanned scene. S5: Perform rich information calibration based on the height map output by visualization.
2. The method of claim 1, wherein, The mapping relationship between the depth camera and the vehicle coordinate system is determined by extrinsic parameter calibration of the depth camera; the calibration process is as follows: A1: Fix the feature object to the ground plane; A2: Obtain the pose relationship between the depth camera and the feature object by using the feature information of the feature object, and obtain the mapping from the depth camera to the feature object; A3: The mapping from the depth camera coordinate system to the vehicle coordinate system is completed through hand-eye calibration.
3. The method of claim 1, wherein, The process of extracting effective information is as follows: After obtaining real-time depth camera data, the ground plane determined by the calibration value is used as the reference plane to fit the ground plane of the current data; After fitting the ground plane, the ground plane data is removed, and the point clouds of objects above and below the ground are extracted as valid data.
4. The method of claim 1, wherein, The visualization output process of the height map is as follows: The height values below ground level in the elevation map are normalized to a certain filled image series and displayed as an image. The height values above ground in the elevation map are normalized to another filled image series and displayed as an image.
5. The method of claim 4, wherein, Positive and negative height values are assigned to different fill image series, and different height gratings are assigned to different fill images in the fill image series.
6. The method of claim 1 or 5, wherein, The rich information calibration process is as follows: Select the rich information type to be calibrated, and automatically filter out the raster that matches the characteristics for highlighting; Identify the highlighted areas and mark the locations of the rich information; Generate a rich information file based on the selected area.
7. A depth map based rich information calibration device, which adopts a depth map based rich information calibration method according to any one of claims 1-6, characterized in that, include: Depth camera, acquires depth camera data of the probe area in the scanned scene; The data transformation and fusion module extracts effective information from the depth camera data and transforms it into the global map through the mapping relationship between the depth camera and the vehicle coordinate system and the current positioning value of the vehicle. All valid information is fused using a probabilistic statistical approach to ultimately generate a height map of the scanned scene. The visual interactive platform outputs a height map, displays the corresponding area of the height map based on the selected rich information type, and generates a calibration file by selecting and calibrating the area.
8. The apparatus according to claim 7, wherein, The depth camera is mounted on the top of the vehicle body, with the camera lens pointing downwards and the angle between the camera lens and the horizontal plane being 30° to 60°.