A method for detecting low obstacles

Abstract

This invention relates to the field of environmental perception technology for mobile robots, and particularly to a method for detecting low-lying obstacles. The method includes: acquiring LiDAR data and depth camera data; performing instance segmentation on RGB images using a trained YOLOv5 model to obtain ground segmentation maps and obstacle segmentation maps; further obtaining ground mask maps and obstacle mask maps, and combining them with the depth map to obtain panoramic point cloud and obstacle point cloud data with the ground removed; converting these into panoramic pseudo-radar data and obstacle pseudo-radar data with the ground removed using a point cloud to pseudo-radar data conversion algorithm; fusing this data with LiDAR data; and performing obstacle detection based on the fused data to obtain the detection result. This invention achieves low-lying obstacle detection by visually analyzing RGB images to remove ground point cloud data, thus accurately removing ground point cloud data and accurately identifying low-lying obstacles, resulting in more accurate environmental perception.

Description

Technical Field

[0001] This invention relates to the field of environmental perception technology for mobile robots, and in particular to a method for detecting low-lying obstacles. Background Technology

[0002] In recent years, with the rapid development of technologies such as artificial intelligence and machine learning, mobile robot research has entered a new stage, giving rise to various new technologies and algorithms. However, robotics still faces a series of problems when dealing with challenges in complex environments and specific scenarios. Therefore, improving the perception and decision-making systems of robots, especially their autonomous navigation capabilities in complex environments, is of great significance for meeting people's growing needs.

[0003] Traditional SLAM (Simultaneous Localization and Mapping) has limitations, failing to handle special scenarios such as obstacles below the radar scanning plane. Low-lying obstacle detection is a crucial aspect of environmental perception for mobile robots, significantly impacting their stable operation. In environmental perception, depth camera point clouds are typically categorized into two types: obstacle points and ground points. After target detection, a large portion of the point cloud data consists of ground points, exhibiting a textured appearance. This affects subsequent target point cloud processing, necessitating the removal of ground points from the target point cloud. Due to the massive volume of 3D target point cloud data, researching robust and accurate real-time ground segmentation algorithms is essential for ensuring efficient segmentation.

[0004] Existing ground segmentation techniques mainly fall into categories such as geometric feature methods, grid map methods, and deep learning methods. RANSAC (Random Sample Consensus) plane fitting continuously updates and iteratively optimizes the ground model based on the distances from other points to the initial seed plane; SVD (Singular Value Decomposition) plane fitting extracts ground point cloud features based on normal vectors; LeGo-LOAM extracts ground features using the angle between beams below the horizontal laser beam and the curvature of each point; SLOPE segmentation uses the slope between adjacent beams and the laser point relative to the reference point to improve the robustness and real-time performance of ground segmentation; scanline segmentation achieves efficient and accurate ground segmentation for various environments, including flat and undulating terrains, through vertical and horizontal bidirectional segmentation of the point cloud; furthermore, Euclidean clustering and multi-plane segmentation can also achieve efficient ground segmentation for non-flat terrains. In summary, existing ground segmentation methods generally cannot simultaneously address issues of robustness, real-time performance, and segmentation accuracy. Meanwhile, traditional point cloud segmentation algorithms tend to identify noise as ground and remove some unnecessary point clouds, which makes it impossible for low obstacle detection algorithms to accurately remove ground point cloud data, resulting in errors in perception. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention uses the YOLOv5 algorithm to perform 2D instance segmentation on RGB images. Then, a transformation matrix is ​​used to fuse the RGB images with the depth map to calculate depth camera point cloud data after removing ground point cloud data. This significantly improves the accuracy of ground point cloud segmentation and avoids directly processing point cloud data, reducing computational load and increasing processing speed.

[0006] To achieve the above objectives, the present invention provides a method for detecting low-lying obstacles, comprising:

[0007] (1) Acquire lidar data and depth camera data, wherein the depth camera data includes RGB images and depth maps;

[0008] (2) For the RGB image, use the trained YOLOv5 model to perform instance segmentation to obtain the ground segmentation map and the obstacle segmentation map;

[0009] (3) Based on the RGB image, a ground mask image and an obstacle mask image are obtained by using the ground segmentation image and the obstacle segmentation image;

[0010] (4) Combine the culled ground mask map and obstacle mask map with the depth map to obtain the culled ground panoramic view cloud and obstacle point cloud;

[0011] (5) The point cloud of the panoramic view and the point cloud of obstacles with the ground removed are converted into panoramic pseudo radar data and obstacle pseudo radar data with the ground removed by the point cloud to pseudo radar data algorithm.

[0012] (6) The panoramic pseudo-radar data and obstacle pseudo-radar data that have had their ground removed are fused with the lidar data, and obstacle detection is performed based on the fused data to obtain the detection results.

[0013] Furthermore, the YOLOv5 model is the YOLOv5s model, and the training process utilizes the transfer learning steps of YOLOv5s.

[0014] Furthermore, step (3) specifically involves:

[0015] (3.1) Generate a binary mask image of the same size as the RGB image;

[0016] (3.2) Set the region corresponding to the ground segmentation map in the Mask image to 0 and set the other regions to 1 to obtain the ground removal mask image;

[0017] (3.3) Set the region corresponding to the obstacle segmentation map in the Mask image to 1 and set the other regions to 0 to obtain the obstacle mask map.

[0018] Furthermore, step (4) specifically involves:

[0019] (4.1) Multiply the depth map and the culled ground mask map pixel by pixel to obtain the depth map of the culled ground, and use the PCL library and depth2cloud package to convert it into point cloud data to obtain the panoramic point cloud of the culled ground;

[0020] (4.2) Multiply the depth map and the obstacle mask map pixel by pixel to obtain the depth map of the obstacle, and use the PCL library and depth2cloud package to convert it into point cloud data to obtain the obstacle point cloud.

[0021] Furthermore, the point cloud to pseudo radar data conversion algorithm is specifically as follows:

[0022] (5.1) Map any point M(u, v) in the input point cloud image coordinate system to its coordinates in the world coordinate system using the following formula: M(X) w ,Y w Z w );

[0023]

[0024] Where: u and v are the coordinates of any point in the input point cloud image coordinate system; u0 and v0 are the coordinates of the center point of the input point cloud image; X w Y w Z w Z represents the three-dimensional coordinates of point M in the world coordinate system. c This indicates the distance between the target and the vertical plane where the camera is located;

[0025] (5.2) Define an array laser[n] to represent pseudo radar data. The index n of the point M projected onto the laser array is calculated by the formula:

[0026] n = N(θ - α) / (β - α)

[0027] Where: [α,β] represents the maximum and minimum range of the virtual laser beam; N represents the number of laser points converted by the virtual laser beam within the maximum and minimum range; θ represents the included angle AOC calculated from the coordinates of point M;

[0028] (5.3) Calculate the distance r between the projection point C of point M on the x-axis and the camera origin O; obtain the distance set of n laser points, and form the pseudo radar data array laser[n];

[0029]

[0030] Where: x represents the x-coordinate of point M in the world coordinate system, and z represents the y-coordinate of point M in the world coordinate system;

[0031] (5.4) Perform the operations on all pixels in the input point cloud image according to steps (5.1) to (5.3) to obtain multiple layers of pseudo laser points. Then, select the effective value with the smallest distance in each column and save it into the pseudo radar data array to complete the conversion from point cloud to pseudo radar data.

[0032] Furthermore, the specific steps of fusing the ground-removed panoramic pseudo-radar data and obstacle pseudo-radar data with the lidar data are as follows:

[0033] When the i-th pseudo radar data of the obstacle satisfies laser_obj[i] = 0:

[0034] laser_filter[i]=laser_scan[i]

[0035] Otherwise, calculate the difference between the i-th panoramic pseudo-radar data after removing the ground and the i-th lidar data;

[0036] diff=|laser_panorama[i]-laser_scan[i]|

[0037] The difference is compared with the threshold to obtain the i-th fused data:

[0038]

[0039] By iterating through all the points as described above, the fused data is obtained.

[0040] Where: i = 1, 2, 3...n; the array laser_obj represents the obstacle pseudo radar data, the array laser_panorama represents the panoramic pseudo radar data after removing the ground, the array laser_scan represents the lidar data, and the array laser_filter represents the fused data; threshold is the threshold value; and diff is the difference value.

[0041] The present invention also provides a low-profile obstacle detection device, comprising:

[0042] The data acquisition module is used to acquire LiDAR data and depth camera data, wherein the depth camera data includes RGB images and depth maps;

[0043] The instance segmentation module is used to perform instance segmentation on the RGB image using a trained YOLOv5 model to obtain a ground segmentation map and an obstacle segmentation map.

[0044] The mask generation module is used to obtain a ground mask image and an obstacle mask image by using the ground segmentation image and the obstacle segmentation image based on the RGB image;

[0045] The point cloud generation module is used to combine the culled ground mask map and the obstacle mask map with the depth map to obtain the panoramic point cloud and obstacle point cloud with the culled ground.

[0046] The pseudo radar data generation module is used to convert the panoramic view cloud and obstacle point cloud, which have had their ground removed, into panoramic pseudo radar data and obstacle pseudo radar data, respectively, through a point cloud to pseudo radar data conversion algorithm.

[0047] The detection module is used to fuse the panoramic pseudo-radar data and obstacle pseudo-radar data (excluding ground) with the lidar data, and to perform obstacle detection based on the fused data to obtain the detection result.

[0048] The present invention also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the low obstacle detection method as described above.

[0049] The present invention also provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the low obstacle detection method as described above.

[0050] The beneficial effects of this invention are:

[0051] 1. This invention implements a low-profile obstacle detection algorithm by visually analyzing RGB images to remove ground point cloud data. This algorithm can accurately remove ground point cloud data and accurately identify low-profile obstacles, thus achieving more accurate environmental perception.

[0052] 2. By converting obstacle point clouds and ground-level indoor environment point clouds into pseudo-radar data, perception of low-lying and hollowed-out obstacles can be achieved.

[0053] 3. By using a multi-sensor fusion algorithm, radar data and depth camera pseudo-radar data are spatially filtered and fused to improve the efficiency of data computation. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the low obstacle detection method according to an embodiment of the present invention.

[0055] Figure 2 This is a comparison chart of the mapping effect of the method in this embodiment of the invention and the mapping effect of a single-line lidar. Detailed Implementation

[0056] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0057] like Figure 1 As shown, the present invention provides a method for detecting low-lying obstacles, comprising the following steps:

[0058] S101. Acquire lidar data and depth camera data, wherein the depth camera data includes RGB images and depth maps;

[0059] This invention employs both lidar data and depth camera data (RGB-D) to detect low-lying obstacles. By fusing these two types of data, more efficient and accurate environmental information can be obtained, resulting in more accurate detection results.

[0060] S102. For the RGB image, use the trained YOLOv5 model to perform instance segmentation to obtain a ground segmentation map and an obstacle segmentation map;

[0061] Given that the YOLO series algorithms have the advantages of fast detection speed and high detection accuracy, this embodiment of the invention selects the YOLOv5s model for instance segmentation.

[0062] The training process utilizes the transfer learning steps of YOLOv5s. First, a new dataset was prepared, specifically targeting indoor scene floors and common obstacle types. Self-made training and test sets were created, containing segmented images of over ten common indoor obstacles such as tables, chairs, doors, and boxes, as well as indoor floors. The format was ensured to conform to YOLOv5s requirements. Then, the pre-trained weights file yolov5s-seg.pt was downloaded, and the network structure, number of classes, and dataset location information were modified in the configuration file. During transfer learning, setting the `freeze` parameter to 10 freezes the backbone of the pre-trained model, training only the subsequent classifier part to better adapt to the instance segmentation task in this paper. Through training the YOLOv5s model, a high-performing instance segmentation model was finally obtained.

[0063] S103. Based on the RGB image, obtain a ground mask image and an obstacle mask image by using the ground segmentation image and the obstacle segmentation image;

[0064] (1) Generate a binary mask image of the same size as the RGB image.

[0065] (2) Set the region corresponding to the ground segmentation map in the Mask image to 0 and set the other regions to 1 to obtain the ground mask image.

[0066] (3) Set the region corresponding to the obstacle segmentation map in the Mask image to 1 and set the other regions to 0 to obtain the obstacle mask map.

[0067] S104. Combine the ground mask map and obstacle mask map with the depth map to obtain the panoramic view cloud and obstacle point cloud with the ground removed;

[0068] (1) Multiply the depth map and the ground removal mask image pixel by pixel to obtain the ground removal depth map. Use the PCL library and depth2cloud package to convert it into point cloud data to obtain the panoramic point cloud with the ground removed.

[0069] (2) Multiply the depth map and the obstacle mask map pixel by pixel to obtain the depth map of the obstacle. Use the PCL library and depth2cloud package to convert it into point cloud data to obtain the obstacle point cloud.

[0070] The generated point cloud is filtered and reconstructed to reduce the impact of noise in the point cloud on data processing.

[0071] S105. The panoramic view cloud and obstacle point cloud with the ground removed are converted into panoramic pseudo-radar data and obstacle pseudo-radar data with the ground removed through the point cloud to pseudo-radar data algorithm.

[0072] After processing the depth camera data, the Region of Interest (ROI) of the point cloud data under the depth camera observation model has been successfully extracted (excluding the panoramic point cloud of the ground and the obstacle point cloud). This includes 3D obstacle information observed during the movement of the mobile robot, including the length, width, and height information of the obstacles. To effectively extract obstacle information under the depth camera observation model, this invention proposes a 3D point cloud to pseudo-radar data conversion algorithm.

[0073] By projecting and transforming the point cloud data of the depth image, the distance between each projected pixel in the image and the camera origin is obtained. The minimum distance between each column of projected pixels and the camera origin is then taken to obtain the distance set as pseudo-radar data. Compared with lidar data, the transformed pseudo-radar data compresses information in the height direction, improving obstacle detection capabilities.

[0074] The process of converting depth camera point cloud data into pseudo radar data is equivalent to calculating the distance between the projection points of 3D points along the y-axis in space corresponding to the pixels on the camera and the depth image point cloud data. The specific process of the point cloud to pseudo radar data conversion algorithm is as follows:

[0075] (1) The coordinates of any point M(u,v) in the depth image coordinate system are M(X,v) in the world coordinate system. w ,Y w Z w The mapping process from pixels in a depth image to three-dimensional points in space is shown in the figure.

[0076] The transformation expression for any point M on the depth map to the world coordinate system is shown in the formula:

[0077]

[0078] Where: u and v are the coordinates of any point in the depth image coordinate system; u0 and v0 are the coordinates of the center point of the depth image; X w Y w Z w Z represents the three-dimensional coordinates of point M in the world coordinate system. c R represents the distance between the target and the vertical plane where the camera is located; R is a 3×3 extrinsic rotation matrix, and T is a 3×1 extrinsic translation matrix.

[0079] Since the world coordinate origin coincides with the camera origin, there is no translation or rotation. The extrinsic parameter matrices R and T are set as follows:

[0080]

[0081] The origins of the camera coordinate system and the world coordinate system coincide, therefore the same object in both systems has the same depth, i.e., Z. c =Zw Image points [u,v] T To world coordinates [X] w ,Y w Z w ] T The transformation is shown by the formula:

[0082]

[0083] (2) Define an array laser[n] to represent the pseudo radar data array. The index n of the projection of point M onto the laser array is calculated by the following formula:

[0084] n = N(θ - α) / (β - α)

[0085] Where: [α,β] represents the maximum and minimum range of the virtual laser beam; N represents the number of laser points that the virtual laser beam can convert within the maximum and minimum range; θ represents the included angle AOC calculated from the coordinates of point M.

[0086] (3) Calculate the distance r between the projection point C of point M on the x-axis and the camera origin O; obtain the distance set of n laser points and form the pseudo radar data array laser[n].

[0087]

[0088] Where: x represents the x-coordinate of point M in the world coordinate system, and z represents the y-coordinate of point M in the world coordinate system.

[0089] (4) Perform the operation on all pixels on the input point cloud image according to steps (1) to (3) to obtain multiple layers of pseudo laser points. Then, select the effective value with the smallest distance in each column and save it into the pseudo radar data array to complete the conversion from point cloud to pseudo radar data.

[0090] The algorithm described above transforms the 3D point cloud data (excluding the panoramic point cloud and obstacle point cloud on the ground) scanned by the depth camera into 2D LiDAR data (excluding the panoramic pseudo-radar data and obstacle pseudo-radar data on the ground) that can be used for SLAM mapping.

[0091] The obtained pseudo-LiDAR data retains information about the nearest obstacle in the longitudinal direction. Typically, a single-line LiDAR measures at 10Hz, and a depth camera samples at 30 frames per second. The converted 2D LiDAR data has a maximum output frequency of 30Hz. In this embodiment, the maximum scanning angle is (-0.521567881107, 0.524276316166), and the scanning angle resolution is set to 0.00163668883033, which calculates to generate a total of 640 LiDAR point data points.

[0092] S106. The panoramic pseudo-radar data and obstacle pseudo-radar data after removing the ground are fused with the lidar data, and obstacle detection is performed based on the fused data to obtain the detection result.

[0093] By fusing two types of data (surface pseudo-radar data and obstacle pseudo-radar data excluding ground-based data) with lidar data, more efficient and accurate lidar data reflecting environmental information can be obtained. The fusion process is as follows:

[0094] Assuming that obstacle pseudo-radar data is stored in the array laser_obj, panoramic pseudo-radar data is stored in the array laser_panorama, and lidar data is stored in the array laser_scan, the final fused data is stored in the array laser_filter.

[0095] The point cloud-to-pseudo-radar data conversion algorithm determines that pseudo-radar data and lidar data with the same index have the same θ angle. The laser_obj array is then iterated through.

[0096] When the pseudo radar data of the i-th obstacle satisfies laser_obj[i] = 0:

[0097] laser_filter[i]=laser_scan[i]:

[0098] Otherwise, calculate the difference between the i-th panoramic pseudo-radar data (with the ground removed) and the i-th lidar data.

[0099] diff=|laser_panorama[i]-laser_scan[i]|

[0100] The difference is compared with the threshold to obtain the i-th fused data.

[0101]

[0102] Where: i = 1, 2, 3...n.

[0103] After data fusion is completed, the fused data is used for obstacle detection.

[0104] To verify the performance of the embodiments of the present invention, the embodiments of the present invention use scanning accuracy to measure the degree of perception of the real environment by the fused data. The accuracy of a single test is obtained by averaging the differences between the actual distances to panoramic obstacles and the distances measured in the sensor data. The following formula is used:

[0105]

[0106] Where: acc represents precision, dsensor d represents the distance measured by the sensor. real This represents the actual distance. If the lidar does not acquire altitude information, the accuracy information is not calculated. n represents the number of data points.

[0107] Table 1 shows a comparison of the results of obstacle detection using the method (sensor fusion) of the present invention and using a depth camera or lidar alone.

[0108] Table 1

[0109]

[0110] The results show that the fused data obtained in the embodiments of the present invention can better reflect real environmental information.

[0111] This invention also compares mapping using the fused data of this invention with mapping using only LiDAR data through the GMAPping algorithm, such as... Figure 2 As shown, (1) represents the result of mapping using only lidar data, and (2) represents the result of mapping using the fused data of the present invention. The results also show that the method of the embodiment of the present invention achieves better results.

[0112] This invention also provides a low-profile obstacle detection device, comprising:

[0113] The data acquisition module is used to acquire LiDAR data and depth camera data, wherein the depth camera data includes RGB images and depth maps;

[0114] The instance segmentation module is used to perform instance segmentation on the RGB image using a trained YOLOv5 model to obtain a ground segmentation map and an obstacle segmentation map.

[0115] The mask generation module is used to obtain a ground mask image and an obstacle mask image by using the ground segmentation image and the obstacle segmentation image based on the RGB image;

[0116] The point cloud generation module is used to combine the culled ground mask map and the obstacle mask map with the depth map to obtain the panoramic point cloud and obstacle point cloud with the culled ground.

[0117] The pseudo radar data generation module is used to convert the panoramic view cloud and obstacle point cloud, which have had their ground removed, into panoramic pseudo radar data and obstacle pseudo radar data, respectively, through a point cloud to pseudo radar data conversion algorithm.

[0118] The detection module is used to fuse the panoramic pseudo-radar data and obstacle pseudo-radar data (excluding ground) with the lidar data, and to perform obstacle detection based on the fused data to obtain the detection result.

[0119] This invention also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the low obstacle detection method described above.

[0120] This invention also provides a computer-readable storage medium comprising a stored computer program, wherein the computer program, when running, controls the device containing the computer-readable storage medium to perform the low obstacle detection method as described above.

[0121] It should be understood that the above description of the preferred embodiments is quite detailed and should not be construed as a limitation on the scope of protection of this application. Those skilled in the art, under the guidance of this invention, may make substitutions or modifications without departing from the scope of protection of the claims of this application, all of which fall within the scope of protection of this application. The scope of protection claimed in this application shall be determined by the appended claims.

Claims

1. A method for detecting low-lying obstacles, characterized in that, Includes the following steps: (1) Acquire lidar data and depth camera data, wherein the depth camera data includes RGB images and depth maps; (2) For the RGB image, use the trained YOLOv5 model to perform instance segmentation to obtain the ground segmentation map and the obstacle segmentation map; (3) Based on the RGB image, a ground mask image and an obstacle mask image are obtained using the ground segmentation image and the obstacle segmentation image; (3.1) Generate a binary mask image of the same size as the RGB image; (3.2) Set the region corresponding to the ground segmentation map in the Mask image to 0 and set the other regions to 1 to obtain the ground removal mask image; (3.3) Set the region corresponding to the obstacle segmentation map in the Mask image to 1 and set the other regions to 0 to obtain the obstacle mask map; (4) Combine the culled ground mask map and obstacle mask map with the depth map to obtain the panoramic view cloud and obstacle point cloud with the culled ground; (4.1) Multiply the depth map and the culled ground mask map pixel by pixel to obtain the depth map of the culled ground, and use the PCL library and depth2cloud package to convert it into point cloud data to obtain the panoramic point cloud of the culled ground; (4.2) Multiply the depth map and the obstacle mask map pixel by pixel to obtain the depth map of the obstacle, and use the PCL library and depth2cloud package to convert it into point cloud data to obtain the obstacle point cloud; (5) The point cloud of the panoramic view and the point cloud of obstacles with the ground removed are converted into panoramic pseudo radar data and obstacle pseudo radar data with the ground removed by the point cloud to pseudo radar data algorithm. (6) The panoramic pseudo-radar data and obstacle pseudo-radar data after removing the ground are fused with the lidar data, and obstacle detection is performed based on the fused data to obtain the detection result; The specific steps of fusing the panoramic pseudo-radar data and obstacle pseudo-radar data (after removing the ground) with the lidar data are as follows: When the i The obstacle pseudo-radar data satisfies laser_obj [ i When ]=0: Otherwise calculate the first i The panoramic pseudo-radar data with ground removed mentioned above and the first i Differences in LiDAR data; By iterating through all the points as described above, the fused data is obtained. in: array laser_obj The array represents the pseudo radar data of the obstacle. laser panorama This represents the panoramic pseudo-radar data after removing the ground, array laser_scan The array represents the lidar data. laser_filter The fused data is represented by threshold and the difference is represented by diff.

2. The method for detecting low obstacles according to claim 1, characterized in that: The YOLOv5 model used is the YOLOv5s model, and the training process utilizes the transfer learning steps of YOLOv5s.

3. The method for detecting low obstacles according to claim 1, characterized in that, The algorithm for converting point cloud data into pseudo radar data is as follows: (5.1) The coordinates of any point M(u, v) in the input point cloud image coordinate system are mapped to the world coordinate system using the following formula: ; Where u and v are the coordinates of any point in the input point cloud image coordinate system; , The coordinates of the center point of the input point cloud image; , , The three-dimensional coordinate point corresponding to point M in the world coordinate system; This indicates the distance between the target and the vertical plane where the camera is located; (5.2) Define array The pseudo-radar data array, where the index n of point M projected onto the laser array is calculated by the following formula: in: This represents the maximum and minimum range of the virtual laser beam; N represents the number of laser points that the virtual laser beam converts within the maximum and minimum ranges. This represents the included angle AOC calculated using the coordinates of point M; (5.3) Calculate the distance r between the projection point C of point M on the x-axis and the camera origin O; obtain the distance set of n laser points, and form the pseudo-radar data array. ; Where: x represents the x-coordinate of point M in the world coordinate system, and z represents the y-coordinate of point M in the world coordinate system; (5.4) Perform the operations on all pixels on the input point cloud image according to steps (5.1) to (5.3) to obtain multiple layers of pseudo laser points. Then, select the effective value with the smallest distance in each column and save it into the pseudo radar data array to complete the conversion from point cloud to pseudo radar data.

4. A low-lying obstacle detection device, characterized in that, include: The data acquisition module is used to acquire LiDAR data and depth camera data, wherein the depth camera data includes RGB images and depth maps; The instance segmentation module is used to perform instance segmentation on the RGB image using a trained YOLOv5 model to obtain a ground segmentation map and an obstacle segmentation map. The mask generation module is used to obtain a ground mask image and an obstacle mask image by using the ground segmentation image and the obstacle segmentation image based on the RGB image; (3.1) Generate a binary mask image of the same size as the RGB image; (3.2) Set the region corresponding to the ground segmentation map in the Mask image to 0 and set the other regions to 1 to obtain the ground removal mask image; (3.3) Set the region corresponding to the obstacle segmentation map in the Mask image to 1 and set the other regions to 0 to obtain the obstacle mask map; The point cloud generation module is used to combine the culled ground mask map and the obstacle mask map with the depth map to obtain the panoramic point cloud and obstacle point cloud with the culled ground. (4.1) Multiply the depth map and the culled ground mask map pixel by pixel to obtain the depth map of the culled ground, and use the PCL library and depth2cloud package to convert it into point cloud data to obtain the panoramic point cloud of the culled ground; (4.2) Multiply the depth map and the obstacle mask map pixel by pixel to obtain the depth map of the obstacle, and use the PCL library and depth2cloud package to convert it into point cloud data to obtain the obstacle point cloud; The pseudo radar data generation module is used to convert the panoramic view cloud and obstacle point cloud, which have had their ground removed, into panoramic pseudo radar data and obstacle pseudo radar data, respectively, through a point cloud to pseudo radar data conversion algorithm. The detection module is used to fuse the panoramic pseudo-radar data and obstacle pseudo-radar data (excluding ground) with the lidar data, and to perform obstacle detection based on the fused data to obtain the detection result. The specific steps of fusing the panoramic pseudo-radar data and obstacle pseudo-radar data (after removing the ground) with the lidar data are as follows: When the i The obstacle pseudo-radar data satisfies laser_obj [ i When ]=0: Otherwise calculate the first i The panoramic pseudo-radar data with ground removed mentioned above and the first i Differences in LiDAR data; By iterating through all the points as described above, the fused data is obtained. in: array laser_obj The array represents the pseudo radar data of the obstacle. laser panorama This represents the panoramic pseudo-radar data after removing the ground, array laser_scan The array represents the lidar data. laser_filter The fused data is represented by threshold and the difference is represented by diff.

5. A terminal device, characterized in that, The device includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the low obstacle detection method as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the low obstacle detection method as described in any one of claims 1 to 3.

Images