Method, device and computer equipment for detecting undefined category obstacles
By performing rasterization, ground segmentation, and clustering on point cloud data, the problem of insufficient detection of undefined obstacle categories in existing technologies has been solved, achieving accurate detection of undefined obstacle categories and improving the safety of autonomous driving.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN DEEPROUTE AI CO LTD
- Filing Date
- 2021-04-28
- Publication Date
- 2026-07-14
AI Technical Summary
Existing deep learning-based object detection methods cannot effectively detect obstacles of undefined categories, such as tripods and fences that are not commonly found on roads, resulting in lower safety for autonomous driving.
By acquiring current frame point cloud data, historical obstacle detection data, and map data, and processing them into a raster, the data is input into the detection model. Combined with ground segmentation and clustering algorithms, the occupancy information of undefined category obstacles is extracted, thereby enabling the detection of undefined category obstacles.
It improves the robustness of detecting undefined obstacle categories, enhances the autonomous vehicle's ability to perceive all obstacles, and improves the safety of autonomous driving.
Smart Images
Figure CN115917357B_ABST
Abstract
Description
Technical Field
[0001] This application relates to a method, apparatus, computer equipment, storage medium, and vehicle for detecting undefined categories of obstacles. Background Technology
[0002] In autonomous driving, it is necessary to detect obstacles in the surrounding environment in real time and mark the areas where obstacles are located in order to better plan reasonable routes, avoid obstacles, and comply with traffic rules, thereby ensuring the safety of autonomous driving. Existing object detection methods, such as deep learning-based object detection methods, require detection based on the labeled obstacle categories, thus necessitating the pre-definition of deterministic categories, such as pedestrians and vehicles. However, deep learning-based object detection methods cannot handle some undefined categories, such as obstacles that are not frequently encountered on roads, such as tripods and fences, resulting in lower safety levels in autonomous driving. Summary of the Invention
[0003] According to various embodiments disclosed in this application, a method, apparatus, computer device, storage medium, and vehicle for detecting undefined categories of obstacles are provided.
[0004] A method for detecting undefined category obstacles, comprising:
[0005] Acquire current frame point cloud data, historical obstacle detection data, and map data; the current frame point cloud data contains areas of obstacles with undefined categories.
[0006] The current frame point cloud data is rasterized to obtain an initial raster map;
[0007] The initial grid map is input into the detection model, and the target detection results and obstacle segmentation results corresponding to the obstacle regions with undefined categories are output.
[0008] Ground segmentation is performed on the current frame point cloud data to determine the ground point cloud data;
[0009] A target raster map is determined based on the target detection results, the obstacle segmentation results, the ground point cloud data, the historical obstacle detection data, the map data, and the initial raster map; and
[0010] Clustering is performed on the target grid map to obtain the detection results of undefined category obstacles.
[0011] A detection device for undefined category obstacles, comprising:
[0012] A computer device includes a memory and one or more processors, the memory storing computer-readable instructions that, when executed by the processors, cause the one or more processors to perform the following steps:
[0013] Acquire current frame point cloud data, historical obstacle detection data, and map data; the current frame point cloud data contains areas of obstacles with undefined categories.
[0014] The current frame point cloud data is rasterized to obtain an initial raster map;
[0015] The initial grid map is input into the detection model, and the target detection results and obstacle segmentation results corresponding to the obstacle regions with undefined categories are output.
[0016] Ground segmentation is performed on the current frame point cloud data to determine the ground point cloud data;
[0017] A target raster map is determined based on the target detection results, the obstacle segmentation results, the ground point cloud data, the historical obstacle detection data, the map data, and the initial raster map; and
[0018] Clustering is performed on the target grid map to obtain the detection results of undefined category obstacles.
[0019] One or more computer storage media storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the following steps:
[0020] Acquire current frame point cloud data, historical obstacle detection data, and map data; the current frame point cloud data contains areas of obstacles with undefined categories.
[0021] The current frame point cloud data is rasterized to obtain an initial raster map;
[0022] The initial grid map is input into the detection model, and the target detection results and obstacle segmentation results corresponding to the obstacle regions with undefined categories are output.
[0023] Ground segmentation is performed on the current frame point cloud data to determine the ground point cloud data;
[0024] A target raster map is determined based on the target detection results, the obstacle segmentation results, the ground point cloud data, the historical obstacle detection data, the map data, and the initial raster map; and
[0025] Clustering is performed on the target grid map to obtain the detection results of undefined category obstacles.
[0026] A means of transport includes the steps of performing the above-described method for detecting undefined categories of obstacles.
[0027] Details of one or more embodiments of this application are set forth in the following drawings and description. Other features and advantages of this application will become apparent from the specification, drawings, and claims. Attached Figure Description
[0028] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0029] Figure 1 This is an application environment diagram for a method of detecting obstacles of undefined categories in one or more embodiments.
[0030] Figure 2 This is a flowchart illustrating a method for detecting undefined obstacle categories in one or more embodiments.
[0031] Figure 3 This is a flowchart illustrating the steps in one or more embodiments of inputting an initial grid map into a detection model and outputting target detection results and obstacle segmentation results corresponding to obstacle regions with undefined categories.
[0032] Figure 4 This is a flowchart illustrating the steps of determining a target raster map based on target detection results, obstacle segmentation results, ground point cloud data, historical obstacle detection data, map data, and an initial raster map in one or more embodiments.
[0033] Figure 5 This is a block diagram of a detection device for obstacles of undefined categories in one or more embodiments.
[0034] Figure 6 This is a block diagram of a computer device in one or more embodiments. Detailed Implementation
[0035] To make the technical solutions and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0036] The method for detecting undefined obstacle categories provided in this application can be applied to, for example... Figure 1In the application environment shown, the vehicle-mounted sensor 102 communicates with the vehicle-mounted computer device 104 via a network. There can be one or more vehicle-mounted sensors. The vehicle-mounted computer device can be simply referred to as the computer device. The vehicle-mounted sensor 102 sends the collected point cloud data to the computer device 104. The computer device 104 can save the point cloud data frame by frame. When obstacle detection is required, it acquires the current frame point cloud data, historical obstacle detection data, and map data. The current frame point cloud data contains obstacle areas with undefined categories. The current frame point cloud data is rasterized to obtain an initial raster map. The initial raster map is input into the detection model, which outputs the target detection results and obstacle segmentation results corresponding to the undefined category obstacle areas. This allows for ground segmentation of the current frame point cloud data, determining the ground point cloud data. Based on the target detection results, obstacle segmentation results, ground point cloud data, historical obstacle detection data, and map data, occupancy information is extracted from the initial raster map to obtain the target raster map. The target raster map is then clustered to obtain the undefined category obstacle detection results. The vehicle-mounted sensor 102 may be, but is not limited to, a lidar or a laser scanner.
[0037] In one embodiment, such as Figure 2 As shown, a method for detecting undefined category obstacles is provided, which can be applied to... Figure 1 Taking a computer device as an example, the explanation includes the following steps:
[0038] Step 202: Obtain the current frame point cloud data, historical obstacle detection data, and map data; the current frame point cloud data contains areas with undefined obstacle categories.
[0039] Point cloud data is data recorded in point cloud form by onboard sensors scanning the surrounding environment. Specifically, point cloud data can include the three-dimensional coordinates (x, y, z) of each point, laser reflection intensity, and color information (RGB). The three-dimensional coordinates represent the positional information of obstacles on the surface of the surrounding environment. For example, the three-dimensional coordinates can be the coordinates of a point in a Cartesian coordinate system, specifically including the horizontal, vertical, and triangular coordinates of the point in the Cartesian coordinate system. The Cartesian coordinate system is a three-dimensional spatial coordinate system established with the onboard sensor as the origin, and includes a horizontal axis (x-axis), a vertical axis (y-axis), and a triangular axis (z-axis). The three-dimensional spatial coordinate system established with the onboard sensor as the origin satisfies the right-hand rule.
[0040] During autonomous driving, vehicles can scan their surroundings using onboard sensors to obtain point cloud data, which is then transmitted to a computer. For example, the onboard sensors could be LiDAR. The computer can save the point cloud data frame by frame, recording information such as the data acquisition time for each frame. The computer can then use the current frame of point cloud data for obstacle detection. The current frame of point cloud data may contain areas marked with undefined obstacle categories. These undefined obstacle categories can be manually labeled. An undefined category refers to obstacles not labeled in the obstacle category classification, such as targets whose category cannot be defined. Undefined obstacle categories refer to the point cloud areas corresponding to obstacles of other categories that are not labeled in the current frame of point cloud data. Specifically, this can be the area formed by the remaining point cloud data after removing ground point cloud data and point cloud data for obstacles of defined categories.
[0041] The computer device stores historical obstacle detection data and map data. When obstacle detection is required, the computer device retrieves this stored historical obstacle detection data and map data. Historical obstacle detection data refers to the area where each undefined category obstacle is located, detected based on historical frame point cloud data. Map data refers to information on permanent obstacles generated offline, such as tall buildings and railings. Permanent obstacles are fixed obstacles whose location and area remain unchanged.
[0042] Step 204: Rasterize the current frame point cloud data to obtain an initial raster map.
[0043] Rasterization refers to quantizing the data space corresponding to the current frame's point cloud data from a top-down viewpoint according to a preset quantization resolution, resulting in a raster image. The top-down viewpoint maps the data space corresponding to the current frame's point cloud data to a horizontal plane (xy plane), and this raster image can be called the initial raster image. Rasterization does not consider the height information of the current frame's point cloud data. For example, when the preset quantization resolution is 0.2, and the planar space of the current frame's point cloud data in the top-down viewpoint is 100m x 50m, then the current frame's point cloud data will be quantized into a 500m x 250m raster image. The initial raster image includes multiple grids, and each grid can include points from the current frame's point cloud data.
[0044] Step 206: Input the initial grid map into the detection model and output the target detection results and the obstacle segmentation results corresponding to the obstacle regions with undefined categories.
[0045] The computer device contains pre-trained detection models, which are used to predict the category and location of obstacles with defined categories and to segment obstacles with undefined categories. The detection models are trained using a large amount of sample data.
[0046] Since the current frame point cloud data contains undefined obstacle regions, the initial raster map obtained after rasterizing the current frame point cloud data also includes undefined obstacle regions. It is understandable that the unidentified obstacle regions in the initial raster map are also represented in raster form.
[0047] After obtaining the initial raster map, a detection model is invoked. The initial raster map is input into the detection model, which performs object detection on the initial raster map and segments undefined obstacle regions within it. This yields the object detection results for the initial raster map and the segmented obstacle regions corresponding to the undefined obstacle regions. The object detection results include the category of defined obstacles and their corresponding regions. The regions containing defined obstacle categories refer to the bounding boxes corresponding to those categories. The obstacle segmentation results include point cloud data corresponding to different undefined obstacle categories.
[0048] Step 208: Perform ground segmentation on the current frame point cloud data to determine the ground point cloud data.
[0049] Ground segmentation refers to separating ground point cloud data from non-ground point cloud data in the current frame point cloud data.
[0050] Since the current frame point cloud data includes a large amount of ground point cloud data, and the amount of useful information contained in the current frame point cloud data that can be used to complete obstacle detection is relatively small, it is necessary to perform ground segmentation on the current frame point cloud data so that obstacle detection can be completed based on the remaining point cloud data after removing the ground point cloud data.
[0051] Computer equipment can segment the current frame's point cloud data into ground segments using either traditional ground segmentation methods or deep learning-based ground segmentation algorithms. Traditional ground segmentation methods can include horizontal plane calibration methods, grid height difference methods, normal vector methods, absolute height methods, and average height methods. Deep learning-based ground segmentation algorithms can be, but are not limited to, those based on Logistic Regression (LR), Support Vector Machine (SVM), or Convolutional Neural Networks (CNN). The aforementioned LR model is a simple and efficient classification model in machine learning with a wide range of applications. The aforementioned SVM model is one of the most robust and accurate methods among all well-known data mining algorithms; it belongs to the binary classification algorithm and can support both linear and non-linear classification. The aforementioned CNN model is a type of feedforward neural network with convolutional computation and a deep structure, and is one of the representative algorithms of deep learning. Convolutional neural networks have representation learning capabilities, enabling them to classify input information invariantly according to their hierarchical structure.
[0052] Step 210: Extract occupancy information from the initial grid map based on the target detection results, obstacle segmentation results, ground point cloud data, historical obstacle detection data, and map data to obtain the target grid map.
[0053] Since the detection model can only segment obstacles of different undefined categories in the initial raster map, the obstacle segmentation results are not accurate enough. Obstacle occupancy information can be extracted from the initial raster map based on the target detection results, ground point cloud data, and map data. Obstacle occupancy information refers to the probability of an undefined obstacle category existing in each raster. The obstacle segmentation results are then fused with the extracted obstacle occupancy information and historical obstacle detection data to obtain the target raster map. The target raster map is a raster map containing obstacle occupancy information extracted from multiple information sources.
[0054] Step 212: Cluster the target grid map to obtain the detection results of undefined category obstacles.
[0055] Each grid cell in the target grid map has a probability of containing an undefined category obstacle. This probability is obtained by fusing multiple information sources. The computer device identifies the regions corresponding to the grid cells containing undefined category obstacles in the target grid map. These regions are then clustered, grouping points belonging to the same undefined category obstacle into one class. The bounding box for each undefined category obstacle is calculated, resulting in the region corresponding to each undefined category obstacle and thus the undefined category obstacle detection result. The clustering method can be any of the following clustering algorithms: connected component clustering, DBSCAN (Density-Based Spatial Clustering of Applications with Noise), etc.
[0056] In this embodiment, current frame point cloud data, historical obstacle detection data, and map data are acquired. The current frame point cloud data contains obstacle regions with undefined categories. The current frame point cloud data is rasterized to obtain an initial raster map. The initial raster map is input into the detection model, which outputs the target detection results and obstacle segmentation results corresponding to the obstacle regions with undefined categories. This process performs ground segmentation on the current frame point cloud data to determine the ground point cloud data. Based on the target detection results, obstacle segmentation results, ground point cloud data, historical obstacle detection data, map data, and the initial raster map, a target raster map is determined. The target raster map is then clustered to obtain the obstacle detection results with undefined categories. Based on target detection results, ground point cloud data, and map data, obstacle occupancy information of undefined category obstacles in the initial grid map can be extracted. This obstacle occupancy information is then fused with obstacle segmentation results and historical obstacle detection data. Clustering is performed based on the target grid map, enabling accurate detection of undefined category obstacles. This allows autonomous vehicles to perceive all obstacle targets. Furthermore, since the target grid map is obtained through the fusion of multiple information sources, it can reliably detect undefined category obstacles, improving the robustness of undefined category obstacle detection and thus enhancing the safety of autonomous driving.
[0057] In one embodiment, such as Figure 3 As shown, the steps for inputting the initial grid map into the detection model and outputting the target detection results and obstacle segmentation results corresponding to undefined obstacle regions include:
[0058] Step 302: Input the initial grid map into the detection model, which includes a parallel target detection network and an obstacle segmentation network.
[0059] Step 304: Perform target detection on the initial raster image using a target detection network and output the target detection results.
[0060] Step 306: The obstacle regions in the initial grid map that have no defined categories are segmented using an obstacle segmentation network to obtain the obstacle segmentation results.
[0061] The detection model comprises two parallel network branches: an object detection network and an obstacle segmentation network. For example, the object detection network can be any of the following: Convolutional Neural Networks (CNN), PointNet, PointPillar, PolarNet, etc. The obstacle segmentation network can also be a CNN, PointNet, etc. The detection process is performed offline. After the computer device inputs an initial grid map into the detection model, since the initial grid map includes obstacle regions with undefined categories, the object detection network and the obstacle segmentation network of the detection model perform object detection on the initial grid map and segmentation on the obstacle regions with undefined categories, respectively. The object detection network can extract features from the initial grid map, predict the category and corresponding bounding box of defined-category obstacles based on the extracted feature maps, and output the object detection result. The obstacle segmentation network can predict the point cloud data corresponding to undefined-category obstacles in the obstacle regions with undefined categories and output the obstacle segmentation result.
[0062] In this embodiment, the detection network includes a target detection network and an obstacle segmentation network. The target detection network is used to pre-define obstacle categories, while the obstacle segmentation network is used to segment obstacles in areas with undefined obstacle categories. This allows for the simultaneous detection of obstacles of defined categories and the segmentation of obstacles of undefined categories. Furthermore, the detection process of this model is performed offline, enabling rapid segmentation of undefined obstacle categories in the offline state.
[0063] In one embodiment, ground segmentation is performed on the current frame point cloud data to determine the ground point cloud data, which includes: dividing the point cloud region corresponding to the current frame point cloud data into multiple grids; calculating the ground corresponding to each grid according to a preset plane equation; and determining the points in each grid whose distance value to the corresponding ground is less than a distance threshold as ground point cloud data.
[0064] Ground segmentation refers to extracting ground point cloud data from the current frame's point cloud data to separate ground point cloud data from non-ground point cloud data. A point cloud region refers to the data space containing point cloud data from each frame.
[0065] Computer equipment can project the current frame's point cloud region onto a horizontal plane (xy plane) and divide the horizontal plane into multiple grids. Specifically, the computer equipment can divide the point cloud region corresponding to the current frame's point cloud data into grids according to preset parameters. Preset parameters can be the dimensions of each grid. For example, preset parameters can be length * width, indicating the length and width of each grid obtained after grid division. The length and width can be the same or different. Preset parameters can also be equal division, where multiple grids have the same height. The computer equipment can first divide the point cloud region corresponding to the current frame's point cloud data along the x-axis and y-axis respectively according to the preset parameters.
[0066] The computer device acquires a preset plane equation, which is a traditional equation used to calculate a plane based on point coordinates. Based on the preset plane equation, the least squares method is used to calculate the ground corresponding to each grid cell's point cloud, thus obtaining the ground corresponding to each grid cell. The ground corresponding to each grid cell can be represented as a three-variable linear equation. The computer device iterates through the input point coordinates of each point in the corresponding grid cell into the ground-corresponding equation, calculating the distance value between each point and the corresponding ground. The computer device pre-stores distance thresholds for determining point categories. Point categories can include ground points and non-ground points. The computer device compares the distance value with the distance threshold; when the distance value is less than the distance threshold, it indicates that the point is a ground point. When the distance value is greater than or equal to the distance threshold, it indicates that the point is a non-ground point. Points with distance values less than the distance threshold are thus grouped into ground point cloud data.
[0067] In this embodiment, the point cloud region corresponding to the current frame point cloud data is divided into multiple grids, thereby calculating the ground corresponding to the point cloud data in each grid. Points in each grid whose distance to the corresponding ground is less than a threshold are identified as ground point cloud data. Since grid division only requires dividing the point cloud region corresponding to the point cloud data in the x-axis and y-axis directions, it can quickly perform ground segmentation processing in autonomous driving mode when the computing resources of the computer equipment are limited and the real-time requirements are high.
[0068] Furthermore, the calculation of the ground corresponding to each grid cell according to the preset plane equation includes: selecting the point with the smallest height value among the points corresponding to each grid cell; calculating the height difference between each point in each grid cell and the point with the smallest height value; extracting points whose height difference is less than the height difference threshold; and performing plane fitting on the selected points according to the preset plane equation to obtain the ground corresponding to each grid cell.
[0069] The computer device divides the point cloud region corresponding to the current frame of point cloud data into multiple grids, each grid containing points with corresponding height values. Within each grid, the point with the smallest height value is selected, and the height difference between each point in the corresponding grid and the point with the smallest height value is calculated. The computer device pre-stores a height difference threshold for determining whether a point is a plane fitting point. When the height difference is less than the threshold, it indicates that the point corresponding to that height difference is a plane fitting point. The computer device compares the height difference values with the height difference threshold, selects points with height differences less than the threshold, and performs plane fitting on the selected points according to a preset plane equation to obtain the ground corresponding to each grid.
[0070] In this embodiment, the computer device selects plane fitting points by calculating the height difference between each point in each grid and the point with the smallest height value, and comparing the height difference with a height difference threshold. Since the point with the smallest height value has the highest probability of being a ground point, by calculating the height difference between each point and the point with the smallest height value, the plane fitting points can be determined more accurately, thereby improving the efficiency of ground segmentation.
[0071] In one embodiment, the step of determining the target raster map based on the target detection result, obstacle segmentation result, ground point cloud data, historical obstacle detection data, map data, and initial raster map includes:
[0072] Step 402: Remove defined category obstacles, ground point cloud data, and fixed obstacles from the target detection results in the initial raster map to obtain the raster map after removal.
[0073] Step 404: Extract obstacle occupancy information from the removed raster map.
[0074] Step 406: The obstacle occupancy information, obstacle segmentation results and historical obstacle detection data are fused to obtain the target grid map.
[0075] The target detection results define the categories of obstacles and their locations. The obstacle segmentation results include point cloud data for different undefined obstacle categories. Ground point cloud data includes ground points from the current frame's point cloud data. Historical obstacle detection data refers to the regions where each undefined obstacle category is located, detected based on historical frame point cloud data. Map data refers to information on permanently generated obstacles offline, such as tall buildings and railings. Permanent obstacles are fixed obstacles whose locations remain unchanged. The initial raster map is the raster map obtained by quantizing the current raster map in a top-down view.
[0076] To facilitate the detection of undefined obstacle categories, it is necessary to remove data from the initial raster map that is irrelevant to the detection of undefined obstacle categories, such as defined obstacle categories and ground points. Fixed obstacles in the map can also be considered defined obstacles and need to be removed. Specifically, the computer device can remove defined obstacle categories from the target detection results in the initial raster map, remove ground point cloud data, and remove fixed obstacles from the map data, thus removing data irrelevant to the detection of undefined obstacle categories and obtaining a raster map after the removal process.
[0077] Obstacle occupancy information refers to the occupancy rate of each grid cell, that is, the probability that each grid cell contains an undefined category obstacle (occupied). Computer equipment can extract obstacle occupancy information from the deprocessed grid map. This allows for the fusion of obstacle occupancy information, obstacle segmentation results, and historical obstacle detection data. Since obstacle occupancy information, obstacle segmentation results, and historical obstacle detection data are all represented through a grid map, and each grid cell contains the probability of containing an undefined category obstacle, the fusion method can be to accumulate the probabilities of obstacle occupancy information, obstacle segmentation results, and historical obstacle detection data.
[0078] Furthermore, extracting obstacle occupancy information from the removed raster map includes: predicting the probability of an obstacle existing in each grid cell of the removed raster map, and obtaining obstacle occupancy information based on the probability. The removed raster map contains multiple grid cells, and the computer device can use a ray tracing algorithm to determine the probability of an undefined category obstacle existing in each grid cell of the removed raster map. When the probability is greater than a preset threshold, it indicates that the grid cell contains an undefined category obstacle. The state of a grid cell containing an undefined category obstacle can be defined as Occupied, and the state of a grid cell without an undefined category obstacle can be defined as Free. The computer device can then obtain obstacle occupancy information based on the probability of obstacles existing in all grid cells of the removed raster map.
[0079] In this embodiment, defined obstacle categories, ground point cloud data, and fixed obstacles in the map data are removed from the initial raster map. This process removes data irrelevant to the detection of undefined obstacle categories, facilitating the extraction of more accurate obstacle occupancy information. The obstacle segmentation result is obtained through offline state segmentation, while the historical obstacle detection data is the region where each undefined obstacle category is located, detected based on historical frame point cloud data. By fusing these three data sources, the detection accuracy of undefined obstacle categories is effectively improved.
[0080] In one embodiment, clustering the target grid map to obtain undefined category obstacle detection results includes: determining the occupied area in the target grid map; performing connected component detection on the occupied area to obtain multiple connected components and a point set corresponding to each connected component; calculating the bounding box of the corresponding undefined category obstacle based on the point set corresponding to each connected component to obtain the undefined category obstacle detection results.
[0081] A target raster map is a raster map that incorporates information from multiple sources regarding undefined obstacle categories. An occupied area refers to the region within the target raster map containing undefined obstacle categories.
[0082] Each grid cell in the target grid map contains the probability of the presence of an undefined category obstacle, and each cell is marked with either an Occupied or Free state. The Occupied or Free state is determined by comparing the probability of an undefined category obstacle in each cell with a preset threshold. Grids with a probability greater than the preset threshold are marked as Occupied, and grids with a probability less than or equal to the preset threshold are marked as Free. An Occupied state indicates the presence of an undefined category obstacle in that grid cell, while a Free state indicates the absence of undefined category obstacles. The computer device can define the area corresponding to the grid cells marked with an Occupied state in the target grid map as the occupied area. Typically, all points belonging to an obstacle are distributed around its center point; therefore, it is necessary to perform correlation clustering on the points in the occupied area, grouping points belonging to the same obstacle together. Specifically, the computer device can perform connected component detection on the occupied area to obtain multiple connected components and the corresponding point set for each connected component. Each connected component corresponds to an undefined category obstacle. The point set corresponding to each connected component refers to all points corresponding to the corresponding undefined category obstacle. Connected component detection can be performed by traversing the obstacle grids in the occupied area, finding the first unmarked obstacle grid, marking it initially, finding other obstacle grids in the 8-neighborhood of this obstacle grid and marking them in the same way, recording the positions of all obstacle grids, and merging the points at these positions to obtain the point set corresponding to the connected component. It can be understood that computer devices can obtain the point set corresponding to other connected components in the same way.
[0083] After obtaining the point set for each undefined obstacle category, the bounding box of the undefined obstacle category can be calculated based on the point set, including its center position, size, and orientation. This bounding box can then be used to determine the region corresponding to each undefined obstacle category. The bounding box can be calculated using any method such as L-shape fitting or principal component analysis. The bounding box can include the center point coordinates, size, and orientation of the undefined obstacle category. By identifying the bounding boxes corresponding to undefined obstacles, different undefined obstacle categories can be accurately distinguished.
[0084] In this embodiment, the occupied area in the target grid map is determined; connected component detection is performed on the occupied area to obtain multiple connected components and a point set corresponding to each connected component; the bounding box of the corresponding undefined category obstacle is calculated based on the point set corresponding to each connected component to obtain the undefined category obstacle detection result. This allows for the prediction of the area where the undefined category obstacle is located.
[0085] In one embodiment, such as Figure 5 As shown, a device for detecting undefined obstacle categories is provided, comprising: a data acquisition module 502, a rasterization processing module 504, a detection module 506, a ground segmentation module 508, a determination module 510, and a clustering module 512, wherein:
[0086] The data acquisition module 502 is used to acquire current frame point cloud data, historical obstacle detection data, and map data; the current frame point cloud data contains areas of obstacles with undefined categories.
[0087] The rasterization processing module 504 is used to rasterize the current frame point cloud data to obtain an initial raster map.
[0088] The detection module 506 is used to input the initial grid map into the detection model and output the target detection results and the obstacle segmentation results corresponding to the obstacle areas with undefined categories.
[0089] The ground segmentation module 508 is used to segment the current frame point cloud data to determine the ground point cloud data.
[0090] The determination module 510 is used to determine the target raster map based on the target detection results, obstacle segmentation results, ground point cloud data, historical obstacle detection data, map data and initial raster map.
[0091] Clustering module 512 is used to cluster the target raster map to obtain the detection results of undefined category obstacles.
[0092] In one embodiment, the detection module 506 is used to input an initial grid map into a detection model, which includes a parallel target detection network and an obstacle segmentation network; to perform target detection on the initial grid map through the target detection network and output the target detection result; and to segment the obstacle regions in the initial grid map that have no defined category through the obstacle segmentation network to obtain the obstacle segmentation result.
[0093] In one embodiment, the ground segmentation module 508 is further configured to divide the point cloud region corresponding to the current frame point cloud data into multiple grids; calculate the ground corresponding to each grid according to a preset plane equation; and determine the points in each grid whose distance value to the corresponding ground is less than a distance threshold as ground point cloud data.
[0094] In one embodiment, the determining module 510 is further configured to remove defined category obstacles, ground point cloud data and fixed obstacles in the map data from the target detection results in the initial grid map to obtain a grid map after removal processing; extract obstacle occupancy information in the grid map after removal processing; and fuse the obstacle occupancy information, obstacle segmentation results and historical obstacle detection data to obtain a target grid map.
[0095] In one embodiment, the determining module 510 is further configured to predict the probability that each grid cell in the removed grid map contains an obstacle, and obtain obstacle occupancy information based on the probability.
[0096] In one embodiment, the clustering module 512 is further configured to determine the occupied area in the target grid map; perform connected component detection on the occupied area to obtain multiple connected components and a point set corresponding to each connected component; calculate the bounding box of the corresponding undefined category obstacle based on the point set corresponding to each connected component, and obtain the undefined category obstacle detection result.
[0097] Specific limitations regarding the detection device for undefined obstacle categories can be found in the limitations of the detection method for undefined obstacle categories mentioned above, and will not be repeated here. Each module in the aforementioned detection device for undefined obstacle categories can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the operations corresponding to each module.
[0098] In one embodiment, a computer device is provided, the internal structure of which can be shown in the figure below. Figure 6As shown, the computer device includes a processor, memory, communication interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer-readable instructions, and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the non-volatile storage medium. The database stores data for a method of detecting undefined obstacle categories. The communication interface is used to communicate with an external terminal. When the computer-readable instructions are executed by the processor, they implement a method for detecting undefined obstacle categories.
[0099] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0100] A computer device includes a memory and one or more processors. The memory stores computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps described in the various method embodiments above.
[0101] One or more computer storage media storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps in the various method embodiments described above.
[0102] The computer storage medium is a readable storage medium, which can be either non-volatile or volatile.
[0103] In one embodiment, a means of transportation is provided, which may specifically include an autonomous vehicle. The means of transportation includes the aforementioned computer equipment and can perform the steps in the above-described undefined category obstacle method embodiment.
[0104] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a non-volatile computer-readable storage medium. When executed, these computer-readable instructions can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0105] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0106] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A method for detecting undefined category obstacles, comprising: Acquire current frame point cloud data, historical obstacle detection data, and map data; The current frame point cloud data contains areas of obstacles with undefined categories; The historical obstacle detection data refers to the area where each undefined category obstacle is located, detected based on historical frame point cloud data. The map data refers to the information of permanent obstacles generated offline. The permanent obstacle refers to a fixed obstacle whose location and area remain unchanged. The undefined category refers to other categories that are not marked in the obstacle category labeling. The current frame point cloud data is rasterized to obtain an initial raster map; The initial grid map is input into the detection model, and the target detection results and obstacle segmentation results corresponding to the obstacle regions with undefined categories are output. The target detection results include the category of the defined obstacle and the area where it is located; Ground segmentation is performed on the current frame point cloud data to determine the ground point cloud data; The target grid map is determined based on the target detection results, the obstacle segmentation results, the ground point cloud data, the historical obstacle detection data, the map data, and the initial grid map; and Clustering the target grid map yields undefined category obstacle detection results; The step of determining the target raster map based on the target detection result, the obstacle segmentation result, the ground point cloud data, the historical obstacle detection data, the map data, and the initial raster map includes: Remove defined category obstacles, ground point cloud data, and fixed obstacles from the target detection results in the initial grid map to obtain the grid map after removal processing; Extract obstacle occupancy information from the raster map after the removal process, whereby obstacle occupancy information refers to the probability that each raster contains an undefined category obstacle; and The obstacle occupancy information, the obstacle segmentation results, and the historical obstacle detection data are fused to obtain the target grid map.
2. The method according to claim 1, characterized in that, The step of inputting the initial grid map into the detection model and outputting the target detection result and the obstacle segmentation result corresponding to the obstacle region with undefined category includes: The initial grid map is input into the detection model, which includes a parallel target detection network and an obstacle segmentation network; The target detection network is used to perform target detection on the initial raster map, and the target detection results are output; and The obstacle segmentation network is used to segment undefined obstacle regions in the initial grid map to obtain obstacle segmentation results.
3. The method according to claim 1, characterized in that, The step of performing ground segmentation on the current frame point cloud data to determine the ground point cloud data includes: The point cloud region corresponding to the current frame point cloud data is divided into multiple grids; Calculate the ground corresponding to each grid cell based on the preset plane equation; and Points in each grid whose distance to the corresponding ground is less than a distance threshold are identified as ground point cloud data.
4. The method according to claim 3, characterized in that, The step of calculating the ground corresponding to each grid cell based on a preset plane equation includes: Select the point with the smallest height value from the points corresponding to each grid; Calculate the height difference between each point in each grid and the point with the smallest height value; Extract points whose height difference is less than the height difference threshold; The selected points are fitted with a plane according to a preset plane equation to obtain the ground corresponding to each grid.
5. The method according to claim 1, characterized in that, The extraction of obstacle occupancy information from the removed raster map includes: Predict the probability that each grid cell in the removed grid map contains an obstacle, and obtain obstacle occupancy information based on the probability.
6. The method according to claim 1, characterized in that, The clustering of the target grid map to obtain undefined category obstacle detection results includes: Determine the occupied area in the target grid map; Connectivity detection is performed on the occupied region to obtain multiple connected components and a set of points corresponding to each connected component; and The bounding box of the corresponding undefined category obstacle is calculated based on the point set corresponding to each connected component, and the detection result of the undefined category obstacle is obtained.
7. A detection device for undefined category obstacles, comprising: The data acquisition module is used to acquire current frame point cloud data, historical obstacle detection data, and map data; The current frame point cloud data is marked with obstacle areas of undefined categories; the historical obstacle detection data refers to the area where each undefined category obstacle is located, detected based on the historical frame point cloud data; the map data refers to the information of permanent obstacles generated offline; the permanent obstacles refer to fixed obstacles whose location and area remain unchanged; and the undefined categories refer to other categories that are not marked in the obstacle category labeling. The rasterization processing module is used to rasterize the current frame point cloud data to obtain an initial raster map; The detection module is used to input the initial grid map into the detection model and output the target detection results and the obstacle segmentation results corresponding to the obstacle areas with undefined categories. The target detection results include the category of the defined obstacle and the area where it is located; The ground segmentation module is used to perform ground segmentation on the current frame point cloud data to determine the ground point cloud data. The determination module is used to determine the target raster map based on the target detection result, the obstacle segmentation result, the ground point cloud data, the historical obstacle detection data, the map data, and the initial raster map; and The clustering module is used to cluster the target raster map to obtain the detection results of undefined category obstacles; The determining module is further configured to remove defined category obstacles, ground point cloud data, and fixed obstacles from the target detection results in the initial grid map to obtain a grid map after removal processing; extract obstacle occupancy information from the grid map after removal processing, wherein the obstacle occupancy information refers to the probability that each grid cell contains an undefined category obstacle; and fuse the obstacle occupancy information, the obstacle segmentation results, and the historical obstacle detection data to obtain a target grid map.
8. The apparatus according to claim 7, characterized in that, The detection module is used to input the initial grid map into the detection model, which includes a parallel target detection network and an obstacle segmentation network; to perform target detection on the initial grid map through the target detection network and output the target detection result; and to segment the obstacle regions in the initial grid map that have no defined category through the obstacle segmentation network to obtain the obstacle segmentation result.
9. The apparatus according to claim 7, characterized in that, The ground segmentation module is also used to divide the point cloud region corresponding to the current frame point cloud data into multiple grids; calculate the ground corresponding to each grid according to a preset plane equation; and determine the points in each grid whose distance value to the corresponding ground is less than a distance threshold as ground point cloud data.
10. The apparatus according to claim 9, characterized in that, The ground segmentation module is also used to select the point with the smallest height value among the points corresponding to each grid; calculate the height difference between each point in each grid and the point with the smallest height value; extract the points whose height difference is less than the height difference threshold; and perform plane fitting on the selected points according to the preset plane equation to obtain the ground corresponding to each grid.
11. A computer device comprising a memory and one or more processors, the memory storing computer-readable instructions that, when executed by the one or more processors, cause the one or more processors to perform the following steps: The system acquires current frame point cloud data, historical obstacle detection data, and map data. The current frame point cloud data includes areas marked with undefined obstacle categories. The historical obstacle detection data refers to the areas where each undefined obstacle category is located, detected based on the historical frame point cloud data. The map data refers to information on permanent obstacles generated offline. Permanent obstacles refer to fixed obstacles whose locations remain unchanged. Undefined categories refer to other categories that are not marked in the obstacle category labeling. The current frame point cloud data is rasterized to obtain an initial raster map; The initial grid map is input into the detection model, and the target detection results and obstacle segmentation results corresponding to the obstacle regions with undefined categories are output. Ground segmentation is performed on the current frame point cloud data to determine the ground point cloud data; The target grid map is determined based on the target detection results, the obstacle segmentation results, the ground point cloud data, the historical obstacle detection data, the map data, and the initial grid map; The target detection results include the category of the defined obstacle and the area where it is located; and Clustering the target grid map yields undefined category obstacle detection results; The step of determining the target raster map based on the target detection result, the obstacle segmentation result, the ground point cloud data, the historical obstacle detection data, the map data, and the initial raster map includes: Remove defined category obstacles, ground point cloud data, and fixed obstacles from the target detection results in the initial grid map to obtain the grid map after removal processing; Extract obstacle occupancy information from the raster map after the removal process, whereby obstacle occupancy information refers to the probability that each raster contains an undefined category obstacle; and The obstacle occupancy information, the obstacle segmentation results, and the historical obstacle detection data are fused to obtain the target grid map.
12. The computer device according to claim 11, characterized in that, When the processor executes the computer-readable instructions, it further performs the following steps: inputting the initial grid map into a detection model, the detection model including a parallel target detection network and an obstacle segmentation network; performing target detection on the initial grid map through the target detection network and outputting target detection results; and segmenting undefined obstacle regions in the initial grid map through the obstacle segmentation network to obtain obstacle segmentation results.
13. The computer device according to claim 11, characterized in that, When the processor executes the computer-readable instructions, it also performs the following steps: dividing the point cloud region corresponding to the current frame point cloud data into multiple grids; calculating the ground corresponding to each grid according to a preset plane equation; and determining the points in each grid whose distance value to the corresponding ground is less than a distance threshold as ground point cloud data.
14. The computer device according to claim 13, characterized in that, When the processor executes the computer-readable instructions, it also performs the following steps: selecting the point with the smallest height value among the points corresponding to each grid; calculating the height difference between each point in each grid and the point with the smallest height value; extracting the points whose height difference is less than a height difference threshold; and performing plane fitting on the selected points according to a preset plane equation to obtain the ground corresponding to each grid.
15. The computer device according to claim 11, characterized in that, When the processor executes the computer-readable instructions, it also performs the following steps: predicting the probability that each grid cell in the de-processed grid map contains an obstacle, and obtaining obstacle occupancy information based on the probability.
16. One or more computer storage media storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the following steps: The system acquires current frame point cloud data, historical obstacle detection data, and map data. The current frame point cloud data includes areas marked with undefined obstacle categories. The historical obstacle detection data refers to the areas where each undefined obstacle category is located, detected based on the historical frame point cloud data. The map data refers to information on permanent obstacles generated offline. Permanent obstacles refer to fixed obstacles whose locations remain unchanged. Undefined categories refer to other categories that are not marked in the obstacle category labeling. The current frame point cloud data is rasterized to obtain an initial raster map; The initial grid map is input into the detection model, and the target detection results and obstacle segmentation results corresponding to the obstacle regions with undefined categories are output. The target detection results include the category of the defined obstacle and the area where it is located; Ground segmentation is performed on the current frame point cloud data to determine the ground point cloud data; The target grid map is determined based on the target detection results, the obstacle segmentation results, the ground point cloud data, the historical obstacle detection data, the map data, and the initial grid map; and Clustering the target grid map yields undefined category obstacle detection results; The step of determining the target raster map based on the target detection result, the obstacle segmentation result, the ground point cloud data, the historical obstacle detection data, the map data, and the initial raster map includes: Remove defined category obstacles, ground point cloud data, and fixed obstacles from the target detection results in the initial grid map to obtain the grid map after removal processing; Extract obstacle occupancy information from the raster map after the removal process, whereby obstacle occupancy information refers to the probability that each raster contains an undefined category obstacle; and The obstacle occupancy information, the obstacle segmentation results, and the historical obstacle detection data are fused to obtain the target grid map.
17. The storage medium according to claim 16, characterized in that, When the computer-readable instructions are executed by the processor, the following steps are also performed: inputting the initial grid map into a detection model, the detection model including a parallel object detection network and an obstacle segmentation network; The target detection network is used to detect targets in the initial grid map and output the target detection results; and the obstacle segmentation network is used to segment the obstacle regions in the initial grid map that have no defined categories to obtain the obstacle segmentation results.
18. The storage medium according to claim 16, characterized in that, When the computer-readable instructions are executed by the processor, the following steps are also performed: dividing the point cloud region corresponding to the current frame point cloud data into multiple grids; calculating the ground corresponding to each grid according to a preset plane equation; and determining the points in each grid whose distance value to the corresponding ground is less than a distance threshold as ground point cloud data.
19. The storage medium according to claim 18, characterized in that, When the computer-readable instructions are executed by the processor, the following steps are also performed: selecting the point with the smallest height value among the points corresponding to each grid; calculating the height difference between each point in each grid and the point with the smallest height value; extracting the points whose height difference is less than the height difference threshold; and performing plane fitting on the selected points according to a preset plane equation to obtain the ground corresponding to each grid.
20. A means of transport comprising a method for detecting undefined categories of obstacles according to any one of claims 1-6.