Obstacle detection method and device, storage medium and computer equipment
Through the improvement of obstacle detection in parking scenes, the reference frame point cloud and coordinate conversion of multi-frame timing point clouds are used to generate bird's-eye view and extract point cloud features, identify and fuse the foreground grid, and judge obstacle types, the problem of inaccurate dynamic obstacle recognition in traditional detection solutions is solved, and higher accuracy of obstacle recognition and type judgment is achieved.
Patent Information
- Application Number
- CN202311088484.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-28
- Publication Date
- 2025-02-28
AI Technical Summary
In parking scenarios, due to small space, many blind spots, weak echoes, traditional obstacle detection scheme based on nearby time detection results can easily identify dynamic obstacles as static obstacles, resulting in poor recognition effects and even difficult to enter the warehouse.
By determining the reference frame point cloud in the multi-frame timing point cloud and performing coordinate conversion on other frame point clouds, a bird's-eye view of the target point clouds is generated for each frame, point cloud features are extracted, foreground grids are identified and predicted raster velocities are predicted, foreground grids are fused foreground grids of multi-frame bird's-eye view, target foreground objects are determined, and obstacle types are judged based on their speed.
It effectively reduces the interference of unrelated obstacles such as roofs to the detection results, improves the accuracy of obstacle identification and obstacle type judgment, and ensures that the vehicle can accurately identify and avoid dynamic obstacles and successfully enter the warehouse.
Smart Images
Figure CN119540903A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of obstacle detection, and in particular to an obstacle detection method and device, a storage medium, and a computer device. Background Art
[0002] LiDAR (laser radar) is an indispensable sensor in the current L3 / L4 autonomous driving system. LiDAR sensors provide high-resolution data by emitting thousands of laser signals. These lasers are reflected back to the sensor by the object, and the distance of the object can be determined by calculating the time required for the signal to return. The perception algorithm based on LiDAR point cloud has become one of the focuses of autonomous driving research and development in recent years.
[0003] Taking parking scenarios as an example, due to the small space, many blind spots, weak echoes and other reasons, traditional obstacle detection solutions based on nearby time detection results tend to identify dynamic obstacles as static obstacles, which in turn easily leads to many ghost images in the space, poor recognition effect, and even makes it difficult for vehicles to enter the parking lot. Summary of the invention
[0004] In view of this, the present application provides an obstacle detection method and apparatus, a storage medium, and a computer device, which help to improve the accuracy of obstacle recognition and obstacle type judgment.
[0005] According to one aspect of the present application, a method for obstacle detection is provided, the method comprising:
[0006] Determine a reference frame point cloud in the multi-frame time-series point cloud collected by the current vehicle, and perform coordinate conversion on other frame point clouds to the reference frame point cloud to obtain a multi-frame target point cloud in the coordinate system of the reference frame point cloud, wherein the other frame point cloud is the part of the multi-frame time-series point cloud excluding the reference frame point cloud;
[0007] Generate a bird's-eye view corresponding to each frame of the target point cloud, and determine the point cloud features corresponding to each frame of the bird's-eye view;
[0008] identifying foreground grids in the bird's-eye view based on the point cloud features, and predicting the speed of each grid corresponding to the bird's-eye view based on the point cloud features;
[0009] The foreground grids of the multiple frames of the bird's-eye view are fused to determine at least one target foreground object, the foreground object speed corresponding to the target foreground object is determined based on the grid speeds corresponding to the bird's-eye view, and the obstacle type of each target foreground object is determined based on the foreground object speed.
[0010] Optionally, generating a bird's-eye view corresponding to each frame of the target point cloud, and determining point cloud features corresponding to each frame of the bird's-eye view, includes:
[0011] Down-sampling the target point cloud of each frame based on voxels in the three-dimensional space, projecting the sampled voxels onto a horizontal plane to obtain the bird's-eye view, and counting the voxel projection data corresponding to each pixel point in the bird's-eye view as the pixel parameters of the bird's-eye view, wherein the pixel parameters include the maximum height of the voxel corresponding to each pixel point, the point cloud intensity value corresponding to the maximum height of the voxel, the average intensity value of the voxel, and the number of point clouds;
[0012] The pixel parameters of the bird's-eye view are subjected to feature extraction through a feature extraction model to obtain point cloud features corresponding to the bird's-eye view.
[0013] Optionally, fusing foreground grids of multiple frames of the bird's-eye view to determine at least one target foreground object comprises:
[0014] Determine a target bird's-eye view among the multiple frames of the bird's-eye view, and fuse foreground grids in other bird's-eye views into the target bird's-eye view to obtain a target bird's-eye view including all foreground grids in the multiple frames of the bird's-eye view;
[0015] At least one target foreground object is determined based on the foreground grid contained in the target bird's-eye view.
[0016] Optionally, the determining at least one target foreground object based on the foreground grid contained in the target bird's-eye view includes:
[0017] Clustering the foreground grids contained in the target bird's-eye view, and determining at least one first target foreground object based on the foreground grids in each cluster, wherein the target foreground objects include the first target foreground object;
[0018] The foreground grids that do not belong to any category are regarded as disordered grids, and contour recognition is performed on the disordered grids. At least one second target foreground object is determined based on the recognized disordered point contours, wherein the target foreground objects include the second target foreground object.
[0019] Optionally, after determining the point cloud features corresponding to each frame of the bird's-eye view, the method further includes:
[0020] Predicting the moving speed of the current vehicle based on the point cloud features;
[0021] Accordingly, determining the foreground object speed corresponding to the target foreground object based on the grid speeds corresponding to the bird's-eye view, and determining the obstacle type of each target foreground object based on the foreground object speed, includes:
[0022] Acquire a target grid speed of a grid corresponding to the target foreground object based on each grid speed corresponding to the bird's-eye view, and determine a foreground object speed of the target foreground object based on the target grid speed;
[0023] The absolute speed of each of the target foreground objects is determined according to the speed of each of the foreground objects and the moving speed of the current vehicle, and the obstacle type of each of the target foreground objects is determined based on the absolute speed.
[0024] Optionally, determining a reference frame point cloud in a multi-frame time-sequential point cloud collected by the current vehicle includes:
[0025] Acquire the real-time point cloud collected by the current vehicle, and whenever the real-time point cloud meets a preset condition, divide the multiple frames of real-time point cloud meeting the preset condition into a time-series point cloud group, wherein the preset condition includes a preset frame number condition or a preset time length condition;
[0026] Determine a reference frame point cloud among the multiple frames of sequential point clouds in the sequential point cloud group;
[0027] Accordingly, after determining the foreground object speed of the target foreground object based on the target grid speed, the method further includes:
[0028] A foreground object calculated based on a historical time-series point cloud group is obtained as a reference foreground object, and a foreground object speed of the target foreground object is calibrated based on a reference foreground object speed corresponding to the reference foreground object.
[0029] Optionally, the method further comprises:
[0030] The target bird's-eye view is displayed, and the target foreground object in the target bird's-eye view is framed and selected, and the target foreground objects of different obstacle types are displayed separately.
[0031] According to another aspect of the present application, there is provided an obstacle detection device, the device comprising:
[0032] A point cloud acquisition module, used to determine a reference frame point cloud in the multi-frame time-series point cloud collected by the current vehicle, and to perform coordinate conversion on other frame point clouds to the reference frame point cloud, so as to obtain a multi-frame target point cloud in the coordinate system of the reference frame point cloud, wherein the other frame point cloud is the part of the multi-frame time-series point cloud excluding the reference frame point cloud;
[0033] A feature determination module, used to generate a bird's-eye view corresponding to each frame of the target point cloud, and determine the point cloud features corresponding to each frame of the bird's-eye view;
[0034] A prediction module, configured to identify foreground grids in the bird's-eye view based on the point cloud features, and predict the speed of each grid corresponding to the bird's-eye view based on the point cloud features;
[0035] The obstacle recognition module is used to fuse the foreground grids of multiple frames of the bird's-eye view to determine at least one target foreground object, determine the foreground object speed corresponding to the target foreground object based on the grid speeds corresponding to the bird's-eye view, and determine the obstacle type of each target foreground object based on the foreground object speed.
[0036] Optionally, the feature determination module is further used to:
[0037] Down-sampling the target point cloud of each frame based on voxels in the three-dimensional space, projecting the sampled voxels onto a horizontal plane to obtain the bird's-eye view, and counting the voxel projection data corresponding to each pixel point in the bird's-eye view as the pixel parameters of the bird's-eye view, wherein the pixel parameters include the maximum height of the voxel corresponding to each pixel point, the point cloud intensity value corresponding to the maximum height of the voxel, the average intensity value of the voxel, and the number of point clouds;
[0038] The pixel parameters of the bird's-eye view are subjected to feature extraction through a feature extraction model to obtain point cloud features corresponding to the bird's-eye view.
[0039] Optionally, the obstacle recognition module is further used to:
[0040] Determine a target bird's-eye view among the multiple frames of the bird's-eye view, and fuse foreground grids in other bird's-eye views into the target bird's-eye view to obtain a target bird's-eye view including all foreground grids in the multiple frames of the bird's-eye view;
[0041] At least one target foreground object is determined based on the foreground grid contained in the target bird's-eye view.
[0042] Optionally, the obstacle recognition module is further used to:
[0043] Clustering the foreground grids contained in the target bird's-eye view, and determining at least one first target foreground object based on the foreground grids in each cluster, wherein the target foreground objects include the first target foreground object;
[0044] The foreground grids that do not belong to any category are regarded as disordered grids, and contour recognition is performed on the disordered grids. At least one second target foreground object is determined based on the recognized disordered point contours, wherein the target foreground objects include the second target foreground object.
[0045] Optionally, the prediction module is further used to predict the moving speed of the current vehicle based on the point cloud features;
[0046] Accordingly, the obstacle recognition module is further used for:
[0047] Acquire a target grid speed of a grid corresponding to the target foreground object based on each grid speed corresponding to the bird's-eye view, and determine a foreground object speed of the target foreground object based on the target grid speed;
[0048] The absolute speed of each of the target foreground objects is determined according to the speed of each of the foreground objects and the moving speed of the current vehicle, and the obstacle type of each of the target foreground objects is determined based on the absolute speed.
[0049] Optionally, the point cloud acquisition module is further used to:
[0050] Acquire the real-time point cloud collected by the current vehicle, and whenever the real-time point cloud meets a preset condition, divide the multiple frames of real-time point cloud meeting the preset condition into a time-series point cloud group, wherein the preset condition includes a preset frame number condition or a preset time length condition;
[0051] Determine a reference frame point cloud among the multiple frames of sequential point clouds in the sequential point cloud group;
[0052] Accordingly, the obstacle recognition module is further used for:
[0053] A foreground object calculated based on a historical time-series point cloud group is obtained as a reference foreground object, and a foreground object speed of the target foreground object is calibrated based on a reference foreground object speed corresponding to the reference foreground object.
[0054] Optionally, the device further comprises:
[0055] The display module is used to display the target bird's-eye view, select the target foreground object in the target bird's-eye view, and distinguish and display the target foreground objects of different obstacle types.
[0056] According to another aspect of the present application, a storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above obstacle detection method is implemented.
[0057] According to another aspect of the present application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the above-mentioned obstacle detection method when executing the program.
[0058] By means of the above technical solution, the present application provides an obstacle detection method and device, storage medium, and computer equipment, which obtain multi-frame time-series point clouds and convert the multi-frame time-series point clouds into multi-frame target point clouds in the same coordinate system, thereby using the target point cloud to generate a bird's-eye view and extract point cloud features of the bird's-eye view, identify the foreground grid in the bird's-eye view based on the point cloud features and predict the grid speed, and then fuse the foreground grids of the multi-frame bird's-eye view to determine at least one target foreground object, and determine the obstacle type according to the speed of the target foreground object. The present application is based on a data-driven approach, by generating a bird's-eye view and using the bird's-eye view to extract point cloud features for obstacle detection, which can effectively reduce the interference of irrelevant obstacles such as roofs on the detection results and improve the accuracy of obstacle identification and obstacle type judgment.
[0059] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0061] Figure 1 A schematic diagram of a process of an obstacle detection method provided in an embodiment of the present application is shown;
[0062] Figure 2 A schematic diagram of a process flow of another obstacle detection method provided in an embodiment of the present application is shown;
[0063] Figure 3 A schematic diagram of the structure of an obstacle detection device provided in an embodiment of the present application is shown;
[0064] Figure 4 A schematic diagram of the device structure of a computer device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0065] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.
[0066] In this embodiment, an obstacle detection method is provided. Figure 1 As shown, the method includes:
[0067] Step 101, determine the reference frame point cloud in the multi-frame time-series point cloud collected by the current vehicle, and perform coordinate conversion on other frame point clouds to the reference frame point cloud to obtain a multi-frame target point cloud in the coordinate system of the reference frame point cloud, wherein the other frame point cloud is the part of the multi-frame time-series point cloud excluding the reference frame point cloud.
[0068] The embodiments of the present application are applied to the field of vehicle obstacle detection, using point cloud data collected by the vehicle's lidar to identify different types of obstacles near the vehicle, and are particularly suitable for parking scenarios with small spaces, many blind spots, and weak echoes.
[0069] First, the laser radar of the current vehicle is used to collect data, and a multi-frame time-series point cloud is generated through time-series aggregation. One frame is selected from the continuous multi-frame time-series point cloud as the reference frame point cloud, for example, the last frame in the multi-frame is selected as the reference frame, so that other frame power sources are spatially synchronized with the reference frame point cloud through posture conversion, and all frame point clouds in the coordinate system corresponding to the reference frame point cloud are obtained. In this way, the coordinate system of the multi-frame point cloud is synchronized, and the subsequent processing and fusion of the multi-frame point cloud is more accurate, avoiding the accuracy of obstacle detection affected by different spatial coordinates.
[0070] Step 102: Generate a bird's-eye view corresponding to each frame of the target point cloud, and determine the point cloud features corresponding to each frame of the bird's-eye view.
[0071] Secondly, the target point cloud is converted into BEV (Bird's-Eye View) input to obtain the bird's-eye view corresponding to each frame of the target point cloud, and the point cloud features are extracted using the bird's-eye view of each frame. Taking advantage of the simple structure of the bird's-eye view, the roof can be deducted and the obstacle edge can be directly output using the bird's-eye view.
[0072] Optionally, step 102 includes: down-sampling the target point cloud in each frame based on voxels in three-dimensional space, projecting the sampled voxels onto a horizontal plane to obtain the bird's-eye view, and counting the voxel projection data corresponding to each pixel point in the bird's-eye view as pixel parameters of the bird's-eye view, wherein the pixel parameters include the maximum height of the voxel corresponding to each pixel point, the point cloud intensity value corresponding to the maximum height of the voxel, the average intensity value of the voxel, and the number of point clouds; performing feature extraction on the pixel parameters of the bird's-eye view through a feature extraction model to obtain point cloud features corresponding to the bird's-eye view.
[0073] In this embodiment, voxel refers to dividing the three-dimensional space into cuboids of fixed size, and each cuboid (Δl*Δw*Δh) is a voxel, that is, dividing the three-dimensional space into voxels. For each frame of the target point cloud, the target point cloud is downsampled using voxels, and then each voxel is projected onto the horizontal plane as a point to obtain a bird's-eye view. At the same time, the pixel coordinates of the BEV view can be obtained when the voxel is projected, and the feature value of each pixel can be obtained by a statistical method, which is called hand-crafted feature extraction. The features extracted based on the bird's-eye view are the above-mentioned pixel parameters. The pixel parameters of the pixel points are determined based on all voxels projected onto this pixel point. The pixel parameters of a certain pixel point include the maximum height of the voxels corresponding to all voxels projected onto this pixel point, the point cloud intensity value of the point cloud contained in the maximum height voxel, the average intensity value of the voxels containing the point cloud of all voxels, and the number of point clouds contained in all voxels. Then the pixel parameters of the bird's-eye view are input into the pre-trained feature extraction model for point cloud feature extraction, so as to perform feature enhancement to obtain point cloud features. Among them, the general backbone network backbone model is selected to extract point cloud features of lidar.
[0074] Step 103: identifying foreground grids in the bird's-eye view based on the point cloud features, and predicting the speed of each grid corresponding to the bird's-eye view based on the point cloud features.
[0075] Then, using the pre-designed detection task, after extracting the point cloud features, the point cloud features are used to identify the foreground and background of the bird's-eye view. Specifically, the point cloud features can be input into the pre-trained recognition model to determine whether each grid in the bird's-eye view is a foreground grid or a background grid according to the output of the model. In addition, the pre-designed detection task includes not only the grid unit classification task for foreground grid identification mentioned above, but also a speed prediction task. After obtaining the point cloud features, the point cloud features can also be used to predict the speed of each grid in the bird's-eye view. Specifically, the point cloud features can be input into the pre-trained speed prediction model to determine the speed of each grid in the bird's-eye view according to the output of the model.
[0076] Step 104, fusing foreground grids of multiple frames of the bird's-eye view to determine at least one target foreground object, determining a foreground object speed corresponding to the target foreground object based on the grid speeds corresponding to the bird's-eye view, and determining an obstacle type of each target foreground object based on the foreground object speed.
[0077] Finally, after obtaining the foreground grids and grid speeds in each frame of the bird's-eye view, the comprehensiveness and accuracy of foreground object recognition are improved. The foreground grids in each frame of the bird's-eye view can also be fused, and the fused foreground grids can be determined as at least one target foreground object by clustering the foreground grid fusion results. The grid speed of the grid covered by the target foreground object is obtained from the predicted bird's-eye view grid speed data, so as to determine the foreground object speed of the target foreground object according to the grid speed corresponding to the target foreground object, for example, the average or median value of the grid speed corresponding to the target foreground object is taken as the foreground object speed. Finally, the obstacle type corresponding to each target foreground object is determined according to the foreground object speed. If the speed of the target foreground object relative to the ground is zero, then the target foreground object is determined as a static obstacle, otherwise it is determined as a dynamic obstacle.
[0078] By applying the technical solution of this embodiment, a multi-frame time-series point cloud is obtained, and the multi-frame time-series point cloud is converted into a multi-frame target point cloud in the same coordinate system, so as to generate a bird's-eye view using the target point cloud and extract the point cloud features of the bird's-eye view, identify the foreground grid in the bird's-eye view based on the point cloud features and predict the grid speed, and then fuse the foreground grids of the multi-frame bird's-eye view to determine at least one target foreground object, and determine the obstacle type according to the speed of the target foreground object. This application is based on a data-driven approach, by generating a bird's-eye view, and using the bird's-eye view to extract point cloud features for obstacle detection, which can effectively reduce the interference of irrelevant obstacles such as roofs on the detection results, and improve the accuracy of obstacle identification and obstacle type judgment.
[0079] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to fully illustrate the specific implementation process of this embodiment, another obstacle detection method is provided, such as Figure 2 As shown, the method includes:
[0080] Step 201, obtain the real-time point cloud collected by the current vehicle, and whenever the real-time point cloud meets the preset conditions, divide the multiple frames of real-time point cloud that meet the preset conditions into a time-series point cloud group, wherein the preset conditions include preset frame number conditions or preset time length conditions.
[0081] In the embodiment of the present application, during the data collection process by the laser radar of the current vehicle, each time a certain number of real-time point clouds or a certain length of real-time point clouds are obtained, this part of the real-time point clouds is regarded as a time-series point cloud group. Therefore, each time a time-series point cloud group is obtained, the following steps are performed for the multiple frames of time-series point clouds in the time-series point cloud group.
[0082] Step 202, determine the reference frame point cloud in the multi-frame time-series point cloud in the time-series point cloud group; perform coordinate conversion on other frame point clouds to the reference frame point cloud to obtain multi-frame target point clouds in the coordinate system of the reference frame point cloud, wherein the other frame point clouds are the parts of the multi-frame time-series point clouds excluding the reference frame point cloud.
[0083] Step 203 , generating a bird's-eye view corresponding to each frame of the target point cloud, and determining point cloud features corresponding to each frame of the bird's-eye view.
[0084] Step 204: identifying foreground grids in the bird's-eye view based on the point cloud features, and predicting the speed of each grid corresponding to the bird's-eye view based on the point cloud features; and predicting the moving speed of the current vehicle based on the point cloud features.
[0085] Step 205 , determining a target bird's-eye view in the multiple frames of the bird's-eye view, and fusing foreground grids in other bird's-eye views into the target bird's-eye view, to obtain a target bird's-eye view including all foreground grids in the multiple frames of the bird's-eye view.
[0086] In an embodiment of the present application, each time a time-series point cloud group is obtained, a reference frame point cloud is selected from the multiple frames of the time-series point cloud group, and the coordinates of the other frame point clouds are converted to the reference frame point cloud so that all the frame point clouds in the group maintain a consistent coordinate system, and then a bird's-eye view corresponding to the target point cloud of each frame is generated and point cloud features are extracted based on the bird's-eye view. Next, foreground grid recognition, grid speed prediction, and current vehicle moving speed prediction are performed based on the point cloud features. Further, after determining the foreground grids of each frame of the bird's-eye view, any one of the frames can be selected as the target bird's-eye view, and the foreground grids in other bird's-eye views can be fused into the target bird's-eye view, so that the target bird's-eye view contains the foreground grids identified in all the bird's-eye views. For example, if a grid in the target bird's-eye view does not belong to the foreground grid, but the grid in other bird's-eye views belongs to the foreground grid, the foreground grid in other bird's-eye views will be fused into the target bird's-eye view.
[0087] Specifically, the pre-trained recognition model can be used to predict the grid intensity based on the point cloud features and classify the grid as foreground or background based on the intensity prediction value. In addition, after the foreground grids corresponding to each frame of the bird's-eye view are subsequently fused, the loss value corresponding to the recognition model can be calculated based on the fused foreground grid fusion intensity value and the foreground grid intensity prediction value corresponding to each frame of the bird's-eye view through the loss function of the recognition model, and the recognition model can be optimized when the loss value is greater than the preset value, so as to achieve dynamic optimization of the recognition model and improve the model accuracy. Similarly, the speed of each grid can be predicted through the pre-trained speed prediction model, and the loss value corresponding to the speed prediction model can be calculated based on the fused foreground grid fusion speed and the foreground grid speed corresponding to each frame of the bird's-eye view through the loss function of the Sun Dan prediction model, and the speed prediction model can be optimized when the loss value is greater than the preset value, so as to achieve dynamic optimization of the speed prediction model and improve the model accuracy.
[0088] Step 206 , clustering the foreground grids included in the target bird's-eye view, and determining at least one first target foreground object based on the foreground grids in each cluster, wherein the target foreground objects include the first target foreground object.
[0089] Step 207: treating foreground grids that do not belong to any category as unordered grids, and performing contour recognition on the unordered grids, and determining at least one second target foreground object based on the recognized unordered point contours, wherein the target foreground objects include the second target foreground object.
[0090] Furthermore, after the foreground grids of all bird's-eye views are fused into the target bird's-eye view, the foreground grids can be divided into target foreground objects by clustering the foreground grids. Specifically, the foreground grids contained in each cluster can be regarded as a target foreground object, i.e., the first target foreground object. In addition, the foreground grids may also contain some disordered points, which may be small obstacles. These disordered points cannot be aggregated into one class by clustering. At this time, contour recognition can be used to perform contour recognition on these disordered grids that do not belong to any class. For example, the alpha shapes boundary point extraction algorithm is used to perform contour recognition on the disordered grids, and the identified disordered point contours are used to determine another part of the target foreground objects, i.e., the second target foreground objects. Specifically, each identified contour can be regarded as a second target foreground object, or the disordered point contours that meet certain rules can be set as a second target foreground object.
[0091] Step 208: acquiring a target grid speed of a grid corresponding to the target foreground object based on the grid speeds corresponding to the bird's-eye view, and determining a foreground object speed of the target foreground object based on the target grid speed.
[0092] After determining the target foreground object, the target grid speed of the grid covered by the target foreground object is obtained based on the grid speed of the bird's-eye view predicted above. For the foreground grid that belongs to the target bird's-eye view, the grid speed predicted for the target bird's-eye view can be directly selected. For the foreground grid that does not belong to the target bird's-eye view, it can be determined in which other bird's-eye view the grid belongs to the foreground grid, and then the grid speed predicted for the other bird's-eye view can be obtained. After obtaining the target grid speed of the grid covered by the target foreground object, the foreground object speed of the target foreground object is calculated based on the target grid speed. Specifically, the foreground object speed can be determined by taking the mean or median.
[0093] In the embodiment of the present application, optionally, it also includes: obtaining a foreground object calculated based on a historical time-series point cloud group as a reference foreground object, and calibrating the foreground object speed of the target foreground object based on the reference foreground object speed corresponding to the reference foreground object.
[0094] In this embodiment, if the above calculation process is for the first sequential point cloud group, then directly enter step 209 to classify the target foreground object into obstacle types using the calculated foreground object speed. If it is not the first sequential point cloud group, the target foreground object calculated by the current sequential point cloud group can also be tracked using the target foreground object calculated by the historical sequential point cloud group (i.e., the reference foreground object), so as to calibrate the foreground object speed of the target foreground object. Since adjacent sequential point cloud groups are continuous in time, the speed of the same foreground object will not suddenly change, and calibration can be performed based on this.
[0095] Step 209 , determining the absolute speed of each of the target foreground objects according to the speed of each of the foreground objects and the moving speed of the current vehicle, and determining the obstacle type of each of the target foreground objects based on the absolute speed.
[0096] After determining the speed of the target foreground object, the absolute speed of each target foreground object is calculated based on the current vehicle's moving speed and the foreground object speed of each target foreground object, that is, the speed relative to the stationary ground, so as to judge the obstacle type of each target foreground object according to the absolute speed of each target foreground object. The obstacle with an absolute speed of 0 belongs to a static obstacle, otherwise it belongs to a dynamic obstacle. In this way, the recognition of different types of obstacles can be realized.
[0097] In the embodiment of the present application, optionally, it also includes: displaying the target bird's-eye view, and selecting the target foreground object in the target bird's-eye view, and distinguishing and displaying the target foreground objects of different obstacle types.
[0098] In this embodiment, after different types of obstacles are identified, they can also be displayed. Specifically, a target bird's-eye view can be displayed, and each target foreground object in the target bird's-eye view can be framed, and different types of target foreground objects can be displayed separately. For example, only dynamic obstacles can be framed, and static obstacles can be displayed in special colors, or different types of obstacles can be framed in different colors, so that users can observe obstacles intuitively and better control vehicle parking or driving.
[0099] Further, as Figure 1 The specific implementation of the method, the embodiment of the present application provides an obstacle detection device, such as Figure 3 As shown, the device comprises:
[0100] A point cloud acquisition module, used to determine a reference frame point cloud in the multi-frame time-series point cloud collected by the current vehicle, and to perform coordinate conversion on other frame point clouds to the reference frame point cloud, so as to obtain a multi-frame target point cloud in the coordinate system of the reference frame point cloud, wherein the other frame point cloud is the part of the multi-frame time-series point cloud excluding the reference frame point cloud;
[0101] A feature determination module, used to generate a bird's-eye view corresponding to each frame of the target point cloud, and determine the point cloud features corresponding to each frame of the bird's-eye view;
[0102] A prediction module, configured to identify foreground grids in the bird's-eye view based on the point cloud features, and predict the speed of each grid corresponding to the bird's-eye view based on the point cloud features;
[0103] The obstacle recognition module is used to fuse the foreground grids of multiple frames of the bird's-eye view to determine at least one target foreground object, determine the foreground object speed corresponding to the target foreground object based on the grid speeds corresponding to the bird's-eye view, and determine the obstacle type of each target foreground object based on the foreground object speed.
[0104] Optionally, the feature determination module is further used to:
[0105] Down-sampling the target point cloud of each frame based on voxels in the three-dimensional space, projecting the sampled voxels onto a horizontal plane to obtain the bird's-eye view, and counting the voxel projection data corresponding to each pixel point in the bird's-eye view as the pixel parameters of the bird's-eye view, wherein the pixel parameters include the maximum height of the voxel corresponding to each pixel point, the point cloud intensity value corresponding to the maximum height of the voxel, the average intensity value of the voxel, and the number of point clouds;
[0106] The pixel parameters of the bird's-eye view are subjected to feature extraction through a feature extraction model to obtain point cloud features corresponding to the bird's-eye view.
[0107] Optionally, the obstacle recognition module is further used to:
[0108] Determine a target bird's-eye view among the multiple frames of the bird's-eye view, and fuse foreground grids in other bird's-eye views into the target bird's-eye view to obtain a target bird's-eye view including all foreground grids in the multiple frames of the bird's-eye view;
[0109] At least one target foreground object is determined based on the foreground grid contained in the target bird's-eye view.
[0110] Optionally, the obstacle recognition module is further used to:
[0111] Clustering the foreground grids contained in the target bird's-eye view, and determining at least one first target foreground object based on the foreground grids in each cluster, wherein the target foreground objects include the first target foreground object;
[0112] The foreground grids that do not belong to any category are regarded as disordered grids, and contour recognition is performed on the disordered grids. At least one second target foreground object is determined based on the recognized disordered point contours, wherein the target foreground objects include the second target foreground object.
[0113] Optionally, the prediction module is further used to predict the moving speed of the current vehicle based on the point cloud features;
[0114] Accordingly, the obstacle recognition module is further used for:
[0115] Acquire a target grid speed of a grid corresponding to the target foreground object based on each grid speed corresponding to the bird's-eye view, and determine a foreground object speed of the target foreground object based on the target grid speed;
[0116] The absolute speed of each of the target foreground objects is determined according to the speed of each of the foreground objects and the moving speed of the current vehicle, and the obstacle type of each of the target foreground objects is determined based on the absolute speed.
[0117] Optionally, the point cloud acquisition module is further used to:
[0118] Acquire the real-time point cloud collected by the current vehicle, and whenever the real-time point cloud meets a preset condition, divide the multiple frames of real-time point cloud meeting the preset condition into a time-series point cloud group, wherein the preset condition includes a preset frame number condition or a preset time length condition;
[0119] Determine a reference frame point cloud among the multiple frames of sequential point clouds in the sequential point cloud group;
[0120] Accordingly, the obstacle recognition module is further used for:
[0121] A foreground object calculated based on a historical time-series point cloud group is obtained as a reference foreground object, and a foreground object speed of the target foreground object is calibrated based on a reference foreground object speed corresponding to the reference foreground object.
[0122] Optionally, the device further comprises:
[0123] The display module is used to display the target bird's-eye view, select the target foreground object in the target bird's-eye view, and distinguish and display the target foreground objects of different obstacle types.
[0124] It should be noted that for other corresponding descriptions of the functional units involved in the obstacle detection device provided in the embodiment of the present application, reference can be made to Figure 1 to Figure 2 The corresponding description in the method will not be repeated here.
[0125] The present application also provides a computer device, which may be a personal computer, a server, a network device, etc. Figure 4 As shown, the computer device includes a bus, a processor, a memory and a communication interface, and may also include an input and output interface and a display device. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store location information. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in each method embodiment are implemented.
[0126] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0127] In one embodiment, a computer-readable storage medium is provided. The computer-readable storage medium may be non-volatile or volatile, and stores a computer program thereon. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0128] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0129] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.
[0130] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.
[0131] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0132] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. An obstacle detection method, characterized in that: The method comprises: Determine a reference frame point cloud in the multi-frame time-series point cloud collected by the current vehicle, and perform coordinate conversion on other frame point clouds to the reference frame point cloud to obtain a multi-frame target point cloud in the coordinate system of the reference frame point cloud, wherein the other frame point cloud is the part of the multi-frame time-series point cloud excluding the reference frame point cloud; Generate a bird's-eye view corresponding to each frame of the target point cloud, and determine the point cloud features corresponding to each frame of the bird's-eye view; identifying foreground grids in the bird's-eye view based on the point cloud features, and predicting the speed of each grid corresponding to the bird's-eye view based on the point cloud features; The foreground grids of the multiple frames of the bird's-eye view are fused to determine at least one target foreground object, the foreground object speed corresponding to the target foreground object is determined based on the grid speeds corresponding to the bird's-eye view, and the obstacle type of each target foreground object is determined based on the foreground object speed.
2. The method according to claim 1, characterized in that The generating of the bird's-eye view corresponding to each frame of the target point cloud and determining the point cloud features corresponding to each frame of the bird's-eye view includes: Down-sampling the target point cloud of each frame based on voxels in the three-dimensional space, projecting the sampled voxels onto a horizontal plane to obtain the bird's-eye view, and counting the voxel projection data corresponding to each pixel point in the bird's-eye view as the pixel parameters of the bird's-eye view, wherein the pixel parameters include the maximum height of the voxel corresponding to each pixel point, the point cloud intensity value corresponding to the maximum height of the voxel, the average intensity value of the voxel, and the number of point clouds; The pixel parameters of the bird's-eye view are subjected to feature extraction through a feature extraction model to obtain point cloud features corresponding to the bird's-eye view.
3. The method according to claim 1, characterized in that The step of fusing foreground grids of multiple frames of the bird's-eye view to determine at least one target foreground object comprises: Determine a target bird's-eye view among the multiple frames of the bird's-eye view, and fuse foreground grids in other bird's-eye views into the target bird's-eye view to obtain a target bird's-eye view including all foreground grids in the multiple frames of the bird's-eye view; At least one target foreground object is determined based on the foreground grid contained in the target bird's-eye view.
4. The method according to claim 3, characterized in that The determining at least one target foreground object based on the foreground grid contained in the target bird's-eye view comprises: Clustering the foreground grids contained in the target bird's-eye view, and determining at least one first target foreground object based on the foreground grids in each cluster, wherein the target foreground objects include the first target foreground object; The foreground grids that do not belong to any category are regarded as disordered grids, and contour recognition is performed on the disordered grids. At least one second target foreground object is determined based on the recognized disordered point contours, wherein the target foreground objects include the second target foreground object.
5. The method according to any one of claims 1 to 4, characterized in that After determining the point cloud features corresponding to each frame of the bird's-eye view, the method further includes: Predicting the moving speed of the current vehicle based on the point cloud features; Accordingly, the determining of the foreground object speed corresponding to the target foreground object based on the grid speeds corresponding to the bird's-eye view, and determining the obstacle type of each target foreground object based on the foreground object speed, includes: Acquire a target grid speed of a grid corresponding to the target foreground object based on each grid speed corresponding to the bird's-eye view, and determine a foreground object speed of the target foreground object based on the target grid speed; The absolute speed of each of the target foreground objects is determined according to the speed of each of the foreground objects and the moving speed of the current vehicle, and the obstacle type of each of the target foreground objects is determined based on the absolute speed.
6. The method according to claim 5, characterized in that The step of determining a reference frame point cloud in a multi-frame sequential point cloud collected by the current vehicle includes: Acquire the real-time point cloud collected by the current vehicle, and whenever the real-time point cloud meets a preset condition, divide the multiple frames of real-time point cloud meeting the preset condition into a time-series point cloud group, wherein the preset condition includes a preset frame number condition or a preset time length condition; Determine a reference frame point cloud among the multiple frames of sequential point clouds in the sequential point cloud group; Accordingly, after determining the foreground object speed of the target foreground object based on the target grid speed, the method further includes: A foreground object calculated based on a historical time-series point cloud group is obtained as a reference foreground object, and a foreground object speed of the target foreground object is calibrated based on a reference foreground object speed corresponding to the reference foreground object.
7. The method according to claim 6, characterized in that The method further comprises: The target bird's-eye view is displayed, and the target foreground object in the target bird's-eye view is framed and selected, and the target foreground objects of different obstacle types are displayed separately.
8. An obstacle detection device, characterized in that: The device comprises: A point cloud acquisition module, used to determine a reference frame point cloud in the multi-frame time-series point cloud collected by the current vehicle, and to perform coordinate conversion on other frame point clouds to the reference frame point cloud, so as to obtain a multi-frame target point cloud in the coordinate system of the reference frame point cloud, wherein the other frame point cloud is the part of the multi-frame time-series point cloud excluding the reference frame point cloud; A feature determination module, used to generate a bird's-eye view corresponding to each frame of the target point cloud, and determine the point cloud features corresponding to each frame of the bird's-eye view; A prediction module, configured to identify foreground grids in the bird's-eye view based on the point cloud features, and predict the speed of each grid corresponding to the bird's-eye view based on the point cloud features; The obstacle recognition module is used to fuse the foreground grids of multiple frames of the bird's-eye view to determine at least one target foreground object, determine the foreground object speed corresponding to the target foreground object based on the grid speeds corresponding to the bird's-eye view, and determine the obstacle type of each target foreground object based on the foreground object speed.
9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
10. A computer device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.