Obstacle detection method, device, equipment and medium

Abstract

The invention discloses an obstacle detection method, device and equipment and a medium, and relates to the technical field of obstacle detection, and the method comprises the steps: obtaining a three-dimensional point cloud of the surrounding environment of a vehicle, and obtaining a first obstacle detection frame obtained through the target detection of the three-dimensional point cloud; performing semantic segmentation on the three-dimensional point cloud to obtain a ground point cloud and at least one foreground obstacle point cloud, and determining a foreground obstacle detection frame corresponding to each foreground obstacle point cloud; determining a second obstacle detection frame from the foreground obstacle detection frames according to the ground point cloud; the second obstacle detection frame is a foreground obstacle detection frame located in the ground area; and fusing the first obstacle detection frame and the second obstacle detection frame to obtain a target obstacle detection frame, and determining an obstacle in the target obstacle detection frame as a target obstacle. According to the invention, the accuracy of obstacle detection is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of obstacle detection, and in particular to an obstacle detection method, device, equipment and medium. Background Art

[0002] The perception system plays a vital role in the autonomous driving system. The perception system can collect data through multiple sensors such as lidar, millimeter-wave radar and cameras, and fuse the disabled data to identify obstacles.

[0003] At present, the existing technology mainly uses LiDAR as the source of perception data, and usually adopts the method of target detection and geometric clustering to remove the ground point cloud and determine the remaining point cloud as the obstacle point cloud. This existing technology is prone to many false detections, such as trees, splashing water, rain or snow falling in rainy and snowy weather, etc., which can be mistakenly detected as obstacles, thus affecting the vehicle's automatic driving. Automatic driving poses a risk to ground traffic safety. Summary of the invention

[0004] The present invention provides an obstacle detection method, device, equipment and medium to improve the accuracy of detected obstacles.

[0005] In a first aspect, the present invention provides an obstacle detection method, comprising:

[0006] Acquire a three-dimensional point cloud of the vehicle's surroundings, and acquire a first obstacle detection frame obtained by performing target detection on the three-dimensional point cloud;

[0007] Performing semantic segmentation on the three-dimensional point cloud to obtain a ground point cloud and at least one foreground obstacle point cloud, and determining a foreground obstacle detection frame corresponding to each foreground obstacle point cloud;

[0008] According to the ground point cloud, a second obstacle detection frame is determined from each foreground obstacle detection frame; the second obstacle detection frame is a foreground obstacle detection frame located in the ground area;

[0009] The first obstacle detection frame and the second obstacle detection frame are fused to obtain a target obstacle detection frame, and the obstacle in the target obstacle detection frame is determined as the target obstacle.

[0010] In a second aspect, the present invention further provides an obstacle detection device, comprising:

[0011] A point cloud acquisition module, used to acquire a three-dimensional point cloud of the vehicle's surrounding environment and to acquire a first obstacle detection frame obtained by performing target detection on the three-dimensional point cloud;

[0012] A ground point cloud determination module is used to perform semantic segmentation on the three-dimensional point cloud to obtain a ground point cloud and at least one foreground obstacle point cloud, and determine a foreground obstacle detection frame corresponding to each foreground obstacle point cloud;

[0013] A detection frame determination module, used to determine a second obstacle detection frame from each foreground obstacle detection frame according to the ground point cloud; the second obstacle detection frame is a foreground obstacle detection frame located in the ground area;

[0014] The obstacle determination module is used to fuse the first obstacle detection frame and the second obstacle detection frame to obtain a target obstacle detection frame, and determine the obstacle in the target obstacle detection frame as the target obstacle.

[0015] In a third aspect, an embodiment of the present invention further provides an electronic device, including:

[0016] at least one processor; and

[0017] a memory communicatively coupled to at least one processor; wherein

[0018] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can execute the obstacle detection method provided by any embodiment of the present invention.

[0019] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the obstacle detection method of any embodiment of the present invention when executed by a processor.

[0020] The technical solution of the embodiment of the present invention can obtain the detection frame of the foreground obstacle moving on the ground and the ground point cloud by semantically segmenting the point cloud; then determine the detection frame of the foreground obstacle in the ground area through the ground point cloud, so as to fuse the detection frame of the foreground obstacle in the ground area and the detection frame of the target detection obstacle, accurately obtain the moving foreground obstacle in the ground area, and improve the accuracy of the detected obstacles.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0023] Figure 1A is a flow chart of an obstacle detection method provided according to Embodiment 1 of the present invention;

[0024] Figure 1B is a flow chart of another obstacle detection method provided according to Embodiment 2 of the present invention;

[0025] Figure 2 is a flow chart of an obstacle detection method provided according to Embodiment 2 of the present invention;

[0026] Figure 3A is a flow chart of an obstacle detection method provided according to Embodiment 3 of the present invention;

[0027] Figure 3B is a flow chart of determining real semantic information provided according to Embodiment 3 of the present invention;

[0028] Figure 3C is a structural schematic diagram of an obstacle detection model provided according to Embodiment 3 of the present invention;

[0029] Figure 4 is a structural schematic diagram of an obstacle detection device provided according to a fourth embodiment of the present invention;

[0030] Figure 5 Schematic diagram of the structure of an electronic device for implementing the obstacle detection method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first" and "second" and the like in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] In the technical solution of the embodiment of the present invention, the acquisition, storage and application of the three-dimensional point cloud and the first obstacle detection frame involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0034] Embodiment 1

[0035] Figure 1A A flowchart of an obstacle detection method provided in Embodiment 1 of the present invention. This embodiment is applicable to the case of detecting obstacle point clouds on the ground. The method can be performed by an obstacle detection device, which can be implemented in the form of hardware and / or software and specifically configured in an electronic device, such as a server.

[0036] See also Figure 1A The obstacle detection method shown includes:

[0037] S101, obtaining a three-dimensional point cloud of the vehicle's surrounding environment, and obtaining a first obstacle detection frame obtained by performing target detection on the three-dimensional point cloud.

[0038] In this embodiment, the first obstacle detection frame may be a detection frame of an obstacle obtained by three-dimensional target detection, and the first obstacle detection frame may be obtained by performing three-dimensional target detection on a three-dimensional point cloud. The three-dimensional point cloud may be obtained by a sensor configured on the vehicle, and the present invention does not limit the type of the sensor.

[0039] S102: semantically segment the three-dimensional point cloud to obtain a ground point cloud and at least one foreground obstacle point cloud, and determine a foreground obstacle detection frame corresponding to each foreground obstacle point cloud.

[0040] In this embodiment, the ground point cloud may be a point cloud composed of ground points. The foreground obstacle detection frame may be a detection frame of a foreground obstacle in motion. Specifically, a certain algorithm is used to perform semantic segmentation on the three-dimensional point cloud to obtain a ground point cloud and at least one foreground obstacle point cloud; for each foreground obstacle point cloud, a three-dimensional frame containing the foreground obstacle point cloud is determined, and the determined three-dimensional frame is used as the foreground obstacle detection frame corresponding to the foreground obstacle point cloud.

[0041] S103. Determine a second obstacle detection frame from each foreground obstacle detection frame according to the ground point cloud; the second obstacle detection frame is a foreground obstacle detection frame located in the ground area.

[0042] Specifically, a certain algorithm is used to determine the second obstacle detection frame from each foreground obstacle detection frame according to the ground point cloud.

[0043] S104: Fusing the first obstacle detection frame and the second obstacle detection frame to obtain a target obstacle detection frame, and determining the obstacle in the target obstacle detection frame as the target obstacle.

[0044] Specifically, a certain algorithm is used to fuse the first obstacle detection frame and the second obstacle detection frame to obtain a target obstacle detection frame, and the obstacle in the target obstacle detection frame is determined as the target obstacle.

[0045] Optionally, the first obstacle detection frame and the second obstacle detection frame are merged to obtain a target obstacle detection frame, and the obstacle in the target obstacle detection frame is determined as the target obstacle, including: for each first obstacle detection frame, according to the position of the first obstacle detection frame and the positions of each second obstacle detection frame, determining a neighboring detection frame of the first obstacle detection frame from each second obstacle detection frame; for each neighboring detection frame, according to the position and size of the neighboring detection frame and the position and size of the first obstacle detection frame, determining a detection frame overlap ratio between the neighboring detection frame and the first obstacle detection frame; according to the detection frame overlap ratio, classifying the neighboring detection frames to obtain an overlap type of each neighboring detection frame; fusing a neighboring detection frame whose overlap type is overlap with the first obstacle detection frame, and determining the fusion result as the target obstacle detection frame; deleting a neighboring detection frame whose overlap type is duplicate, and determining a neighboring detection frame whose overlap type is non-overlap as the target obstacle detection frame.

[0046] The neighboring detection frame may be a second obstacle detection frame whose distance from the first obstacle detection frame is less than or equal to the set distance threshold. The detection frame overlap ratio may refer to the overlap ratio between the area surrounded by the neighboring detection frame and the area surrounded by the first obstacle detection frame. The overlap type may refer to the type of overlap between the neighboring detection frame and the first obstacle detection frame.

[0047] Specifically, for each first obstacle detection frame, the position distance between the position of the first obstacle detection frame and the positions of the second obstacle detection frames is determined according to the position of the first obstacle detection frame and the positions of the second obstacle detection frames; a second obstacle detection frame whose position distance is less than a set distance threshold is determined as a neighboring detection frame of the first obstacle detection frame; for each neighboring detection frame, the overlapping area between the enclosing area of ​​the neighboring detection frame and the enclosing area of ​​the first obstacle detection frame, and the union area between the neighboring detection frame and the first obstacle detection frame are determined according to the position and size of the neighboring detection frame and the position and size of the first obstacle detection frame; a ratio between the volume of the overlapping area and the volume of the union area is determined, and the ratio is determined as the detection frame overlap ratio.

[0048] If the detection frame overlap ratio is greater than or equal to the first ratio threshold, the overlap type of the neighboring detection frame is determined to be repeated; if the detection frame overlap ratio is less than the first ratio threshold, greater than or equal to the first ratio threshold, the overlap type of the neighboring detection frame is determined to be overlap; if the detection frame overlap ratio is less than the first ratio threshold, the overlap type of the neighboring detection frame is determined to be non-overlap; wherein the first ratio threshold is greater than the first ratio threshold; the union detection frame of the neighboring detection frame with the overlap type of overlap and the first obstacle detection frame is determined to be the target obstacle detection frame; the neighboring detection frame with the overlap type of duplicate is deleted, and the neighboring detection frame with the overlap type of non-overlap is determined to be the target obstacle detection frame. It should be noted that the distance threshold, the first ratio threshold, and the first ratio threshold can be set independently by the technician according to actual needs or practical experience, and the present invention does not limit this.

[0049] It can be understood that, by adopting the above technical solution, the neighboring detection frames of the first obstacle detection frame can be determined; the overlap type of each neighboring detection frame can be determined according to the position and size of the first obstacle detection frame and the position and size of the neighboring detection frame; the neighboring detection frames that are repeated with the first obstacle detection frame are deleted, the neighboring detection frames that overlap with the first obstacle detection frame are merged with the first obstacle detection frame, and the non-overlapping neighboring detection frames are determined as target obstacle detection frames, thereby improving the accuracy of the target obstacle detection frame.

[0050] In an optional embodiment, the three-dimensional point cloud can also be converted into a BEV image (Bird's-Eye-View) to obtain a target detection obstacle result of the BEV image; the BEV image is semantically segmented to obtain a ground semantic result and at least one foreground semantic obstacle result; based on the ground semantic result, the target detection obstacle result located in the ground area is determined from each target detection obstacle result; based on the ground point cloud, the foreground semantic obstacle result located in the ground area is determined from each foreground obstacle detection frame; the target detection obstacle result and the foreground semantic obstacle result are fused to obtain a target obstacle result.

[0051] Optional, Figure 1B is a flow chart of another obstacle detection method; Figure 1B As shown, the obstacle detection model can be used to perform 3D target detection on the 3D point cloud to obtain a first obstacle detection frame, and semantic segmentation can be performed to obtain a foreground obstacle point cloud and a ground point cloud; the obstacle detection frame corresponding to the foreground obstacle point cloud is fused with the first obstacle detection frame to obtain a fused obstacle detection frame; according to the fused obstacle detection frame and the ground point cloud, the fused obstacle detection frame located outside the ground area is filtered to obtain a fused obstacle detection frame located in the ground area, and the obstacle corresponding to the fused obstacle detection frame located in the ground area is determined as the target obstacle.

[0052] The technical solution of the embodiment of the present invention can obtain the detection frame of the foreground obstacle moving on the ground and the ground point cloud by semantically segmenting the point cloud; then determine the detection frame of the foreground obstacle in the ground area through the ground point cloud, so as to fuse the detection frame of the foreground obstacle in the ground area and the detection frame of the target detection obstacle, accurately obtain the moving foreground obstacle in the ground area, and improve the accuracy of the detected obstacles.

[0053] Embodiment 2

[0054] Figure 2 This is a flow chart of an obstacle detection method provided in the second embodiment of the present invention. Based on the technical solutions of the above embodiments, the embodiment of the present invention optimizes and improves the operation of determining the second obstacle detection frame.

[0055] Furthermore, “determine the second obstacle detection frame from each foreground obstacle detection frame based on the ground point cloud” is refined into “for each foreground obstacle detection frame, determine whether the foreground obstacle detection frame is the second obstacle detection frame based on the position and size of the ground point cloud and the position and size of the foreground obstacle detection frame” to improve the determination operation of the second obstacle detection frame.

[0056] It should be noted that, for the parts not described in detail in the embodiments of the present invention, reference may be made to the descriptions in the aforementioned embodiments.

[0057] See also Figure 2 The obstacle detection method shown includes:

[0058] S201, obtaining a three-dimensional point cloud of the vehicle's surrounding environment, and obtaining a first obstacle detection frame obtained by performing target detection on the three-dimensional point cloud.

[0059] S202: semantically segment the three-dimensional point cloud to obtain a ground point cloud and at least one foreground obstacle point cloud, and determine a foreground obstacle detection frame corresponding to each foreground obstacle point cloud.

[0060] S203. For each foreground obstacle detection frame, determine whether the foreground obstacle detection frame is a second obstacle detection frame according to the position and size of the ground point cloud and the position and size of the foreground obstacle detection frame; the second obstacle detection frame is a foreground obstacle detection frame located in the ground area.

[0061] Specifically, a certain algorithm is used to determine whether the foreground obstacle detection frame is the second obstacle detection frame according to the position and size of the ground point cloud and the position and size of the foreground obstacle detection frame.

[0062] Optionally, determining whether the foreground obstacle detection frame is a second obstacle detection frame based on the position and size of the ground point cloud and the position and size of the foreground obstacle detection frame includes: determining whether the foreground obstacle detection frame is located in the ground point cloud based on the position and size of each foreground obstacle detection frame and the position and size of the ground point cloud; if the foreground obstacle detection frame is located in the ground point cloud, determining that the foreground obstacle detection frame is the second obstacle detection frame; otherwise, determining that the foreground obstacle detection frame is not the second obstacle detection frame.

[0063] Specifically, for each foreground obstacle detection frame, the overlapping area between the foreground obstacle detection frame and the ground point cloud is determined according to the position and size of the foreground obstacle detection frame and the position and size of the ground point cloud; the volume ratio between the overlapping area and the area surrounded by the foreground obstacle detection frame is determined; if the volume ratio is greater than or equal to a third ratio threshold, it is determined that the foreground obstacle detection frame is not a second obstacle detection frame; if the volume ratio is less than the third ratio threshold, it is determined that the foreground obstacle detection frame is a second obstacle detection frame.

[0064] It is understandable that, by adopting the above technical solution, the foreground obstacle detection frame can be determined as the second obstacle detection frame by determining whether the foreground obstacle detection frame is located in the ground point cloud, thereby improving the accuracy of the determined second obstacle detection frame.

[0065] S204: Fusing the first obstacle detection frame and the second obstacle detection frame to obtain a target obstacle detection frame, and determining the obstacle in the target obstacle detection frame as the target obstacle.

[0066] This embodiment determines, for each foreground obstacle detection frame, whether the foreground obstacle detection frame is the second obstacle detection frame according to the position and size of the ground point cloud and the position and size of the foreground obstacle detection frame. It is possible to accurately determine the second obstacle detection frame located in the ground area from each foreground obstacle detection frame, thereby improving the accuracy of determining the second obstacle detection frame.

[0067] Embodiment 3

[0068] Figure 3A This is a flow chart of an obstacle detection method provided in Embodiment 3 of the present invention. Based on the technical solutions of the above embodiments, this embodiment of the present invention optimizes and improves the determination operations of the foreground obstacle detection frame and the ground point cloud.

[0069] Furthermore, "performing semantic segmentation on the three-dimensional point cloud to obtain a ground point cloud and at least one foreground obstacle point cloud, and determining a foreground obstacle detection frame corresponding to each foreground obstacle point cloud" is refined into "performing semantic segmentation on the three-dimensional point cloud through a semantic segmentation model to obtain a ground point cloud and at least one foreground obstacle point cloud, and determining a foreground obstacle detection frame corresponding to each foreground obstacle point cloud" to improve the determination operation of the foreground obstacle detection frame and the ground point cloud.

[0070] It should be noted that, for the parts not described in detail in the embodiments of the present invention, reference may be made to the descriptions in the aforementioned embodiments.

[0071] See also Figure 3A The obstacle detection method shown includes:

[0072] S301, obtaining a three-dimensional point cloud of the vehicle's surrounding environment, and obtaining a first obstacle detection frame obtained by performing target detection on the three-dimensional point cloud.

[0073] S302. Perform semantic segmentation on the three-dimensional point cloud through a semantic segmentation model to obtain a ground point cloud and at least one foreground obstacle point cloud, and determine a foreground obstacle detection frame corresponding to each foreground obstacle point cloud; wherein the semantic segmentation model is trained through sample point clouds and annotated detection frames of the sample point clouds.

[0074] In this embodiment, the sample point cloud may be a point cloud for training the model; the annotated detection frame of the sample point cloud may be an obstacle detection frame in a manually annotated sample point cloud.

[0075] Specifically, the semantic segmentation model can be used to perform semantic segmentation on the three-dimensional point cloud to obtain the semantic segmentation results of each point in the three-dimensional point cloud; the detection frame composed of the points of the foreground obstacles whose semantic segmentation results are determined as the foreground obstacle detection frame; the detection frame composed of the points of the ground whose semantic segmentation results are determined as the ground point cloud.

[0076] In an optional embodiment, the three-dimensional point cloud can be cut in a voxel manner; the three-dimensional point cloud after voxel cutting is semantically segmented through a semantic segmentation model to obtain a semantic segmentation result of each voxel in the three-dimensional point cloud; the detection frame composed of points in the voxels of the foreground obstacle whose semantic segmentation result is determined as the foreground obstacle detection frame; the detection frame composed of points in the voxels of the ground whose semantic segmentation result is determined as the ground point cloud.

[0077] Optionally, the semantic segmentation model is trained in the following manner: determining the true semantic information of each point in the sample point cloud based on the labeled detection box; performing semantic segmentation on the sample point cloud through the initial semantic model to obtain the predicted semantic information of each point in the sample point cloud; training the initial semantic model based on each predicted semantic information and each true semantic information to obtain the semantic segmentation model.

[0078] The real semantic information may be real and correct semantic information. The predicted semantic information may be the result of predicting the semantic information of points in the sample point cloud by using the initial semantic model. The initial semantic model may be an untrained deep learning model.

[0079] Specifically, a certain algorithm is used to determine the true semantic information of each point in the sample point cloud according to the labeled detection box; the sample point cloud is semantically segmented through the initial semantic model to obtain the predicted semantic information of each point in the sample point cloud; the loss value of the model is determined according to each predicted semantic information and each true semantic information; the initial semantic model is trained according to the loss value to obtain a semantic segmentation model.

[0080] It can be understood that by adopting the above technical solution, the true semantic information of each point in the sample point cloud can be determined according to the labeled detection box. Compared with the true semantic information of each point manually labeled by technicians, the efficiency of determining the true semantic information is improved, thereby improving the efficiency of training the model.

[0081] Optionally, the annotation detection box includes a foreground obstacle annotation detection box; the true semantic information includes ground type information and object type information; based on the annotation detection box, the true semantic information of each point in the sample point cloud is determined, including: by identifying ground points in the sample point cloud, the sample point cloud is divided into a ground point cloud and a non-ground point cloud, and the ground identification information is determined as the ground type information of each point in the ground point cloud, and the non-ground identification information is determined as the ground type information of each point in the non-ground point cloud; from the non-ground point cloud, the object type information of the points enclosed in the foreground obstacle annotation detection box is determined as the foreground identification information.

[0082] The foreground obstacle annotation detection frame may be a detection frame of the foreground obstacle determined by target detection, etc. Ground type information may include but is not limited to ground identification information and non-ground identification information; object type information may include but is not limited to foreground identification information; ground identification information may be used to characterize that a point is located in the ground area; non-ground identification information may be used to characterize that a point is located outside the ground area; foreground identification information may be used to characterize that a point is a point in a foreground obstacle.

[0083] Specifically, a ground point recognition algorithm is adopted to identify the ground points in the sample point cloud, determine the ground points in the sample point cloud, and determine the points in the sample point cloud other than the ground points as non-ground points; determine the ground identification information as the ground type information of each ground point, and determine the non-ground identification information as the ground type information of each ground point; and determine the object type information of the points surrounded by the foreground obstacle annotation detection frame as the foreground identification information.

[0084] In an optional embodiment, points in the sample point cloud, except for the points enclosed in the ground point cloud and the foreground obstacle annotation detection box, can be determined as background obstacle points; the background representation information can be determined as the object type information of the background obstacle points, and the point cloud composed of the background obstacle points can be determined as the background obstacle point cloud.

[0085] It can be understood that by adopting the above technical solution, the sample point cloud can be automatically divided into ground point cloud and non-ground point cloud according to the foreground obstacle annotation detection box and the ground point recognition algorithm, and the sample point cloud can be divided into foreground obstacle point cloud, thereby automatically determining the ground type information and object type information of each point in the sample point cloud, thereby improving the accuracy and efficiency of the real semantic information.

[0086] Optional, Figure 3B is a flow chart for determining true semantic information. In this optional embodiment, the sample point cloud can be divided into voxel or grid formats, and the true semantic information of each voxel or grid can be determined. Figure 3BAs shown, the sample point cloud and the detection frame annotation file are read; the manually annotated universal obstacle detection frame is read in the detection frame annotation file; wherein the universal obstacle detection frame refers to the detection frame of common obstacle categories, wherein the common obstacle categories may be, for example, engineering vehicles and concrete mixer trucks; the universal obstacle detection frame includes the foreground obstacle annotation detection frame; the ground point cloud in the sample point cloud is identified by the ground point recognition algorithm; in the sample point cloud, the point cloud contained in each universal obstacle detection frame is determined; according to the identified ground point cloud and the point cloud contained in each universal obstacle detection frame, the real semantic information of each voxel or grid is determined.

[0087] S303 . Determine a second obstacle detection frame from each foreground obstacle detection frame according to the ground point cloud; the second obstacle detection frame is a foreground obstacle detection frame located in the ground area.

[0088] S304: Fusing the first obstacle detection frame and the second obstacle detection frame to obtain a target obstacle detection frame, and determining the obstacle in the target obstacle detection frame as the target obstacle.

[0089] Optional, Figure 3C It is a structural diagram of an obstacle detection model. Figure 3C As shown, the obstacle detection model includes a preprocessing module, a backbone network, a target detection neck, a target detection head, a semantic segmentation neck, a semantic segmentation head and a result output module; wherein the preprocessing module includes a point cloud preprocessing (PointPillarsPreProcess) unit, a feature extraction (PillarFeatureNet) unit and a feature mapping (PointPillarScatter) unit. The output result of the feature mapping unit is input to the backbone network for processing; the output result of the backbone network is respectively input to the target detection neck and the semantic segmentation neck for processing; the output result of the target detection neck is input to the target detection head for processing; the processing result of the semantic segmentation neck is input to the semantic segmentation head for processing; the result output module includes a target detection frame output unit and a semantic segmentation mask output unit; wherein the target detection frame output unit can be used to output the target detection frame determined by the target detection head; the semantic segmentation mask output unit can be used to output the semantic segmentation mask determined by the semantic segmentation head, and the semantic detection frame can be further determined according to the semantic segmentation mask. After obtaining the target detection frame and the semantic detection frame, the semantic detection frame and the target detection frame are fused to obtain the final detection frame, and the obstacles in the detection mine are determined. Among them, the backbone network can adopt HybridNetss (multi-task end-to-end perception network).

[0090] The embodiment of the present invention uses a semantic segmentation model to perform semantic segmentation on a three-dimensional point cloud, obtains a ground point cloud and at least one foreground obstacle point cloud, and determines a foreground obstacle detection frame corresponding to each foreground obstacle point cloud; wherein the semantic segmentation model is trained by a sample point cloud and annotated detection frames of the sample point cloud. The embodiment of the present invention uses a semantic segmentation model obtained by training a sample point cloud and annotated detection frames of the sample point cloud to perform semantic segmentation on a three-dimensional point cloud, thereby improving the semantic segmentation accuracy of the foreground obstacle detection frame and the ground point cloud, and thereby improving the obstacle detection accuracy.

[0091] Embodiment 4

[0092] Figure 4 This is a schematic diagram of the structure of an obstacle detection device provided in Embodiment 4 of the present invention. This embodiment can be applied to the case of detecting obstacle point clouds on the ground, the device can execute the obstacle detection method, the obstacle detection device can be implemented in the form of hardware and / or software, and the device can be configured in an electronic device.

[0093] See also Figure 4 The obstacle detection device shown includes a point cloud acquisition module 401, a ground point cloud determination module 402, a second obstacle detection frame determination module 403 and an obstacle determination module 404, wherein:

[0094] The point cloud acquisition module 401 is used to acquire a three-dimensional point cloud of the vehicle's surrounding environment and to acquire a first obstacle detection frame obtained by performing target detection on the three-dimensional point cloud;

[0095] The ground point cloud determination module 402 is used to perform semantic segmentation on the three-dimensional point cloud to obtain a ground point cloud and at least one foreground obstacle point cloud, and determine a foreground obstacle detection frame corresponding to each foreground obstacle point cloud;

[0096] A second obstacle detection frame determination module 403 is used to determine a second obstacle detection frame from each foreground obstacle detection frame according to the ground point cloud; the second obstacle detection frame is a foreground obstacle detection frame located in the ground area;

[0097] The obstacle determination module 404 is configured to fuse the first obstacle detection frame and the second obstacle detection frame to obtain a target obstacle detection frame, and determine the obstacle in the target obstacle detection frame as the target obstacle.

[0098] The embodiment of the present invention obtains a three-dimensional point cloud of the vehicle surrounding environment through a point cloud acquisition module, and obtains a first obstacle detection frame obtained by performing target detection on the three-dimensional point cloud; semantically segmenting the three-dimensional point cloud through the ground point cloud to obtain a ground point cloud and at least one foreground obstacle point cloud, and determining a foreground obstacle detection frame corresponding to each foreground obstacle point cloud; determining a second obstacle detection frame from each foreground obstacle detection frame according to the ground point cloud through a second obstacle detection frame determination module; the second obstacle detection frame is a foreground obstacle detection frame located in the ground area; fusing the first obstacle detection frame and the second obstacle detection frame through the obstacle determination module to obtain a target obstacle detection frame, and determining the obstacle in the target obstacle detection frame as the target obstacle, and obtaining a foreground obstacle detection frame and a ground point cloud moving on the ground by semantically segmenting the point cloud; and then determining a detection frame of a foreground obstacle on the ground area through the ground point cloud, thereby fusing the detection frame of the foreground obstacle on the ground area and the detection frame of the target detection obstacle, accurately obtaining a moving foreground obstacle in the ground area, and improving the accuracy of the detected obstacles.

[0099] Optionally, the second obstacle detection frame determination module 403 includes:

[0100] The second obstacle detection frame determination unit is used to determine, for each foreground obstacle detection frame, whether the foreground obstacle detection frame is a second obstacle detection frame according to the position and size of the ground point cloud and the position and size of the foreground obstacle detection frame.

[0101] Optionally, the second obstacle detection frame determining unit includes:

[0102] The detection subunit is used to determine whether the foreground obstacle detection frame is located within the ground point cloud according to the position and size of the ground point cloud and the position and size of the foreground obstacle detection frame;

[0103] a foreground determination subunit, configured to determine the foreground obstacle detection frame as a second obstacle detection frame if the foreground obstacle detection frame is located within the ground point cloud;

[0104] The second determining subunit is used to determine that the foreground obstacle detection frame is not the second obstacle detection frame otherwise.

[0105] Optionally, the obstacle determination module 404 includes:

[0106] a neighboring detection frame determining unit, configured to determine, for each first obstacle detection frame, a neighboring detection frame of the first obstacle detection frame from among the second obstacle detection frames according to a position of the first obstacle detection frame and positions of the second obstacle detection frames;

[0107] a ratio determination unit, configured to determine, for each neighboring detection frame, a detection frame overlap ratio between the neighboring detection frame and the first obstacle detection frame according to the position and size of the neighboring detection frame and the position and size of the first obstacle detection frame;

[0108] An overlap type determination unit, used to classify adjacent detection frames according to the overlap ratio of the detection frames, and obtain the overlap type of each adjacent detection frame;

[0109] a foreground determination unit, configured to fuse the neighboring detection frame whose overlap type is overlap with the first obstacle detection frame, and determine the fusion result as the target obstacle detection frame;

[0110] The first determination unit is used to delete the neighboring detection frame whose overlap type is repeated, and determine the neighboring detection frame point cloud whose overlap type is non-overlapping as the target obstacle detection frame.

[0111] Optionally, the ground point cloud determination module 402 includes:

[0112] The ground point cloud determination unit is used to perform semantic segmentation on the three-dimensional point cloud through a semantic segmentation model to obtain a ground point cloud and at least one foreground obstacle point cloud, and determine a foreground obstacle detection frame corresponding to each foreground obstacle point cloud; wherein the semantic segmentation model is trained through a sample point cloud and an annotated detection frame of the sample point cloud.

[0113] Optionally, the device further comprises:

[0114] The semantic information determination module is used to determine the true semantic information of each point in the sample point cloud according to the labeled detection box;

[0115] The semantic information prediction module is used to perform semantic segmentation on the sample point cloud through the initial semantic model to obtain the predicted semantic information of each point in the sample point cloud;

[0116] The model training module is used to train the initial semantic model according to each predicted semantic information and each real semantic information to obtain a semantic segmentation model.

[0117] Optionally, the annotated detection frame includes a foreground obstacle annotated detection frame; the real semantic information includes ground type information and object type information;

[0118] The semantic information determination module includes:

[0119] a first information determination unit, configured to divide the sample point cloud into a ground point cloud and a non-ground point cloud by identifying ground points in the sample point cloud, and to determine the ground identification information as ground type information of each point in the ground point cloud, and to determine the non-ground identification information as ground type information of each point in the non-ground point cloud;

[0120] The second information determining unit is used to determine the object type information of the points enclosed in the foreground obstacle label detection frame as the foreground identification information.

[0121] The obstacle detection device provided in the embodiment of the present invention can execute the obstacle detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the obstacle detection method.

[0122] Embodiment 5

[0123] Figure 5 A schematic diagram of the structure of an electronic device 500 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0124] like Figure 5 As shown, the electronic device 500 includes at least one processor 501, and a memory connected to the at least one processor 501 in communication, such as a read-only memory (ROM) 502, a random access memory (RAM) 503, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 501 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 502 or the computer program loaded from the storage unit 508 to the random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 can also be stored. The processor 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0125] Multiple components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0126] The processor 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 501 performs the various methods and processes described above, such as the obstacle detection method.

[0127] In some embodiments, the obstacle detection method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the processor 501, one or more steps of the obstacle detection method described above may be performed. Alternatively, in other embodiments, the processor 501 may be configured to perform the obstacle detection method in any other appropriate manner (e.g., by means of firmware).

[0128] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0129] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable obstacle detection device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0130] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0131] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0132] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0133] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS (Virtual Private Server) services.

[0134] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0135] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An obstacle detection method, characterized in that: The method comprises: Acquire a three-dimensional point cloud of the vehicle's surroundings, and acquire a first obstacle detection frame obtained by performing target detection on the three-dimensional point cloud; Performing semantic segmentation on the three-dimensional point cloud to obtain a ground point cloud and at least one foreground obstacle point cloud, and determining a foreground obstacle detection frame corresponding to each foreground obstacle point cloud; According to the ground point cloud, determining a second obstacle detection frame from each of the foreground obstacle detection frames; the second obstacle detection frame is a foreground obstacle detection frame located in the ground area; The first obstacle detection frame and the second obstacle detection frame are fused to obtain a target obstacle detection frame, and an obstacle in the target obstacle detection frame is determined as a target obstacle.

2. The method according to claim 1, characterized in that The determining, according to the ground point cloud, a second obstacle detection frame from each of the foreground obstacle detection frames comprises: For each of the foreground obstacle detection frames, it is determined whether the foreground obstacle detection frame is a second obstacle detection frame according to the position and size of the ground point cloud and the position and size of the foreground obstacle detection frame.

3. The method according to claim 2, characterized in that The determining, according to the position and size of the ground point cloud and the position and size of the foreground obstacle detection frame, whether the foreground obstacle detection frame is a second obstacle detection frame comprises: Determining whether the foreground obstacle detection frame is located within the ground point cloud according to the position and size of the ground point cloud and the position and size of the foreground obstacle detection frame; If the foreground obstacle detection frame is located in the ground point cloud, determining the foreground obstacle detection frame as a second obstacle detection frame; Otherwise, it is determined that the foreground obstacle detection frame is not the second obstacle detection frame.

4. The method according to claim 1, characterized in that: The fusing the first obstacle detection frame and the second obstacle detection frame to obtain a target obstacle detection frame includes: For each of the first obstacle detection frames, determining a neighboring detection frame of the first obstacle detection frame from each of the second obstacle detection frames according to a position of the first obstacle detection frame and a position of each of the second obstacle detection frames; For each neighboring detection frame, determining a detection frame overlap ratio between the neighboring detection frame and the first obstacle detection frame according to a position and a size of the neighboring detection frame and a position and a size of the first obstacle detection frame; Classifying the adjacent detection frames according to the overlap ratio of the detection frames to obtain the overlap type of each of the adjacent detection frames; Fusing the neighboring detection frame whose overlap type is overlap with the first obstacle detection frame, and determining the fusion result as the target obstacle detection frame; The neighboring detection frames with the overlap type of duplicate are deleted, and the neighboring detection frame point cloud with the overlap type of non-overlap is determined as the target obstacle detection frame.

5. The method according to claim 1, characterized in that The performing semantic segmentation on the three-dimensional point cloud to obtain a ground point cloud and at least one foreground obstacle point cloud, and determining a foreground obstacle detection frame corresponding to each foreground obstacle point cloud, includes: The three-dimensional point cloud is semantically segmented by a semantic segmentation model to obtain a ground point cloud and at least one foreground obstacle point cloud, and a foreground obstacle detection frame corresponding to each foreground obstacle point cloud is determined; wherein the semantic segmentation model is trained by a sample point cloud and an annotated detection frame of the sample point cloud.

6. The method according to claim 5, characterized in that The semantic segmentation model is trained in the following way: Determining true semantic information of each point in the sample point cloud according to the labeled detection frame; Performing semantic segmentation on the sample point cloud through an initial semantic model to obtain predicted semantic information of each point in the sample point cloud; The initial semantic model is trained according to each of the predicted semantic information and each of the true semantic information to obtain a semantic segmentation model.

7. The method according to claim 6, characterized in that The annotated detection frame includes a foreground obstacle annotated detection frame; the real semantic information includes ground type information and object type information; Determining the true semantic information of each point in the sample point cloud according to the labeled detection frame includes: By identifying ground points in the sample point cloud, the sample point cloud is divided into a ground point cloud and a non-ground point cloud, and ground identification information is determined as ground type information of each point in the ground point cloud, and non-ground identification information is determined as ground type information of each point in the non-ground point cloud; The object type information of the points enclosed in the foreground obstacle annotation detection frame is determined as the foreground identification information.

8. An obstacle detection device, characterized in that: The device comprises: A point cloud acquisition module, used to acquire a three-dimensional point cloud of the vehicle's surrounding environment and to acquire a first obstacle detection frame obtained by performing target detection on the three-dimensional point cloud; A ground point cloud determination module, used to perform semantic segmentation on the three-dimensional point cloud to obtain a ground point cloud and at least one foreground obstacle point cloud, and determine a foreground obstacle detection frame corresponding to each foreground obstacle point cloud; A detection frame determination module, configured to determine a second obstacle detection frame from each of the foreground obstacle detection frames according to the ground point cloud; the second obstacle detection frame is a foreground obstacle detection frame located in the ground area; The obstacle determination module is used to fuse the first obstacle detection frame and the second obstacle detection frame to obtain a target obstacle detection frame, and determine the obstacle in the target obstacle detection frame as a target obstacle.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the obstacle detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the obstacle detection method according to any one of claims 1 to 7 when executed.

Images