Obstacle detection method and device and vehicle

Abstract

The invention relates to an obstacle detection method and device and a vehicle, and the method comprises the steps: obtaining image data collected by a camera and point cloud data collected by a laser radar; determining a first detection result according to the image data and the point cloud data; determining a second detection result according to the point cloud data; determining obstacle information according to the first detection result and the second detection result; wherein the first detection result and the second detection result both comprise the three-dimensional bounding box information, the category and the confidence of the detected obstacle.

Description

Technical Field

[0001] The present disclosure relates to autonomous driving technology, and more specifically, to an obstacle detection method, device and vehicle. Background Art

[0002] The development of autonomous driving technology is in full swing. In autonomous driving technology, the accuracy of obstacle detection is a prerequisite for the safe driving of the vehicle.

[0003] At present, the sensors used for obstacle detection are mainly cameras and laser radars. Cameras collect image data. The advantages of using cameras to detect obstacles are low cost and very rich semantic information. The disadvantages are that the image quality is poor in strong light, at night, and in extreme weather conditions. Laser radar collects point cloud data. Using laser radar to detect obstacles has good ranging accuracy, ranging range, and adaptability to temperature and light. The disadvantages are that it is expensive, lacks semantic information, and it is difficult to identify obstacle categories.

[0004] Therefore, a new technical solution needs to be provided to improve the accuracy of obstacle detection. Summary of the invention

[0005] An object of the present invention is to provide a new technical solution for a method for detecting obstacles.

[0006] According to a first aspect of the present invention, there is provided a method for detecting an obstacle, comprising:

[0007] Obtain image data collected by the camera and point cloud data collected by the lidar;

[0008] Determine a first detection result according to the image data and the point cloud data;

[0009] Determine a second detection result according to the point cloud data;

[0010] Determine obstacle information according to the first detection result and the second detection result; wherein,

[0011] The first detection result and the second detection result both include three-dimensional bounding box information, category, and confidence level of the detected obstacle.

[0012] Optionally, determining a first detection result according to the image data and the point cloud data includes:

[0013] Determining, according to the image data and the point cloud data, point cloud clustering information of the obstacle, a category corresponding to the point cloud clustering information of the obstacle, and a confidence level corresponding to the point cloud clustering information of the obstacle;

[0014] A first detection result is generated according to the point cloud clustering information of the obstacle, the category corresponding to the point cloud clustering information of the obstacle, and the confidence level corresponding to the point cloud clustering information of the obstacle.

[0015] Optionally, determining the point cloud clustering information of the obstacle, the category corresponding to the point cloud clustering information of the obstacle, and the confidence level corresponding to the point cloud clustering information of the obstacle according to the image data and the point cloud data includes:

[0016] Determining two-dimensional bounding box coordinate information of the obstacle, the category of the obstacle, and the confidence level of the obstacle based on the image data;

[0017] Determining point cloud clustering information of obstacles according to the point cloud data;

[0018] According to the two-dimensional bounding box coordinate information of the obstacle, the category of the obstacle, the confidence of the obstacle and the point cloud clustering information of the obstacle, the category corresponding to the point cloud clustering information of the obstacle and the confidence corresponding to the point cloud clustering information of the obstacle are determined.

[0019] Optionally, determining the category corresponding to the point cloud clustering information of the obstacle and the confidence corresponding to the point cloud clustering information of the obstacle according to the two-dimensional bounding box coordinate information of the obstacle, the category of the obstacle, the confidence of the obstacle and the point cloud clustering information of the obstacle includes:

[0020] Determining whether the obstacles are the same obstacle according to the two-dimensional bounding box coordinate information of the obstacle and the point cloud clustering information of the obstacle;

[0021] In the case of determining that the obstacle is the same, the category of the obstacle is determined as the category corresponding to the point cloud clustering information of the obstacle, and the confidence of the obstacle is determined as the confidence corresponding to the point cloud clustering information of the obstacle.

[0022] Optionally, before determining point cloud clustering information of obstacles according to the point cloud data, the method further includes:

[0023] The vehicle body point cloud data, the ground point cloud data and the noise point data are respectively cropped from the point cloud data to obtain the first preprocessed point cloud data.

[0024] Optionally, determining obstacle information according to the first detection result and the second detection result includes:

[0025] Based on each category of obstacles, all obstacles are sorted from high to low according to the confidence level to obtain an obstacle sequence;

[0026] Traversing each obstacle in the obstacle sequence in turn, and determining the overlap degree between the current obstacle and each obstacle arranged after the current obstacle;

[0027] When the overlap exceeds a preset overlap, the corresponding obstacle information arranged after the current obstacle is deleted.

[0028] Optionally, determining the degree of overlap between the current obstacle and obstacles arranged after the current obstacle includes:

[0029] The degree of overlap between the current obstacle and the obstacles arranged behind the current obstacle is determined according to the three-dimensional bounding box information of the current obstacle and the three-dimensional bounding box information of the obstacles arranged behind the current obstacle.

[0030] Optionally, the method further includes:

[0031] The camera and the laser radar are subjected to time synchronization processing and joint calibration processing.

[0032] According to a second aspect of the present invention, there is provided an obstacle detection device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer program is used to control the processor to operate so as to execute the obstacle detection method according to any one of the first aspect of the present invention.

[0033] According to a third aspect of the present invention, there is provided a vehicle comprising a camera, a laser radar and an obstacle detection device as described in the second aspect of the present invention.

[0034] In the obstacle detection method provided by the present invention, the first detection result is a detection result obtained by fusing image data and point cloud data, and the second detection result is a detection result obtained based on the point cloud data. By fusing the first detection result and the second detection result, the accuracy of the spatial information and orientation information of the obstacle can be improved, the missed detection rate of the obstacle can be reduced, and thus the detection accuracy of the obstacle can be improved.

[0035] Features and advantages of the embodiments of the present specification will become apparent from the following detailed description of exemplary embodiments of the present specification with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the specification and, together with the description, serve to explain the principles of the embodiments of the specification.

[0037] Figure 1 is a processing flow chart of a method for detecting an obstacle according to an embodiment of the present invention.

[0038] Figure 2 is a processing flow chart of a method for detecting an obstacle according to an embodiment of the present invention.

[0039] Figure 3 4 is a principle block diagram of an obstacle detection device according to an embodiment of the present invention.

[0040] Figure 4 Schematic diagram of the hardware structure of an obstacle detection device according to an embodiment of the present invention.

[0041] Figure 5 is a functional block diagram of a vehicle according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] Various exemplary embodiments of the present specification will now be described in detail with reference to the accompanying drawings.

[0043] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the embodiments of the present specification and its application or uses.

[0044] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0045] In one embodiment of the present invention, a method for detecting an obstacle is provided. Figure 1 As shown, the obstacle detection method of this embodiment may include the following steps S110 to S140.

[0046] Step S110, acquiring image data collected by the camera and point cloud data collected by the lidar.

[0047] In one embodiment, before executing step S110, the method further includes: performing time synchronization processing and joint calibration processing on the camera and the lidar.

[0048] The camera and lidar are time-synchronized to ensure that they can capture data at the same moment, providing a reliable data basis for subsequent data fusion and obstacle detection.

[0049] The purpose of joint calibration of the camera and the laser radar is to determine the relative position and posture relationship between the camera and the laser radar. During the joint calibration process, the intrinsic parameter matrix of the camera and the extrinsic parameter matrix of the laser radar are obtained. The intrinsic parameter matrix of the camera includes focal length, principal point coordinates, and distortion coefficients. The extrinsic parameter matrix of the laser radar describes the position and posture of the laser radar relative to the camera. The extrinsic parameter matrix of the laser radar includes two parts: the rotation matrix and the translation matrix. The rotation matrix represents the rotation angle of the laser radar coordinate system relative to the camera coordinate system, and the translation matrix represents the spatial distance between the two. Through joint calibration, the image data collected by the camera and the point cloud data collected by the laser radar can be unified into the same coordinate system to achieve data alignment, which provides a basis for subsequent data fusion. It should be noted that the same coordinate system can be a two-dimensional coordinate system or a three-dimensional coordinate system.

[0050] Step S120: determining a first detection result according to the image data and the point cloud data.

[0051] The first detection result includes the three-dimensional bounding box information, category and confidence level of the detected obstacle.

[0052] In one embodiment, step S120 includes the following steps S121 - S122 .

[0053] Step S121 : determining point cloud clustering information of the obstacle, the category corresponding to the point cloud clustering information of the obstacle, and the confidence level corresponding to the point cloud clustering information of the obstacle according to the image data and the point cloud data.

[0054] The point cloud cluster information of the obstacle includes the coordinate information of each point in the point cloud cluster.

[0055] The category corresponding to the point cloud clustering information of the obstacle is any one of pedestrians, vehicles, animals, and roadblocks.

[0056] The confidence level corresponding to the point cloud clustering information of the obstacle is the degree of certainty that the target corresponding to the point cloud clustering information of the obstacle is an obstacle.

[0057] Step S122: Generate a first detection result according to the point cloud clustering information of the obstacle, the category corresponding to the point cloud clustering information of the obstacle, and the confidence level corresponding to the point cloud clustering information of the obstacle.

[0058] In one embodiment, step S121 specifically includes: determining the two-dimensional bounding box coordinate information of the obstacle, the category of the obstacle and the confidence of the obstacle according to the image data; determining the point cloud clustering information of the obstacle according to the point cloud data; determining the category corresponding to the point cloud clustering information of the obstacle and the confidence corresponding to the point cloud clustering information of the obstacle according to the two-dimensional bounding box coordinate information of the obstacle, the category of the obstacle, the confidence of the obstacle and the point cloud clustering information of the obstacle.

[0059] In this embodiment, the results determined based on the image data and the results determined based on the point cloud data are integrated, which can avoid the inherent limitations of each algorithm and improve the obstacle detection accuracy.

[0060] The 2D bounding box of an obstacle is a rectangular area that surrounds the detected object. The coordinate information of the 2D bounding box of the obstacle is used to locate the position and size of the detected object in the image. There are two ways to annotate the 2D bounding box: one is to use (x 1 ,y 1 ,x 2 ,y 2 ) to represent the upper left and lower right two-dimensional coordinate points; the other is to use (x, y, w, h) to represent the center point coordinates, width and height of the rectangular area.

[0061] The type of obstacle is any one of pedestrians, vehicles, animals, and roadblocks.

[0062] The confidence of an obstacle is the degree of certainty that the target within the two-dimensional bounding box is an obstacle.

[0063] For example, the image data is input into the 2D object detection network model to obtain the two-dimensional bounding box coordinate information of the obstacle, the category of the obstacle, and the confidence of the obstacle.

[0064] In this example, the algorithm used by the 2D object detection network model is YOLOX (You Only Look Once).

[0065] In one embodiment, before determining point cloud clustering information of obstacles based on point cloud data, the point cloud data is preprocessed to obtain first preprocessed point cloud data, and the point cloud clustering information of obstacles is determined based on the preprocessed point cloud data.

[0066] Specifically, the vehicle body point cloud data, the ground point cloud data and the noise point data are respectively cropped from the point cloud data to obtain the first preprocessed point cloud data.

[0067] To crop the body point cloud data from the point cloud data, first, the body area is determined from the point cloud data, and then the point cloud data within this area is removed using a cropping algorithm.

[0068] To crop ground point cloud data from point cloud data, the ground model can be used to segment the ground points in the point cloud data.

[0069] To remove noise data from point cloud data, point cloud voxelization filtering can be used. The point cloud data is divided into a series of 3D cubes (voxels), and then the points in each voxel are counted or processed to reduce the number of points and remove noise.

[0070] The body point cloud data and ground point cloud data do not reflect the characteristics of the surrounding environment, and do not contribute to the subsequent obstacle detection task. Instead, they increase the calculation amount of the detection task and may even interfere with the subsequent obstacle detection. Therefore, cutting the body point cloud data and ground point cloud data from the point cloud data can improve the speed and accuracy of subsequent obstacle detection. Filter the point cloud data to remove noise and outliers, and improve the accuracy of subsequent obstacle detection.

[0071] In one embodiment, the point cloud clustering information of the obstacle is determined based on the preprocessed point cloud data using the Euclidean clustering algorithm. The principle of the Euclidean clustering algorithm is to classify similar points as the same detection target.

[0072] In one embodiment, whether they are the same obstacle is determined based on the two-dimensional bounding box coordinate information of the obstacle and the point cloud clustering information of the obstacle; if it is determined to be the same obstacle, the category of the obstacle is determined as the category corresponding to the point cloud clustering information of the obstacle, and the confidence of the obstacle is determined as the confidence corresponding to the point cloud clustering information of the obstacle.

[0073] Specifically, according to the point cloud clustering information of the obstacle, the region coordinate information corresponding to the point cloud clustering information is determined. According to the two-dimensional bounding box coordinate information of the obstacle and the region coordinate information corresponding to the point cloud clustering information, it is determined whether the obstacle detected based on the image data and the obstacle detected based on the point cloud data are the same obstacle. In the case of determining that they are the same obstacle, the category of the obstacle is determined as the category corresponding to the point cloud clustering information of the obstacle, and the confidence of the obstacle is determined as the confidence corresponding to the point cloud clustering information of the obstacle.

[0074] The area coordinate information corresponding to the point cloud clustering information and the two-dimensional bounding box coordinate information of the obstacle are in the same coordinate system. According to the two-dimensional bounding box coordinate information of the obstacle and the area coordinate information corresponding to the point cloud clustering information, the overlap ratio value of the two-dimensional bounding box of the obstacle and the area corresponding to the point cloud clustering information is determined. When the overlap degree of the two-dimensional bounding box of the obstacle and the area corresponding to the point cloud clustering information exceeds a preset overlap ratio value, it is determined that the obstacle detected based on the image data and the obstacle detected based on the point cloud data are the same obstacle.

[0075] When the number of two-dimensional bounding boxes of obstacles and the number of areas corresponding to point cloud clustering information are both multiple, the overlap ratio value is calculated based on the two-dimensional bounding box of each obstacle and the area corresponding to each point cloud clustering information, or the overlap ratio value is calculated based on the area corresponding to each point cloud clustering information and the two-dimensional bounding box of each obstacle.

[0076] Step S130: determining a second detection result according to the point cloud data.

[0077] The second detection result includes the three-dimensional bounding box information, category and confidence level of the detected obstacle.

[0078] In one embodiment, step S130 specifically includes: inputting the point cloud data into a 3D object detection network model to obtain three-dimensional bounding box information, category, and confidence of the obstacle.

[0079] The 3D object detection network model is the CenterPoint network model. Specifically, the point cloud data is input into the CenterPoint network model. The CenterPoint network model processes through multiple layers within it (such as feature extraction layer, convolution layer, pooling layer, etc.), predicts the center point coordinates, size and direction of the target, and outputs the 3D bounding box information, category and confidence of the detected obstacle.

[0080] In one embodiment, before determining the second detection result based on the point cloud data, the point cloud data is preprocessed to obtain second preprocessed point cloud data, and the second detection result is determined based on the second preprocessed point cloud data.

[0081] Specifically, the vehicle body point cloud data is cropped and the noise data is removed from the point cloud data to obtain preprocessed point cloud data.

[0082] The body point cloud data is cropped separately from the point cloud data. First, the body area is determined from the point cloud data, and then the point cloud data located in this area is removed using a cropping algorithm.

[0083] To remove noise data from point cloud data, point cloud voxelization filtering can be used. The point cloud data is divided into a series of 3D cubes (voxels), and then the points in each voxel are counted or processed to reduce the number of points and remove noise.

[0084] The body point cloud data does not reflect the characteristics of the surrounding environment and does not contribute to the subsequent obstacle detection task. Instead, it increases the computational complexity of the detection task and may even interfere with the subsequent obstacle detection. Therefore, cutting the body point cloud data separately from the point cloud data can improve the speed and accuracy of subsequent obstacle detection. The point cloud data is filtered to remove noise and outliers to improve the accuracy of subsequent obstacle detection.

[0085] Step S140, determining obstacle information according to the first detection result and the second detection result; wherein the first detection result and the second detection result both include three-dimensional bounding box information, category and confidence of the detected obstacle.

[0086] In one embodiment, step S140 specifically includes the following steps S141 to S143.

[0087] Step S141 : based on each category of obstacles, all obstacles are sorted from high to low according to the confidence level to obtain an obstacle sequence.

[0088] Step S142: traverse each obstacle in the obstacle sequence in turn to determine the overlap between the current obstacle and each obstacle arranged after the current obstacle.

[0089] Specifically, the degree of overlap between the current obstacle and the obstacles arranged behind the current obstacle is determined according to the three-dimensional bounding box information of the current obstacle and the three-dimensional bounding box information of the obstacles arranged behind the current obstacle.

[0090] Step S143: when the overlap degree exceeds a preset overlap degree, the corresponding obstacle information arranged after the current obstacle is deleted.

[0091] When the overlap exceeds the preset overlap, it means that the current obstacle and the obstacle corresponding to the overlap arranged after the current obstacle are the same obstacle. Therefore, the obstacle corresponding to the overlap arranged after the current obstacle is deleted, so that the final obstacle information is more accurate and the problem of duplicate obstacle information is avoided.

[0092] Taking the obstacle types including pedestrians, vehicles, animals, and roadblocks as an example, the above steps S141 to S143 are specifically described.

[0093] According to the first detection result and the second detection result, it is determined that the vehicle category includes 4 obstacles. The 4 obstacles in the category are sorted from high to low according to the confidence level to obtain an obstacle sequence, namely obstacle a, obstacle b, obstacle c, and obstacle d.

[0094] First, based on obstacle a, determine the overlap between obstacle a and obstacle b, obstacle c, and obstacle d, so that three overlaps can be obtained. These three overlaps are compared with the preset overlaps to obtain three comparison results. The three comparison results are that the overlap between obstacle a and obstacle b does not exceed the preset overlap, the overlap between obstacle a and obstacle c exceeds the preset overlap, and the overlap between obstacle a and obstacle d does not exceed the preset overlap. In this way, obstacle c is deleted and the obstacle sequence is adjusted to obstacle a, obstacle b, and obstacle d.

[0095] Then, based on obstacle b, the overlap between obstacle b and obstacle d is determined. The overlap is compared with the preset overlap to obtain a comparison result. The comparison result is that the overlap between obstacle b and obstacle d exceeds the preset overlap. In this way, obstacle d is deleted and the obstacle sequence is adjusted to obstacle a and obstacle b.

[0096] In this way, based on the category of vehicle, the obstacle information obtained includes information corresponding to obstacle a and information corresponding to obstacle b. Similarly, according to the above steps S141 to S143, the obstacle information corresponding to the other three categories can be obtained.

[0097] In the obstacle detection method provided by the present invention, the first detection result is a detection result obtained by fusing image data and point cloud data, and the second detection result is a detection result obtained based on the point cloud data. By fusing the first detection result and the second detection result, the accuracy of the spatial information and orientation information of the obstacle can be improved, the missed detection rate of the obstacle can be reduced, and thus the detection accuracy of the obstacle can be improved.

[0098] The obstacle detection method provided by the present invention is described below with a specific embodiment. Figure 2 , the obstacle detection method includes steps S201 to S211.

[0099] Step S201, performing time synchronization and joint calibration processing on the camera and the lidar.

[0100] Step S202, obtaining image data collected by the camera and point cloud data collected by the laser radar.

[0101] Step S203, cutting the vehicle body point cloud data from the point cloud data.

[0102] Step S204, cropping ground point cloud data from the point cloud data.

[0103] Step S205 , using a point cloud voxelization filtering method to remove noise data from the point cloud data.

[0104] Step S206: Determine point cloud clustering information of obstacles using a Euclidean clustering algorithm based on the preprocessed point cloud data.

[0105] Step S207: input the image data into the 2D object detection network model, and use the YOLOX algorithm to obtain the two-dimensional bounding box coordinate information of the obstacle, the category of the obstacle, and the confidence of the obstacle.

[0106] Step S208 , determining the category corresponding to the point cloud clustering information of the obstacle and the confidence corresponding to the point cloud clustering information of the obstacle according to the two-dimensional bounding box coordinate information of the obstacle, the category of the obstacle, the confidence of the obstacle and the point cloud clustering information of the obstacle.

[0107] Step S209, generating a first detection result according to the point cloud clustering information of the obstacle, the category corresponding to the point cloud clustering information of the obstacle, and the confidence corresponding to the point cloud clustering information of the obstacle, wherein the first detection result includes the three-dimensional bounding box information, category and confidence of the detected obstacle.

[0108] After step S203, step S210 is further performed to remove noise data from the point cloud data using a point cloud voxelization filtering processing method.

[0109] Step S211, input the preprocessed point cloud data into the CenterPoint network model to obtain a second detection result, wherein the second detection result includes the three-dimensional bounding box information, category and confidence of the detected obstacle.

[0110] After steps S209 and S211, step S212 is executed to determine obstacle information according to the first detection result and the second detection result.

[0111] An embodiment of the present invention provides an obstacle detection device, such as Figure 3 The obstacle detection device 300 includes an acquisition module 310 , a first detection result determination module 320 , a second detection result determination module 330 and an obstacle information determination module 340 .

[0112] The acquisition module 310 is used to acquire image data collected by the camera and point cloud data collected by the laser radar.

[0113] The first detection result determination module 320 is used to determine the first detection result according to the image data and the point cloud data.

[0114] The second detection result determination module 330 is used to determine the second detection result according to the point cloud data.

[0115] The obstacle information determination module 340 is used to determine obstacle information according to the first detection result and the second detection result; wherein the first detection result and the second detection result both include the three-dimensional bounding box information, category and confidence of the detected obstacle.

[0116] In one embodiment, the first detection result determination module 320 is used to determine the point cloud clustering information of the obstacle, the category corresponding to the point cloud clustering information of the obstacle, and the confidence level corresponding to the point cloud clustering information of the obstacle based on the image data and the point cloud data; and generate the first detection result based on the point cloud clustering information of the obstacle, the category corresponding to the point cloud clustering information of the obstacle, and the confidence level corresponding to the point cloud clustering information of the obstacle.

[0117] In one embodiment, the first detection result determination module 320 is used to determine the two-dimensional bounding box coordinate information of the obstacle, the category of the obstacle and the confidence of the obstacle based on the image data; determine the point cloud clustering information of the obstacle based on the point cloud data; determine the category corresponding to the point cloud clustering information of the obstacle and the confidence corresponding to the point cloud clustering information of the obstacle based on the two-dimensional bounding box coordinate information of the obstacle, the category of the obstacle, the confidence of the obstacle and the point cloud clustering information of the obstacle.

[0118] In one embodiment, the first detection result determination module 320 is used to determine whether they are the same obstacle based on the two-dimensional bounding box coordinate information of the obstacle and the point cloud clustering information of the obstacle; if it is determined to be the same obstacle, the category of the obstacle is determined as the category corresponding to the point cloud clustering information of the obstacle, and the confidence of the obstacle is determined as the confidence corresponding to the point cloud clustering information of the obstacle.

[0119] In one embodiment, the obstacle detection device further includes a preprocessing module. The preprocessing module is used to determine the point cloud clustering information of the obstacle according to the point cloud data. The method further includes: cutting the vehicle body point cloud data, the ground point cloud data and removing the noise data from the point cloud data to obtain the first preprocessed point cloud data.

[0120] In one embodiment, the obstacle information determination module 340 is used to sort all obstacles from high to low according to the confidence level based on each category of obstacles to obtain an obstacle sequence; traverse each obstacle in the obstacle sequence in turn to determine the overlap between the current obstacle and the obstacles arranged after the current obstacle; if the overlap exceeds a preset overlap, delete the corresponding obstacle information arranged after the current obstacle.

[0121] In one embodiment, the obstacle information determination module 340 is further configured to determine the degree of overlap between the current obstacle and each obstacle arranged after the current obstacle based on the 3D bounding box information of the current obstacle and the 3D bounding box information of each obstacle arranged after the current obstacle.

[0122] In one embodiment, the obstacle detection device further includes a calibration processing module. The calibration processing module is used to perform time synchronization processing and joint calibration processing on the camera and the laser radar.

[0123] An embodiment of the present invention provides an obstacle detection device, such as Figure 4 The obstacle detection device 400 includes a memory 420 and a processor 410. The memory 420 stores a computer program, and the computer program is used to control the processor 410 to operate so as to execute the obstacle detection method in any of the above embodiments.

[0124] One embodiment of the present invention provides a vehicle, including a camera, a laser radar, and an obstacle detection device such as any of the above embodiments. Figure 5 .

[0125] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. For the device embodiment, its related parts can be referred to the partial description of the method embodiment.

[0126] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0127] The embodiments of the present specification may be systems, methods and / or computer program products. The computer program product may include a computer-readable storage medium carrying computer instructions for causing a processor to implement various aspects of the embodiments of the present specification.

[0128] A computer-readable storage medium may be a tangible device that can hold and store computer instructions used by a computer instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which computer instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0129] The computer instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device through a network layer, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network layer may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network layer adapter card or network layer interface in each computing / processing device receives the computer instructions from the network layer and forwards the computer instructions for storage in a computer-readable storage medium in each computing / processing device.

[0130] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to the multiple embodiments of this specification. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a computer instruction, and a part of a module, a program segment or a computer instruction contains one or more executable computer instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0131] The embodiments of the present specification have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art without departing from the scope of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for detecting an obstacle, characterized in that: include: Obtain image data collected by the camera and point cloud data collected by the lidar; Determine a first detection result according to the image data and the point cloud data; Determine a second detection result according to the point cloud data; Determine obstacle information according to the first detection result and the second detection result; wherein, The first detection result and the second detection result both include three-dimensional bounding box information, category, and confidence level of the detected obstacle.

2. The method according to claim 1, characterized in that The determining a first detection result according to the image data and the point cloud data includes: Determining, according to the image data and the point cloud data, point cloud clustering information of the obstacle, a category corresponding to the point cloud clustering information of the obstacle, and a confidence level corresponding to the point cloud clustering information of the obstacle; A first detection result is generated according to the point cloud clustering information of the obstacle, the category corresponding to the point cloud clustering information of the obstacle, and the confidence level corresponding to the point cloud clustering information of the obstacle.

3. The method according to claim 2, characterized in that The determining, based on the image data and the point cloud data, the point cloud clustering information of the obstacle, the category corresponding to the point cloud clustering information of the obstacle, and the confidence level corresponding to the point cloud clustering information of the obstacle includes: Determining two-dimensional bounding box coordinate information of the obstacle, the category of the obstacle, and the confidence level of the obstacle based on the image data; Determining point cloud clustering information of obstacles according to the point cloud data; According to the two-dimensional bounding box coordinate information of the obstacle, the category of the obstacle, the confidence of the obstacle and the point cloud clustering information of the obstacle, the category corresponding to the point cloud clustering information of the obstacle and the confidence corresponding to the point cloud clustering information of the obstacle are determined.

4. The method according to claim 3, characterized in that The determining, according to the two-dimensional bounding box coordinate information of the obstacle, the category of the obstacle, the confidence of the obstacle and the point cloud clustering information of the obstacle, the category corresponding to the point cloud clustering information of the obstacle and the confidence corresponding to the point cloud clustering information of the obstacle includes: Determining whether the obstacles are the same obstacle according to the two-dimensional bounding box coordinate information of the obstacle and the point cloud clustering information of the obstacle; In the case of determining that the obstacle is the same, the category of the obstacle is determined as the category corresponding to the point cloud clustering information of the obstacle, and the confidence of the obstacle is determined as the confidence corresponding to the point cloud clustering information of the obstacle.

5. The method according to claim 3, characterized in that: Before determining point cloud clustering information of obstacles according to the point cloud data, the method further includes: The vehicle body point cloud data, the ground point cloud data and the noise point data are respectively cropped from the point cloud data to obtain the first preprocessed point cloud data.

6. The method according to any one of claims 1 to 5, characterized in that: The determining obstacle information according to the first detection result and the second detection result includes: Based on each category of obstacles, all obstacles are sorted from high to low according to the confidence level to obtain an obstacle sequence; Traversing each obstacle in the obstacle sequence in turn, and determining the overlap degree between the current obstacle and each obstacle arranged after the current obstacle; When the overlap exceeds a preset overlap, the corresponding obstacle information arranged after the current obstacle is deleted.

7. The method according to claim 6, characterized in that The determining the degree of overlap between the current obstacle and each obstacle arranged after the current obstacle includes: The degree of overlap between the current obstacle and the obstacles arranged behind the current obstacle is determined according to the three-dimensional bounding box information of the current obstacle and the three-dimensional bounding box information of the obstacles arranged behind the current obstacle.

8. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: The camera and the laser radar are subjected to time synchronization processing and joint calibration processing.

9. An obstacle detection device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the computer program is used to control the processor to operate so as to execute the obstacle detection method according to any one of claims 1 to 8.

10. A vehicle, characterized in that: The device comprises a camera, a laser radar and an obstacle detection device as claimed in claim 9.

Images