An obstacle detection method, device, electronic equipment and storage medium

Abstract

The application discloses an obstacle detection method and device, electronic equipment and storage medium, comprising: acquiring multiple frames of initial point clouds collected by a multi-line laser radar and pose data collected by an IMU when a vehicle is driving, de-distorting and fusing the multiple frames of initial point clouds based on the pose data to obtain target point clouds; inputting the target point clouds into a preset obstacle recognition model to obtain orientation information and size information of small obstacles, the small obstacles being obstacles with a height within a preset height range. The multiple frames of initial point clouds are de-distorted based on the pose data, which can improve the accuracy of the point cloud data and avoid the difficulty in detecting small obstacles due to point cloud distortion. The multiple frames of corrected point clouds are fused into target point clouds, which improves the point cloud density of the target point clouds and further improves the point cloud density of small obstacles in the target point clouds, so that the characteristics of small obstacles can be more completely expressed, and small obstacles can be detected.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to an obstacle detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] In the field of autonomous driving, environmental perception is a prerequisite for autonomous vehicles to perform their tasks safely and reliably, and obstacle detection is a basic function of the environmental perception system.

[0003] As a primary sensor for obstacle detection, the effectiveness of lidar in obstacle detection is generally related to the stability of the radar detection. Solid-state lidar employs a MEMS micro-mirror scanning scheme, requiring very few laser emitters and receivers. During operation, only the micro-mirrors oscillate, resulting in high stability in obstacle detection. However, its cost is higher and its technological maturity is not as advanced as that of multi-line lidar. Therefore, in current technologies, object detection is mainly carried out using multi-line lidar.

[0004] The adjacent beams of a multi-beam lidar point cloud are relatively sparse. For large obstacles, due to their large size, a relatively large number of points can be collected, resulting in better detection of large obstacles. However, for small obstacles, a relatively small number of points can be collected, which may prevent the detection system from identifying the point cloud corresponding to the small obstacle from the collected point cloud, leading to unstable detection results for small obstacles. Summary of the Invention

[0005] This invention provides an obstacle detection method, apparatus, electronic device, and storage medium to solve the problem of unstable detection performance for small obstacles when using low-beam lidar point clouds for object detection.

[0006] In a first aspect, the present invention provides an obstacle detection method, comprising:

[0007] While the vehicle is in motion, acquire multiple frames of initial point cloud data collected by multi-line lidar and pose data collected by IMU.

[0008] Based on the pose data, the initial point cloud of multiple frames is subjected to distortion removal processing to obtain multiple frames of corrected point cloud.

[0009] The corrected point clouds from multiple frames are fused into a target point cloud;

[0010] The target point cloud is input into a preset obstacle recognition model to obtain the orientation and size information of small obstacles, which are obstacles whose height is within a preset height range.

[0011] In a second aspect, the present invention provides an obstacle detection device, comprising:

[0012] The data acquisition module is used to acquire multiple frames of initial point cloud data collected by the multi-line lidar and pose data collected by the IMU while the vehicle is in motion.

[0013] The distortion correction module is used to perform distortion correction processing on multiple frames of the initial point cloud based on the pose data to obtain multiple frames of corrected point cloud.

[0014] The point cloud fusion module is used to fuse the corrected point clouds from multiple frames into a target point cloud;

[0015] The target information extraction module is used to input the target point cloud into a preset obstacle recognition model to obtain the orientation and size information of small obstacles, wherein the small obstacles are obstacles whose height is within a preset height range.

[0016] Thirdly, the present invention provides an electronic device, the electronic device comprising:

[0017] At least one processor; and

[0018] A memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the obstacle detection method according to the first aspect of the present invention.

[0020] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the obstacle detection method described in the first aspect of the present invention.

[0021] The obstacle detection method provided in this embodiment of the invention acquires multiple frames of initial point cloud data collected by a multi-line lidar and pose data collected by an IMU while the vehicle is in motion; performs distortion correction processing on the multiple frames of initial point cloud data based on the pose data to obtain multiple frames of corrected point cloud data; fuses the multiple frames of corrected point cloud data to obtain target point cloud data; inputs the target point cloud data into a preset obstacle recognition model to obtain the orientation and size information of small obstacles, wherein the small obstacles are obstacles whose height is within a preset height range. By performing distortion correction on multiple initial point clouds based on pose data, the accuracy of the point cloud data can be improved, enabling it to more realistically reflect environmental information and preventing small obstacles from being difficult to detect due to point cloud distortion. Then, the corrected point clouds from multiple frames are fused into a target point cloud, increasing the point cloud density of the target point cloud and consequently increasing the point cloud density of small obstacles within it. This allows for a more complete representation of the characteristics of small obstacles, facilitating the obstacle recognition model to identify the point cloud corresponding to the small obstacle from the target point cloud. Based on this point cloud, the orientation and size information of the small obstacle can be determined, thus pinpointing its specific location in the scene and improving the stability of small obstacle detection.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of an obstacle detection method provided in Embodiment 1 of the present invention;

[0025] Figure 2 This is a flowchart of an obstacle detection method provided in Embodiment 2 of the present invention;

[0026] Figure 3 This is an XY plane view of a radar image labeled with a minimum cube, provided in Embodiment 2 of the present invention;

[0027] Figure 4 This is a schematic diagram of the structure of an obstacle detection device provided in Embodiment 3 of the present invention;

[0028] Figure 5 This is a schematic diagram of the structure of the electronic device provided in Embodiment 4 of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] Example 1

[0031] Figure 1 This is a flowchart of an obstacle detection method provided in Embodiment 1 of the present invention. This embodiment is applicable to the detection of small obstacles on the road for autonomous vehicles. The method can be executed by an obstacle detection device, which can be implemented in hardware and / or software and can be configured in an electronic device, such as the vehicle's onboard computer. Figure 1 As shown, the obstacle detection method includes:

[0032] S101. While the vehicle is in motion, acquire multiple frames of initial point cloud data collected by the multi-line lidar and pose data collected by the IMU.

[0033] The vehicle can be an autonomous vehicle or an autonomous sweeper, with multi-line lidar and IMU installed on it.

[0034] Multi-line lidar is a type of radar system and an active sensor. LiDAR measures point data representing the surface characteristics of a target object, forming a point cloud—a dataset of points in a coordinate system. The target detection performance of lidar is typically positively correlated with the number of lines in the radar beam. However, given the high cost of high-beam lidar, existing technologies primarily employ low-beam lidar. In this embodiment, the vehicle-mounted lidar uses a low-beam lidar, and the point cloud it acquires is a single frame of environmental point cloud data. In this embodiment, while the vehicle is in motion, the multi-line lidar can acquire point cloud data of the road surface, obtaining multiple initial point clouds. Generally, these multiple initial point clouds are acquired within a continuous time period, which can include multiple acquisition cycles.

[0035] An IMU (Inertial Measurement Unit) is used to measure the pose data of an object, including angular velocity and acceleration.

[0036] During vehicle operation, the multi-frame point cloud collected by the current multi-line lidar is used as the initial point cloud. Similarly, the pose data is also a point cloud collected in a continuous time period, and the collection time of the pose data covers the collection time of the initial point cloud.

[0037] S102. Based on the pose data, perform distortion correction processing on the initial point clouds of multiple frames to obtain corrected point clouds of multiple frames.

[0038] For most LiDAR systems, due to the delay during scanning, each point in the point cloud is not generated at the same time. For example, in a 360-degree multi-line LiDAR, each frame of the point cloud is obtained by scanning around the radar. If a vehicle is traveling at a low speed (1.5 m / s) in a straight line, and within one scan cycle (100 ms), the vehicle and the multi-line LiDAR move 0.15 m, the accumulated data within that scan cycle will be output as a single frame of point cloud. In this frame, the coordinate system of each point will be different. Intuitively, this frame of point cloud data will be "distorted" and cannot accurately correspond to the detected environmental information; this is called the LiDAR's motion distortion.

[0039] To correct distortion in a point cloud frame that has experienced self-motion distortion, the coordinates of all points in the frame can be transformed to the same coordinate system. Pose data represents the position and orientation of each point in the point cloud. By comparing the pose data of different points, the transformation relationship between the coordinate systems of the points can be determined. This transformation relationship is then used to transform all points to the same coordinate system, resulting in a corrected point cloud, thus achieving point cloud distortion correction. Here, the pose data of a point refers to the pose data corresponding to the timestamp of the point. Based on existing IMU-acquired position data, the pose data corresponding to each point can be obtained using neighborhood interpolation or bilinear interpolation.

[0040] Specifically, a reference point can be determined, and the transformation relationship between the coordinate system of the other points and the coordinate system of the reference point can be calculated based on the pose data of the other points and the pose data of the reference point. Then, the coordinates of the other points are transformed to the coordinate system of the reference point, thereby obtaining a multi-frame corrected point cloud.

[0041] S103. Fuse the multi-frame corrected point cloud into the target point cloud.

[0042] After distortion correction processing of the point cloud, each frame of the point cloud can accurately reflect the relevant environmental information. However, the environmental information reflected by a single frame of point cloud is relatively small. When the point cloud is sparse, the accuracy of detecting various obstacles is low, especially the accuracy of detecting small obstacles. This is because the scanning range of multi-line lidar is limited, and the points obtained by scanning are generally local data of the object, rather than global data. For small obstacles with low height (5-25cm), their volume is small, and the point cloud of small obstacles that multi-line lidar can collect is relatively small. It is difficult to determine the full picture of small obstacles based on the collected point cloud, making it difficult to detect small obstacles. Therefore, in this embodiment, the point clouds after distortion correction processing of multiple frames are fused to increase the point cloud density and thus improve the detection rate of small obstacles.

[0043] For point clouds from different frames, the acquisition time for each frame is different. Due to vehicle movement, the relative position of the same object and the multi-line lidar is constantly changing. Directly fusing multiple point clouds results in a radar image with ghosting, failing to enhance the features of small obstacles. Therefore, point clouds from different frames can be converted into point clouds in the same coordinate system before fusion. In this embodiment, the transformation relationship between different point clouds can be calculated using point cloud registration algorithms such as ICP (Iterative Closest Point) or NDT (Normal Distribution Transform). Based on this transformation relationship, point clouds from different frames are converted into point clouds in the same coordinate system. Point clouds in the same coordinate system can be directly fused to obtain the target point cloud.

[0044] S104. Input the target point cloud into the preset obstacle recognition model to obtain the orientation and size information of small obstacles.

[0045] Small obstacles are those with a height within a preset range, which can be set to 5-25cm. Small obstacles are generally low-height, small objects on the road where vehicles travel, such as aluminum cans, stones, and small animals.

[0046] The pre-defined obstacle recognition model can be obtained through training. For example, a training point cloud set can be pre-acquired, which includes point cloud data of small obstacles. The point cloud data containing small obstacles is labeled with information and confidence level. The label information includes the orientation and size information of the small obstacles. Then, the point cloud in the training point cloud set is input into the initialized obstacle recognition model to obtain output data. The parameters of the obstacle recognition model are adjusted according to the output data and the label information and confidence level of the input point cloud, and finally, an obstacle recognition model that meets the training completion standard is obtained.

[0047] Once the obstacle recognition model is trained, it can be used to extract the orientation and size information of small obstacles from the target point cloud, and obstacle avoidance or clearing can be performed based on the orientation and size information of small obstacles.

[0048] The obstacle detection method provided in this embodiment of the invention acquires multiple frames of initial point cloud data collected by a multi-line lidar and pose data collected by an IMU while the vehicle is in motion; performs distortion correction processing on the multiple frames of initial point cloud data based on the pose data to obtain multiple frames of corrected point cloud data; fuses the multiple frames of corrected point cloud data to obtain target point cloud data; inputs the target point cloud data into a preset obstacle recognition model to obtain the orientation and size information of small obstacles, wherein the small obstacles are obstacles whose height is within a preset height range. By performing distortion correction on multiple initial point clouds based on pose data, the accuracy of the point cloud data can be improved, enabling it to more realistically reflect environmental information and preventing small obstacles from being difficult to detect due to point cloud distortion. Then, the corrected point clouds from multiple frames are fused into a target point cloud, increasing the point cloud density of the target point cloud and consequently increasing the point cloud density of small obstacles within it. This allows for a more complete representation of the characteristics of small obstacles, facilitating the obstacle recognition model to identify the point cloud corresponding to the small obstacle from the target point cloud. Based on this point cloud, the orientation and size information of the small obstacle can be determined, thus pinpointing its specific location in the scene and improving the stability of small obstacle detection.

[0049] Example 2

[0050] Figure 2 This is a flowchart of an obstacle detection method provided in Embodiment 2 of the present invention. This embodiment of the present invention is an optimization based on Embodiment 1 described above, such as... Figure 2 As shown, the obstacle detection method includes:

[0051] S201. While the vehicle is in motion, acquire multiple frames of initial point cloud data collected by the multi-line lidar and pose data collected by the IMU.

[0052] The initial point cloud typically consists of multiple frames acquired by a multi-line LiDAR over a continuous time period, while the pose data is acquired by an IMU over a continuous time period. The acquisition time range of the pose data is greater than or equal to the acquisition time range of the initial point cloud. Both the multi-line LiDAR and the IMU, as sensors for the vehicle, can connect to the onboard computer. The multi-line LiDAR and IMU send the acquired data to the onboard computer, which then uses this data for obstacle recognition.

[0053] S202. For each frame of the initial point cloud, determine the reference point from the initial point cloud, and take the other points in the initial point cloud other than the reference point as distortion points.

[0054] Because the position of the multi-line LiDAR installed on the vehicle changes rapidly during normal vehicle operation, and a point cloud frame is formed within a scanning cycle, the changing position of the multi-line LiDAR within a scanning cycle means that the spatial position of each point in a point cloud frame is different, i.e., the coordinate system of each point is different, causing self-distortion of the point cloud. To correct the distortion of the point cloud, the coordinates of all points can be transformed to the same coordinate system, i.e., the points in the point cloud are projected to the same pose, so that all points in a point cloud frame appear as if they were acquired at the same time. Therefore, for each initial point cloud frame, a reference point can be determined from the initial point cloud, and the other points can be used as distortion points. The coordinates of the distortion points can then be transformed to the coordinate system of the reference point.

[0055] S203. Motion compensation is performed on the points in the initial point cloud using pose data to obtain interpolated pose data for each point.

[0056] For each point in the point cloud, motion compensation is a method to describe the difference between adjacent frames. Specifically, it describes how each small block in the previous frame moves to a certain position in the current frame. LiDAR cannot perform motion compensation through optics itself because there is no velocity in the point cloud data, that is, there is no correlation between points before and after. IMU can be used to detect the pose data of the LiDAR itself, determine the correlation between points through the pose data, and then perform motion compensation on the point cloud.

[0057] For example, the pose of the current frame can be considered as the pose of the lidar itself when the last point in the current frame point cloud is sampled. Then, the last point in the initial point cloud can be used as the reference point, and the pose data can be used to perform motion compensation on the points in the initial point cloud to obtain the interpolated pose data of each point. This includes: for each point in the initial point cloud, the pose data of the two frames before and after the timestamp corresponding to the point is used as the reference pose data, and the motion change of the two frames of reference pose data is used as the interpolated pose data of the point.

[0058] Since IMU and multi-line lidar are two different sensors, their acquisition start time and acquisition period may be different. The acquisition frequency of IMU is much higher than that of multi-line lidar. For example, the acquisition frequency of multi-line lidar is 10Hz, while the acquisition frequency of IMU can be 200Hz.

[0059] S204. For each distortion point, calculate the spatial pose relationship between the distortion point and the reference point based on the interpolated pose data between the timestamp of the distortion point and the timestamp of the reference point.

[0060] After knowing the interpolated pose data for each point, for each distorted point, the spatial pose relationship between the distorted point and the reference point is calculated based on the interpolated position data within the start and end times. For example, the spatial pose relationship can be calculated by integrating the interpolated pose data.

[0061] S205. In the initial point cloud of each frame, the distorted points are transformed into the coordinate system of the reference point according to the spatial pose relationship corresponding to the distorted points, so as to obtain the corrected point cloud.

[0062] In each initial point cloud frame, after obtaining the spatial pose relationships between points, the coordinates of each distorted point can be transformed to the coordinate system of the reference point, resulting in multiple undistorted points. These undistorted points, along with the reference point, form a corrected point cloud frame. This process is repeated to obtain multiple corrected point clouds from multiple initial point clouds. Since point clouds do not include velocity data, motion compensation (correction) cannot be performed based on the point cloud itself. However, pose data includes angular velocity and acceleration, and there is a correlation between the reference pose data of two consecutive frames. Therefore, motion compensation can be performed on the point based on the reference pose data of the two consecutive frames, achieving the distortion removal effect.

[0063] S206. Fuse the multiple frames of corrected point clouds to obtain the target point cloud.

[0064] Because the point cloud collected by multi-line lidar is relatively sparse, when the point cloud is sparse, a single frame of point cloud can reflect less environmental information, resulting in lower accuracy in detecting various obstacles, especially small obstacles. Therefore, in this embodiment, the point clouds after distortion correction of multiple frames are fused to increase the point cloud density and thus improve the detection rate of small obstacles. Small obstacles are obstacles with a height within a preset height range, which can be 5-25cm.

[0065] For point clouds of different frames, the acquisition time of each frame is different. Due to vehicle movement, the relative position of the same object and the multi-line lidar is constantly changing. The radar image obtained by directly fusing multiple frame point clouds will have ghosting and will not be able to enhance the features of small obstacles. Therefore, point clouds of different frames can be converted into point clouds in the same coordinate system before fusion. In this embodiment, a first transformation matrix from the coordinate system of the second point cloud to the map coordinate system of the vehicle and a second transformation matrix from the map coordinate system to the coordinate system of the first point cloud can be obtained. Based on the first transformation matrix, multiple frames of the second point cloud are converted into point clouds in the map coordinate system to obtain multiple frames of the fourth point cloud. Based on the second transformation matrix, multiple frames of the fourth point cloud are converted into point clouds in the coordinate system of the first point cloud to obtain multiple frames of the third point cloud. The multiple frames of the third point cloud are then fused with the first point cloud to form the target point cloud. The first transformation matrix can be obtained using a matching method or a fusion localization method. As for the second transformation matrix, an intermediate transformation matrix from the coordinate system of the first point cloud to the map coordinate system of the vehicle can be obtained in advance, and an inverse matrix transformation can be performed on the intermediate transformation matrix to obtain the second transformation matrix.

[0066] The target point cloud includes point clouds of small obstacles from multiple frames, increasing the point cloud density of these small obstacles. Point cloud density is an indicator of data resolution; for the same object, higher point cloud density means more information or higher image resolution, while lower point cloud density means less information or lower image resolution. More information or higher image resolution can more fully reflect the characteristics of small obstacles, thus improving the accuracy and stability of small obstacle detection. Furthermore, the point cloud set in this embodiment includes the latest frame of point cloud data, i.e., the latest obstacle data, which can maintain the real-time performance of obstacle detection during point cloud fusion.

[0067] S207. Input the target point cloud into the preset obstacle recognition model to obtain label information and confidence level.

[0068] The label information consists of the location and size information of small obstacles, and the confidence level is the confidence level of the label information.

[0069] The pre-defined obstacle recognition model is trained in the following way:

[0070] Obtain a training point cloud set containing multiple point cloud subsets. When the point cloud subset is a preset type of point cloud subset, the point cloud subset includes the point cloud corresponding to small obstacles.

[0071] For each preset type of point cloud subset, a first label and a first confidence level are assigned. The first label includes the first orientation and first size information of small obstacles; the first confidence level is the confidence level of the first label information.

[0072] Initialize the obstacle recognition model;

[0073] A random subset of point cloud data is extracted and input into the obstacle recognition model to obtain the second label information and the second confidence level of the point cloud subset. The second label information includes the second orientation information and the second size information of small obstacles; the second confidence level is the confidence level of the second label information.

[0074] The error is calculated using the first label information, the first confidence level, the second label information, and the second confidence level.

[0075] Determine whether the error is less than a preset error threshold;

[0076] If so, stop training the obstacle recognition model and obtain the trained obstacle recognition model;

[0077] If not, the model parameters of the obstacle recognition model are adjusted using error correction, and the process returns to the step of randomly extracting training images and inputting them into the obstacle recognition model.

[0078] The point cloud data in the training point cloud set is collected in multiple scenarios, such as vehicle driving roads, and these scenarios need to contain various small obstacles to facilitate obtaining point cloud data corresponding to these small obstacles, while ensuring the richness of the point cloud data. Each subset of the training point cloud can be viewed as a multi-frame point cloud of an image. The number of frames in a subset can be set according to the vehicle speed; for example, at a vehicle speed of 1.5 m / s, a subset could include 10 frames. To ensure the model's training effect, the number of point cloud subsets is no less than the preset number of training images, such as 6000 images. It should be noted that some subsets of the training point cloud set contain point cloud data corresponding to small obstacles. When using the training point cloud set for training, the subsets can be divided into training, validation, and test sets for the training, validation, and testing phases of the small obstacle model, respectively. Furthermore, TensorRT can be used to accelerate model inference, thereby speeding up inference and reducing memory resource consumption.

[0079] The point cloud in the training point cloud set can be acquired using solid-state radar or multi-line lidar. When using multi-line lidar, the acquired point cloud can be processed using steps S101-S103 in Example 1 or steps S201-S203 in this example, making the point cloud in the training point cloud set a dense point cloud that better reflects the characteristics of small obstacles. In the above training method, the label information includes the orientation and size information of the small obstacles. The label information can fully and accurately represent the characteristics of the small obstacles. The obstacle recognition model is trained using the error of the first label information, the first confidence level, the second label information, and the second confidence level. This allows the obstacle recognition model to learn the characteristics of small obstacles more fully and accurately extract the point cloud data corresponding to the small obstacles from the training point cloud set.

[0080] In an optional embodiment of the present invention, labeling each subset of point clouds with first label information includes: generating a radar image based on each subset of point clouds; marking the smallest cube surrounding the small obstacle in the radar image, and using the orientation information and size information of the smallest cube as the first label information, wherein the orientation information includes the center coordinates and direction angle of the smallest cube.

[0081] The radar image can be a 3D radar image. Labeling the smallest cube surrounding small obstacles in the radar image involves annotating the radar image based on the known point cloud data of the small obstacles. Specifically, the smallest cube is drawn in the radar image based on the known coordinates of the center point of the small obstacle and its length, width, and height. Then, the azimuth angle is determined based on the coordinate axes of the smallest cube and the radar image. It should be noted that a preset type of point cloud subset may include point clouds corresponding to multiple small obstacles, so the generated radar image may include multiple smallest cubes corresponding to small obstacles. For each cube, a label and a confidence level need to be assigned. Therefore, a preset type of point cloud subset may correspond to multiple labels and multiple confidence levels. After labeling each preset type of point cloud subset with the first label, the confidence level corresponding to that first label is 1.

[0082] The azimuth angle, or cube orientation, is the direction of the cube's length in the radar image. Since a cube cannot be uniquely determined solely by its length, width, height, and center point coordinates, adding the azimuth angle parameter ensures that each small obstacle corresponds to a unique cube, thus facilitating the determination of the small obstacle's specific location.

[0083] After inputting the target point cloud into a preset obstacle recognition model to obtain label information and confidence levels, the smallest cube (minimum bounding box) corresponding to small obstacles can be marked in the radar image based on the label information, such as... Figure 3The image shown is an XY plane view of a radar image. This view displays the locations of all small obstacles in the environment corresponding to the target image, facilitating the vehicle's autonomous driving system's identification of these obstacles. Furthermore, the confidence level of the label information corresponding to the bounding box of the view can be annotated.

[0084] S208. When the confidence level is greater than the preset confidence threshold, the label information is used as the orientation and size information of the small obstacle.

[0085] The confidence threshold is used to measure the credibility of the tag information. When the confidence level is greater than the preset confidence threshold, the tag information is considered to be highly credible, and this tag information can be used as information about the corresponding small obstacle, namely its location and size. After obtaining the location and size information of the small obstacle, the vehicle can avoid or clear it based on this information.

[0086] The obstacle detection method provided in this invention addresses the issue that, since point clouds do not include velocity data, motion compensation (correction) cannot be performed based on the point clouds themselves. However, pose data includes angular velocity and acceleration, and there is a correlation between the reference pose data of two consecutive frames. Motion compensation can be performed on the points based on the reference pose data of the two consecutive frames, achieving distortion reduction and improving the accuracy of the point cloud data. This allows the point cloud data to more realistically reflect environmental information and avoids the difficulty in detecting small obstacles due to point cloud distortion. On the other hand, by using the error of the first label information, the first confidence level, the second label information, and the second confidence level to train the obstacle recognition model, the obstacle recognition model can learn the features of small obstacles more fully. This allows the model to accurately extract the point cloud data corresponding to small obstacles from the training point cloud set, thereby improving the detection rate of small obstacles.

[0087] Example 3

[0088] Figure 4 This is a schematic diagram of the structure of an obstacle detection device provided in Embodiment 3 of the present invention. Figure 4 As shown, the obstacle detection device includes:

[0089] The data acquisition module 401 is used to acquire multiple frames of initial point cloud data collected by the multi-line lidar and pose data collected by the IMU when the vehicle is in motion.

[0090] The distortion correction module 402 is used to perform distortion correction processing on multiple frames of the initial point cloud based on the pose data to obtain multiple frames of corrected point cloud.

[0091] Point cloud fusion module 403 is used to fuse the corrected point clouds from multiple frames into a target point cloud;

[0092] The target information extraction module 404 is used to input the target point cloud into a preset obstacle recognition model to obtain the orientation and size information of small obstacles, wherein the small obstacles are obstacles whose height is within a preset height range.

[0093] In an optional embodiment of the present invention, the distortion correction module 402 includes:

[0094] The reference point determination submodule is used to determine a reference point from the initial point cloud for each frame, and to treat other points in the initial point cloud other than the reference point as distortion points.

[0095] The interpolation pose data calculation submodule is used to perform motion compensation on the points in the initial point cloud using the pose data to obtain the interpolation pose data of each point.

[0096] The spatial pose relationship calculation submodule is used to calculate the spatial pose relationship between the distorted point and the reference point for each distorted point based on the interpolated pose data between the timestamp of the distorted point and the timestamp of the reference point.

[0097] The corrected point cloud acquisition submodule is used to transform the distorted points into the coordinate system of the reference point in each frame of the initial point cloud according to the spatial pose relationship corresponding to the distorted points, so as to obtain the corrected point cloud.

[0098] In an optional embodiment of the present invention, the interpolation pose data calculation submodule includes:

[0099] The reference pose data acquisition unit is used to take the pose data of the two frames before and after the timestamp corresponding to each point in the initial point cloud as reference pose data.

[0100] The interpolation pose data calculation unit is used to use the motion change of the two frames of the reference pose data as the interpolation pose data of the point.

[0101] In an optional embodiment of the present invention, the point cloud fusion module 403 includes:

[0102] The reference point cloud determination submodule is used to determine a first point cloud and a second point cloud acquired before the first point cloud from multiple frames of the correction point cloud;

[0103] The point cloud conversion submodule is used to convert the second point cloud into a point cloud in the coordinate system of the first point cloud to obtain multiple frames of the third point cloud.

[0104] The target point cloud acquisition submodule is used to fuse the third point cloud from multiple frames with the first point cloud to form a target point cloud.

[0105] In an optional embodiment of the present invention, the point cloud conversion submodule includes:

[0106] The transformation relationship acquisition unit is used to acquire a first transformation matrix from the second point cloud coordinate system to the map coordinate system of the vehicle in each frame, and a second transformation matrix from the map coordinate system to the first point cloud coordinate system.

[0107] The fourth point cloud acquisition unit is used to convert multiple frames of the second point cloud into point clouds in the map coordinate system based on the first transformation matrix, so as to obtain multiple frames of the fourth point cloud;

[0108] The third point cloud acquisition unit is used to convert multiple frames of the fourth point cloud into point clouds in the first point cloud coordinate system based on the second transformation matrix, so as to obtain multiple frames of the third point cloud.

[0109] In an optional embodiment of the present invention, the preset obstacle recognition model is trained in the following manner:

[0110] Obtain a training point cloud set containing multiple point cloud subsets. When the point cloud subset is a preset type of point cloud subset, the point cloud subset includes point clouds corresponding to small obstacles.

[0111] For each preset type of point cloud subset, a first label and a first confidence level are assigned. The first label includes the first orientation and first size information of small obstacles; the first confidence level is the confidence level of the first label.

[0112] Initialize the obstacle recognition model;

[0113] A subset of the point cloud is randomly extracted and input into the obstacle recognition model to obtain the second label information and the second confidence level of the point cloud subset. The second label information includes the second orientation information and the second size information of the small obstacle. The second confidence level is the confidence level of the second label information.

[0114] The error is calculated using the first label information, the first confidence level, the second label information, and the second confidence level.

[0115] Determine whether the error is less than a preset error threshold;

[0116] If so, then stop training the obstacle recognition model and obtain a trained obstacle recognition model;

[0117] If not, the error is used to adjust the model parameters of the obstacle recognition model, and the process returns to the step of randomly extracting the training image and inputting it into the obstacle recognition model.

[0118] The step of labeling each subset of the point cloud with first label information includes:

[0119] For each of the point cloud subsets, a radar image is generated based on the point cloud subset;

[0120] The smallest cube surrounding the small obstacle is marked in the radar image;

[0121] The orientation and size information of the smallest cube are used as the first label information, and the orientation information includes the center coordinates and orientation angle of the smallest cube.

[0122] In an optional embodiment of the present invention, the target information extraction module 404 includes:

[0123] The target point cloud recognition submodule is used to input the target point cloud into a preset obstacle recognition model to obtain label information and confidence level. The label information is the orientation and size information of small obstacles, and the confidence level is the confidence level of the label information.

[0124] The target information confirmation submodule is used to use the tag information as the location and size information of small obstacles when the confidence level is greater than a preset confidence threshold.

[0125] The obstacle detection device provided in the embodiments of the present invention can execute the obstacle detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0126] Example 4

[0127] Figure 5 A schematic diagram of an electronic device 50 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0128] like Figure 5As shown, the electronic device 50 includes at least one processor 51 and a memory, such as a read-only memory (ROM) 52 and a random access memory (RAM) 53, communicatively connected to the at least one processor 51. The memory stores computer programs executable by the at least one processor. The processor 51 can perform various appropriate actions and processes based on the computer program stored in the ROM 52 or loaded into the RAM 53 from storage unit 58. The RAM 53 can also store various programs and data required for the operation of the electronic device 50. The processor 51, ROM 52, and RAM 53 are interconnected via a bus 54. An input / output (I / O) interface 55 is also connected to the bus 54.

[0129] Multiple components in electronic device 50 are connected to I / O interface 55, including: input unit 56, such as keyboard, mouse, etc.; output unit 57, such as various types of monitors, speakers, etc.; storage unit 58, such as disk, optical disk, etc.; and communication unit 59, such as network card, modem, wireless transceiver, etc. Communication unit 59 allows electronic device 50 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0130] Processor 51 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 51 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 51 performs the various methods and processes described above, such as obstacle detection methods.

[0131] In some embodiments, the obstacle detection method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 58. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 50 via ROM 52 and / or communication unit 59. When the computer program is loaded into RAM 53 and executed by processor 51, one or more steps of the obstacle detection method described above may be performed. Alternatively, in other embodiments, processor 51 may be configured to perform the obstacle detection method by any other suitable means (e.g., by means of firmware).

[0132] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0133] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0134] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0135] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0136] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0137] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0138] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0139] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An obstacle detection method, characterized in that, include: While the vehicle is in motion, acquire multiple frames of initial point cloud data collected by multi-line lidar and pose data collected by IMU. Based on the pose data, the initial point cloud of multiple frames is subjected to distortion removal processing to obtain multiple frames of corrected point cloud. The corrected point clouds from multiple frames are fused into a target point cloud; The target point cloud is input into a preset obstacle recognition model to obtain the orientation and size information of small obstacles, wherein the small obstacles are obstacles whose height is within a preset height range; The preset obstacle recognition model is trained in the following way: Obtain a training point cloud set containing multiple point cloud subsets. When the point cloud subset is a preset type of point cloud subset, the point cloud subset includes point clouds corresponding to small obstacles. For each preset type of point cloud subset, first label information and first confidence level are labeled. The first label information includes the first orientation information and first size information of small obstacles. The first confidence level is the confidence level of the first tag information. Initialize the obstacle recognition model; Randomly extract a subset of the point cloud and input it into the obstacle recognition model to obtain the second label information and the second confidence level of the point cloud subset. The second label information includes the second orientation information and the second size information of the small obstacle. The second confidence level is the confidence level of the second tag information; The error is calculated using the first label information, the first confidence level, the second label information, and the second confidence level. Determine whether the error is less than a preset error threshold; If so, then stop training the obstacle recognition model and obtain a trained obstacle recognition model; If not, the error is used to adjust the model parameters of the obstacle recognition model, and the process returns to the step of randomly extracting the point cloud subset and inputting it into the obstacle recognition model. The step of labeling each subset of the point cloud with first label information includes: For each of the point cloud subsets, a radar image is generated based on the point cloud subset; The smallest cube surrounding the small obstacle is marked in the radar image; The orientation and size information of the smallest cube are used as the first label information, and the orientation information includes the center coordinates and orientation angle of the smallest cube.

2. The method as described in claim 1, characterized in that, The process of performing distortion correction on multiple frames of the initial point cloud based on the pose data to obtain multiple frames of corrected point cloud includes: For each frame of the initial point cloud, a reference point is determined from the initial point cloud, and other points in the initial point cloud other than the reference point are taken as distortion points; The pose data is used to perform motion compensation on the points in the initial point cloud to obtain interpolated pose data for each point. For each of the distortion points, the spatial pose relationship between the distortion point and the reference point is calculated based on the interpolated pose data between the timestamp of the distortion point and the timestamp of the reference point. In each frame of the initial point cloud, the distorted point is transformed into the coordinate system of the reference point according to the spatial pose relationship corresponding to the distorted point, thus obtaining the corrected point cloud.

3. The method as described in claim 2, characterized in that, The reference point is the point at the last moment in the initial point cloud. The motion compensation of the points in the initial point cloud using the pose data to obtain interpolated pose data for each point includes: For each point in the initial point cloud, the pose data of the two frames before and after the timestamp corresponding to the point are used as reference pose data; The motion changes of the two frames of the reference pose data are used as the interpolated pose data of the points.

4. The method as described in claim 1, characterized in that, The step of fusing multiple frames of the corrected point cloud into a target point cloud includes: A first point cloud and a second point cloud acquired before the first point cloud are determined from the multiple frames of the corrected point cloud; The multiple frames of the second point cloud are converted into point clouds in the coordinate system of the first point cloud to obtain multiple frames of the third point cloud; The third point cloud from multiple frames is fused with the first point cloud to form the target point cloud.

5. The method as described in claim 4, characterized in that, The step of converting multiple frames of the second point cloud into point clouds in the coordinate system of the first point cloud to obtain multiple frames of the third point cloud includes: Obtain a first transformation matrix from the coordinate system of the second point cloud in each frame to the map coordinate system of the vehicle, and a second transformation matrix from the map coordinate system to the coordinate system of the first point cloud. Based on the first transformation matrix, the multiple frames of the second point cloud are converted into point clouds in the map coordinate system to obtain multiple frames of the fourth point cloud; Based on the second transformation matrix, the fourth point cloud in multiple frames is transformed into a point cloud in the coordinate system of the first point cloud to obtain the third point cloud in multiple frames.

6. The method according to any one of claims 1-5, characterized in that, The step of inputting the target point cloud into a preset obstacle recognition model to obtain the orientation and size information of small obstacles includes: The target point cloud is input into a preset obstacle recognition model to obtain label information and confidence level. The label information is the orientation and size information of small obstacles, and the confidence level is the confidence level of the label information. When the confidence level is greater than a preset confidence threshold, the label information is used as the location and size information of the small obstacle.

7. An obstacle detection device, characterized in that, An obstacle detection method for performing any one of claims 1-6 includes: The data acquisition module is used to acquire multiple frames of initial point cloud data collected by the multi-line lidar and pose data collected by the IMU while the vehicle is in motion. The distortion correction module is used to perform distortion correction processing on multiple frames of the initial point cloud based on the pose data to obtain multiple frames of corrected point cloud. The point cloud fusion module is used to fuse the corrected point clouds from multiple frames into a target point cloud; The target information extraction module inputs the target point cloud into a preset obstacle recognition model to obtain the orientation and size information of small obstacles, wherein the small obstacles are obstacles whose height is within a preset height range.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the obstacle detection method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the obstacle detection method according to any one of claims 1-6.

Images