Dynamic obstacle detection method for autonomous driving

Abstract

The application relates to a dynamic obstacle detection method for automatic driving and belongs to the technical field of automatic driving. The method combines two-dimensional image data and three-dimensional point cloud data, utilizes a deep learning algorithm, compares and fuses detection results of the two kinds of data, and realizes three-dimensional obstacle detection. A semantic segmentation network is utilized to process two-dimensional images collected by a camera, extracts lane and obstacle data, and extracts the relative position of the obstacle on the image. Image coordinates are converted into world coordinates. Meanwhile, point cloud data is utilized to detect obstacles through a threeD target detection network, the relative position of the obstacle and a laser radar is extracted, and the position of the obstacle on a laser scanner coordinate is calculated. Finally, the laser scanner coordinate is rotated and translated, compared with the world coordinate, and the detection results of the two kinds of data are matched. The application can reduce redundant data, reduce detection time, and improve detection precision.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving technology and relates to a dynamic obstacle detection method for autonomous driving. Background Technology

[0002] In the field of autonomous driving technology, road obstacle detection is a crucial technology. Intelligent vehicles typically use onboard sensors such as cameras, radar (e.g., LiDAR, millimeter-wave radar), GPS, and IMU (Inertial Measurement Unit) to perceive their surroundings. They then employ appropriate obstacle detection algorithms to detect obstacles in their path and control the vehicle to take avoidance measures. Specifically, when using LiDAR for obstacle detection, the LiDAR emits a laser beam and constructs a laser point cloud based on the reflected beams from object surfaces, thus identifying obstacles in the environment. When using cameras for obstacle detection, the camera captures environmental images and uses image recognition technology to identify obstacles within those images.

[0003] However, due to the limitations of the working principles of LiDAR and cameras, LiDAR can only detect obstacles in a small area, while the detection error of obstacle distance is large when using cameras for identification, which affects the accuracy of obstacle detection.

[0004] In obstacle detection based on collected data, 3D point cloud data acquired by LiDAR has high accuracy, but its sparse point cloud makes it difficult to identify 3D targets. While 2D image data acquired by a monocular camera combined with deep learning algorithms can achieve accurate target recognition, the lack of 3D positional information in 2D images prevents 3D target localization. Images acquired by binocular cameras can generate dense 3D point clouds through disparity calculations for 3D target recognition and localization, but the complexity of disparity calculations means current algorithms are either too slow or inaccurate. Even after classification and clustering, the sparsity of the point cloud still leads to inaccurate bounding boxes in the regression. Therefore, combining 2D image information and 3D point cloud information for obstacle detection is helpful for enabling automatic obstacle avoidance in intelligent vehicles. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a dynamic obstacle detection method, which achieves the effects of reducing redundant data, reducing detection time, and improving detection accuracy by matching the detection results of two-dimensional image data from a camera with the three-dimensional point cloud data from a lidar.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A dynamic obstacle detection method for autonomous driving is proposed. This method combines two-dimensional image data and three-dimensional point cloud data, and uses a deep learning algorithm to compare and fuse the detection results of the two-dimensional image data and the three-dimensional point cloud data to achieve obstacle detection. The details are as follows:

[0008] S1. Acquire two-dimensional and three-dimensional point cloud images of the road;

[0009] S2. Use a semantic segmentation network to process the two-dimensional image, extract lane data and obstacle data, as well as the relative position of the obstacles on the image, and convert the image coordinates into world coordinates.

[0010] S3. Use a 3D target detection network to detect obstacles in the 3D point cloud image and calculate the position of the obstacle on the coordinates of the laser scanner to obtain the obstacle coordinate value.

[0011] S4. Map the world coordinates obtained in step S2 and the coordinate values ​​obtained in step S3 to the YOZ plane coordinates and perform matching. If the matching is successful, output the three-dimensional information of the obstacle.

[0012] Furthermore, in step S2, the image processing process is as follows: the two-dimensional image is input into the convolutional neural network, features are extracted through convolutional and pooling layers, then upsampled, and finally a segmented image with the same size as the original image is output.

[0013] In step S2, the coordinate transformation process is as follows:

[0014] 1) Obstacles are represented by pixel coordinates on the image, so they are first converted to image coordinates, as shown in the following formula:

[0015]

[0016] In the formula, u and v represent the coordinates of the obstacle in pixel coordinates, u0 and v0 represent the coordinates of the center point of the image in pixel coordinates, and dx and dy represent the scale values ​​of each column and each row, respectively, in millimeters.

[0017] 2) Convert the image coordinates to camera coordinates, as shown in the following formula:

[0018]

[0019] In the formula, X c Y c Z c These represent the coordinates of the obstacle in the camera coordinate system, and f represents the camera focal length.

[0020] 3) Rotate and translate the camera coordinates to convert them into world coordinates, as shown in the following formula:

[0021]

[0022] In the formula, X w Y w and Z w Let R and T represent the coordinates of the obstacle in world coordinates, respectively. Let R represent the rotation matrix and T represent the translation matrix.

[0023] Furthermore, in step S3, the process of calculating the obstacle coordinates is as follows:

[0024] 1) Extract the distance ρ and the included angle Θ between the obstacle and the lidar.

[0025] 2) Calculate the obstacle coordinates based on the distance and the included angle, as shown in the following formula:

[0026]

[0027] In the formula, X socs Y scos and Z socs These represent the coordinates of the obstacle in the laser scanner coordinate system.

[0028] Furthermore, in step S4, the final output of the obstacle's three-dimensional information is specifically the obstacle's three-dimensional information obtained through the 3D target detection network.

[0029] The beneficial effects of this invention are as follows: This invention uses a deep learning algorithm to compare and fuse the detection results of two-dimensional image data and three-dimensional point cloud data to achieve three-dimensional obstacle detection, which can reduce redundant data, reduce detection time, and improve detection accuracy.

[0030] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0031] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0032] Figure 1 This is a flowchart of a dynamic obstacle detection method.

[0033] Figure 2 This is a schematic diagram of coordinate transformation and comparison fusion. Detailed Implementation

[0034] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0035] like Figure 1 , Figure 2 As shown, this is a dynamic obstacle detection method for autonomous driving. This method combines 2D image data and 3D point cloud data, utilizing deep learning algorithms to compare and fuse the detection results from both datasets to detect 3D obstacles. Specifically, a semantic segmentation network processes the 2D images captured by the camera, extracting lane and obstacle data and determining the relative positions of obstacles on the image, then converting the image coordinates to world coordinates. A 3D object detection network then detects obstacles in the 3D point cloud data acquired by the LiDAR, extracting the relative positions of obstacles to the LiDAR and calculating the obstacle's position in the LiDAR scanner's coordinate system. Finally, the LiDAR scanner's coordinates are rotated and translated, compared with the world coordinates, and the detection results from the 2D image data and 3D point cloud data are matched. If a match is successful, the 3D information of the obstacle is output through the 3D object detection network.

[0036] The specific process for acquiring and processing two-dimensional image data is as follows:

[0037] First, a camera captures an image of the front of the vehicle, which is then used as input. A deep learning-based semantic segmentation network is used to segment the lane and obstacles. The input image is processed by a convolutional neural network, where features are extracted through convolutional and pooling layers. The image is then upsampled to obtain a segmented image of the same size as the original image, which is then output as the segmented image.

[0038] When an obstacle is detected, its position in the image is extracted, and then a coordinate transformation is performed. The obstacle's position on the image is represented by pixel coordinates, which need to be converted to image coordinates, as shown in the following formula:

[0039]

[0040] In the formula, u and v represent the coordinates of the obstacle in pixel coordinates, u0 and v0 represent the coordinates of the center point of the image in pixel coordinates, and dx and dy represent the scale values ​​of each column and each row, respectively, in millimeters.

[0041] After obtaining the position of the obstacle in image coordinates, the image coordinates are then converted to camera coordinates, as shown in the following formula:

[0042]

[0043] In the formula, X c Y c Z c Let represent the coordinates of the obstacle in camera coordinates, and f represent the camera focal length. Based on this, rotating and translating the camera coordinates yields the coordinates of the obstacle in world coordinates, as shown in the following formula:

[0044]

[0045] In the formula, X w Y w and Z w These represent the coordinates of the obstacle in world coordinates; R represents a 1×3 rotation matrix, obtained by multiplying the rotation matrices R1, R2, and R3 around the X, Y, and Z axes; T represents a 3×1 translation matrix. This represents a 3×3 matrix where all elements are 0.

[0046] The specific steps for acquiring and processing 3D point cloud data are as follows:

[0047] Point cloud data of the area directly in front of the vehicle is acquired using LiDAR. A 3D target detection network is then used to process the point cloud data, performing obstacle detection and extracting the distance ρ and the included angle Θ between the obstacle and the LiDAR. After obtaining the distance and angle values, calculate the coordinates of the obstacle in the laser scanner coordinate system using the following formula:

[0048]

[0049] In the formula, X socs Y scos and Z socs These represent the coordinates of the obstacle in the laser scanner coordinate system.

[0050] After obtaining the coordinates of the obstacle in the laser scanner coordinates, the obstacle information in the laser scanner coordinates is mapped to the YOZ plane coordinates. At the same time, the obstacle information in the world coordinates is mapped to the YOZ plane coordinates. The detection results of the two types of data are matched. If the match is successful, the three-dimensional information of the obstacle obtained by the 3D target detection network is output.

[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A dynamic obstacle detection method for autonomous driving, characterized in that: This method combines two-dimensional image data and three-dimensional point cloud data, and uses a deep learning algorithm to compare and fuse the detection results of the two-dimensional image data and the three-dimensional point cloud data to achieve obstacle detection, as detailed below: S1. Acquire two-dimensional images and three-dimensional point cloud images of the road; S2. Use a semantic segmentation network to process the two-dimensional image, extract lane data and obstacle data, as well as the relative position of the obstacles on the image, and convert the image coordinates into world coordinates. S3. Use a 3D target detection network to detect obstacles in the 3D point cloud image and calculate the position of the obstacle on the coordinates of the laser scanner to obtain the obstacle coordinate value. S4. Map the world coordinates obtained in step S2 and the coordinate values ​​obtained in step S3 to the YOZ plane coordinates and perform matching. If the matching is successful, output the three-dimensional information of the obstacle. The three-dimensional information of the obstacle is the distance and angle between the obstacle and the lidar obtained by the 3D target detection network.

2. The obstacle detection method according to claim 1, characterized in that: In step S2, the image processing process is as follows: the two-dimensional image is input into the convolutional neural network, features are extracted through convolutional and pooling layers, then upsampled, and finally a segmented image with the same size as the original image is output.

3. The obstacle detection method according to claim 1, characterized in that: In step S2, the coordinate transformation process is as follows: 1) Obstacles are represented by pixel coordinates on the image, so they are first converted to image coordinates, as shown in the following formula: In the formula, u , v These represent the coordinates of the obstacle in pixel coordinates. , These represent the pixel coordinates of the image's center point. dx , dy These represent the scale values ​​for each column and each row, in millimeters. 2) Convert the image coordinates to camera coordinates, as shown in the following formula: In the formula, , , These represent the coordinates of the obstacles in the camera coordinate system. f Indicates the camera's focal length; 3) Rotate and translate the camera coordinates to convert them into world coordinates, as shown in the following formula: In the formula, X w , Y w and Z w These represent the coordinates of the obstacles in world coordinates. R Represents the rotation matrix. T Represents the translation matrix. This represents a matrix with elements equal to 0.

4. The obstacle detection method according to claim 1, characterized in that: In step S3, the process of calculating the obstacle coordinates is as follows: 1) Extract the distance between obstacles and the lidar and included angle and ; 2) Calculate the obstacle coordinates based on the distance and the included angle, as shown in the following formula: In the formula, X socs , Y socs and Z socs These represent the coordinates of the obstacle in the laser scanner coordinate system.

Images