Obstacle sensing method and device based on vision and laser radar fusion

Abstract

The invention relates to an obstacle sensing method and device based on vision and laser radar fusion, and belongs to the technical field of automatic driving environment perception, and the method comprises the steps: obtaining point cloud data through laser radar scanning, obtaining a camera image through camera shooting, and projecting each point in the point cloud data to a two-dimensional plane to obtain a plurality of point pixel coordinates; screening the point pixel coordinates according to the resolution of the camera image to obtain effective point pixel coordinates, obtaining a fusion feature map according to the camera image and the effective point pixel coordinates, and performing obstacle recognition according to the fusion feature map; according to the invention, fusion recognition is carried out through the camera and the radar, recognition errors caused by changes of environmental factors can be effectively avoided, the reliability of obstacle recognition is improved, and the category of the obstacle can be recognized at the same time.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving environmental perception technology, specifically relating to an obstacle perception method and device based on the fusion of vision and lidar. Background Technology

[0002] In the autonomous driving technology system, environmental perception is one of the core components, and its performance directly determines the safety and reliability of vehicle operation. Single sensors have significant limitations: while cameras can provide rich semantic information, such as obstacle color, texture, and category characteristics, they are easily affected by factors such as lighting and occlusion, and cannot directly obtain the target's three-dimensional spatial position; LiDAR can accurately collect three-dimensional point cloud data of targets, possessing all-weather, high-precision ranging capabilities, but point cloud data lacks semantic information, making it difficult to directly distinguish obstacle categories, such as pedestrians, vehicles, and guardrails. Therefore, a reliable obstacle recognition method with high category recognition accuracy is urgently needed. Summary of the Invention

[0003] To address this issue, the present invention provides an obstacle perception method and apparatus based on the fusion of vision and lidar, in order to solve the problem that existing single-sensor identification is prone to low reliability or low category recognition due to environmental factors.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides an obstacle perception method based on the fusion of vision and lidar, comprising: Point cloud data is obtained by scanning with LiDAR, and camera images are obtained by capturing images with a camera. Projecting each point in the point cloud data onto a two-dimensional plane yields multiple point pixel coordinates; The effective pixel coordinates are obtained by filtering the pixel coordinates based on the resolution of the camera image; A fused feature map is obtained based on the camera image and the effective pixel coordinates; Obstacle identification is performed based on the fused feature map.

[0005] Further, the step of projecting each point in the point cloud data onto a two-dimensional plane to obtain multiple point pixel coordinates includes: Based on preset extrinsic parameters, the points in the point cloud data are transformed to the camera coordinate system to obtain the camera coordinates corresponding to the points in the point cloud data. The pixel coordinates obtained by projecting the camera coordinate system coordinates onto the camera image are used as point pixel coordinates.

[0006] Further, the step of transforming the points in the point cloud data to the camera coordinate system based on preset extrinsic parameters to obtain the camera coordinates corresponding to the points in the point cloud data includes: The camera coordinates of the points in the point cloud data are obtained using a first transformation formula, which is: Pc = R·Pi + T; Where Pc is the camera coordinate system coordinate corresponding to point i in the point cloud data, Pc=(xc,yc,zc); Pi is the coordinate of point i in the point cloud data, Pi= (xi, yi, zi); R and T are extrinsic parameters, that is, the extrinsic parameters are (R,T).

[0007] Further, the step of projecting the camera coordinate system coordinates onto the camera image to obtain the pixel coordinates as point pixel coordinates includes: The pixel coordinates of the point are obtained using a second conversion formula, which is: ; Where u and v are the coordinate values ​​of the pixel point, the pixel point coordinates are P(u,v); the camera coordinate system coordinates are Pc=(xc,yc,zc); cx is the x-coordinate of the principal point, cy is the y-coordinate of the principal point, and the camera principal point coordinates are (cx, cy); fx and fy are the focal lengths of the camera on the X-axis and Y-axis, respectively.

[0008] Further, obtaining the effective pixel coordinates based on the resolution of the camera image and the pixel coordinates includes: Obtain the pixel coordinates of the point to be confirmed as the pixel coordinates of the point to be confirmed. If u∈[0, width-1] and v∈[0, height-1] in the pixel coordinates of the point to be confirmed, then the pixel coordinates of the point to be confirmed are valid pixel coordinates; where width and height are the camera image resolution.

[0009] Further, obtaining the fused feature map based on the camera image and the effective point pixel coordinates includes: Based on the camera image, a fusion feature matrix is ​​formed by superimposing the three-dimensional coordinates and reflectivity of the corresponding point in the point cloud data at each effective point pixel coordinate position, which is then used as the fusion feature map.

[0010] Further, the obstacle recognition based on the fused feature map includes: Multiple obstacle target regions are extracted from the fused feature map using a pre-trained semantic segmentation model; Multiple obstacle target regions are selected as obstacle candidate regions based on the semantic label matrix of the obstacle target regions; Extract the point cloud data corresponding to the obstacle candidate region from the feature fusion map as a subset of the point cloud; The point cloud subset is clustered using the Euclidean clustering algorithm to obtain multiple clusters; each cluster corresponds to an independent obstacle. The geometric features of the clusters are obtained based on the minimum and maximum values ​​of the clusters in the x, y, and z axes.

[0011] Furthermore, the method also includes: The category distribution in the semantic label matrix corresponding to the cluster is statistically analyzed, and the category with the highest frequency of occurrence is selected as the final category of the obstacle corresponding to the cluster.

[0012] Furthermore, the method also includes: The preset external parameters of the lidar and camera are obtained by manual calibration using a standard checkerboard calibration board.

[0013] Secondly, the present invention provides an obstacle perception device based on the fusion of vision and lidar, comprising: The acquisition module is used to obtain point cloud data through LiDAR scanning and to obtain camera images through camera capture. The feature fusion module is used to project each point in the point cloud data onto a two-dimensional plane to obtain multiple point pixel coordinates; filter the point pixel coordinates according to the resolution of the camera image to obtain valid point pixel coordinates; and obtain a fused feature map based on the camera image and the valid point pixel coordinates. The recognition module is used to identify obstacles based on the fused feature map.

[0014] The present invention, by adopting the above technical solution, has at least the following beneficial effects: This invention provides an obstacle perception method and apparatus based on the fusion of vision and lidar. Point cloud data is obtained through lidar scanning, and camera images are obtained through camera capture. Each point in the point cloud data is projected onto a two-dimensional plane to obtain multiple point pixel coordinates. The point pixel coordinates are filtered according to the resolution of the camera images to obtain valid point pixel coordinates. A fused feature map is obtained based on the camera images and the valid point pixel coordinates, and obstacle recognition is performed based on the fused feature map. This application uses camera and lidar fusion recognition, which can effectively avoid recognition errors caused by changes in environmental factors, improve the reliability of obstacle recognition, and simultaneously identify the type of obstacle.

[0015] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating an obstacle perception method based on the fusion of vision and lidar, as shown in an exemplary embodiment of the present invention. Figure 2 This is a schematic block diagram illustrating an obstacle perception device based on the fusion of vision and lidar, as shown in an exemplary embodiment of the present invention.

[0018] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0020] In the autonomous driving technology system, environmental perception is one of the core components, and its performance directly determines the safety and reliability of vehicle operation. Single sensors have obvious limitations: although cameras can provide rich semantic information (such as obstacle color, texture, and category features), they are easily affected by factors such as lighting (such as strong light, heavy rain, and nighttime) and occlusion, and cannot directly obtain the three-dimensional spatial position of the target; LiDAR can accurately collect the three-dimensional point cloud data of the target and has all-weather, high-precision ranging capabilities, but point cloud data lacks semantic information and it is difficult to directly distinguish obstacle categories (such as pedestrians, vehicles, and guardrails).

[0021] This invention provides an obstacle perception method and device based on the fusion of vision and lidar. This application uses camera and radar for fusion recognition, which can effectively avoid recognition errors caused by changes in environmental factors, improve the reliability of obstacle recognition, and identify the type of obstacle, thus adapting to more application scenarios.

[0022] The methods and apparatus of the present invention will be described below through specific embodiments.

[0023] Please see Figure 1 , Figure 1This is a flowchart illustrating an obstacle perception method based on the fusion of vision and lidar, as shown in an exemplary embodiment of the present invention. See also... Figure 1 The method includes: Step S11: Obtain point cloud data by scanning with LiDAR and obtain camera images by taking pictures with a camera; Step S12: Project each point in the point cloud data onto a two-dimensional plane to obtain multiple point pixel coordinates; Step S13: Filter the point pixel coordinates according to the resolution of the camera image to obtain the valid point pixel coordinates; Step S14: Obtain the fused feature map based on the camera image and the effective pixel coordinates; Step S15: Obstacle recognition is performed based on the fused feature map.

[0024] It should be noted that the technical solution provided in this embodiment is applicable to scenarios including but not limited to obstacle perception in autonomous driving.

[0025] It is understood that the method provided in this embodiment obtains point cloud data through LiDAR scanning, obtains camera images through camera capture, projects each point in the point cloud data onto a two-dimensional plane to obtain multiple point pixel coordinates, filters the point pixel coordinates according to the resolution of the camera image to obtain valid point pixel coordinates, obtains a fused feature map based on the camera image and the valid point pixel coordinates, and performs obstacle recognition based on the fused feature map. This application performs fusion recognition through camera and radar, which can effectively avoid recognition errors caused by changes in environmental factors, improve the reliability of obstacle recognition, and at the same time identify the category of obstacle.

[0026] In practice, step S11, "obtaining point cloud data through LiDAR scanning and obtaining camera images through camera shooting," includes: LiDAR and camera are fixedly installed on the autonomous vehicle to ensure that their fields of view are complementary.

[0027] Specifically, the selection and installation of the lidar: Select a lidar with ≥128 lines, ranging accuracy ≤±2cm, and horizontal field of view ≥120°, and install it in the center of the front of the vehicle roof. The lidar optical axis should be parallel to the vehicle's direction of travel to ensure coverage of the three-dimensional space within a range of 10-200m in front of the vehicle.

[0028] Specifically, camera selection and installation: Select an industrial camera with a resolution ≥1920×1080, frame rate ≥30fps, and dynamic range ≥120dB. Select a lens focal length of 8-12mm. Install it directly below or to the side of the LiDAR (horizontal distance ≤10cm). The angle between the camera optical axis and the LiDAR optical axis should be ≤5°. Ensure that the overlap rate between the camera's field of view and the LiDAR's field of view is ≥80% to avoid blind spots caused by installation position deviations.

[0029] It should be noted that an aluminum alloy bracket is used to rigidly connect the LiDAR and the camera. The bracket is fastened to the vehicle body with bolts to ensure that the sensor does not shift relative to the vehicle during driving. A waterproof, dustproof and shockproof protective cover is installed on the outside of the sensor. The light-transmitting part of the protective cover is made of optical glass with high light transmittance (≥90%) to avoid affecting the accuracy of sensor data acquisition.

[0030] It should be noted that the data communication and scheduling between the LiDAR, camera and sensing nodes are realized based on the ROS (Robot Operating System) architecture to ensure data real-time performance and synchronization.

[0031] In practice, step S12, "projecting each point in the point cloud data onto a two-dimensional plane to obtain multiple point pixel coordinates", includes: transforming the points in the point cloud data to the camera coordinate system based on preset extrinsic parameters to obtain the camera coordinate system coordinates corresponding to the points in the point cloud data; and using the pixel coordinates obtained by projecting the camera coordinate system coordinates onto the camera image as point pixel coordinates.

[0032] It should be noted that the extrinsic parameter calibration of the preset extrinsic parameters is carried out manually. The extrinsic parameters of the lidar and camera are solved using a standard checkerboard calibration board. The solution is calculated as rotation matrix R and translation vector T.

[0033] Specifically, the preset extrinsic parameters of the LiDAR and camera are obtained through manual calibration using a standard checkerboard calibration board. The specific steps are as follows: 1. Preparation of calibration board and equipment: A checkerboard calibration board is selected, with a side length a = 100mm and 11×8 corner points (11 horizontally and 8 vertically). The calibration board is made of highly reflective metal plate (LiDAR reflectivity ≥ 80%), and the surface is printed with a high-precision black and white checkerboard pattern (corner point positioning accuracy ≤ 0.1mm). Connect the LiDAR, camera, and host computer (industrial computer, CPU ≥ i7-12700H, memory ≥ 32GB) to ensure that the sensors output data normally; install calibration software (developed based on C++ and OpenCV) for data acquisition, feature extraction, and extrinsic parameter calculation. 2. Synchronous Data Acquisition: Place the calibration board within 5-15m in front of the sensor. Set the angles between the calibration board plane and the sensor's optical axis to 0°, 30°, 45°, 60°, and 90° (5 angles in total). At each angle, move the calibration board along the sensor's horizontal field of view and acquire synchronous data (images and point clouds) at 10 different locations, for a total of 50 sets of synchronous data. Data Synchronization Mechanism: Synchronization is achieved through sensor hardware trigger signals. Both the LiDAR and the camera receive a 1PPS (1 pulse per second) trigger signal sent by the host computer, ensuring that the timestamp deviation of each set of data is ≤1ms. During acquisition, record the timestamp, calibration board angle, and position information of each set of data for subsequent data filtering and verification. 3. Camera intrinsic parameter calculation: Extracting checkerboard corner features from each set of acquired camera images: The `findChessboardCorners` function in OpenCV is used to detect interior corners, and then the `cornerSubPix` function is used for sub-pixel optimization to obtain the coordinates of the interior corners in the image plane: Ui = (ui, vi), i = 1, 2, ..., N, where N is the total number of interior corners. In this scheme, N = 11 × 8 - 4 = 84, and invalid corners at the edges are removed. Define the checkerboard world coordinate system: With the first interior corner point at the top left corner of the calibration board as the origin Ow, the Xw axis is along the horizontal direction of the calibration board, the Yw axis is along the vertical direction of the calibration board, and the Zw axis is perpendicular to the calibration board plane (right-hand coordinate system). Then, the world coordinates of each interior corner point are Pwi = (xwi, ywi, 0), where xwi = (icol - 1) × a, ywi = (irow - 1) × a, icol is the horizontal index of the interior corner point, and irow is the vertical index. Based on the pinhole camera model, the relationship between image coordinates and world coordinates is: s·[ui; vi; 1]= K·[Rc·Pwi+ Tc]; where s is the scale factor, K is the camera intrinsic parameter matrix, and Rc and Tc are the rotation matrix and translation vector of the camera relative to the world coordinate system.The Zhang Zhengyou calibration method was used to perform nonlinear optimization on the interior corner point data of 50 sets of images to solve the camera intrinsic parameter matrix K: K = [[fx, 0, cx], [0, fy, cy], [0, 0, 1]]; where fx and fy are the focal lengths of the camera in the x-axis and y-axis directions, respectively, in pixels, and cx and cy are the coordinates of the camera principal point, in pixels. 4. Solving extrinsic parameters of LiDAR and camera, identifying the calibration board reflective surface from each set of synchronized point clouds: In the LiDAR point cloud, the point cloud of the calibration board region has high reflectivity (≥70%) and is planar. The calibration board point cloud is extracted through the following steps: Filtering points with reflectivity ≥70% in the point cloud to obtain a candidate point set S; using the RANSAC (Random Sample Consensus) algorithm to fit a plane from S, the plane equation is ax + by + cz + d = 0, setting the in-plane threshold ≤2mm, and retaining the in-plane point set Splane; calculating the minimum bounding rectangle of Splane to determine the region of the calibration board in the point cloud, and extracting the point cloud coordinates Pli = (xli, yli, zli) corresponding to the vertices of the rectangle, i=1,2,...,84, which correspond one-to-one with the inner corner points of the image. Spatial matching point set construction: The optimized interior corner coordinates Ui in the image are back-projected to the camera coordinate system through the camera intrinsic parameter matrix K, resulting in point coordinates Pci = K^{-1}·s·[ui; vi; 1] in the camera coordinate system. Assuming Zc=1, further optimization is performed using the PnP algorithm. The point coordinates Pli and Pci in the LiDAR coordinate system form a spatial matching point pair (Pli, Pci). Extrinsic parameters are solved based on the PnP (Perspective-n-Point) algorithm: The coordinate transformation relationship between the LiDAR and the camera is Pci = R·Pli + T, where R is a 3×3 rotation matrix and T is a 3×1 translation vector (extrinsic parameter). The solution is obtained through the following steps: Initialization: The initial values ​​of R and T are calculated using the Efficient PnP algorithm; Nonlinear optimization: With the goal of minimizing the reprojection error of the matching point pairs, a loss function is constructed: Loss(R, T) = Σ||Pci - (R·Pli + T)||²; The Levenberg-Marquardt algorithm is used to iteratively optimize R and T until the loss function converges (convergence threshold ≤ 0.1 mm), thus obtaining the final preset extrinsic parameters (R, T).5. Visualize and verify the calibration results. In 3D point cloud visualization software (such as PCL Viewer), transform the LiDAR point cloud to the camera coordinate system using extrinsic parameters (R, T), and then project the camera image onto the image plane corresponding to the point cloud using intrinsic parameters K to generate a "point cloud-image overlay". Observe the alignment of the inner corner points of the calibration board in the overlay and calculate the offset between the point cloud and the image for each inner corner point. If the average offset is ≤1 pixel, the calibration result is qualified; otherwise, re-collect data and repeat the above calibration steps until the accuracy requirements are met.

[0034] Specifically, the point cloud data is transformed to the camera coordinate system based on preset extrinsic parameters to obtain the camera coordinate system coordinates corresponding to the points in the point cloud data. This includes: obtaining the camera coordinate system coordinates corresponding to the points in the point cloud data through a first transformation formula, which is: Pc = R·Pi + T; where Pc is the camera coordinate system coordinate corresponding to point i in the point cloud data, Pc=(xc,yc,zc); Pi is the coordinates of point i in the point cloud data, Pi= (xi, yi, zi); R and T are extrinsic parameters, i.e., the extrinsic parameters are (R,T).

[0035] It should be noted that the camera coordinate system is a three-dimensional Cartesian coordinate system established with the camera's optical center as the origin, used to determine the position of points in point cloud data relative to the camera.

[0036] Specifically, projecting the camera coordinate system coordinates onto the camera image to obtain the pixel coordinates as the point pixel coordinates includes: obtaining the point pixel coordinates through a second transformation formula, which is: Where u and v are the coordinate values ​​of the pixel point, and the pixel point coordinates are P(u,v); the camera coordinate system coordinates are Pc=(xc,yc,zc); cx is the x-coordinate of the principal point, cy is the y-coordinate of the principal point, and the camera principal point coordinates are (cx, cy); fx and fy are the focal lengths of the camera on the X-axis and Y-axis, respectively.

[0037] It should be noted that the second conversion formula yields multiple pixel coordinates, and the pixels corresponding to these pixel coordinates form a two-dimensional plane.

[0038] In practice, step S13, "filtering point pixel coordinates based on the resolution of the camera image to obtain valid point pixel coordinates", includes: obtaining point pixel coordinates as point pixel coordinates to be confirmed; if u∈[0,width-1] and v∈[0,height-1] in the point pixel coordinates to be confirmed, then the point pixel coordinates to be confirmed are valid point pixel coordinates; where width and height are the camera image resolution.

[0039] It should be noted that the validity of point pixel coordinates is determined as follows: if u∈[0, width-1] and v∈[0, height-1], then the point is a valid projection point, and its point cloud information is retained, including three-dimensional coordinates and reflectivity; the image pixel information corresponding to the point pixel coordinates, i.e., the RGB value, is also retained; otherwise, it is considered an invalid point and is discarded.

[0040] In practice, step S14, "obtaining a fusion feature map based on camera images and effective point pixel coordinates", includes: using camera images as a basis, and superimposing the three-dimensional coordinates and reflectivity of the corresponding points in the point cloud data at each effective point pixel coordinate position to form a fusion feature matrix as a fusion feature map.

[0041] It should be noted that, based on the camera image, the corresponding three-dimensional coordinates of the point cloud and reflectance are superimposed on the pixel coordinates of each valid point to form a fusion feature matrix F = [Frgb, Fxyz, Freflect], where Frgb is the RGB feature of the image, Fxyz is the three-dimensional coordinate feature of the point cloud, and Freflect is the reflectance feature of the point cloud. The fusion feature matrix is ​​used as the fusion feature map.

[0042] It is understood that the technical solution provided in this embodiment combines camera images with radar point cloud images to form a fused feature map, which can effectively reflect location information and semantic information, providing data support for subsequent identification.

[0043] In practice, step S15, "obstacle recognition based on the fused feature map," includes: extracting multiple obstacle target regions from the fused feature map using a pre-trained semantic segmentation model; selecting multiple obstacle target regions as obstacle candidate regions based on the semantic label matrix of the obstacle target regions; extracting point cloud data corresponding to the obstacle candidate regions from the feature fusion map as a subset of the point cloud; using the Euclidean clustering algorithm to cluster the point cloud subset to obtain multiple clusters; each cluster corresponds to an independent obstacle; and obtaining the geometric features of the clusters based on the minimum and maximum values ​​of the clusters in the x, y, and z axes.

[0044] It should be noted that a semantic segmentation model is used to extract obstacle target regions in images. The model selection and training are as follows: Model architecture: A lightweight semantic segmentation model (such as MobileNetV3+DeepLabV3+) is selected to balance segmentation accuracy and inference speed, ensuring that the inference time on embedded platforms (such as NVIDIA Jetson AGX Orin) is ≤30ms; Dataset training: The model is trained using public datasets (such as KITTI, Cityscapes) and a self-made dataset (containing 10 common obstacle categories such as pedestrians, vehicles, cyclists, guardrails, and traffic signs). Data augmentation (such as random flipping, rotation, and brightness adjustment) is used during training to improve the model's generalization ability. The final model achieves an mIoU (mean intersection-over-union) ≥85% on the validation set; Semantic segmentation inference: The corrected camera image is input into the trained semantic segmentation model, and the semantic segmentation result image is output. Each pixel in the image is labeled with the corresponding obstacle category (such as "pedestrian", "vehicle", "background"), resulting in a semantic label matrix L = [L11, L12,..., Lwh] (Lij is pixel (i,j)). The category labels are determined. Based on the semantic label matrix L, pixel regions categorized as "non-background" are selected, i.e., obstacle candidate regions, and the pixel coordinate range (umin, umax, vmin, vmax) of each candidate region is obtained. The point cloud subset Sobj corresponding to the candidate regions is extracted from the fused feature map F. Sobj is clustered using Euclidean clustering, with a cluster radius of 0.3m (based on common obstacle sizes). Each cluster corresponds to an independent obstacle. The geometric features of each cluster are calculated: 3D bounding box: By calculating the minimum and maximum values ​​of the clusters in the x, y, and z axes, the bounding box coordinates (xmin, xmax, ymin, ymax, zmin, zmax) are obtained, and the bounding box center coordinates (xcenter, ycenter, zcenter) = ((xmin+xmax) / 2, (ymin+ymax) / 2, (zmin+zmax) / 2); Dimensions: Length L = xmax - xmin, Width W = ymax - ymin, Height H = zmax - zmin; Distance: The distance from the center of the obstacle to the vehicle, D = sqrt(xcenter² + ycenter² + zcenter²).

[0045] Specifically, the category distribution in the semantic label matrix corresponding to the cluster is statistically analyzed, and the category with the highest frequency of occurrence is selected as the final category of the obstacle corresponding to the cluster.

[0046] It should be noted that the category distribution in the semantic label matrix L corresponding to each cluster is statistically analyzed, and the category with the highest frequency is selected as the final category of the obstacle. For example, if the proportion of the "vehicle" label in the corresponding pixel of the cluster is ≥90%, it is determined to be a "vehicle" obstacle. If there are multiple categories with similar proportions, such as "pedestrian" and "cyclist" both accounting for 45%, then geometric features are used to assist in the judgment. For example, if the height H ≥1.5m and the width W ≤0.6m, it is determined to be a "pedestrian".

[0047] It is understood that the technical solution provided in this embodiment, by using camera and radar for fusion recognition, can effectively avoid recognition errors caused by changes in environmental factors, improve the reliability of obstacle recognition, and at the same time identify the type of obstacle.

[0048] Please see Figure 2 , Figure 2 This is a schematic block diagram illustrating an obstacle perception device based on the fusion of vision and lidar, as shown in an exemplary embodiment of the present invention. See also... Figure 2 The obstacle perception device 100 based on the fusion of vision and lidar includes: The acquisition module 101 is used to obtain point cloud data by scanning with LiDAR and to obtain camera images by capturing images with a camera. The fusion feature module 102 is used to project each point in the point cloud data onto a two-dimensional plane to obtain multiple point pixel coordinates; filter the point pixel coordinates according to the resolution of the camera image to obtain valid point pixel coordinates; and obtain a fusion feature map based on the camera image and the valid point pixel coordinates. The recognition module 103 is used to recognize obstacles based on the fused feature map.

[0049] It should be noted that the device provided in this embodiment is applicable to scenarios including but not limited to: obstacle recognition in autonomous driving.

[0050] It is understood that the device provided in this embodiment obtains point cloud data through LiDAR scanning, obtains camera images through camera capture, projects each point in the point cloud data onto a two-dimensional plane to obtain multiple point pixel coordinates, filters the point pixel coordinates according to the resolution of the camera image to obtain valid point pixel coordinates, obtains a fused feature map based on the camera image and the valid point pixel coordinates, and performs obstacle recognition based on the fused feature map. This application performs fusion recognition through camera and radar, which can effectively avoid recognition errors caused by changes in environmental factors, improve the reliability of obstacle recognition, and at the same time identify the category of obstacle.

[0051] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0052] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0053] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0054] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0055] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0056] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An obstacle perception method based on the fusion of vision and lidar, characterized in that, The method includes: Point cloud data is obtained by scanning with LiDAR, and camera images are obtained by capturing images with a camera. Projecting each point in the point cloud data onto a two-dimensional plane yields multiple point pixel coordinates; The effective pixel coordinates are obtained by filtering the pixel coordinates based on the resolution of the camera image; A fused feature map is obtained based on the camera image and the effective pixel coordinates; Obstacle identification is performed based on the fused feature map.

2. The method according to claim 1, characterized in that, The step of projecting each point in the point cloud data onto a two-dimensional plane to obtain multiple point pixel coordinates includes: Based on preset extrinsic parameters, the points in the point cloud data are transformed to the camera coordinate system to obtain the camera coordinates corresponding to the points in the point cloud data. The pixel coordinates obtained by projecting the camera coordinate system coordinates onto the camera image are used as point pixel coordinates.

3. The method according to claim 2, characterized in that, The step of transforming the points in the point cloud data to the camera coordinate system based on preset extrinsic parameters to obtain the camera coordinates corresponding to the points in the point cloud data includes: The camera coordinates of the points in the point cloud data are obtained using a first transformation formula, which is: Pc = R·Pi + T; Where Pc is the camera coordinate system coordinate corresponding to point i in the point cloud data, Pc=(xc,yc,zc); Pi is the coordinate of point i in the point cloud data, Pi= (xi, yi, zi); R and T are preset extrinsic parameters, that is, the preset extrinsic parameters are (R,T).

4. The method according to claim 3, characterized in that, The step of projecting the camera coordinate system coordinates onto the camera image to obtain the pixel coordinates as point pixel coordinates includes: The pixel coordinates of the point are obtained using a second conversion formula, which is: ； Where u and v are the coordinate values ​​of the pixel point, the pixel point coordinates are P(u,v); the camera coordinate system coordinates are Pc=(xc,yc,zc); cx is the x-coordinate of the principal point, cy is the y-coordinate of the principal point, and the camera principal point coordinates are (cx, cy); fx and fy are the focal lengths of the camera on the X-axis and Y-axis, respectively.

5. The method according to claim 4, characterized in that, The step of obtaining the effective pixel coordinates based on the resolution of the camera image and the pixel coordinates includes: Obtain the pixel coordinates of the point to be confirmed as the pixel coordinates of the point to be confirmed. If u∈[0, width-1] and v∈[0, height-1] in the pixel coordinates of the point to be confirmed, then the pixel coordinates of the point to be confirmed are valid pixel coordinates; where width and height are the camera image resolution.

6. The method according to claim 5, characterized in that, The process of obtaining the fused feature map based on the camera image and the effective pixel coordinates includes: Based on the camera image, a fusion feature matrix is ​​formed by superimposing the three-dimensional coordinates and reflectivity of the corresponding point in the point cloud data at each effective point pixel coordinate position, and this matrix is ​​used as the fusion feature map.

7. The method according to claim 6, characterized in that, The obstacle recognition based on the fused feature map includes: Multiple obstacle target regions are extracted from the fused feature map using a pre-trained semantic segmentation model; Multiple obstacle target regions are selected as obstacle candidate regions based on the semantic label matrix of the obstacle target regions; Extract the point cloud data corresponding to the obstacle candidate region from the feature fusion map as a subset of the point cloud; The point cloud subset is clustered using the Euclidean clustering algorithm to obtain multiple clusters; each cluster corresponds to an independent obstacle. The geometric features of the clusters are obtained based on the minimum and maximum values ​​of the clusters in the x, y, and z axes.

8. The method according to claim 7, characterized in that, The method further includes: The category distribution in the semantic label matrix corresponding to the cluster is statistically analyzed, and the category with the highest frequency of occurrence is selected as the final category of the obstacle corresponding to the cluster.

9. The method according to claim 2, characterized in that, The method further includes: The preset external parameters of the lidar and camera are obtained by manual calibration using a standard checkerboard calibration board.

10. An obstacle perception device based on the fusion of vision and lidar, characterized in that, The device includes: The acquisition module is used to obtain point cloud data through LiDAR scanning and to obtain camera images through camera capture. The feature fusion module is used to project each point in the point cloud data onto a two-dimensional plane to obtain multiple point pixel coordinates; filter the point pixel coordinates according to the resolution of the camera image to obtain valid point pixel coordinates; and obtain a fused feature map based on the camera image and the valid point pixel coordinates. The recognition module is used to identify obstacles based on the fused feature map.

Images