Vehicle passable area detection method and device fusing 3D point cloud and panoramic image

Abstract

The invention discloses a vehicle passable area detection method and device fusing 3D point cloud and a panoramic image. The method comprises the steps that the panoramic image in a designated area near a vehicle and 3D point cloud data obtained through laser radar scanning are acquired respectively; performing ground segmentation on the acquired 3D point cloud data to obtain obstacle point cloud; scanning the obstacle point cloud based on polar coordinate rays in the laser radar coordinate system to obtain the range of a passable area; generating a new passable area point cloud according to the range of the passable area; projecting the generated new passable area point cloud to the panoramic image according to the relation between the image pixel coordinate system and the laser radar coordinate system; and detecting the passable area point cloud projected to the panoramic image, and determining the boundary of the passable area in the image pixel space. The method has the advantages of being simple in implementation method, low in cost, high in detection efficiency, high in environmental adaptability, convenient to observe and the like.

Description

Technical Field

[0001] The present invention relates to the field of vehicle intelligent driving control technology, and in particular to a vehicle passable area detection method and device that integrates 3D point cloud and panoramic image. Background Art

[0002] The traversable area detection is to detect the area around the vehicle that can be planned and controlled. It is an important perception function for realizing intelligent driving control. The traversable area detection methods in the prior art can be divided into the following two categories:

[0003] One is a detection method based on 2D image space, such as image column-based or semantic segmentation-based methods. This method obtains the passable area by segmenting the image in 2D space. However, this type of detection method based on 2D image space cannot obtain accurate depth information, needs to rely on a large amount of training data, and is prone to false detection.

[0004] The other type is a 3D space-based detection method, which is usually based on the 3D point cloud data scanned by LiDAR and the ground segmentation based on deep learning methods to obtain the passable area. The point cloud data of LiDAR can provide accurate 3D position measurement information, but cannot provide rich image texture and semantic information. In the scenario of assisted driving, the passable area obtained by this type of detection method is not convenient for the driver to observe. Summary of the invention

[0005] The technical problem to be solved by the present invention is: in response to the technical problems existing in the prior art, the present invention provides a vehicle passable area detection method and device that integrates 3D point clouds and panoramic images, which has a simple implementation method, low cost, high detection efficiency and accuracy, strong environmental adaptability and is easy to observe.

[0006] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0007] A vehicle passable area detection method integrating 3D point cloud and panoramic image, comprising the following steps:

[0008] Obtain panoramic images of the designated area near the vehicle and 3D point cloud data obtained by laser radar scanning respectively; perform ground segmentation on the obtained 3D point cloud data to obtain obstacle point cloud;

[0009] Scanning the obstacle point cloud based on polar coordinate rays in the laser radar coordinate system to obtain the range of the passable area; generating a new passable area point cloud according to the range of the passable area;

[0010] According to the relationship between the image pixel coordinate system and the laser radar coordinate system, the generated new traversable area point cloud is projected onto the panoramic image;

[0011] The point cloud of the passable area projected onto the panoramic image is detected to determine the boundary of the passable area in the image pixel space.

[0012] Furthermore, a progressive morphological segmentation method is used to perform ground segmentation on the acquired 3D point cloud data to obtain an obstacle point cloud, and the steps include:

[0013] Select the point cloud with the smallest value in the height axis direction from the 3D point cloud data to construct a minimum elevation surface grid;

[0014] The minimum elevation surface grid is input into an opening operation filter for opening operation and multiple iterations are performed to obtain an obstacle point cloud. In each iteration, the result of the previous iteration is used as the input of the next iteration, and the window size of the opening operation filter is gradually increased.

[0015] Furthermore, the step of inputting the minimum elevation surface mesh into an open operation filter to perform multiple iterations to obtain an obstacle point cloud comprises:

[0016] In the first iteration, the minimum elevation surface grid is input into the opening operation filter, and the opening operation filter uses the initial filtering window to perform an opening operation on the minimum elevation surface grid. The opening operation is to first perform an erosion operation and then perform an expansion operation. The erosion operation is used to find the neighboring point with the lowest elevation value in the current filtering window, and the expansion operation is used to find the neighboring point with the highest elevation value in the current filtering window.

[0017] After increasing and adjusting the filter window of the current open operation filter and calculating the elevation difference threshold, the result of the previous iteration is input into the open operation filter, and the open operation filter performs the open operation according to the current increased and adjusted filter window;

[0018] Determine whether the difference between the height of the point cloud data obtained by the current iteration and the height of the corresponding point cloud data after the opening operation is less than the currently calculated elevation difference threshold. If so, the corresponding point cloud is determined to be a ground point cloud, otherwise it is determined to be an obstacle point cloud;

[0019] After repeating the iterative operation for many times, the obstacle point cloud is obtained.

[0020] Furthermore, the calculation expression for calculating the elevation difference threshold is:

[0021]

[0022] Among them, dh k is the elevation difference threshold, dh 0 is the initial value of the elevation difference threshold, s is the slope, dh max is the maximum elevation difference threshold, w kIndicates the window size at the kth iteration, n 0 represents the preset window size threshold, and c is the grid size of the minimum elevation surface grid.

[0023] Furthermore, the obstacle point cloud is scanned based on polar coordinate rays in the laser radar coordinate system to obtain the range of the passable area including:

[0024] The obstacle point cloud is obtained by scanning a fan perpendicular to the xoy plane of the laser radar coordinate system within a specified range at a preset angle interval, and an effective detection distance of the passable area is set;

[0025] During the scanning process, the distance from the origin of the LiDAR coordinate system to the nearest obstacle point cloud at each angle is recorded;

[0026] Determine the state of the obstacle point cloud scanned at each angle. If it is determined that there is still no obstacle point cloud beyond the effective detection distance of the passable area, then record the effective detection distance of the passable area at the corresponding angle;

[0027] The range of the traversable area at each angle is obtained from the distance recorded at each angle.

[0028] Furthermore, when recording the distance from the origin of the laser radar coordinate system to the nearest obstacle point cloud at each angle during the scanning process, it also includes performing an open operation filtering on each recorded distance.

[0029] Furthermore, the detecting of the passable area point cloud projected onto the panoramic image to determine the boundary of the passable area in the image pixel space includes:

[0030] The point cloud of the traversable area projected onto the panoramic image is traversed using a sliding window;

[0031] During the traversal process, find the point cloud with the maximum y value and the minimum point cloud in the image pixels in each window;

[0032] The boundary of the passable area in the image pixel space is obtained based on the maximum point cloud and the minimum point cloud found.

[0033] Furthermore, when generating a new passable area point cloud based on the range of the passable area, it also includes dividing the passable area point cloud into two-dimensional grids, projecting the identified ground point cloud into the divided grids according to the x, y coordinates to obtain the terrain undulations of each grid, and assigning the corresponding ground height estimation value to the passable area point cloud by solving the elevation value of the passable area point cloud, thereby forming a passable area point cloud with ground height estimation.

[0034] Furthermore, the elevation value of the passable area point cloud is the minimum elevation value in each divided grid or the average height of the ground points, or after the ground curve equation is formed by curve fitting, the z value in the ground curve equation is solved according to the x and y values ​​of the passable area point cloud as the elevation value of the passable area point cloud.

[0035] A vehicle passable area detection device that integrates 3D point cloud and panoramic image includes a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0036] A computer-readable storage medium storing a computer program, wherein the computer program implements the above method when executed by a processor.

[0037] Compared with the prior art, the advantages of the present invention are as follows: the present invention obtains panoramic images near the vehicle and 3D point cloud data obtained by laser radar scanning respectively, first performs ground segmentation on the 3D point cloud data and then scans it in combination with the polar coordinate ray method to obtain the range of the passable area in 3D space, thereby generating a new passable area point cloud, and uses the panoramic image to project the generated passable area point cloud to the image space, so that the 3D passable area detection result can be fused with the 2D image space corresponding to the panoramic image, which can not only provide accurate 3D position measurement information, but also provide rich image texture and semantic information, and provide the driver with a visualization effect of the passable area under a wide field of view, thereby achieving the effect of assisted driving, and combined with the fusion method of 3D point cloud and panoramic image, it can also improve environmental adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the implementation process of the vehicle passable area detection method integrating 3D point cloud and panoramic image in this embodiment.

[0039] Figure 2 It is a schematic diagram of the visualization effect obtained by segmenting a 3D laser point cloud in a specific application embodiment of the present invention.

[0040] Figure 3 A schematic diagram of the principle of point cloud scanning based on polar coordinate rays in a specific application embodiment of the present invention.

[0041] Figure 4 It is a schematic diagram of the effect of the traversable area point cloud generated in a specific application embodiment of the present invention.

[0042] Figure 5 It is a schematic diagram of the passable area detection effect obtained in a specific application embodiment of the present invention. DETAILED DESCRIPTION

[0043] The present invention is further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the protection scope of the present invention is not limited thereby.

[0044] like Figure 1 As shown, the steps of detecting the vehicle passable area by fusing 3D point cloud and panoramic image in this embodiment include:

[0045] Step S01. Obtain a panoramic image of a specified area near the vehicle and 3D point cloud data obtained by laser radar scanning.

[0046] In this embodiment, a semi-solid laser radar and four monocular cameras of the same model are pre-arranged on the vehicle. The field of view of the stitched image generated by stitching the images of the four single cameras is 120*60°. The cameras are fixed on the same plane, and the cameras have the same angles. The semi-solid laser radar is fixed on the four cameras. During the driving process of the vehicle, the laser radar scans the area directly in front of the vehicle to obtain a 3D point cloud, and the four cameras respectively collect images within the four field of view angles directly in front of the vehicle, and a panoramic image directly in front is generated through the image stitching algorithm.

[0047] It is understandable that the type of LiDAR, the number and type of cameras can be configured according to actual needs, or a single large field of view camera can be used directly to obtain a panoramic image. Panoramic images and 3D point cloud data can be collected in real time or read offline, and can be configured according to actual needs.

[0048] Optionally, after obtaining the 3D point cloud data, the point cloud data is further preprocessed by performing point cloud ROI (region of interest) segmentation, downsampling, statistical outlier filtering, etc., so as to retain the point cloud features of the target area while reducing the noise and quantity of the point cloud. The above-mentioned point cloud downsampling method can adopt random downsampling, voxel center downsampling, voxel centroid downsampling, etc.

[0049] Step S02: performing ground segmentation on the acquired 3D point cloud data to obtain an obstacle point cloud.

[0050] Morphological ground segmentation uses the spatial features of point cloud data to separate ground and non-ground points. The morphological ground segmentation algorithm includes erosion and dilation operations. The erosion operation can remove burrs on the contour and retain the main information of the object. The dilation operation increases the volume of the object on the basis of the original object, which can repair the holes on the contour and make the details fuller. The operation of first erosion and then dilation is called opening operation. By performing the opening operation of first erosion and then dilation on the point cloud of the target area, the point cloud belonging to the obstacle part on the ground can be removed.

[0051] In this embodiment, a progressive morphological segmentation method is used to perform ground segmentation on the acquired 3D point cloud data to obtain an obstacle point cloud. By performing an opening operation of first corroding and then dilating the minimum elevation surface of the point cloud, and gradually increasing the size of the opening operation window, obstacle point cloud segmentation of different scales can be achieved, thereby improving the accuracy of obstacle point cloud detection.

[0052] As an optional implementation, the step of using a progressive morphological segmentation method to perform ground segmentation on the acquired 3D point cloud data to obtain an obstacle point cloud includes:

[0053] Step S201. Select the point cloud with the smallest value in the height axis direction from the 3D point cloud data to construct a minimum elevation surface grid.

[0054] Specifically, the point cloud coordinates measured by the laser radar can be read and divided into two-dimensional grids. The point cloud with the smallest value in the height axis direction can be selected in each grid to construct the minimum elevation surface grid. The grid size of the minimum elevation surface grid is c. Each grid contains the (x, y, z) coordinates of the point cloud. If there is no corresponding point cloud in a grid, the point cloud information closest to it can be stored.

[0055] Step S202: Input the minimum elevation surface mesh into the opening operation filter for opening operation and perform multiple iterations to obtain the obstacle point cloud. In each iteration, the result of the previous iteration is used as the input of the next iteration, and the window size of the opening operation filter is gradually increased.

[0056] Specifically, in the first iteration, the minimum elevation surface and the initial filter window size can be used as inputs of the opening operation filter. In the next iteration, the filter surface obtained from the previous iteration and the increased and adjusted filter window size are used as inputs of the opening operation filter. After each iteration, a further smoothed surface after the opening operation and a non-ground point cloud detected based on the elevation difference threshold are obtained.

[0057] For example, for a point p in the radar coordinate system, the corrosion operation is to find the neighboring point with the lowest elevation value in a window of size w at point p in the x, y dimensions. The calculation formula can be expressed as:

[0058]

[0059] The expansion operation is to find the neighboring point with the highest elevation value in the w-size window of point p in the x and y dimensions. The calculation formula can be expressed as:

[0060]

[0061] The non-ground point cloud covered by the window can be obtained through the corrosion operation, and the shape of the ground can be restored through the expansion operation. Then, by gradually increasing the size of the window, the extraction of non-ground point clouds of different scales can be achieved, thereby segmenting the point cloud belonging to the obstacle.

[0062] The step of inputting the minimum elevation surface mesh into the open operation filter to perform multiple iterations to obtain the obstacle point cloud in the above step S202 may include:

[0063] Step S221. In the first iteration, the minimum elevation surface mesh is input into the opening operation filter, and the opening operation filter uses the initial filter window to perform an opening operation on the minimum elevation surface mesh. The opening operation is to first perform an erosion operation and then perform an expansion operation. The erosion operation is used to find the neighboring point with the lowest elevation value in the current filter window, and the expansion operation is used to find the neighboring point with the highest elevation value in the current filter window.

[0064] Step S222. Increase and adjust the filter window of the current open operation filter, calculate the elevation difference threshold, input the last iteration result to the open operation filter, and the open operation filter performs the open operation according to the current increased and adjusted filter window;

[0065] Step S223. Determine whether the difference between the height (z coordinate value) of the point cloud data obtained by the current iteration and the height value (z coordinate value) of the corresponding point cloud data after the opening operation is less than the currently calculated elevation difference threshold. If so, determine that the corresponding point cloud is a ground point cloud, otherwise determine that it is an obstacle point cloud;

[0066] Step S2223. Repeat steps S222 and S223 until a preset number of iterations is reached or the preset filter window size is reached. After multiple iterations, the final obstacle point cloud is obtained.

[0067] When the filter window is small, obstacles such as buildings, hills, trees, and people will be detected. For example, the tops of some small hills may be extracted when the filter window is small. Considering that the elevation of obstacles such as buildings, hills, trees, and people in the window is abrupt, while the ground elevation is gradual, if during the iteration process, the z value of a point in the original radar point cloud is compared with the z value of the ground point cloud obtained in the iteration, if the difference is less than the elevation difference threshold, the point cloud can be considered to be a ground point. This embodiment can effectively identify obstacle point clouds by introducing an elevation difference threshold, thereby solving the problem of obstacle identification when the window size is small.

[0068] As an optional implementation, when calculating the elevation difference threshold in step S222, if the filtering window w during iteration is k Smaller, such as w 寿 ≤n 0, n 0 represents the preset window size threshold, k represents the number of iterations, and the elevation difference threshold dh for the current iteration is k Take the preset initial value of the elevation difference threshold. If the filtering window w during iteration k Larger, such as w k >n 0 , then according to the difference w between the two filtering windows k -w k-1 Modify the initial value of the elevation difference threshold to obtain the elevation difference threshold dh for the current iteration k , if the elevation difference threshold dh of the current iteration k Exceeding the preset maximum elevation difference threshold dh max , then the maximum elevation difference threshold dh max As the elevation difference threshold dh for the current iteration k , the specific calculation formula can be expressed as follows:

[0069]

[0070] Among them, dh k is the elevation difference threshold, dh 0 is the initial value of the elevation difference threshold, c is the grid size of the minimum elevation surface grid, s is the slope, and the specific slope value can be set according to the radar and the actual application scenario, dh max is the maximum elevation difference threshold, w k Indicates the window size at the kth iteration.

[0071] The above window size threshold n 0 The specific value can be configured according to actual needs. For example, it can be configured to 3, that is, when w k When the elevation difference threshold is ≤3, dh k Take the initial value of the elevation difference threshold dh 0 , if w k >3, then according to s(w k -w k-1 )c+dh 0 Calculate the elevation difference threshold dh k , until the maximum elevation difference threshold dh is reached max The window size w for each iteration is k It can be determined by linear increase, for example, w k =2kb+1, b is the initial window size, which can also be determined by exponential increase, for example, w k =2b k +1, the initial window size and the maximum window size can be adjusted according to the specific situation. During the iteration process, the window size does not exceed the pre-entered maximum window size.

[0072] In a specific application example, the above method is used in the 3D visualization tool rviz to segment the 3D laser point cloud to obtain a visualization effect as follows: Figure 2 As shown, (a) corresponds to the original point cloud, and (b) corresponds to the non-ground point cloud segmented by the above method.

[0073] It is understandable that, in addition to using the progressive morphological segmentation method for ground segmentation, other ground segmentation methods can also be used according to actual needs, such as a ground segmentation method based on line fitting, a ground segmentation method based on plane fitting, etc.

[0074] This embodiment uses a progressive morphological segmentation method to perform ground segmentation, which can not only segment a normal flat road surface, but also achieve better segmentation for an unstructured road surface with a certain slope, and can effectively improve the segmentation precision and accuracy.

[0075] Step S03: Scan the obstacle point cloud based on polar coordinate rays in the laser radar coordinate system to obtain the range of the passable area.

[0076] In order to detect the part of the point cloud that does not belong to the passable area, based on the polar coordinate ray method, the passable area grows around with the lidar as the center. Once the point cloud belonging to the inaccessible area is encountered, the growth of the passable area is stopped. After a complete process, the passable area at the current vehicle position can be obtained.

[0077] As an optional implementation, the step of scanning the obstacle point cloud based on polar coordinate rays in the laser radar coordinate system to obtain the range of the passable area may include:

[0078] Step S301. Obstacle point clouds are obtained by scanning a fan perpendicular to the xoy plane of the laser radar coordinate system within a specified range at a preset angle interval (for example, within the range of 0-360°, with each 1° progressive), and the effective detection distance of the passable area is set;

[0079] Step S302. During the scanning process, the distance from the origin of the laser radar coordinate system to the nearest obstacle point cloud at each angle is recorded;

[0080] Step S303: Determine the status of the obstacle point cloud scanned at each angle. If it is determined that there is still no obstacle point cloud beyond the effective detection distance of the passable area, then record the effective detection distance of the passable area at the corresponding angle;

[0081] Step S304: Obtain the range of the traversable area at each angle from the distance recorded at each angle.

[0082] by Figure 3For example, the gray filled area is a scanning sector, the height and length of the sector are fixed values, which can be set according to the vehicle parameters, etc. The black points in the scanning sector are the obstacle point cloud detected in step S02. Figure 3 Scan the sector in the middle, and record the distance of the nearest obstacle point cloud within the scanning range for each sector. If there is no obstacle point cloud at this angle, that is, there is still no obstacle point cloud after exceeding the effective detection distance of the passable area, then record the length of the scanning sector. The length of the scanning sector is the effective detection distance of the passable area. The distance recorded at each angle (the distance of the nearest obstacle point cloud or the effective detection distance of the passable area) can be used to obtain the range of the passable area at each angle.

[0083] Preferably, when recording the distance from the origin of the laser radar coordinate system to the nearest obstacle point cloud at each angle during the scanning process, it also includes performing an opening operation filter on the recorded distances to eliminate burrs caused by discontinuity of the point cloud.

[0084] Step S04: Generate a new traversable area point cloud according to the range of the traversable area.

[0085] After determining the range of the passable area, point clouds are generated according to the range, so that the generated point clouds are within the range of the passable area. For example, within the passable area, new passable area point clouds can be generated at equal intervals according to the preset resolution. The x, y values ​​of each passable area point cloud (x, y, z) do not exceed the passable area range, and the z value can be preset to 0. The points (x, y, z) are evenly arranged within the passable area at equal intervals to obtain the point cloud within the passable area.

[0086] Preferably, when generating a new passable area point cloud according to the range of the passable area, it also includes performing two-dimensional grid division on the passable area point cloud, assigning a ground height estimation value to the passable area point cloud according to the grid division result, and forming a passable area point cloud with a ground height estimation. By performing ground height estimation based on grid division and generating a passable area point cloud in a 3D space with a ground height estimation, the passable area can have an approximate height value.

[0087] As an optional implementation, the following steps can be used to form a passable area point cloud with ground height estimation: the currently generated passable area point cloud only determines the x and y values ​​(z is the default value), so a rectangular area can be first delineated using the maximum and minimum x and y values ​​of the passable area range; the rectangular area can be divided into a number of small squares according to a preset grid resolution; the ground point cloud obtained above is then projected into the small squares according to its x and y coordinates to obtain the terrain undulations of each small square; and the z value (elevation value) of the passable area point cloud is then solved to form a passable area point cloud with ground height estimation.

[0088] Optionally, when solving the z value (elevation value) of the passable area point cloud, the minimum elevation surface grid can be constructed by selecting the point cloud with the smallest value in the height axis direction in each grid, and the two-dimensional grid of the passable area point cloud can be divided. The corresponding elevation value in the grid is used as the z value of these passable area point clouds, that is, the corresponding minimum elevation value in the divided two-dimensional grid is used as the z value of the generated new passable area point cloud. After the assignment is completed, the passable area point cloud with ground height estimation is obtained. In addition to the above method, the method of taking the average height of the ground points in the corresponding square grid can also be adopted. It is also possible to use a cubic curve fitting based on the ground points to obtain the ground curve equation, and substitute the x and y values ​​of the passable area point cloud into the equation for solution.

[0089] In a specific application embodiment, the traversable area point cloud generated according to the above method is as follows Figure 4 As shown, the red point cloud is the point cloud of the traversable area.

[0090] Step S05: Based on the relationship between the image pixel coordinate system and the laser radar coordinate system, the generated new traversable area point cloud is projected onto the panoramic image.

[0091] Optionally, the relationship between the image pixel coordinate system and the lidar coordinate system can be determined by pre-calibrating the lidar and each single camera external parameters. At the same time, by obtaining a lookup table of the corresponding relationship between the single camera image pixels and the panoramic image pixels, the generated 3D passable area point cloud can be projected into the pixel coordinate system of the panoramic image.

[0092] Specifically, all the point clouds of the passable area can be projected into the pixel coordinate system of a single camera in turn by using the conversion relationship between the pixel coordinate system of a single camera and the laser radar point cloud coordinate system. If the pixel value after projection calculation is within the resolution range of the image, it means that the projection is successful, and this point cloud will no longer be repeatedly projected to the pixel coordinate system of other cameras. Then, the correspondence between the pixels of a single camera and the pixels of the panoramic stitched image is used to further project the point cloud of the passable area onto the panoramic stitched image. Finally, the correspondence between the pixels of a single camera and the pixels of the panoramic stitched image is used to further project the point cloud of the passable area onto the panoramic stitched image.

[0093] Step S06: Detect the passable area point cloud projected onto the panoramic image to determine the boundary of the passable area in the image pixel space.

[0094] Optionally, when detecting the passable area point cloud projected onto the panoramic image, a sliding window algorithm may be used to determine the boundary of the passable area in the image pixel space, and the steps may include:

[0095] Step S601. Traverse the traversable area point cloud projected onto the panoramic image using a sliding window;

[0096] Step S602. During the traversal process, find the point cloud with the maximum y value and the minimum point cloud in the image pixels in each window; Step S603. Obtain the boundary of the passable area in the image pixel space based on the found maximum point cloud and minimum point cloud.

[0097] Specifically, when the sliding window algorithm is used to traverse the passable area point cloud projected onto the panoramic stitching image, the window traverses the image in the positive direction of the image along the x-axis, and finds the point cloud with the maximum and minimum y values ​​in each window in the image pixels, thereby obtaining the boundary of the passable area on the image. Further, the function in opencv is used to complete the filling of this area on the image, that is, the final passable area detection area can be obtained. The passable area detection effect obtained in the specific application embodiment is as follows Figure 5 shown.

[0098] The present invention obtains panoramic images near the vehicle and 3D point cloud data obtained by laser radar scanning respectively, first performs ground segmentation on the 3D point cloud data and then scans it in combination with the polar coordinate ray method to obtain the range of the traversable area in 3D space, thereby generating a new traversable area point cloud, and uses the panoramic image to project the generated traversable area point cloud to the image space, so that the 3D traversable area detection result can be fused with the 2D image space corresponding to the panoramic image, which can not only provide accurate 3D position measurement information, but also provide rich image texture and semantic information, and provide the driver with a traversable area visualization effect under a wide field of view, thereby achieving the effect of assisted driving, and combined with the fusion method of 3D point cloud and panoramic image, it can also improve the accuracy and environmental adaptability of the traversable area detection result, and does not need to rely on pre-collected data, and can be flexibly applied to different environments.

[0099] This embodiment further provides a vehicle passable area detection device that integrates 3D point cloud and panoramic image, including a processor and a memory, the memory is used to store computer programs, and the processor is used to execute the computer program to perform the above method.

[0100] It is understandable that the above method of this embodiment can be executed by a single device, such as a computer or server, etc., and can also be applied to a distributed scenario and completed by multiple devices in cooperation with each other. In the case of a distributed scenario, one of the multiple devices can only execute one or more steps in the above method of this embodiment, and multiple devices interact to complete the above method. The processor can be implemented in the form of a general-purpose CPU, a microprocessor, an application-specific integrated circuit, or one or more integrated circuits, etc., for executing related programs to implement the above method of this embodiment. The memory can be implemented in the form of a read-only memory ROM, a random access memory RAM, a static storage device, and a dynamic storage device. The memory can store an operating system and other applications. When the above method of this embodiment is implemented by software or firmware, the relevant program code is stored in the memory and called and executed by the processor.

[0101] This embodiment further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the above method is implemented.

[0102] Those skilled in the art will appreciate that the above-mentioned embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0103] The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Therefore, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. A vehicle passable area detection method integrating 3D point cloud and panoramic image, characterized in that the steps include: Obtain panoramic images of the designated area near the vehicle and 3D point cloud data obtained by laser radar scanning; Perform ground segmentation on the acquired 3D point cloud data to obtain obstacle point cloud; Scan the obstacle point cloud based on polar coordinate rays in the laser radar coordinate system to obtain the range of the passable area; Generate a new passable area point cloud according to the range of the passable area; According to the relationship between the image pixel coordinate system and the laser radar coordinate system, the generated new traversable area point cloud is projected onto the panoramic image; The point cloud of the passable area projected onto the panoramic image is detected to determine the boundary of the passable area in the image pixel space.

2. The vehicle passable area detection method by fusing 3D point cloud and panoramic image according to claim 1, characterized in that: The progressive morphological segmentation method is used to perform ground segmentation on the acquired 3D point cloud data to obtain obstacle point clouds. The steps include: Select the point cloud with the smallest value in the height axis direction from the 3D point cloud data to construct a minimum elevation surface grid; The minimum elevation surface grid is input into an opening operation filter for opening operation and multiple iterations are performed to obtain an obstacle point cloud. In each iteration, the result of the previous iteration is used as the input of the next iteration, and the window size of the opening operation filter is gradually increased.

3. The vehicle passable area detection method by fusing 3D point cloud and panoramic image according to claim 2 is characterized in that: The step of inputting the minimum elevation surface mesh into an open operation filter to perform multiple iterations to obtain an obstacle point cloud comprises: In the first iteration, the minimum elevation surface grid is input into the opening operation filter, and the opening operation filter uses the initial filtering window to perform an opening operation on the minimum elevation surface grid, wherein the opening operation is to first perform an erosion operation and then perform an expansion operation, wherein the erosion operation is used to find the neighboring point with the lowest elevation value in the current filtering window, and the expansion operation is used to find the neighboring point with the highest elevation value in the current filtering window; After increasing and adjusting the filter window of the current open operation filter and calculating the elevation difference threshold, the result of the previous iteration is input into the open operation filter, and the open operation filter performs the open operation according to the current increased and adjusted filter window; Determine whether the difference between the height of the point cloud data obtained by the current iteration and the height of the corresponding point cloud data after the opening operation is less than the currently calculated elevation difference threshold. If so, the corresponding point cloud is determined to be a ground point cloud, otherwise it is determined to be an obstacle point cloud; After repeating the iterative operation for many times, the obstacle point cloud is obtained.

4. The vehicle passable area detection method by fusing 3D point cloud and panoramic image according to claim 3 is characterized in that: The calculation expression for calculating the elevation difference threshold is: Among them, dh k is the elevation difference threshold, dh0 is the initial value of the elevation difference threshold, s is the slope, dh max is the maximum elevation difference threshold, w k represents the window size at the kth iteration, n0 represents the preset window size threshold, and c is the grid size of the minimum elevation surface grid.

5. The vehicle passable area detection method by fusing 3D point cloud and panoramic image according to claim 1, characterized in that: The obstacle point cloud is scanned based on polar coordinate rays in the laser radar coordinate system to obtain the range of the passable area including: The obstacle point cloud is obtained by scanning a fan perpendicular to the xoy plane of the laser radar coordinate system within a specified range at a preset angle interval, and an effective detection distance of the passable area is set; During the scanning process, the distance from the origin of the LiDAR coordinate system to the nearest obstacle point cloud at each angle is recorded; Determine the state of the obstacle point cloud scanned at each angle. If it is determined that there is still no obstacle point cloud beyond the effective detection distance of the passable area, then record the effective detection distance of the passable area at the corresponding angle; The range of the traversable area at each angle is obtained from the distance recorded at each angle.

6. The vehicle passable area detection method by fusing 3D point cloud and panoramic image according to any one of claims 1 to 5, characterized in that: The detecting of the passable area point cloud projected onto the panoramic image to determine the boundary of the passable area in the image pixel space includes: The point cloud of the traversable area projected onto the panoramic image is traversed using a sliding window; During the traversal process, find the point cloud with the maximum y value and the minimum point cloud in the image pixels in each window; The boundary of the passable area in the image pixel space is obtained based on the maximum point cloud and the minimum point cloud found.

7. The vehicle passable area detection method by fusing 3D point cloud and panoramic image according to any one of claims 1 to 5, characterized in that: When generating a new passable area point cloud according to the range of the passable area, it also includes dividing the passable area point cloud into two-dimensional grids, projecting the identified ground point cloud into the divided grids according to the x, y coordinates to obtain the terrain undulations of each grid, and assigning the corresponding ground height estimation value to the passable area point cloud by solving the elevation value of the passable area point cloud, thereby forming a passable area point cloud with ground height estimation.

8. The vehicle passable area detection method by fusing 3D point cloud and panoramic image according to claim 7, characterized in that: The elevation value of the passable area point cloud is the minimum elevation value in each divided grid or the average height of the ground points, or after forming the ground curve equation through curve fitting, the z value in the ground curve equation is solved according to the x and y values ​​of the passable area point cloud as the elevation value of the passable area point cloud.

9. A vehicle passable area detection device integrating 3D point cloud and panoramic image, comprising a processor and a memory, wherein the memory is used to store a computer program, characterized in that: The processor is configured to execute the computer program to perform the method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images