A point cloud fusion method, device, computer device, and storage medium

Abstract

An embodiment of the present invention provides a point cloud data fusion method, apparatus, computer device, and storage medium. The method includes: obtaining a first point cloud collected by a lidar and multiple frames of second point clouds collected before the first point cloud, converting the second point clouds into point clouds in the coordinate system of the first point cloud and then fusing them into a third point cloud, where each point in the third point cloud includes the height of the local part of the object corresponding to the point, then determining points with a height greater than a preset height threshold from the third point cloud to generate a fourth point cloud, and determining points with a height less than or equal to the preset height threshold to generate a fifth point cloud, where the preset height threshold is the upper limit value of the height of small objects, screening the points in the fifth point cloud based on the projections of the points in the fourth and fifth point clouds on the X-Y plane in the coordinate system of the first point cloud to obtain a sixth point cloud, and fusing the sixth point cloud and the first point cloud to obtain a target point cloud, where the target point cloud includes the point clouds of multiple frames of small objects, improving the stability of small object detection.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of point cloud data processing, and in particular, to a point cloud fusion method, apparatus, computer device, and storage medium. Background Art

[0002] With the development of autonomous driving technology, lidar, as the main sensor for detecting obstacles, is increasingly widely used in the field of perception of driverless vehicles.

[0003] For object detection based on lidar, the spatial information of the target object, that is, point cloud data, is mainly directly obtained by the lidar, and algorithms for distance analysis are used to extract the target object and calculate relevant parameters. It is not affected by light and is rich in information, playing a very good role in the field of three-dimensional detection.

[0004] However, the effect of target detection is usually positively correlated with the beam number of the lidar. High-beam-number lidars often have disadvantages such as large volume, high price, and difficulty in fixing on the mounting platform. Therefore, in the prior art, object detection is mainly carried out using point clouds of low-beam-number lidars. However, for the point clouds of low-beam-number lidars, their inherent sparsity results in poor target imaging details. For large obstacles, due to their large volume, relatively more points can be collected, and even if the point cloud is sparse, they can still be detected. However, for small obstacle objects, relatively fewer points can be collected, and they may not be recognized by the detection system, resulting in very unstable detection effects for small obstacles. Summary of the Invention

[0005] The embodiments of the present invention propose a point cloud fusion method, apparatus, computer device, and storage medium to solve the problem of very unstable detection effects for small obstacles in object detection using point clouds of low-beam-number lidars at present.

[0006] In a first aspect, the embodiments of the present invention provide a point cloud fusion method applied to an in-vehicle lidar, including:

[0007] Obtain a point cloud set, where the point cloud set includes the first point cloud collected by the lidar and multiple frames of second point clouds collected before the first point cloud;

[0008] Convert the multiple frames of the second point clouds into point clouds in the coordinate system of the first point cloud, and fuse the converted point clouds into a third point cloud, where each point in the third point cloud includes the height of the local part of the object corresponding to the point;

[0009] Determine points with a height greater than a preset height threshold from the third point cloud to generate a fourth point cloud, and determine points with a height less than or equal to the preset height threshold to generate a fifth point cloud, where the preset height threshold is the upper limit of the height of small objects;

[0010] Based on the projections of the points in the fourth point cloud and the fifth point cloud on the X-Y plane in the first point cloud coordinate system, filter the points in the fifth point cloud to generate a sixth point cloud;

[0011] Fuse the sixth point cloud and the first point cloud to obtain a target point cloud.

[0012] Optionally, the obtaining the point cloud set includes:

[0013] During the vehicle driving process, use the point cloud collected by the current lidar as the first point cloud.

[0014] Optionally, the determining points with a height greater than a preset height threshold from the third point cloud to generate a fourth point cloud further includes:

[0015] Remove the ground point cloud from the third point cloud to obtain an obstacle point cloud;

[0016] Determine points with a height greater than a preset height threshold from the obstacle point cloud to generate a fourth point cloud.

[0017] Optionally, the filtering the points in the fifth point cloud based on the projections of the points in the fourth point cloud and the fifth point cloud on the X-Y plane in the first point cloud coordinate system to generate a sixth point cloud includes:

[0018] In a direction parallel to the X-Y plane in the first point cloud coordinate system, expand the area of the points in the fourth point cloud according to a preset magnification parameter to obtain multiple expanded surfaces;

[0019] Filter the points in the fifth point cloud according to the multiple expanded surfaces and the projection of the fifth point cloud on the X-Y plane to generate a sixth point cloud.

[0020] Optionally, the expanding the area of the points in the fourth point cloud according to a preset magnification parameter in a direction parallel to the X-Y plane in the first point cloud coordinate system to obtain multiple expanded surfaces includes:

[0021] For each point in the fourth point cloud, in a direction parallel to the X-Y plane in the first point cloud coordinate system, use a circle with the point as the center and the preset magnification parameter as the radius as the expanded surface of the point.

[0022] Optionally, the filtering the points in the fifth point cloud according to the multiple expanded surfaces and the projection of the fifth point cloud on the X-Y plane includes:

[0023] Project all the extended surfaces onto the X-Y plane to obtain a plurality of closed connected regions, where the connected regions are the union of the projections of the extended surfaces on the X-Y plane;

[0024] Based on the projection of the fifth point cloud on the X-Y plane and the connected regions, filter the points in the fifth point cloud.

[0025] Optionally, the filtering of the points in the fifth point cloud based on the projection of the fifth point cloud on the X-Y plane and the connected regions includes:

[0026] For each point in the fifth point cloud, determine whether the projection of the point on the X-Y plane is located within the connected regions;

[0027] If so, filter the point from the fifth point cloud;

[0028] If not, retain the point.

[0029] In a second aspect, an embodiment of the present invention further provides a point cloud fusion device, including:

[0030] A point cloud set acquisition module, configured to acquire a point cloud set, where the point cloud set includes a first point cloud collected by a lidar and multiple frames of second point clouds collected before the first point cloud;

[0031] A point cloud fusion module, configured to convert multiple frames of the second point clouds into point clouds in the coordinate system of the first point cloud, and fuse the converted point clouds into a third point cloud, where each point in the third point cloud includes the height of the local part of the object corresponding to the point;

[0032] A point cloud segmentation module, configured to determine points with a height greater than a preset height threshold from the third point cloud to generate a fourth point cloud, and determine points with a height less than or equal to the preset height threshold to generate a fifth point cloud;

[0033] A sixth point cloud generation module, configured to filter the points in the fifth point cloud based on the projections of the points in the fourth point cloud and the fifth point cloud on the X-Y plane in the coordinate system of the first point cloud to generate a sixth point cloud;

[0034] A target point cloud generation module, configured to fuse the sixth point cloud and the first point cloud to obtain a target point cloud.

[0035] Optionally, the point cloud set acquisition module includes:

[0036] A point cloud acquisition sub-module, configured to use the point cloud collected by the current lidar as the first point cloud during the driving process of the vehicle.

[0037] Optionally, the point cloud segmentation module further includes:

[0038] An obstacle point cloud segmentation sub-module, configured to remove ground point cloud from the third point cloud to obtain obstacle point cloud;

[0039] A fourth point cloud generation sub-module, configured to determine points with a height greater than a preset height threshold from the obstacle point cloud to generate a fourth point cloud.

[0040] Optionally, the sixth point cloud generation module includes:

[0041] A region expansion sub-module, configured to expand the points in the fourth point cloud in a direction parallel to the X-Y plane in the first point cloud coordinate system according to a preset magnification parameter to obtain a plurality of expanded surfaces;

[0042] A fifth point cloud screening sub-module, configured to screen the points in the fifth point cloud according to the plurality of expanded surfaces and the projection of the fifth point cloud on the X-Y plane to generate a sixth point cloud.

[0043] Optionally, the region expansion sub-module includes:

[0044] An expanded surface formation unit, configured to, for each point in the fourth point cloud, use a circle with the point as the center and the preset magnification parameter as the radius as the expanded surface of the point in a direction parallel to the X-Y plane in the first point cloud coordinate system.

[0045] Optionally, the fifth point cloud screening sub-module includes:

[0046] A connected region formation unit, configured to project all the expanded surfaces onto the X-Y plane to obtain a plurality of closed connected regions, where the connected regions are the union of the projections of the expanded surfaces on the X-Y plane;

[0047] A fifth point cloud screening unit, configured to screen the points in the fifth point cloud based on the projection of the fifth point cloud on the X-Y plane and the connected regions.

[0048] Optionally, the fifth point cloud screening unit includes:

[0049] A point region judgment sub-unit, configured to, for each point in the fifth point cloud, judge whether the projection of the point on the X-Y plane is located within the connected regions; if so, screen the point from the fifth point cloud; if not, retain the point.

[0050] In a third aspect, an embodiment of the present invention further provides a computer device, where the computer device includes:

[0051] One or more processors;

[0052] A memory for storing one or more computer programs,

[0053] When the one or more computer programs are executed by the one or more processors, the one or more processors implement the point cloud fusion method as described in the first aspect.

[0054] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the point cloud fusion method as described in the first aspect is implemented.

[0055] The point cloud fusion method of the embodiment of the present invention is applied to an on-vehicle lidar. First, a first point cloud and a second point cloud after the first point cloud are obtained, and the second point cloud is fused into a dense third point cloud in the coordinate system of the first point cloud. Then, based on the height threshold of small objects, the third point cloud is segmented into a fourth point cloud including only large object point clouds and a fifth point cloud including both large object point clouds and small object point clouds. Then, based on the fourth point cloud, the large object point clouds in the fifth point cloud are filtered out to obtain a sixth point cloud including only multi-frame small object point clouds. Finally, the sixth point cloud and the first point cloud are fused to obtain a target point cloud. The target point cloud includes multi-frame small object point clouds and one frame of large object point clouds, improving the density of small object point clouds, improving the image accuracy of small objects, and increasing the stability of small object detection. Description of the Drawings

[0056] Figure 1 It is a flowchart of a point cloud fusion method provided by Embodiment 1 of the present invention;

[0057] Figure 2 It is a flowchart of a point cloud fusion method provided by Embodiment 2 of the present invention;

[0058] Figure 3 It is a schematic structural diagram of a point cloud fusion device provided by Embodiment 3 of the present invention;

[0059] Figure 4 It is a schematic structural diagram of a computer device provided by Embodiment 4 of the present invention. Detailed Embodiments

[0060] The present invention will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present invention, rather than limiting the present invention. Additionally, it should be noted that for the sake of description, only parts related to the present invention are shown in the drawings, rather than all structures.

[0061] Embodiment 1

[0062] Figure 1The following is a flowchart of a point cloud fusion method provided in the first embodiment of the present invention. This embodiment is applicable to the situation of detecting small objects during vehicle driving. This method can be executed by a point cloud fusion device, which can be implemented by software and / or hardware and can be configured in a vehicle operating system. The point cloud fusion method specifically includes the following steps:

[0063] S101. Obtain a point cloud set, where the point cloud set includes the first point cloud collected by a lidar and multiple frames of second point clouds collected before the first point cloud;

[0064] A lidar is a radar system and an active sensor, and the data formed is in the form of a point cloud. Its working principle is as follows: emit a beam of laser towards the target to be measured, and then measure information such as the time when the reflected or scattered signal reaches the transmitter, the intensity of the signal, and the frequency change, so as to determine the position (distance and angle), shape (size), and state (speed, attitude) of the target to be measured, and achieve the purpose of detecting, identifying, and tracking the target. The information measured by the lidar is the points (data) of the surface characteristics of the target object, and the point cloud is a data set of points in a certain coordinate system.

[0065] For the target object detection based on lidar, the detection effect of the target detection is usually positively correlated with the number of beams of the radar. In view of the high price of lidars with high beam numbers, in the prior art, mainly low-line-number lidars are used. In the vehicle-mounted lidar in this embodiment, the low-line-number lidar, and the point cloud collected by it is single-frame environmental point cloud data.

[0066] The vehicle-mounted lidar generally detects obstacles during vehicle driving. In this embodiment, obtaining the point cloud set means that during vehicle driving, the point cloud collected by the current lidar is used as the first point cloud, and each frame of point cloud within the most recent preset time period is used as the point cloud set. The point cloud set in this embodiment contains the point cloud data of the latest frame, that is, it contains the latest obstacle data, which can maintain the real-time performance of obstacle detection during point cloud fusion.

[0067] The detection times of the second point cloud and the first point cloud are different. The first point cloud is the current latest frame of point cloud, while the second point cloud is the point cloud collected before the first point cloud. In an example of this embodiment, the second point cloud is a subset extracted from the point cloud set, and the extraction method is to extract the points before the first point cloud from the point cloud set within a fixed preset time period as the second point cloud. For example, the points before the first point cloud in the point cloud set can be divided into multiple frames of second point clouds according to a fixed time period. Suppose the preset time period is 50 ms, then the points collected within the 300 ms time period before the first point cloud can be divided into 5 frames of second point clouds, and multiple frames of second point clouds can be extracted from the 5 frames of second point clouds.

[0068] Since the data of the same point cloud is the same, in order to ensure the richness of the point cloud data and avoid wasting computing resources, when extracting multiple frames of the second point cloud, each frame of the second point cloud can only be extracted once. It should be noted that the order of extracting the second point cloud in this embodiment is not restricted.

[0069] S102. Convert multiple frames of the second point cloud into point clouds in the coordinate system of the first point cloud, and fuse the converted point clouds into a third point cloud.

[0070] The point cloud is collected during the vehicle driving process, and the coordinate system of the point cloud changes as the vehicle moves. That is, for the same target object, the relative position between the vehicle and the target object is different, the viewing angle of observing the target object is different, and the obtained point clouds belong to different coordinate systems. However, the point cloud itself has nothing to do with the viewing angle and can be rotated arbitrarily.

[0071] The conversion of the point cloud coordinate system is to transform a point cloud from its own coordinate system to another coordinate system. In this embodiment, the rotation matrix and translation matrix of the second point cloud relative to the first point cloud can be calculated through point cloud registration algorithms such as ICP (Iterative Closest Point) or NDT (Normal Distribution Transform), and then the coordinates of the second point cloud are converted to the coordinate system of the first point cloud according to the following formula to obtain the converted second point cloud.

[0072] P2 = P1 * R + T

[0073] Wherein, P1 is the second point cloud, R is the rotation matrix, T is the translation matrix, and P2 is the converted second point cloud.

[0074] As long as different point clouds are in the same coordinate system, they can be directly fused. Therefore, by directly fusing multiple frames of the converted second point clouds, a third point cloud can be obtained. Compared with each frame of the second point cloud, the point cloud density of the third point cloud is improved.

[0075] Since when the lidar with a low beam scans a target object, its scanning range is limited, and the scanned points are generally data of the local part of the object rather than the global data of the object. For example, when the target detection object is another vehicle, in the collected point cloud, some points represent the data of the car door, while some points represent the data of the car tail. Each point contains the corresponding local data of the object, including the height of the corresponding local part of the object, and each point in the third point cloud also contains the height of the corresponding local part of the object.

[0076] S103. Determine the points with a height greater than the preset height threshold from the third point cloud to generate a fourth point cloud, and determine the points with a height less than or equal to the preset height threshold to generate a fifth point cloud.

[0077] The points in the point cloud contain the height of the corresponding local part of the object, rather than the overall height of the object. Therefore, the points in the third point cloud with a height greater than the height threshold can be determined as the points of the part of the large object with a height greater than the height threshold, while the points with a height less than or equal to the height threshold may be the points corresponding to small objects, the part of the large object less than or equal to the height threshold, or the ground. Generally speaking, during the process of processing vehicle-mounted lidar point cloud data, ground point cloud processing is usually performed on the point cloud data. Therefore, in this embodiment, the third point cloud can be regarded as having undergone ground point cloud processing, that is, the third point cloud does not include ground point cloud.

[0078] Specifically, the height of the corresponding local part of each point in the third point cloud can be obtained first, and then it is determined whether the height is greater than the preset height threshold. If so, the point is classified into the fourth point cloud, that is, the large object point cloud. If not, the point is classified into the fifth point cloud. Then the fourth point cloud is the set of points of the part of the large object with a height greater than the height threshold, and the fifth point cloud is the set of points of small objects and the part of the large object less than or equal to the height threshold.

[0079] Among them, the preset height threshold is the upper limit value of the height of small objects, which can be set according to actual detection requirements. For example, when the objects to be detected are mainly roadblocks, guardrails, etc., the height threshold can be set to 0.5m, and when the objects are mainly wood, stone piers, the height threshold can be set to 0.3m. In addition, when obtaining the point cloud data, the chassis of the vehicle can be used as the origin of the coordinate system of each frame of point cloud, and the height of the obstacle is relative to the height of the vehicle chassis.

[0080] S104. Based on the projections of the points in the fourth point cloud and the fifth point cloud on the X-Y plane in the first point cloud coordinate system, the points in the fifth point cloud are screened to generate the sixth point cloud.

[0081] The fourth point cloud is the set of points of the part of the large object with a height greater than the height threshold, and the part of the large object less than or equal to the height threshold is included in the fifth point cloud. In order to delete the points of the large object in the fifth point cloud, the points in the fourth point cloud can be projected onto the X-Y plane first to obtain the projection area of the large object, and then the points of the fifth point cloud are projected onto the X-Y plane. If the projection of the point of the fifth point cloud falls into this projection area, the point is screened out from the fifth point cloud to remove the point cloud corresponding to the large object in the fifth point cloud, and finally the set of points corresponding to small objects is obtained, and the set of points corresponding to small objects is used as the sixth point cloud. Among them, the third point cloud is the point cloud in the first point cloud coordinate system, so the fourth point cloud and the fifth point cloud are also the point clouds in the first point cloud coordinate system, and the X-Y plane is the X-Y plane in the first point cloud coordinate system.

[0082] S105. The sixth point cloud and the first point cloud are fused to obtain the target point cloud.

[0083] The point cloud in the sixth point cloud includes the point clouds of multiple small objects. The first point cloud is the current latest frame of point cloud. The sixth point cloud and the first point cloud are fused to obtain a target point cloud. The target point cloud includes the point clouds of multiple small objects and the point cloud of the latest frame of large object, which improves the point cloud density of the small objects. The point cloud density is an index of data resolution. For the same object, a higher point cloud density means more information or high-resolution image, and a higher point cloud density means less information or low-resolution image. More information or high-resolution image can more fully reflect the characteristics of small objects, that is, it improves the accuracy and stability of small object detection and facilitates the system or driver to judge the nature of small objects.

[0084] On the other hand, generally speaking, since moving objects such as motor vehicles and pedestrians generally belong to the range of large objects in this embodiment, that is, the point clouds of these moving objects are not included in the sixth point cloud, and only the first point cloud contains the point clouds of moving objects. After fusing the sixth point cloud and the first point cloud to obtain the target point cloud, the target point cloud only contains the point cloud of one frame of moving object, which can avoid the deformation of the moving object caused by the fusion of the point clouds of multiple frames of moving objects.

[0085] After obtaining the target point cloud, the target point cloud can be processed according to the vehicle-mounted operating system, for example, by the lidar depth imaging method, the coordinate measuring machine method, etc. After processing, a depth image is obtained. The depth image is an image with the distance value from the lidar to each point in the environment as the pixel value. It directly reflects the geometric shape of the visible surface of the scene. The vehicle-mounted operating system can judge the small objects on the driving section and then give a small obstacle prompt broadcast, or display it on the display page of the system for the driver to view.

[0086] The point cloud fusion method of the embodiment of the present invention is mainly used to detect small obstacles during vehicle driving. First, a point cloud set including the first point cloud collected by the lidar and multiple frames of second point clouds collected before the first point cloud is obtained. The second point cloud is converted into a point cloud in the coordinate system of the first point cloud and fused into a third point cloud. Each point in the third point cloud contains the height of the local part of the object corresponding to the point. Then, points with a height greater than a preset height threshold are determined from the third point cloud to generate a fourth point cloud, and points with a height less than or equal to the preset height threshold are determined to generate a fifth point cloud. Among them, the preset height threshold is the upper limit value of the height of small objects. Based on the projection of the points in the fourth point cloud and the fifth point cloud on the X-Y plane, the points in the fifth point cloud are screened to generate a sixth point cloud. The sixth point cloud and the first point cloud are fused to obtain a target point cloud. The target point cloud includes the point clouds of multiple frames of small objects and the point cloud of the latest frame of large object, which improves the density of the point cloud of small objects, can obtain a detailed image of the point cloud of small objects, and increases the stability of small object detection. On the other hand, the target point cloud only contains the point cloud of one frame of moving object, which can avoid the deformation of the moving object caused by the fusion of the point clouds of multiple frames of moving objects.

[0087] Example 2

[0088] Figure 2 The figure is a flowchart of a point cloud fusion method provided in Example 2 of the present invention. This embodiment is further refined based on the foregoing Example 1. The method specifically includes the following steps:

[0089] S201. Obtain a point cloud set, where the point cloud set includes the first point cloud collected by the lidar and multiple frames of second point clouds collected before the first point cloud;

[0090] The detection times of the second point cloud and the first point cloud are different. The first point cloud is the current latest frame of point cloud, and the second point cloud is the point cloud collected before the first point cloud. Since the data of the same point cloud is the same, in order to ensure the richness of the point cloud data and avoid wasting computing resources, when extracting multiple frames of second point clouds, each frame of second point cloud can only be extracted once.

[0091] S202. Convert the multiple frames of second point clouds into point clouds in the coordinate system of the first point cloud, and fuse the converted point clouds into a third point cloud.

[0092] The conversion of the coordinate points of two point clouds is to transform a point cloud from its own coordinate system to another coordinate system. In this embodiment, the rotation matrix and translation matrix of the second point cloud relative to the first point cloud can be calculated through point cloud registration algorithms such as ICP or NDT, and then the coordinates of the second point cloud are converted to the coordinate system of the first point cloud according to the rotation matrix and translation matrix to obtain the converted second point cloud. Then, the multiple frames of converted second point clouds are fused into a third point cloud to transform the sparse point cloud into a dense point cloud.

[0093] S203. Determine the points with a height greater than a preset height threshold from the third point cloud to generate a fourth point cloud, and determine the points with a height less than or equal to the preset height threshold to generate a fifth point cloud.

[0094] The points in the point cloud contain the height of the local part of the corresponding object, rather than the overall height of the object. Therefore, the points in the third point cloud with a height greater than the height threshold can be determined as the points of the part of the large object with a height greater than the height threshold, and the points with a height less than or equal to the height threshold may be the points corresponding to small objects, the part of the large object less than or equal to the height threshold, or the ground.

[0095] Generally speaking, during the process of processing vehicle-mounted lidar point cloud data, ground point cloud processing is usually performed on the point cloud data. Since the ground point cloud algorithm also consumes resources, the ground point cloud can be removed from the third point cloud first to obtain the obstacle point cloud, and then the points with a height greater than the preset height threshold can be determined from the obstacle point cloud to generate the fourth point cloud. First, perform ground point cloud segmentation to obtain the obstacle point cloud, and then screen out the points with a height greater than the preset height threshold from the obstacle point cloud. Compared with directly screening out the points with a height greater than the preset height threshold from the third point cloud, the calculation efficiency is higher, and computing resources are also saved.

[0096] Among them, the preset height threshold is the upper limit value of the height of small objects, which can be set according to actual detection requirements.

[0097] When obtaining the fifth point cloud, the height of the corresponding local object of each point in the third point cloud can be obtained first, and the points with a height less than or equal to the preset height threshold are used as the fifth point cloud. Then the fourth point cloud is the set of points in the large object with a height greater than the height threshold, and the fifth point cloud is the set of points in the small object and the large object with a height less than or equal to the height threshold.

[0098] S204. In the direction parallel to the X-Y plane in the first point cloud coordinate system, expand the area of the points in the fourth point cloud according to the preset magnification parameter to obtain multiple expanded surfaces.

[0099] For obstacles, the upper and lower parts are generally continuous entities. The fourth point cloud is the set of points in the large object with a height greater than the height threshold. The points in the large object with a height less than or equal to the height threshold must be included in the fifth point cloud. To delete the points of the large object in the fifth point cloud, the points in the fourth point cloud can be projected onto the X-Y plane first to obtain the projection area of the large object, and then the points of the fifth point cloud are projected onto the X-Y plane. If the points of the fifth point cloud fall into the projection area, the point is screened out from the fifth point cloud to remove the point cloud corresponding to the large object in the fifth point cloud, and finally the set of points corresponding to the small object is obtained as the sixth point cloud. Among them, the X-Y plane is the X-Y plane in the first point cloud coordinate system.

[0100] Since the point cloud in this embodiment is a sparse point cloud and there is a certain distance between points, the area of the points in the fourth point cloud can be expanded in this embodiment so that the projections of the points corresponding to the same object in the fourth point cloud on the X-Y plane can be connected to form a closed area. Specifically, the points in the fourth point cloud can be expanded in the direction parallel to the X-Y plane according to the preset magnification parameter to obtain multiple expanded surfaces. The expanded surface can be circular or a polygon of other shapes.

[0101] In an example of this embodiment, for each point in the fourth point cloud, in a direction parallel to the X-Y plane, a circle with the point as the center and a preset magnification parameter as the radius is used as the expansion surface of the point. The preset magnification parameter can be set according to the size of the object. For example, it can be set according to the width of the object. For an object with a larger width, a larger magnification parameter is correspondingly set, while for an object with a smaller width, a smaller magnification parameter is correspondingly set, so as to magnify the points corresponding to objects with different widths in different proportions.

[0102] S205. Screen the points in the fifth point cloud according to the multiple expansion surfaces and the projection of the fifth point cloud on the X-Y plane to generate a sixth point cloud.

[0103] The expansion surface is the magnification surface of the points in the fourth point cloud in a direction parallel to the X-Y plane. The projections of the multiple expansion surfaces on the X-Y plane are also multiple expansion surfaces. Then, it is possible to determine whether the points in the fifth point cloud are the points of the part of the large object that is less than or equal to the height threshold according to the relationship between the points in the fifth point cloud and the projections of the expansion surfaces on the X-Y plane.

[0104] In an example of this embodiment, screening the points in the fifth point cloud according to the projection of the expansion surface on the X-Y plane includes: projecting all the expansion surfaces onto the X-Y plane to obtain multiple closed connected regions. The connected regions are the union of the projections of the expansion surfaces on the X-Y plane. Based on the projection of the fifth point cloud on the X-Y plane and the connected regions, the points in the fifth point cloud are screened out, and the fifth point cloud after screening is used as the sixth point cloud. Among them, the connected region is the union of the regions of the projections of all the expansion surfaces on the X-Y plane.

[0105] In an example of this embodiment, screening out the points in the fifth point cloud based on the projection of the fifth point cloud on the X-Y plane and the connected regions includes: for each point in the fifth point cloud, judging whether the projection of the point on the X-Y plane is located within the connected region. If so, the point is screened out from the fifth point cloud; if not, the point is retained.

[0106] By expanding the points in the fourth point cloud and then projecting them onto the X-Y plane, the coverage area of the points in the fourth point cloud on the X-Y plane can be increased, that is, the coverage area of the points corresponding to the large object on the X-Y plane is increased, avoiding the situation that the projections of the points of the part of the large object that is less than or equal to the height threshold on the X-Y plane cannot fall into the projections of the points of the part of the large object that is greater than the height threshold on the X-Y plane due to the sparsity of the point cloud. Furthermore, it can ensure that the points of the part of the large object that is less than or equal to the height threshold in the fifth point cloud can be screened out, and only the point cloud of the small object is retained, so as to increase the density and purity of the point cloud of the small object and avoid being mixed with the point cloud of the large object and affecting the detection effect of the small object.

[0107] S206. Fuse the sixth point cloud and the first point cloud to obtain the target point cloud.

[0108] The sixth point cloud includes point clouds of multiple small objects. The first point cloud is the current latest frame of point cloud. The sixth point cloud and the first point cloud are fused to obtain a target point cloud. The target point cloud includes point clouds of multiple small objects and the point cloud of the latest large object, which improves the point cloud density of the small objects. The point cloud density is an index of data resolution. For the same object, a higher point cloud density means more information or a high-resolution image, while a lower point cloud density means less information or a low-resolution image. More information or a high-resolution image can more fully reflect the characteristics of small objects, that is, it improves the accuracy and stability of small object detection and facilitates the system or the driver to judge the nature of small objects.

[0109] On the other hand, generally speaking, since moving objects such as motor vehicles and pedestrians generally belong to the range of large objects in this embodiment, that is, the point clouds of these moving objects are not included in the sixth point cloud, and only the first point cloud contains the point clouds of moving objects. After fusing the sixth point cloud and the first point cloud to obtain the target point cloud, the target point cloud only contains the point cloud of one frame of moving object, which can avoid the situation of deformation of moving objects caused by the fusion of point clouds of multiple frames of moving objects.

[0110] The point cloud fusion method of the embodiment of the present invention first obtains a point cloud set including the first point cloud collected by a lidar and multiple frames of second point clouds collected before the first point cloud, converts the second point clouds into point clouds in the coordinate system of the first point cloud and fuses them into a third point cloud. Each point in the third point cloud contains the height of the local part of the object corresponding to the point. Then, points with a height greater than a preset height threshold are determined from the third point cloud to generate a fourth point cloud, and points with a height less than or equal to the preset height threshold are determined to generate a fifth point cloud. The preset height threshold is the upper limit value of the height of small objects. The points in the fourth point cloud are regionally expanded in a direction parallel to the X-Y plane according to a preset magnification parameter to obtain multiple expanded surfaces. The points in the fifth point cloud are screened according to the expanded surfaces and the projection of the fifth point cloud on the X-Y plane, and the screened fifth point cloud is used as the sixth point cloud. The sixth point cloud and the first point cloud are fused to obtain the target point cloud. By expanding the region of the points in the fourth point cloud and then projecting them onto the X-Y plane, the coverage area of the points in the fourth point cloud on the X-Y plane can be increased, that is, the coverage area of the points corresponding to the large object on the X-Y plane can be increased, avoiding the situation that the projection of the points in the large object with a height less than or equal to the height threshold on the X-Y plane cannot fall into the projection of the points in the large object with a height greater than the height threshold on the X-Y plane due to the sparsity of the point cloud. Furthermore, it can ensure that the points in the large object with a height less than or equal to the height threshold in the fifth point cloud can be screened out, and only the point cloud of small objects is retained, so as to increase the density and purity of the point cloud of small objects and avoid being mixed with the point cloud of large objects and affecting the small object detection effect.

[0111] It should be noted that for the method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the embodiments of the present invention are not limited by the described action sequences, because according to the embodiments of the present invention, certain steps can be carried out in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential for the embodiments of the present invention.

[0112] Embodiment III

[0113] Figure 3 The following is a structural block diagram of a point cloud fusion device provided in Embodiment III of the present invention, which specifically may include the following modules:

[0114] A point cloud set acquisition module 301, configured to acquire a point cloud set, where the point cloud set includes a first point cloud collected by a lidar and multiple frames of second point clouds collected before the first point cloud;

[0115] A point cloud fusion module 302, configured to convert multiple frames of second point clouds into point clouds in the first point cloud coordinate system, and fuse the converted point clouds into a third point cloud, where each point in the third point cloud includes the height of the local part of the object corresponding to the point;

[0116] A point cloud segmentation module 303, configured to determine points with a height greater than a preset height threshold from the third point cloud to generate a fourth point cloud, and determine points with a height less than or equal to the preset height threshold to generate a fifth point cloud;

[0117] A sixth point cloud generation module 304, configured to screen out points in the fifth point cloud based on the projections of the points in the fourth point cloud and the fifth point cloud on the X-Y plane in the first point cloud coordinate system to generate a sixth point cloud;

[0118] A target point cloud generation module 305, configured to fuse the sixth point cloud and the first point cloud to obtain a target point cloud.

[0119] Optionally, the point cloud set acquisition module 301 includes:

[0120] A point cloud acquisition sub-module, configured to use the point cloud collected by the current lidar as the first point cloud during the driving process of the vehicle.

[0121] Optionally, the point cloud segmentation module 303 further includes:

[0122] An obstacle point cloud segmentation sub-module, configured to remove ground point clouds from the third point cloud to obtain obstacle point clouds;

[0123] A fourth point cloud generation sub-module, configured to determine points with a height greater than a preset height threshold from the obstacle point clouds to generate a fourth point cloud.

[0124] Optionally, the sixth point cloud generation module 304 includes:

[0125] A region expansion sub-module, configured to expand the points in the fourth point cloud in a direction parallel to the X-Y plane in the first point cloud coordinate system according to a preset magnification parameter to obtain a plurality of expanded surfaces;

[0126] A fifth point cloud screening sub-module, configured to screen the points in the fifth point cloud according to the plurality of expanded surfaces and the projection of the fifth point cloud on the X-Y plane to generate a sixth point cloud.

[0127] Optionally, the region expansion sub-module includes:

[0128] An expanded surface forming unit, configured to, for each point in the fourth point cloud, use a circle with the point as the center and the preset magnification parameter as the radius as the expanded surface of the point in a direction parallel to the X-Y plane in the first point cloud coordinate system.

[0129] Optionally, the fifth point cloud screening sub-module includes:

[0130] A connected region forming unit, configured to project all the expanded surfaces onto the X-Y plane to obtain a plurality of closed connected regions, where the connected regions are the union of the projections of the expanded surfaces on the X-Y plane;

[0131] A fifth point cloud screening unit, configured to screen the points in the fifth point cloud based on the projection of the fifth point cloud on the X-Y plane and the connected regions.

[0132] Optionally, the fifth point cloud screening unit includes:

[0133] A point region determination sub-unit, configured to, for each point in the fifth point cloud, determine whether the projection of the point on the X-Y plane is located within the connected region; if so, screen the point from the fifth point cloud; if not, retain the point.

[0134] The point cloud fusion device provided by the embodiments of the present invention can execute the point cloud fusion method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

[0135] Embodiment 4

[0136] Refer to Figure 4 , which shows a schematic structural diagram of a computer device in an example of the present invention. As Figure 4 shown, the computer device may specifically include: a processor 401, a memory 402, a display screen 403 with a touch function, an input device 404, an output device 405, and a communication device 406. The number of processors 401 in the computer device may be one or more, Figure 4Take a processor 401 as an example. The number of memories 402 in the computer device can be one or more. Figure 4 Take a memory 402 as an example. The processor 401, memory 402, display screen 403, input device 404, output device 405, and communication device 406 of the device can be connected through a bus or other means. Figure 4 Take the connection through a bus as an example.

[0137] As a computer-readable storage medium, the memory 402 can be used to store software programs and computer-executable programs. The memory 402 mainly includes a program storage area and a data storage area. Among them, the program storage area can store operating devices and application programs required for at least one function; the data storage area can store data created according to the use of the device, etc. In addition, the memory 402 can include high-speed random access memory, and can also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage devices. In some instances, the memory 402 can further include a memory remotely set relative to the processor 401, and these remote memories can be connected to the device through a network. Examples of the above network include but are not limited to the Internet, enterprise intranet, local area network, mobile communication network, and their combinations.

[0138] The display screen 403 is a display screen 403 with a touch function, which can be a capacitive screen, an electromagnetic screen, or an infrared screen. Generally speaking, the display screen 403 is used to display data according to the instructions of the processor 401, and is also used to receive touch operations acting on the display screen 403 and send corresponding signals to the processor 401 or other devices.

[0139] The communication device 406 is used to establish a communication connection with other devices, and it can be a wired communication device and / or a wireless communication device.

[0140] The input device 404 can be used to receive input digital or character information, and generate key signal inputs related to the user settings and function control of the device. The output device 405 can include audio devices such as speakers. It should be noted that the specific composition of the input device 404 and the output device 405 can be set according to the actual situation.

[0141] The processor 401 executes various functional applications and data processing of the device by running software programs, instructions, and modules stored in the memory 402, that is, implements the above point cloud fusion method.

[0142] Specifically, in the embodiment, when the processor 401 executes one or more programs stored in the memory 402, it specifically implements the point cloud fusion method provided by the embodiments of the present invention.

[0143] Embodiment 5

[0144] Embodiment 5 of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the point cloud fusion method in any embodiment of the present invention can be implemented.

[0145] Certainly, for the storage medium provided by the embodiments of the present invention that contains computer-executable instructions, the computer-executable instructions are not limited to the method operations described above, and can also execute the point cloud fusion method provided by any embodiment of the present invention applied to a device.

[0146] It should be noted that for the embodiments of the device, computer device, and storage medium, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, please refer to the partial description of the method embodiments.

[0147] From the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software and necessary general-purpose hardware. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as a floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk, or optical disc of a computer, etc., including several instructions for causing a computer device (which can be a personal computer, computer device, server, or network device, etc.) to execute the point cloud fusion method described in each embodiment of the present invention.

[0148] It is worth noting that in the embodiments of the above-mentioned point cloud fusion device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of mutual distinction and do not limit the protection scope of the present invention.

[0149] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A point cloud fusion method, characterized in that, Applied to vehicle-mounted lidar, including: Obtain a point cloud set, where the point cloud set includes the first point cloud collected by the lidar and multiple frames of second point clouds collected before the first point cloud; Convert multiple frames of the second point clouds into point clouds in the coordinate system of the first point cloud, and fuse the converted point clouds into a third point cloud, where each point in the third point cloud contains the height of the local part of the object corresponding to the point; Determine points with a height greater than a preset height threshold from the third point cloud to generate a fourth point cloud, and determine points with a height less than or equal to the preset height threshold to generate a fifth point cloud, where the preset height threshold is the upper limit of the height of small objects, and the small objects are small obstacles; Based on the projections of the points in the fourth point cloud and the fifth point cloud on the X-Y plane in the coordinate system of the first point cloud, filter the points in the fifth point cloud to generate a sixth point cloud, including: first project the points in the fourth point cloud onto the X-Y plane in the coordinate system of the first point cloud to obtain a projection area, and then project the points in the fifth point cloud onto the X-Y plane. If the projection of the point in the fifth point cloud falls into the projection area, then remove the point from the fifth point cloud to generate a sixth point cloud; Fuse the sixth point cloud and the first point cloud to obtain a target point cloud.

2. The method according to claim 1, wherein The obtaining of the point cloud set includes: During the driving of the vehicle, use the point cloud collected by the current lidar as the first point cloud.

3. The method according to claim 1, characterized in that, The determining of points with a height greater than a preset height threshold from the third point cloud to generate a fourth point cloud includes: Remove the ground point cloud from the third point cloud to obtain an obstacle point cloud; Determine points with a height greater than a preset height threshold from the obstacle point cloud to generate a fourth point cloud.

4. The method according to any one of claims 1 to 3, characterized in that The filtering of the points in the fifth point cloud based on the projections of the points in the fourth point cloud and the fifth point cloud on the X-Y plane in the coordinate system of the first point cloud to generate a sixth point cloud includes: In a direction parallel to the X-Y plane in the coordinate system of the first point cloud, expand the area of the points in the fourth point cloud according to a preset magnification parameter to obtain multiple expanded surfaces; Filter the points in the fifth point cloud according to the multiple expanded surfaces and the projection of the fifth point cloud on the X-Y plane to generate a sixth point cloud.

5. The method according to claim 4, characterized in that The expanding of the area of the points in the fourth point cloud in a direction parallel to the X-Y plane in the coordinate system of the first point cloud according to a preset magnification parameter to obtain multiple expanded surfaces includes: For each point in the fourth point cloud, in a direction parallel to the X-Y plane in the coordinate system of the first point cloud, use a circle with the point as the center and the preset magnification parameter as the radius as the expanded surface of the point.

6. The method according to claim 4, characterized in that The filtering of the points in the fifth point cloud according to the multiple expanded surfaces and the projection of the fifth point cloud on the X-Y plane includes: Project all the expanded surfaces onto the X-Y plane to obtain multiple closed connected regions, where the connected regions are the union of the projections of the expanded surfaces on the X-Y plane; Filter the points in the fifth point cloud based on the projection of the fifth point cloud on the X-Y plane and the connected regions.

7. The method according to claim 6, wherein Sieving the points in the fifth point cloud based on the projection of the fifth point cloud on the X-Y plane and the connected region includes: For each point in the fifth point cloud, determine whether the projection of the point on the X-Y plane is located within the connected region; If so, sieving the point from the fifth point cloud; If not, retaining the point.

8. A point cloud fusion device, characterized in that, Including: A point cloud set acquisition module for acquiring a point cloud set, where the point cloud set includes a first point cloud collected by a lidar and multiple frames of second point clouds collected before the first point cloud; A point cloud fusion module for converting multiple frames of the second point clouds into point clouds in the coordinate system of the first point cloud and fusing the converted point clouds into a third point cloud, where each point in the third point cloud includes the height of the local part of the object corresponding to the point; A point cloud segmentation module for determining the points with a height greater than a preset height threshold from the third point cloud to generate a fourth point cloud, and determining the points with a height less than or equal to the preset height threshold to generate a fifth point cloud, where the preset height threshold is the upper limit of the height of a small object, and the small object is a small obstacle; A sixth point cloud generation module for sieving the points in the fifth point cloud based on the projections of the points in the fourth point cloud and the fifth point cloud on the X-Y plane in the coordinate system of the first point cloud to generate a sixth point cloud, including: first projecting the points in the fourth point cloud onto the X-Y plane in the coordinate system of the first point cloud to obtain a projection region, and then projecting the points of the fifth point cloud onto the X-Y plane. If the projection of the point of the fifth point cloud falls into the projection region, the point is sieved from the fifth point cloud to generate a sixth point cloud; A target point cloud generation module for fusing the sixth point cloud and the first point cloud to obtain a target point cloud.

9. A computer device, characterized in that, The computer device includes: One or more processors; A memory for storing one or more computer programs, When the one or more computer programs are executed by the one or more processors, the one or more processors implement the point cloud fusion method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the point cloud fusion method according to any one of claims 1-7 is implemented.

Images