Dense ground point cloud generation method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention provides a dense ground point cloud generation method and device, a storage medium and electronic equipment, and the method comprises the steps: obtaining an original ground point cloud; generating a plurality of triangular meshes based on the original ground point cloud; each triangular mesh in the plurality of triangular meshes is used for representing a local area on the ground; performing point cloud densification processing on the original ground point cloud to obtain a three-dimensional space point set; determining that each space point in the three-dimensional space point set is projected into a triangular mesh in the plurality of triangular meshes, and obtaining a triangular mesh corresponding to each space point; determining a set of central points of the triangular mesh corresponding to each spatial point as a dense ground point cloud; the point cloud density of the dense ground point cloud is greater than the point cloud density of the original ground point cloud, and the method solves the problems of high calculation complexity and poor precision of a dense ground point cloud generation method in the related technology.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of autonomous driving technology, and more specifically, to a method, device, storage medium, and electronic device for generating a dense ground point cloud. Background Art

[0002] With the development of autonomous driving technology, accurate environmental perception has become the key to achieving safe and reliable autonomous driving functions. When LiDAR is used for lane line perception tasks, the ground point cloud data it outputs is crucial for lane marking and detection. However, the existing ground point clouds are usually sparse and lack the necessary details. Dense point cloud generation methods are usually used to generate dense point clouds.

[0003] Traditional dense point cloud generation methods, such as image-based depth estimation and multi-view stereo matching, can provide relatively dense point clouds, but these methods rely on high-quality image data, have high computational complexity, and poor accuracy. Summary of the invention

[0004] The embodiments of the present application provide a method, device, storage medium and electronic device for generating a dense ground point cloud, so as to at least solve the problems of high computational complexity and poor precision existing in the method for generating dense ground point cloud in the related art.

[0005] According to one aspect of an embodiment of the present application, a method for generating a dense ground point cloud is provided, including: obtaining an original ground point cloud; generating multiple triangular meshes based on the original ground point cloud; each triangular mesh in the multiple triangular meshes is used to represent a local area on the ground; performing point cloud densification processing on the original ground point cloud to obtain a three-dimensional space point set; determining each spatial point in the three-dimensional space point set to be projected into a triangular mesh in the multiple triangular meshes to obtain a triangular mesh corresponding to each spatial point; determining a set of center points of the triangular meshes corresponding to each spatial point as a dense ground point cloud; the point cloud density of the dense ground point cloud is greater than the point cloud density of the original ground point cloud.

[0006] According to another aspect of an embodiment of the present application, a dense ground point cloud generation device is provided, comprising: a laser radar scanning module for acquiring an original ground point cloud; a triangular mesh generation module for generating a plurality of triangular meshes based on the original ground point cloud; each of the plurality of triangular meshes is used to represent a local area on the ground; a point cloud densification module for performing point cloud densification processing on the original ground point cloud to obtain a three-dimensional space point set; a denoising module for determining each spatial point in the three-dimensional space point set to be projected into a triangular mesh in the plurality of triangular meshes to obtain a triangular mesh corresponding to each spatial point; a dense point cloud generation module for determining a set of center points of the triangular meshes corresponding to each spatial point as a dense ground point cloud; the point cloud density of the dense ground point cloud is greater than the point cloud density of the original ground point cloud.

[0007] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned dense ground point cloud generation method when running.

[0008] According to another aspect of an embodiment of the present application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the dense ground point cloud generation method through the computer program.

[0009] According to another aspect of the embodiments of the present application, a computer program product is also provided, including a computer program, which, when executed by a processor, implements the steps in any of the above-mentioned dense ground point cloud generation method embodiments.

[0010] Through the embodiment of the present application, the collected original ground point cloud is triangulated to obtain multiple triangular meshes, thereby dividing the surface of the original ground point cloud into multiple uniform local areas, providing basic data for obtaining a uniform dense ground point cloud. Further, the original ground point cloud is densified to obtain a three-dimensional space point set. The embodiment of the present application does not directly use the three-dimensional space point set obtained after the densification of the original ground point cloud as the dense ground point cloud to be used in the end, but chooses to project each spatial point in the three-dimensional space point set to a uniform triangular mesh, and determines the set of center points of the triangular mesh corresponding to each spatial point as the dense ground point cloud. This method not only effectively filters out noise points that may exist in the original ground point cloud, significantly improves the signal-to-noise ratio of the original ground point cloud, but also successfully converts the sparse and unevenly distributed point cloud data that may exist in the original ground point cloud into a more uniform and dense representation. More importantly, the entire processing process abandons the dependence on high-quality image data, significantly reduces the computational complexity, and thus obtains a dense ground point cloud with higher accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a hardware structure block diagram of an optional dense ground point cloud generation method of an embodiment of the present application;

[0012] Figure 2 is a schematic flow chart of an optional dense ground point cloud generation method according to an embodiment of the present application;

[0013] Figure 3 is a schematic diagram of an optional triangular mesh according to an embodiment of the present application;

[0014] Figure 4 is a schematic diagram of a triangular mesh obtained after meshing an optional original ground point cloud according to an embodiment of the present application;

[0015] Figure 5 is a schematic diagram of an optional ground mask image according to an embodiment of the present application;

[0016] Figure 6 is a schematic diagram of an optional dense ground point cloud according to an embodiment of the present application;

[0017] Figure 7 is a structural block diagram of an optional dense ground point cloud generation device according to an embodiment of the present application;

[0018] Figure 8 It is a structural block diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0020] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0021] According to one aspect of an embodiment of the present application, a method for generating a dense ground point cloud is provided. Optionally, in this embodiment, the method for generating a dense ground point cloud can be applied to Figure 1 In the hardware environment composed of the vehicle terminal 102 and the server 104 shown in FIG. Figure 1 As shown, the server 104 is connected to the vehicle-mounted terminal 102 via a network, and can be used to provide services (such as application services, etc.) for the terminal or a client installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for the server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data computing services for the server 104.

[0022] The above network may include but is not limited to at least one of the following: wired network, wireless network. The above wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the above wireless network may include but is not limited to at least one of the following: WIFI, Bluetooth.

[0023] The dense ground point cloud generation method of the embodiment of the present application can be executed by the vehicle terminal 102, or can be executed by the server 104 and the vehicle terminal 102. Taking the vehicle terminal 102 executing the dense ground point cloud generation method of the embodiment as an example, Figure 2 is a flow chart of an optional dense ground point cloud generation method according to an embodiment of the present application, such as Figure 2As shown, the process of the method may include the following steps:

[0024] Step S202, obtaining the original ground point cloud.

[0025] The original ground point cloud is a three-dimensional spatial data representation directly obtained by scanning the ground with a laser radar (LiDAR). The original ground point cloud contains a large number of discrete three-dimensional point coordinates (x, y, z) on the ground, which together constitute the three-dimensional shape and contour of the lane line on the ground.

[0026] Optionally, the vehicle terminal starts a laser radar to scan the area containing the lane line. The laser radar will automatically emit laser pulses, receive reflected signals, calculate distance and angle information, and record the obtained three-dimensional point data in real time or in storage to form a point cloud data set S, and feed the point cloud data set S back to the vehicle terminal. The vehicle terminal performs ground segmentation and extraction processing on the point cloud data set S to obtain the original ground point cloud (referred to as S_ground).

[0027] In some embodiments, commonly used ground segmentation methods include: a height threshold-based method (i.e., setting a height threshold, and points below the threshold are considered to be ground points), a shape-based method (calculating the normal vector of each point in the point cloud data and its neighborhood points, and using the distribution characteristics of the normal vector to determine whether the point belongs to the ground), and a plane fitting method (selecting a part of possible ground points (such as the point with the smallest z value), and then using these points to fit a plane model (such as least squares fitting), and judging whether other points belong to the ground based on the distance from the point to the plane).

[0028] Step S204, generating a plurality of triangular meshes based on the original ground point cloud; each of the plurality of triangular meshes is used to represent a local area on the ground.

[0029] Among them, triangular mesh is a method of discretizing a continuous surface or volume into a series of small triangles or tetrahedrons. The basic unit of triangular mesh is a triangle, which consists of three vertices and three edges, and can effectively describe the geometric characteristics and topological structure of the surface. Each triangular mesh is used to represent a local area on the ground.

[0030] Optionally, the vehicle terminal can select four adjacent pixel points from the original ground point cloud in turn, and connect three of the pixel points in a clockwise direction, so as to construct multiple triangular meshes one by one. The vehicle terminal can also generate an enclosing space that fits closely to the original ground point cloud. The enclosing space is a virtual three-dimensional structure that is as close to the surface of the original ground point cloud as possible. The surface of the enclosing space (actually, the surface that is as close to the original ground point cloud as possible) is triangularly meshed, thereby dividing the surface of the enclosing space into multiple closely connected triangular meshes.

[0031] In some embodiments, the vehicle-mounted terminal can generate an enclosing space that fits closely to the original ground point cloud by a method based on geometric projection and expansion, specifically including: projecting the original ground point cloud onto a two-dimensional plane (such as a horizontal plane), and using morphological operations (such as variations of erosion and expansion) on the projected two-dimensional image to identify and extract the two-dimensional boundary points of the point cloud, mapping the extracted two-dimensional boundary points to a three-dimensional space based on the height values ​​of the two-dimensional boundary points before projection, and constructing a three-dimensional polyhedron or surface as an enclosing space based on the mapping points of the two-dimensional boundary points in the three-dimensional space.

[0032] In some embodiments, the vehicle terminal can also generate an enclosing space that fits closely to the original ground point cloud by a method based on volume segmentation and shape fitting, specifically including: dividing the three-dimensional space containing the original ground point cloud into multiple small volume units (such as cube or spherical grids). Each unit is assigned different weights or importance according to the number or density of point cloud points inside it. One or more core points are determined by calculating the weighted centroid or density center of all volume units (the core point can be the geometric center, centroid, maximum density point of the point cloud data, or a point calculated according to a specific algorithm (such as a clustering center)), and these core points will serve as the starting point or reference point for the construction of the enclosing space. Based on the core point, initialize a basic shape as the starting point of the boundary, such as a sphere, a cube, or an initial polygon estimated based on the distribution of the point cloud. According to the scale and distribution of the point cloud, adjust the size and position of the initial shape so that it roughly covers the area of ​​the original ground point cloud. Expand the boundary at the same rate in each direction, or expand at different rates in different directions according to the density or characteristics of the point cloud. After each expansion, one or more optimization goals are set based on preset indicators (such as the minimum distance between the boundary and the point cloud, the proportion of the boundary covering the point cloud, the smoothness or shape complexity of the boundary, etc.), such as minimizing the distance metric, maximizing the coverage, or maintaining appropriate shape complexity. The expansion direction and rate are continuously adjusted according to the optimization goal until the optimization goal is reached. The initial shape when the optimization goal is reached is determined as the enclosing space.

[0033] Step S206, performing point cloud densification processing on the original ground point cloud to obtain a three-dimensional space point set.

[0034] Point cloud densification refers to increasing the number of points in the point cloud to make it denser, thereby more accurately representing the surface shape or terrain features of the original ground. The original ground point cloud appears sparse or incomplete due to the limitations of the acquisition equipment, environmental factors or the complexity of the object itself. In order to more accurately reconstruct the 3D model of the ground, the sparse original ground point cloud needs to be densified.

[0035] A 3D point set refers to a set of points that are generated based on the original ground point cloud. The generated new points and the original points together constitute a point set. A 3D point set includes a set of spatial points with coordinates in 3D space, and each spatial point is used to represent the shape, contour or terrain of the ground.

[0036] Optionally, the vehicle terminal increases the number of pixel points in the original ground point cloud by some methods, and stops generating new pixel points when the number of pixel points in the original ground point cloud reaches a certain number or meets other conditions, thereby obtaining a three-dimensional space point set.

[0037] In some embodiments, the vehicle terminal can generate new pixel points between the original ground point cloud by interpolation. Common interpolation methods include: linear interpolation and inverse distance weighted interpolation (IDW), where linear interpolation is to directly connect two adjacent points and insert new points on the line at a certain interval. Inverse distance weighted interpolation (IDW) is to give different weights according to the distance, and the closer the point is, the greater the influence on the new point.

[0038] In some embodiments, the vehicle-mounted terminal can also reconstruct a surface model (such as a grid model) of the ground based on the original ground point cloud, and then sample and generate new points on the reconstructed surface model. For example, the original ground point cloud is approximated by locally fitting a plane or a curved surface, and then new pixel points are sampled and generated on the fitted surface. Alternatively, the original ground point cloud is divided into a grid according to triangulation, and then new pixel points are sampled and generated on the vertices or edges of the grid.

[0039] In some embodiments, the vehicle-mounted terminal can also convert the original ground point cloud into a two-dimensional image, and then densely process the two-dimensional pixel points in the two-dimensional image through interpolation, image enhancement, etc. to obtain a dense two-dimensional point set, and then convert the processed dense two-dimensional point set to three-dimensional space to obtain a three-dimensional space point set.

[0040] Step S208 , determining that each spatial point in the three-dimensional spatial point set is projected into a triangular mesh among the plurality of triangular meshes, and obtaining a triangular mesh corresponding to each spatial point.

[0041] Among them, the three-dimensional space point set obtained after the original ground point cloud is densified may still contain some abnormal points caused by errors or noise in the acquisition process. Therefore, the embodiment of the present application does not directly use the three-dimensional space point set obtained after the original ground point cloud is densified as the dense ground point cloud to be used in the end, but chooses to further process it, that is, projecting each spatial point in the three-dimensional space point set to a triangular mesh, and determining the set of center points of the triangular mesh corresponding to each spatial point as the dense ground point cloud.

[0042] It should be noted that the three-dimensional space point set obtained after the densification of the original ground point cloud may introduce noise or errors in the following aspects: The first is the limitation of the laser radar scanning technology. Specifically, when the laser radar scans the ground to obtain the original ground point cloud, the original ground point cloud contains certain errors due to accuracy, resolution, scanning angle, distance and other reasons. Therefore, there are still abnormal points in the three-dimensional space point set obtained after the original ground points are densified. Secondly, factors such as lighting conditions, obstructions, and reflective surfaces in the acquisition environment interfere with the acquisition process, thereby introducing noise or abnormal points in the original ground point cloud. Finally, the algorithms used in the densification process (such as interpolation, reconstruction, filtering, etc.) may not be able to completely eliminate errors or noise in the original data, or introduce new errors in the processing process.

[0043] It should be noted that the noise, errors and outliers in the original ground point cloud cannot be completely removed before densification. This is mainly because when the laser radar or other scanning equipment directly collects the original ground point cloud, due to the limitations of the equipment itself (such as accuracy, resolution, scanning angle, etc.) and the influence of environmental factors (such as illumination, occlusion, reflection, etc.), these original data already contain certain errors and noise. These errors and noise are inevitable in the data collection process and are often difficult to completely remove in the data collection stage. Secondly, before the densification process, even if some outliers can be identified and removed, there is no guarantee that new outliers will not be introduced during the densification process. Finally, the point cloud data obtained after the densification of the original ground points has a higher density and a more complete structure, so that the filtering and cleaning algorithms can more accurately analyze the local neighborhood characteristics of each point, thereby more effectively identifying outliers. Before the densification process, due to the sparsity and incompleteness of the point cloud data, the effect of the filtering and cleaning algorithms may be limited.

[0044] The embodiment of the present application projects each spatial point in a three-dimensional spatial point set into a triangular mesh in multiple triangular meshes. Through smoothing or filtering, the noise points in the three-dimensional spatial point set can be removed or weakened, thereby improving the signal-to-noise ratio of the three-dimensional spatial point set. By merging similar or redundant triangles, redundant information in the data can be reduced, thereby improving the compactness and efficiency of the data.

[0045] Optionally, for each spatial point in the three-dimensional spatial point set, the vehicle-mounted terminal uses a selected projection method (such as orthogonal projection, perspective projection, etc.) to project each spatial point onto the plane where the triangular mesh is located, and determines the triangular mesh where each projection point is located by calculating the positional relationship between each projection point and the triangular mesh, and records the triangular mesh information of each projection point (such as the index of the triangle, vertex coordinates, etc.), and determines the triangular mesh where each projection point is located as the triangular mesh corresponding to each spatial point.

[0046] In step S210, a set of center points of the triangular mesh corresponding to each spatial point is determined as a dense ground point cloud; the point cloud density of the dense ground point cloud is greater than the point cloud density of the original ground point cloud.

[0047] Among them, the center point of the triangular mesh can usually better represent the geometric center or center of gravity of the triangular mesh area where it is located, thereby reflecting the ground morphology of the area to a certain extent.

[0048] In an embodiment of the present application, the center point of the triangular mesh corresponding to each spatial point is selected as part of the dense ground point cloud, so that the sparse and unevenly distributed point cloud data that may exist in the original ground point cloud can be converted into a more uniform and dense representation, while reducing the impact of single-point errors or noise on the overall ground morphology.

[0049] Optionally, the vehicle-mounted terminal determines the center point of the triangular mesh corresponding to each spatial point, and divides the center point into a set, and determines the set as a dense ground point cloud.

[0050] Through the above steps S202 to S210, the collected original ground point cloud is triangulated to obtain multiple triangular meshes, thereby dividing the surface of the original ground point cloud into multiple uniform local areas, providing basic data for obtaining a uniform dense ground point cloud. Further, the original ground point cloud is densified to obtain a three-dimensional space point set. The embodiment of the present application does not directly use the three-dimensional space point set obtained after the densification of the original ground point cloud as the dense ground point cloud to be used in the end, but chooses to project each spatial point in the three-dimensional space point set to a uniform triangular mesh, and determines the set of center points of the triangular mesh corresponding to each spatial point as the dense ground point cloud. This method not only effectively filters out noise points that may exist in the original ground point cloud, significantly improves the signal-to-noise ratio of the original ground point cloud, but also successfully converts the sparse and unevenly distributed point cloud data that may exist in the original ground point cloud into a more uniform and dense representation. More importantly, the entire processing process abandons the dependence on high-quality image data, significantly reduces the computational complexity, and thus obtains a dense ground point cloud with higher accuracy.

[0051] In an exemplary embodiment, multiple triangular meshes are generated based on the original ground point cloud, including:

[0052] 1. Generate a gridded point cloud based on the original ground point cloud; the gridded point cloud is a new point cloud obtained by determining a grid division space based on the original ground point cloud, dividing the grid division space into multiple grids, and generating pixel points in each of the multiple divided grids.

[0053] Among them, the gridded point cloud is a new point cloud generated in the gridded space determined by the original ground point cloud. The gridded point cloud consists of a series of pixel points (or "grid points") in a regular grid. It should be noted that the pixel points of the gridded point cloud are not directly selected from the original ground point cloud, but are generated in the gridded space. Therefore, the gridded point cloud is a new method to represent the space where the original ground point cloud is located in a regular manner. It may contain more or fewer points than the original point cloud, depending on the size of the grid and the generation strategy.

[0054] Mesh partition space, or "enclosing space", refers to a series of regular areas divided in three-dimensional or two-dimensional space according to the distribution range and characteristics of the original ground point cloud. The mesh partition space covers the spatial range where the original data is located. For example, in a three-dimensional mesh partition space, each grid unit is usually a cube, tetrahedron or other three-dimensional shape, which are closely arranged to cover the entire three-dimensional space. In a two-dimensional mesh partition space, the grid units are simplified to two-dimensional shapes, such as squares, rectangles or triangles, which are tiled on a two-dimensional plane. Whether it is a three-dimensional or two-dimensional mesh partition space, its division method, grid size, shape and arrangement method can be adjusted and optimized according to the specific application scenario and data characteristics.

[0055] Optionally, the vehicle-mounted terminal calculates the minimum and maximum coordinate values ​​of the original ground point cloud in the X, Y, and Z directions based on the spatial distribution of the original ground point cloud to determine the three-dimensional grid division space covering the original ground point cloud, divides the three-dimensional grid division space into multiple uniform three-dimensional grid units, generates new pixel points in each three-dimensional grid unit, and determines the set of new pixel points in each three-dimensional grid unit as a gridded point cloud.

[0056] Alternatively, the vehicle-mounted terminal calculates the minimum and maximum coordinate values ​​of the original ground point cloud on a two-dimensional plane (such as the XY plane) to determine the boundary of the two-dimensional grid division space, divides the three-dimensional grid division space into multiple uniform two-dimensional grid units, generates new pixel points in each two-dimensional grid unit, and projects the new pixel points generated in the two-dimensional grid unit into the three-dimensional space to obtain a three-dimensional pixel point cloud, which is determined as a gridded point cloud.

[0057] 2. Select four pixel points from the gridded point cloud in sequence to form a rectangle, and divide the formed rectangle into triangular grids based on a diagonal line of the formed rectangle to obtain multiple triangular grids; the four grids corresponding to the four pixel points selected each time are four grids on two adjacent columns in two adjacent rows of grids in the multiple grids.

[0058] Optionally, the vehicle-mounted terminal selects pixel points in four grids in two adjacent rows of grids and located in two adjacent columns of the two adjacent rows of grids from the gridded point cloud in turn to form a rectangle. At a diagonal line connecting the rectangle, the object line divides the rectangle into two triangular grids, and traverses other pixel points in the gridded point cloud in turn to obtain multiple triangular grids.

[0059] For example, Figure 3 is a schematic diagram of a triangular mesh in one embodiment, such as Figure 3 As shown, the pixel point P_i, pixel point P_i+1, pixel point P_i+1+Nx and pixel point P_i, pixel point P_i and pixel point P_i+1+Nx, pixel point P_i+Nx in the gridded point cloud are connected in a clockwise direction to form a triangular mesh M. Figure 4 It is a schematic diagram of a triangular mesh obtained after meshing the original ground point cloud in one embodiment.

[0060] Through this embodiment, the ground is densified using triangular meshes, which is applicable to both horizontal ground and ground with slopes and undulations. The triangular meshes are uniformly interpolated, with high density and good smoothness.

[0061] In an exemplary embodiment, generating a gridded point cloud based on an original ground point cloud includes:

[0062] 1. Project the original ground point cloud onto a horizontal plane and determine a grid division space on the horizontal plane; the grid division space is the minimum rectangular range covering the projection points of the original ground point cloud on the horizontal plane.

[0063] The size of the minimum rectangle range in the horizontal and vertical directions is determined according to the coordinate range of the projection points. Specifically, the left boundary of the rectangle is the smallest x-coordinate value of all projection points, and the right boundary is the largest x-coordinate value of all projection points; similarly, the bottom boundary of the rectangle is the smallest y-coordinate value of all projection points, and the top boundary is the largest y-coordinate value of all projection points.

[0064] Optionally, the vehicle-mounted terminal projects each point in the original ground point cloud onto a two-dimensional horizontal plane, and determines a minimum rectangular range covering the projection points of the original ground point cloud on the horizontal plane based on the minimum x-coordinate value, maximum x-coordinate value, minimum y-coordinate value and maximum y-coordinate value of the projection points in the x-direction, and determines the minimum rectangular range as the grid division space.

[0065] Second, the grid partitioning space is divided into multiple grids, and corresponding two-dimensional plane points are generated in each grid to obtain a two-dimensional plane point set.

[0066] Among them, the two-dimensional plane point generated in each grid in the embodiment of the present application is essentially a single position mark for constructing a triangular grid, that is, only one representative point (pixel point) is set in each grid, and this representative point is specially set for the generation of the triangular grid. The process does not involve consideration of the color attribute of the representative point, so this representative point does not carry pixel value information. The two-dimensional plane point is usually located at the center of the grid, but it can also be a corner point of the grid or a position calculated according to a specific algorithm. The embodiment of the present application takes the two-dimensional plane point located at a corner point of the grid as an example.

[0067] Optionally, the vehicle terminal divides the grid partition space into multiple grids according to a fixed step size, generates a new two-dimensional plane point in each grid, and determines the set of two-dimensional plane points in each grid as a two-dimensional plane point set.

[0068] For example, the original ground point cloud is recorded as S_ground, and the original ground point cloud S_ground is projected onto the XOY plane. According to the minimum x coordinate value on the XOY plane (recorded as X min ), the maximum x-coordinate value (denoted as X max ), the minimum y coordinate value (denoted as Y min ) and the maximum y-coordinate value (denoted as Y max ), determine a minimum rectangular range covering the projection points of the original ground point cloud on the horizontal plane, divide the minimum rectangular range into multiple grids according to the step size a, and calculate the coordinates of the two-dimensional plane points generated in each grid according to the following formula:

[0069] X i =min(X max , X min +a*i)

[0070] Y j =min(Y max , Y min +a*j)

[0071] Where i represents the i-th grid in the X direction; j represents the j-th grid in the Y direction; i represents the x-coordinate value of the i-th two-dimensional plane point; Y j Represents the y coordinate value of the jth two-dimensional plane point. The two-dimensional plane point set is denoted as P_dense. The two-dimensional plane points in the two-dimensional plane point set P_dense are arranged evenly, and each row has N x points, each column has N y points.

[0072] 3. Convert each two-dimensional plane point in the two-dimensional plane point set into a corresponding three-dimensional pixel point to obtain a gridded point cloud.

[0073] Optionally, for each two-dimensional plane point, the vehicle-mounted terminal determines the three-dimensional height information corresponding to each two-dimensional plane point, combines the x-coordinate and y-coordinate of the two-dimensional plane point with its corresponding height information, and generates a three-dimensional pixel point. Each three-dimensional pixel point includes coordinate values ​​in the three directions of x, y, and z. The three-dimensional pixel points obtained by converting all two-dimensional plane points are integrated into a gridded point cloud.

[0074] In some embodiments, the three-dimensional height information corresponding to each two-dimensional plane point can be determined by a height map (DEM), an interpolation method (such as bilinear interpolation, inverse distance weighted interpolation, etc.), or the like.

[0075] Through this embodiment, when generating a gridded point cloud, a method is adopted to determine the two-dimensional grid division space based on the projection of the original ground point cloud on the horizontal plane, and convert the two-dimensional plane points into three-dimensional pixel points. Compared with the method of directly dividing the three-dimensional grid in the three-dimensional space and generating three-dimensional plane points, the computational complexity of two-dimensional grid division and processing is much lower than that of three-dimensional grid. Meshing and point cloud projection on a two-dimensional plane can greatly reduce the amount of calculation.

[0076] In an exemplary embodiment, converting each two-dimensional plane point in the two-dimensional plane point set into a corresponding three-dimensional pixel point includes:

[0077] Take each two-dimensional plane point in the two-dimensional plane point set as the current two-dimensional plane point and perform the following operations to obtain the three-dimensional pixel point corresponding to each two-dimensional plane point:

[0078] Determine the projection position of the current two-dimensional plane point in the original ground point cloud, and search for a neighboring ground point set of the projection position in the original ground point cloud in the horizontal direction; the neighboring ground point set includes at least one ground point in the original ground point cloud; based on the height value of each ground point in the neighboring ground point set in the vertical direction, determine the target height value corresponding to the current two-dimensional plane point, and based on the target height value, convert the current two-dimensional plane point into a three-dimensional pixel point corresponding to the current two-dimensional plane point.

[0079] The projection position refers to the specific three-dimensional coordinate point on the surface of the three-dimensional space where the original ground point cloud is located, which is mapped to the current two-dimensional plane point by some method (such as straight line projection or approximate projection based on terrain features). The projection position not only includes the x and y coordinate components of the two-dimensional plane point (i.e. the position in the horizontal direction), but also determines its specific position in the vertical direction (z axis) through the mapping process, thus forming a complete three-dimensional coordinate.

[0080] The neighboring ground point set refers to a set of ground points in the original ground point cloud that are horizontally close to the projection position. The ground points in the neighboring ground point set are usually the points in the original ground point cloud that are closest to the projection position or within a certain search radius.

[0081] The target height value refers to the three-dimensional height value corresponding to the current two-dimensional plane point calculated based on the height value of the ground point in the vertical direction of the neighboring ground point set through an algorithm (such as average, median, minimum height, maximum height or more complex terrain interpolation algorithm).

[0082] Optionally, the vehicle-mounted terminal projects the current two-dimensional plane point onto the surface of the three-dimensional space where the original ground point cloud is located in some way (such as straight line projection or approximate projection based on terrain features), determines the projection position corresponding to the current two-dimensional plane point, records the three-dimensional coordinates of the projection position, searches for a set of neighboring ground points of the projection position in the horizontal direction in the original ground point cloud based on the three-dimensional coordinates of the projection position, calculates a target height value based on the vertical height value of each ground point in the neighboring ground point set through some algorithm (such as average, median, minimum height, maximum height or more complex terrain interpolation algorithm), assigns the height value to the current two-dimensional plane point, combines the x-coordinate and y-coordinate of the current two-dimensional plane point with its corresponding height information, and obtains a three-dimensional pixel point corresponding to the current two-dimensional plane point, the three-dimensional pixel point including coordinate values ​​in the x, y and z directions.

[0083] In some embodiments, in view of the fact that the two-dimensional plane points in this application are mainly used as unique positioning identifiers for constructing triangular meshes, and the triangular mesh is intended to accurately depict the terrain features of a specific area on the ground. Therefore, when determining the projection position of the current two-dimensional plane point in the original ground point cloud, a strategic method is adopted: first, in the original ground point cloud data, accurately screen out all reference point sets whose X and Y coordinates match the two-dimensional plane point; then, select the pixel point with the highest Z coordinate value or the lowest Z coordinate value from the reference point set, and directly use the pixel point as the projection position of the current two-dimensional plane point in the original ground point cloud. This process ensures that each two-dimensional plane point can accurately reflect the highest or lowest point of the terrain in its local area when converted to a three-dimensional pixel point, thereby enhancing the terrain expression accuracy and practicality of the triangular mesh.

[0084] In some embodiments, the average value of the height of each ground point in the neighboring ground point set in the vertical direction may be determined as the target height value corresponding to the current two-dimensional plane point.

[0085] For example, the two-dimensional plane point set is recorded as P_dense, and each two-dimensional plane point in the two-dimensional plane point set P_dense is traversed, and the current two-dimensional plane point is recorded as P i , the current two-dimensional plane point P i Project it into the original ground point cloud to determine the current two-dimensional plane point P iAt the projection position in the original ground point cloud, search for the neighboring ground point set N of the projection position in the original ground point cloud according to the two dimensions of the XOY plane, and use the following formula to calculate the average value of the vertical height value of each ground point in the neighboring ground point set N:

[0086]

[0087] Among them, the weight d i represents the distance from the neighbor point to the point, σ>0; z i represents the height value of the i-th ground point; n represents the number of ground points in the neighboring ground point set N.

[0088] The calculated Z i Assign to the current two-dimensional plane point P i The corresponding target height value, the current two-dimensional plane point P i The two-dimensional coordinates of the corresponding target height are combined to obtain the current two-dimensional plane point P i The corresponding three-dimensional pixel point.

[0089] Through this embodiment, by searching the neighboring ground point set of the projection position in the horizontal direction, the local terrain information related to the projection point can be efficiently obtained, avoiding the global search in the entire original ground point cloud, thereby improving the processing speed, and determining the target height value based on the height value of each ground point in the neighboring ground point set, making the calculation process more centralized and efficient, reducing unnecessary calculation amount, and more accurately reflecting the changes in local terrain.

[0090] In an exemplary embodiment, the original ground point cloud is densified to obtain a three-dimensional space point set, including:

[0091] 1. Based on the intrinsic parameters of the specified camera and the extrinsic parameters between the specified camera and the lidar, the original ground point cloud is projected onto a blank image to obtain a color projection image, and the color projection image is masked to obtain a ground mask image; the original ground point cloud is collected by the lidar, and the blank image is an image that matches the specified camera.

[0092] The designated camera refers to a virtually constructed camera model in the lane line marking scene, which is specifically used to facilitate the accurate conversion between image data and lidar point cloud data.

[0093] The intrinsic parameters of a given camera are a set of parameters that describe the internal optical and geometric properties of a given camera. Intrinsic parameters are crucial for projecting points in the 3D world onto the 2D imaging plane of the camera. Intrinsic parameters usually include focal length, optical center (i.e., image center), distortion coefficients (such as radial distortion and tangential distortion), etc. Through the intrinsic parameters of the camera, the correspondence between 3D world points and pixel points in the camera image can be established.

[0094] The external parameters between the LiDAR and a specified camera describe the relative position and attitude between the LiDAR coordinate system and the camera coordinate system. The external parameters include the rotation matrix and the translation vector, which together define how to transform points in the LiDAR coordinate system to the camera coordinate system. This transformation is crucial for projecting the 3D point cloud data collected by the LiDAR onto the 2D image of the camera.

[0095] A blank image is an image that matches a specified camera but initially has no content (or only a background color, such as black or white). The size, resolution, and color space of the blank image should match the camera's output image so that the projected color information can be correctly placed on it later.

[0096] The color projection image is obtained by projecting the points in the original ground point cloud (collected by the lidar) onto a blank image according to the intrinsic parameters of the camera and the extrinsic parameters between the lidar and the camera, and coloring each pixel according to the color information of the original ground point cloud.

[0097] Masking is an image processing technique used to select or exclude specific areas from an image. On a color projection map, a ground mask image can be generated by identifying the ground area and applying a mask.

[0098] The ground mask image refers to an image obtained by masking the color projection image, retaining only the area related to the ground, and masking or ignoring other non-ground areas (such as buildings, vegetation, sky, etc.) in some way (such as setting to a specific color, transparency or completely removing them). For example, Figure 5 is a schematic diagram of a ground mask image in one embodiment, such as Figure 5 As shown, the ground mask image only contains information of the ground area, and the non-ground area is excluded or set to a specific color (such as black).

[0099] Optionally, the vehicle terminal determines the internal parameter K of the specified camera and the external parameter T between the specified camera and the laser radar, and uses the external parameter T (rotation matrix and translation vector) between the camera and the laser radar to transform each point in the original ground point cloud from the laser radar coordinate system to the camera coordinate system; uses the camera's internal parameter K (focal length, optical center, etc.) and the camera's imaging model (such as a pinhole camera model) to calculate the projection coordinates (i.e., two-dimensional pixel coordinates) of each point after transformation on the camera image plane, and uses the color information of each pixel point in the original ground point cloud to assign a color to each point projected onto the image plane, and renders the projection point and the corresponding color onto the blank image I to generate a color projection map. The vehicle terminal uses image segmentation, classification or other techniques (such as threshold segmentation based on height information, machine learning algorithms, etc.) to identify the ground area in the color projection map, and generates a mask image of the same size as the color projection map based on the identified ground area. In the mask image, the ground area is marked as 1 (or other values ​​indicating "retain"), and the non-ground area is marked as 0 (or other values ​​indicating "ignore"). The mask image is applied to the color projection image to retain only the information of the ground area, and the ground mask image I is obtained. m .

[0100] Second, the ground mask image is expanded, pixel gaps are filled, eroded, and boundaries are reduced to obtain a dense mask image.

[0101] Among them, dilation is a basic operation in morphological operations, which is used to expand the ground area in the ground mask image. Dilation is achieved by finding and adding additional pixels in the neighborhood of each pixel in the ground area. The dilation operation can increase the area of ​​the ground area in the ground mask image, that is, through dilation, the pixel points of the ground area in the original ground point cloud can be densified, so as to obtain more ground points, providing basic data for the subsequent generation of dense ground point cloud.

[0102] Filling pixel gaps is a process specifically designed to deal with small holes or missing parts inside the ground region of an image. While this is similar to the dilation operation in some ways (both increase the area of ​​the ground region), filling gaps is usually more precise and aims to only repair those parts that are indeed part of the ground region but were not correctly identified for some reason (such as noise, occlusion, etc.). Filling pixel gaps can be achieved in a variety of ways, including but not limited to morphological closing operations (dilation followed by erosion) or using specialized image inpainting algorithms.

[0103] Erosion is the inverse operation of dilation and is used to reduce the ground area in the ground mask image. Unlike dilation, which adds pixels, erosion is achieved by deleting the boundaries of pixels in the ground area. Specifically, the neighborhood of the pixel in each ground area is checked and the pixel is decided whether to be retained based on the shape and size of the structural element. The erosion operation can reduce the area of ​​the ground area in the ground mask image, which helps to remove small isolated points or refine the boundaries of the ground area.

[0104] Shrinking the border usually refers to reducing the border range of the ground area in some way. For example, it can be directly achieved through erosion, because erosion reduces the size of the object and shrinks its border inward. Shrinking the border can also involve more complex operations, such as border smoothing based on edge detection or intelligent border adjustment based on image content.

[0105] A dense mask image refers to a ground mask image obtained after dilation, gap filling, erosion, and boundary reduction, in which the pixels of the ground area in the ground mask image are more continuous and dense, and may have smoother boundaries. The dense mask image reduces noise, holes, and unnecessary details while retaining key terrain information.

[0106] Optionally, the vehicle-mounted terminal processes the ground mask image I m After dilation, fill the ground mask image I m The pixel gaps in the image are cleared, and then the edges are eroded and reduced to obtain a dense mask image.

[0107] 3. Determine the mapping point of each pixel in the dense mask image in the color projection image, assign the pixel color of the mapping point corresponding to each pixel in the dense mask image to the corresponding pixel, and obtain a dense ground pixel set.

[0108] The mapping point refers to the pixel point in the color projection image that matches the position of each pixel point in the dense mask image.

[0109] The dense ground pixel set refers to the pixel set obtained after assigning color information to each pixel of the dense mask image. It can be understood that each pixel in the dense ground pixel set not only contains the mask information, but also contains the pixel color of the mapping point of each pixel in the dense mask image in the color projection image.

[0110] Optionally, the vehicle terminal determines the mapping point of each pixel point in the dense mask image in the color projection image by means of space transformation or image matching, and extracts the pixel color of the mapping point corresponding to each pixel point in the dense mask image, and assigns the pixel color to the pixel point in the dense mask image corresponding to the mapping point, so as to obtain a dense ground pixel set I m -ground.

[0111] In some embodiments, a spatial transformation method may be used to determine a mapping point of each pixel in the dense mask image in the color projection image, specifically including the following steps:

[0112] Using the internal and external parameters of the specified camera, the coordinates of each pixel in the color projection image are converted to the three-dimensional world coordinate system; the three-dimensional points in the three-dimensional world coordinate system are reversely projected back to the two-dimensional coordinate system of the dense mask image, thereby establishing a point-to-point correspondence. According to the established correspondence, each pixel in the color projection image is mapped to the corresponding pixel position in the dense mask image.

[0113] In some embodiments, an image matching method may be used to determine a mapping point of each pixel in the dense mask image in the color projection image, specifically including the following steps:

[0114] Extract feature points from the color projection image and the dense mask image respectively; use a feature matching algorithm (such as FLANN, BFMatcher, etc.) to find the feature point pairs that match each other in the dense mask image and the color projection image; the feature point pairs include the first feature point in the dense mask image and the second feature point in the color projection image; remove the mismatched points through algorithms such as RANSAC (random sampling consensus algorithm) to obtain a sparse feature point correspondence relationship; based on the feature point correspondence relationship, determine the neighbor feature point set of the first non-feature point in the dense mask image; the neighbor feature point set includes multiple first feature points; based on the weighted average value of each first feature point in the neighbor feature point set (the weight can be determined based on the distance or other factors), determine the second non-feature point in the color projection image corresponding to the first non-feature point, and generate a dense non-feature point correspondence relationship based on the correspondence relationship between the first non-feature point and the second non-feature point; according to the feature point correspondence relationship and the non-feature point correspondence relationship, map each pixel in the color projection image to the corresponding pixel position in the dense mask image.

[0115] Fourth, convert each pixel point in the dense ground pixel set into a three-dimensional space point to obtain a three-dimensional space point set.

[0116] Among them, it can be known from the above embodiments that the dense ground pixel set, as a set formed by each pixel in the dense mask image being assigned with detailed color information, substantially includes all ground-related pixels in the image captured by the camera. Since the pixels in the dense ground pixel set are directly derived from the imaging process of the camera, each pixel in the dense ground pixel set can be converted into a three-dimensional space point through the precise mathematical mapping relationship between the pixel coordinate system and the world coordinate system, thereby constructing a three-dimensional space representation of the dense ground pixel set (i.e., a three-dimensional space point set).

[0117] Optionally, the vehicle terminal traverses the dense ground pixel set I m _ground, for the dense ground pixel set I m For each pixel point p in the ground, according to the camera model, a corresponding 3D space point I is generated. P , and obtain a three-dimensional space point set.

[0118] Through this embodiment, by fusing the three-dimensional spatial information of the laser radar and the color image information of the camera, a color projection image containing ground geometry and color information is generated, and the color projection image is masked so that the obtained ground mask image only contains the ground part, removing the interference of non-ground elements, and providing a clear and accurate ground area for the subsequent; further, the ground mask image is expanded, pixel gaps are filled, eroded and the boundary is reduced, which can optimize the quality of the ground mask image, eliminate image defects caused by sensor noise, occlusion and other factors, and make the ground area more continuous and complete. At the same time, the pixel points of the ground area in the original ground point cloud can be densified, so as to obtain more ground points, providing basic data for the subsequent generation of dense ground point cloud; further, the dense mask image is mapped back to the color projection image to obtain a dense ground pixel set containing rich ground details (such as geometric shapes, colors and textures, etc.), and each pixel point in the dense ground pixel set is converted into a three-dimensional space point, so as to obtain a three-dimensional space point set with color information, providing a data basis for the subsequent generation of a colored dense ground point cloud. In the above process of densifying the original ground point cloud to obtain a three-dimensional space point set, it can be seen that compared with the traditional method that requires densification of image data obtained by scanning at different angles, the embodiment of the present application can restore dense ground points with high accuracy through only single-frame image data.

[0119] In an exemplary embodiment, determining that each spatial point in the three-dimensional spatial point set is projected into a triangular mesh among a plurality of triangular meshes to obtain a triangular mesh corresponding to each spatial point includes:

[0120] Take each spatial point in the three-dimensional spatial point set as the current spatial point and perform the following operations to obtain a triangular mesh corresponding to each spatial point: determine a ray with the specified camera as the origin and passing through the current spatial point; and determine the first triangular mesh among multiple triangular meshes that intersects with the ray as the triangular mesh corresponding to the current spatial point.

[0121] Optionally, the vehicle terminal moves from the origin O of the specified camera through the current space point I P , send out a ray OP, and according to the principle of ray intersection, determine the first triangular mesh among the multiple triangular meshes M that intersects with the ray OP, and determine the first triangular mesh as the triangular mesh corresponding to the current spatial point.

[0122] In some embodiments, in the process of determining the first triangular mesh among multiple triangular meshes M that intersects with the ray OP, in order to improve computational efficiency, an AABB tree may be used to accelerate the determination.

[0123] It should be noted that the AABB tree is a data structure used to accelerate collision detection and spatial query in computer graphics, and is used to efficiently handle tasks such as collision detection and ray detection in space.

[0124] Through this embodiment, each spatial point can be accurately associated with the nearest (or first intersecting) triangular mesh by detecting the intersection of the ray and the triangular mesh, which is more accurate than simply assigning points to meshes based on spatial positions; in addition, since only the first triangular mesh intersecting with the ray is considered, noise points and outliers far away from the actual ground or object surface are excluded.

[0125] In an exemplary embodiment, it can be seen from the above embodiments that the dense ground pixel set is a set formed by assigning detailed color information to each pixel point in the dense mask image. Therefore, each spatial point of the three-dimensional spatial point set obtained based on the dense ground pixel set also has a pixel color.

[0126] In some embodiments, the ground point cloud generated by the prior art is not only sparse and lacks necessary details, but also the point cloud data lacks color information, which makes it difficult to distinguish different ground features in complex road environments, limits the intuitiveness and readability of the point cloud data, and reduces the accuracy of lane line marking. Usually, the laser points are directly colored using image color information, but this method has poor coloring effect, and the point cloud is too sparse and not smooth. Therefore, in order to solve the point cloud coloring problem, in an embodiment of the present application, before determining the set of center points of the triangular mesh corresponding to each spatial point as a dense ground point cloud, the above-mentioned dense ground point cloud generation method also includes:

[0127] The pixel color of each spatial point is assigned to the center point of the triangular mesh corresponding to each spatial point.

[0128] Optionally, after determining the triangular mesh corresponding to each spatial point in the three-dimensional spatial point set, the vehicle-mounted terminal assigns the pixel color of each spatial point to the center point of the triangular mesh corresponding to each spatial point, and determines the set of center points of the triangular mesh with color information as a dense ground point cloud.

[0129] For example, passing through the current spatial point I P Take the ray OP as an example, the center point of the first triangle mesh intersecting with the ray OP is recorded as M P , the current spatial point I P The color information is given to the center point M P, after assigning the pixel colors of all spatial points to the center points of the corresponding triangular meshes, the set of center points of the triangular meshes corresponding to each spatial point is determined as a dense ground point cloud, where Figure 6 Schematic diagram of a dense ground point cloud in one embodiment.

[0130] It should be noted that, through the above embodiments, it can be seen that the center point of the triangular mesh mainly carries the position attribute and does not contain color information, that is, it only identifies a point in space and does not involve the expression of color data. Therefore, if the above step of assigning the pixel color of each spatial point to the center point of the triangular mesh corresponding to each spatial point is not performed, but the set of center points of the triangular mesh corresponding to each spatial point is directly determined as a dense ground point cloud, in this case, the obtained dense ground point cloud does not have color information. If the above step of assigning the pixel color of each spatial point to the center point of the triangular mesh corresponding to each spatial point is performed, and then the set of center points of the triangular mesh corresponding to each spatial point is determined as a dense ground point cloud, in this case, the obtained dense ground point cloud is not only dense, but also has color information.

[0131] Through this embodiment, the pixel color of each spatial point is assigned to the center point of the triangular mesh corresponding to each spatial point, and the set of center points of the triangular mesh corresponding to each spatial point is determined as a dense ground point cloud. This not only realizes the comprehensive coloring of the dense ground point cloud with rich pixel colors, but also greatly improves the ability of the dense ground point cloud to restore ground details, making it more vivid and realistic, which helps to efficiently and accurately identify and mark lane lines in complex environments.

[0132] In a detailed embodiment, a method for generating a dense ground point cloud is provided, which specifically includes the following steps:

[0133] 1. Obtain the original ground point cloud.

[0134] 2. Project the original ground point cloud onto a horizontal plane and determine a grid division space on the horizontal plane; the grid division space is the minimum rectangular range covering the projection points of the original ground point cloud on the horizontal plane.

[0135] 3. Divide the grid partitioning space into multiple grids, and generate corresponding two-dimensional plane points in each grid to obtain a two-dimensional plane point set.

[0136] 4. Take each two-dimensional plane point in the two-dimensional plane point set as the current two-dimensional plane point and perform the following operations to obtain the three-dimensional pixel point corresponding to each two-dimensional plane point: determine the projection position of the current two-dimensional plane point in the original ground point cloud, and search the original ground point cloud for the neighboring ground point set of the projection position in the horizontal direction; the neighboring ground point set includes at least one ground point in the original ground point cloud; based on the height value of each ground point in the neighboring ground point set in the vertical direction, determine the target height value corresponding to the current two-dimensional plane point, and based on the target height value, convert the current two-dimensional plane point into a three-dimensional pixel point corresponding to the current two-dimensional plane point.

[0137] 5. All three-dimensional pixel points are determined as gridded point clouds; the gridded point cloud is a new point cloud obtained by determining a grid division space based on the original ground point cloud, dividing the grid division space into multiple grids, and generating pixel points in each of the multiple divided grids.

[0138] 6. Select four pixel points from the gridded point cloud in sequence to form a rectangle, and divide the formed rectangle into triangular meshes based on a diagonal line of the formed rectangle to obtain multiple triangular meshes; the four meshes corresponding to the four pixel points selected each time are four meshes on two adjacent columns in two adjacent rows of meshes in the multiple meshes; each triangular mesh in the multiple triangular meshes is used to represent a local area on the ground.

[0139] 7. Based on the intrinsic parameters of the specified camera and the extrinsic parameters between the specified camera and the lidar, the original ground point cloud is projected onto the blank image to obtain a color projection image, and the color projection image is masked to obtain a ground mask image; the original ground point cloud is collected by the lidar, and the blank image is an image that matches the specified camera.

[0140] 8. Dilate the ground mask image, fill pixel gaps, erode and shrink the boundaries to obtain a dense mask image.

[0141] 9. Determine the mapping point of each pixel in the dense mask image in the color projection image, assign the pixel color of the mapping point corresponding to each pixel in the dense mask image to the corresponding pixel, and obtain a dense ground pixel set.

[0142] 10. Convert each pixel point in the dense ground pixel set into a three-dimensional space point to obtain a three-dimensional space point set.

[0143] 11. Take each spatial point in the three-dimensional spatial point set as the current spatial point and perform the following operations to obtain a triangular mesh corresponding to each spatial point: determine a ray with the specified camera as the origin and passing through the current spatial point; determine the first triangular mesh among multiple triangular meshes that intersects with the ray as the triangular mesh corresponding to the current spatial point.

[0144] 12. Assign the pixel color of each spatial point to the center point of the triangular mesh corresponding to each spatial point.

[0145] 13. The set of center points of the triangular mesh corresponding to each spatial point is determined as a dense ground point cloud; the point cloud density of the dense ground point cloud is greater than the point cloud density of the original ground point cloud.

[0146] Through this embodiment, the collected original ground point cloud is triangulated to obtain multiple triangular meshes, thereby dividing the surface of the original ground point cloud into multiple uniform local areas, providing basic data for obtaining a uniform dense ground point cloud. Further, the original ground point cloud is densified to obtain a three-dimensional space point set. The embodiment of the present application does not directly use the three-dimensional space point set obtained after the densification of the original ground point cloud as the dense ground point cloud to be used in the end, but chooses to project each spatial point in the three-dimensional space point set to a uniform triangular mesh, and determines the set of center points of the triangular mesh corresponding to each spatial point as the dense ground point cloud. This method not only effectively filters out noise points that may exist in the original ground point cloud, significantly improves the signal-to-noise ratio of the original ground point cloud, but also successfully converts the sparse and unevenly distributed point cloud data that may exist in the original ground point cloud into a more uniform and dense representation. More importantly, the entire processing process abandons the dependence on high-quality image data, significantly reduces the computational complexity, and thus obtains a dense ground point cloud with higher accuracy.

[0147] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0148] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware server, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM (Read-Only Memory) / RAM (Random Access Memory), a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0149] According to another aspect of the embodiments of the present application, a dense ground point cloud generation device is also provided, which is used to implement the dense ground point cloud generation method provided in the above-mentioned embodiments, and will not be repeated hereafter. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0150] Figure 7 is a structural block diagram of an optional dense ground point cloud generation device according to an embodiment of the present application, such as Figure 7 As shown, the device may include:

[0151] The laser radar scanning module 701 is used to obtain the original ground point cloud;

[0152] A triangular mesh generation module 702 is used to generate a plurality of triangular meshes based on the original ground point cloud; each of the plurality of triangular meshes is used to represent a local area on the ground;

[0153] A point cloud densification module 703 is used to perform point cloud densification processing on the original ground point cloud to obtain a three-dimensional space point set;

[0154] A denoising module 704 is used to determine that each spatial point in the three-dimensional spatial point set is projected into a triangular mesh in the plurality of triangular meshes to obtain a triangular mesh corresponding to each spatial point;

[0155] The dense point cloud generation module 705 is used to determine the set of center points of the triangular mesh corresponding to each spatial point as a dense ground point cloud; the point cloud density of the dense ground point cloud is greater than the point cloud density of the original ground point cloud.

[0156] In an exemplary embodiment, the triangular mesh generation module 702 is also used to generate a gridded point cloud based on the original ground point cloud; the gridded point cloud is a new point cloud obtained by determining a grid division space based on the original ground point cloud, dividing the grid division space into multiple grids, and generating pixel points in each of the multiple divided grids; four pixel points are selected from the gridded point cloud in turn to form a rectangle, and the formed rectangle is divided into triangular meshes based on a diagonal line of the formed rectangle to obtain the multiple triangular meshes; the four meshes corresponding to the four pixel points selected each time are four meshes on two adjacent columns in two adjacent rows of grids in the multiple grids.

[0157] In an exemplary embodiment, the triangular mesh generation module 702 is also used to project the original ground point cloud onto a horizontal plane and determine the grid division space on the horizontal plane; the grid division space is the minimum rectangular range covering the projection points of the original ground point cloud on the horizontal plane; the grid division space is divided into the multiple grids, and corresponding two-dimensional plane points are generated in each grid to obtain a two-dimensional plane point set; each two-dimensional plane point in the two-dimensional plane point set is converted into a corresponding three-dimensional pixel point to obtain the gridded point cloud.

[0158] In an exemplary embodiment, the triangular mesh generation module 702 is also used to perform the following operations on each two-dimensional plane point in the two-dimensional plane point set as a current two-dimensional plane point to obtain a three-dimensional pixel point corresponding to each two-dimensional plane point: determine the projection position of the current two-dimensional plane point in the original ground point cloud, and search for a neighboring ground point set of the projection position in the original ground point cloud in a horizontal direction; the neighboring ground point set includes at least one ground point in the original ground point cloud; based on the height value of each ground point in the neighboring ground point set in the vertical direction, determine a target height value corresponding to the current two-dimensional plane point, and based on the target height value, convert the current two-dimensional plane point into a three-dimensional pixel point corresponding to the current two-dimensional plane point.

[0159] In an exemplary embodiment, the point cloud densification module 703 is also used to project the original ground point cloud onto a blank image based on the intrinsic parameters of a specified camera and the extrinsic parameters between the specified camera and the laser radar to obtain a color projection image, and mask the color projection image to obtain a ground mask image; the original ground point cloud is collected by the laser radar, and the blank image is an image matched with the specified camera; the ground mask image is expanded, pixel gaps are filled, and boundaries are eroded and reduced to obtain a dense mask image; the mapping point of each pixel point in the dense mask image in the color projection image is determined, and the pixel color of the mapping point corresponding to each pixel point in the dense mask image is assigned to the corresponding pixel point to obtain a dense ground pixel set; each pixel point in the dense ground pixel set is converted into a three-dimensional space point to obtain the three-dimensional space point set.

[0160] In an exemplary embodiment, the denoising module 704 is also used to perform the following operations on each spatial point in the three-dimensional spatial point set as a current spatial point to obtain a triangular mesh corresponding to each spatial point: determine a ray with a specified camera as its origin and passing through the current spatial point; and determine the first triangular mesh among the multiple triangular meshes that intersects with the ray as the triangular mesh corresponding to the current spatial point.

[0161] In an exemplary embodiment, each spatial point in the three-dimensional spatial point set has a pixel color, and each pixel point in the dense ground point cloud has a pixel color; the dense point cloud generation module 705 is also used to assign the pixel color of each spatial point to the center point of the triangular mesh corresponding to each spatial point.

[0162] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules as part of the device can be run in Figure 1 In the hardware environment shown, it can be implemented by software or by hardware, wherein the hardware environment includes a network environment.

[0163] According to another aspect of the embodiments of the present application, a storage medium is further provided. Optionally, in this embodiment, the storage medium can be used to execute the program code of any of the above-mentioned dense ground point cloud generation methods in the embodiments of the present application.

[0164] Optionally, in this embodiment, the storage medium may be located on at least one network device among a plurality of network devices in the network shown in the above embodiment.

[0165] Optionally, in this embodiment, the storage medium is configured to store program codes for executing any of the above-mentioned dense ground point cloud generation methods.

[0166] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, which will not be described in detail in this embodiment.

[0167] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a ROM, a RAM, a mobile hard disk, a magnetic disk, or an optical disk.

[0168] According to another aspect of an embodiment of the present application, an electronic device for implementing the above-mentioned dense ground point cloud generation method is also provided. The electronic device may be a smart device, a server, a terminal, or a combination thereof.

[0169] Figure 8 is a structural block diagram of an optional electronic device according to an embodiment of the present application, such as Figure 8 As shown, it includes a processor 802, a communication interface 804, a memory 806 and a communication bus 808, wherein the processor 802, the communication interface 804 and the memory 806 communicate with each other through the communication bus 808, wherein,

[0170] Memory 806, used for storing computer programs;

[0171] The processor 802 is used to implement the program code of any of the above-mentioned dense ground point cloud generation methods when executing the computer program stored in the memory 806.

[0172] Optionally, the communication bus may be a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The communication interface is used for communication between the above electronic device and other devices.

[0173] The memory may include RAM, or may include non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0174] As an example, the memory 806 may include, but is not limited to, each module in the dense ground point cloud generating device. In addition, it may also include, but is not limited to, other module units in the dense ground point cloud generating device, which will not be described in detail in this example.

[0175] The above-mentioned processor can be a general-purpose processor, which can include but not be limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processing), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0176] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0177] It can be understood by those skilled in the art that Figure 8 The structure shown is for illustration only. The device for implementing the above-mentioned dense ground point cloud generation method may be a terminal device, which may be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, and other terminal devices. Figure 8 The structure of the electronic device is not limited. Figure 8 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 8 Different configurations shown.

[0178] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0179] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0180] The embodiments of the present application also provide a computer program, which includes computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps in any one of the above method embodiments.

[0181] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which can include: a flash drive, ROM, RAM, a magnetic disk or an optical disk, etc.

[0182] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0183] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0184] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0185] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0186] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution provided in this embodiment.

[0187] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist physically separately, or at least two units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0188] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for generating dense ground point cloud, characterized in that: include: Get the original ground point cloud; Based on the original ground point cloud, generate multiple triangular meshes; Each of the plurality of triangular meshes is used to represent a local area on the ground; Performing point cloud densification processing on the original ground point cloud to obtain a three-dimensional space point set; Determine that each spatial point in the three-dimensional spatial point set is projected into a triangular mesh in the plurality of triangular meshes to obtain a triangular mesh corresponding to each spatial point; A set of center points of the triangular mesh corresponding to each of the spatial points is determined as a dense ground point cloud; the point cloud density of the dense ground point cloud is greater than the point cloud density of the original ground point cloud.

2. The method according to claim 1, characterized in that The step of generating a plurality of triangular meshes based on the original ground point cloud comprises: Based on the original ground point cloud, a gridded point cloud is generated; the gridded point cloud is a new point cloud obtained after determining a grid division space based on the original ground point cloud, dividing the grid division space into a plurality of grids, and generating pixel points in each of the plurality of divided grids; Four pixel points are selected from the gridded point cloud in sequence to form a rectangle, and the formed rectangle is divided into triangular grids based on a diagonal line of the formed rectangle to obtain the multiple triangular grids; the four grids corresponding to the four pixel points selected each time are four grids on two adjacent columns in two adjacent rows of grids in the multiple grids.

3. The method according to claim 2, characterized in that The step of generating a gridded point cloud based on the original ground point cloud comprises: Projecting the original ground point cloud onto a horizontal plane, and determining the grid division space on the horizontal plane; the grid division space is a minimum rectangular range covering the projection points of the original ground point cloud on the horizontal plane; Dividing the grid partition space into the plurality of grids, and generating corresponding two-dimensional plane points in each of the grids to obtain a two-dimensional plane point set; Each two-dimensional plane point in the two-dimensional plane point set is converted into a corresponding three-dimensional pixel point to obtain the gridded point cloud.

4. The method according to claim 3, characterized in that The step of converting each two-dimensional plane point in the two-dimensional plane point set into a corresponding three-dimensional pixel point comprises: Each two-dimensional plane point in the two-dimensional plane point set is used as the current two-dimensional plane point to perform the following operations to obtain a three-dimensional pixel point corresponding to each two-dimensional plane point: Determine the projection position of the current two-dimensional plane point in the original ground point cloud, and search the original ground point cloud for a set of neighboring ground points of the projection position in a horizontal direction; the set of neighboring ground points includes at least one ground point in the original ground point cloud; Based on the height value of each ground point in the set of neighboring ground points in the vertical direction, a target height value corresponding to the current two-dimensional plane point is determined, and based on the target height value, the current two-dimensional plane point is converted into a three-dimensional pixel point corresponding to the current two-dimensional plane point.

5. The method according to claim 1, characterized in that The densification of the original ground point cloud to obtain a three-dimensional space point set includes: Based on the internal parameters of the designated camera and the external parameters between the designated camera and the laser radar, the original ground point cloud is projected onto a blank image to obtain a color projection image, and the color projection image is subjected to mask processing to obtain a ground mask image; the original ground point cloud is collected by the laser radar, and the blank image is an image matching the designated camera; The ground mask image is expanded, pixel gaps are filled, and boundaries are eroded and reduced to obtain a dense mask image; Determine a mapping point of each pixel point in the dense mask image in the color projection image, assign a pixel color of the mapping point corresponding to each pixel point in the dense mask image to the corresponding pixel point, and obtain a dense ground pixel set; Each pixel point in the dense ground pixel set is converted into a three-dimensional space point to obtain the three-dimensional space point set.

6. The method according to claim 1, characterized in that The step of determining that each spatial point in the three-dimensional spatial point set is projected into a triangular mesh in the plurality of triangular meshes to obtain a triangular mesh corresponding to each spatial point includes: Each spatial point in the three-dimensional spatial point set is used as the current spatial point and the following operations are performed to obtain a triangular mesh corresponding to each spatial point: a ray with a specified camera as its origin and passing through the current spatial point is determined; and the first triangular mesh among the multiple triangular meshes that intersects with the ray is determined as the triangular mesh corresponding to the current spatial point.

7. The method according to any one of claims 1 to 6, characterized in that Each spatial point in the three-dimensional spatial point set has a pixel color, and each pixel point in the dense ground point cloud has a pixel color; Before determining the set of center points of the triangular meshes corresponding to each of the spatial points as a dense ground point cloud, the method further includes: The pixel color of each spatial point is assigned to the center point of the triangular mesh corresponding to each spatial point.

8. A dense ground point cloud generation device, characterized in that: include: LiDAR scanning module, used to obtain the original ground point cloud; A triangular mesh generation module, used to generate multiple triangular meshes based on the original ground point cloud; Each of the plurality of triangular meshes is used to represent a local area on the ground; A point cloud densification module, used for performing point cloud densification processing on the original ground point cloud to obtain a three-dimensional space point set; A denoising module, used for determining that each spatial point in the three-dimensional spatial point set is projected into a triangular mesh in the plurality of triangular meshes, to obtain a triangular mesh corresponding to each spatial point; A dense point cloud generation module is used to determine the set of center points of the triangular mesh corresponding to each spatial point as a dense ground point cloud; the point cloud density of the dense ground point cloud is greater than the point cloud density of the original ground point cloud.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of any one of the methods of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images