A method for extracting and vectorizing road boundaries by fusing vehicle-mounted images and point clouds

Abstract

The application provides a road boundary extraction and vectorization method fusing vehicle-mounted images and point clouds. Based on the point cloud super voxel segmentation result, the application adopts a shrink distance clustering algorithm to extract and denoise the road boundary point cloud; based on the vehicle-mounted image, the application adopts a semantic segmentation model and a region growing algorithm to extract the road boundary pixels, and further converts the boundary pixels to the world coordinate system; a Snake model is constructed to fuse the extraction results of the point cloud and the vehicle-mounted image, an improved internal force matrix is used to solve the model to obtain a road boundary point set; the point set is vectorized and multi-segment line simplified to obtain a three-dimensional vector line road boundary. The road boundary extraction method provided by the application has strong robustness to three-dimensional road boundaries with different shapes and point densities, and can still obtain complete and position-accurate vector road boundaries in scenes where the road boundary is seriously worn or blocked, and can be applied to high-precision map production for automatic driving and update of basic geographic information data.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of computer vision and laser scanning data processing, and specifically relates to a method for extracting and vectorizing road boundaries by fusing vehicle images and point clouds. Background Technology

[0002] Road boundaries are one of the most common and important urban infrastructure features for dividing different road functional zones such as carriageways and sidewalks. They are a crucial component of intelligent transportation systems, autonomous driving systems, and urban planning, playing a vital role in ensuring residents' safe travel and the efficient operation of socio-economic activities. Furthermore, as geographical features constraining the distribution of roads and their ancillary facilities, the geometric, semantic, and topological information of three-dimensional road boundaries is essential for many geospatial applications. The rapid and accurate extraction of road boundaries to facilitate the construction of high-precision maps is currently a research hotspot in "digital twins."

[0003] The continuous advancement of surveying and mapping technologies and sensor devices has provided a wealth of data sources for road boundary extraction research using multi-sensor data. Among these, vehicle-mounted laser scanning technology offers advantages such as high measurement speed, high precision, and accurate recognition, capable of generating high-density 3D point clouds with millimeter-level accuracy. Meanwhile, high-resolution panoramic cameras provide rich texture and color information of road scenes with ultra-wide-angle views, simultaneously acquiring high-precision 3D spatial and attribute information of roads and roadside features, and are widely used in road scene understanding, high-precision 3D reconstruction, and other fields. However, most current research on accurate road boundary extraction is based on only a single data source (imagery or laser point clouds), thus having certain limitations. Furthermore, most current related research only focuses on road area identification or boundary point classification, without further obtaining 3D vector road boundaries, while accurate road boundary models are essential for the information management of urban infrastructure and the production of high-precision maps for autonomous driving. Summary of the Invention

[0004] This invention addresses the shortcomings of existing technologies for vehicle-mounted laser scanning point clouds and vehicle-mounted images by providing a road boundary extraction technology that fuses point clouds and panoramic images.

[0005] This invention proposes a method for extracting road boundaries by fusing point clouds and panoramic images, comprising the following steps:

[0006] Step 1: The mobile laser scanning system uses a 3D laser scanner to collect road 3D point clouds at multiple times in real time, uses a GNSS positioning module to collect vehicle positions at multiple times in real time, uses a panoramic camera to collect vehicle panoramic images at multiple times in real time, and uses an inertial measurement unit to collect roll angle, yaw angle, and pitch angle at multiple times when the panoramic camera is shooting.

[0007] Step 2: Perform super-voxel segmentation on the 3D point cloud of the road at each time step, extract the 3D point cloud of the road boundary at each time step using the Euclidean clustering method based on shrinkage distance, and denoise the 3D point cloud of the road boundary at each time step to obtain the denoised 3D point cloud of the road boundary at each time step.

[0008] Step 3: Semantically segment the vehicle panoramic image at each time step using a hierarchical multi-scale attention mechanism neural network model to obtain the road surface area image at each time step. Further, use the region growing algorithm to extract multiple road boundary pixels in the road surface area image at each time step. Register the vehicle panoramic image at each time step with the denoised road boundary 3D point cloud to obtain the relationship model between pixels and point cloud. Further, transform multiple road boundary pixels in the road surface area image at each time step to the world coordinate system to obtain the coordinate transformed road boundary 3D point cloud at each time step.

[0009] Step 4: Stitch together the denoised 3D point clouds of the road boundary from multiple time points to obtain a stitched 3D point cloud of the road boundary. Calculate the bounding box of the stitched 3D point cloud of the road boundary. Construct an XOY gridded pixel plane based on the bounding box of the stitched 3D point cloud of the road boundary. Project the stitched 3D point cloud of the road boundary onto the XOY gridded pixel plane to obtain a projected road boundary image. Binarize the projected road boundary image to obtain a road boundary feature map. Alternatively, stitch together the coordinate-transformed 3D point clouds of the road boundary from multiple time points by setting a region of interest to obtain a coordinate-transformed 3D point cloud of the road boundary. Calculate the bounding box of the coordinate-transformed 3D point cloud of the road boundary. Construct an XOY gridded pixel plane based on the bounding box of the coordinate-transformed 3D point cloud of the road boundary. Project the stitched coordinate-transformed 3D point cloud of the road boundary onto the XOY gridded pixel plane to obtain a projected road boundary image. Binarize the projected road boundary image to obtain an initial contour image. Construct a Snake model and solve the Snake model using an improved internal force matrix to obtain a set of road boundary points.

[0010] Step 5: Vectorize the road boundary point set to obtain the road boundary polyline, and further simplify the road boundary polyline using the Douglas poker method to obtain the vector line road boundary.

[0011] Preferably, step 3 involves registering the vehicle-mounted panoramic image at each moment with the denoised 3D point cloud of the road boundary to obtain the correspondence between the pixel point clouds. The specific process is as follows:

[0012] Step 3.1: Map the vehicle-mounted 3D laser point cloud coordinate system corresponding to the denoised road boundary 3D point cloud to the panoramic image coordinate system using the roll angle, yaw angle, and pitch angle of the panoramic camera at each moment. For the vehicle-mounted panoramic image at the i-th moment.

[0013] Calculate the rotation matrix R at time i. i The details are as follows:

[0014]

[0015] In the formula, γ i Let β be the roll angle during the panoramic image capture at time i. i Let α be the yaw angle during the panoramic image capture at time i. i Let R be the pitch angle during the panoramic image capture at time i. i Let be the rotation matrix at time i;

[0016] For the panoramic image at time i, the t-th point in the denoised 3D point cloud of the road boundary at time i in the world coordinate system Transformed into the t-th road boundary pixel of the panoramic image at time i in the camera coordinate system The specific formula is as follows:

[0017]

[0018]

[0019]

[0020] In the formula, Let x be the x-axis coordinate of the t-th point in the denoised 3D point cloud of the road boundary at time i in the world coordinate system; Let y be the y-coordinate of the t-th point in the denoised 3D point cloud of the road boundary at time i in the world coordinate system; Let t be the z-axis coordinate of the denoised 3D point cloud of the road boundary at time i in the world coordinate system; Let x be the x-coordinate of the t-th road boundary pixel in the panoramic image at time i in the camera coordinate system; Let y be the y-coordinate of the t-th road boundary pixel in the panoramic image at time i in the camera coordinate system; Let T be the z-axis coordinate of the t-th road boundary pixel in the panoramic image at time i in the camera coordinate system; i Let be the translation vector between the camera coordinate system and the world coordinate system for the panoramic image at time i.

[0021] Step 3.2: Based on the spherical imaging model, the t-th road boundary pixel of the panoramic image at time i is... When mapping to the pixel spherical coordinate system of the panoramic image, the transformation formula is as follows:

[0022]

[0023] In the formula, Let x be the x-coordinate of the t-th road boundary pixel in the panoramic image at time i in the camera coordinate system; Let y be the y-coordinate of the t-th road boundary pixel in the panoramic image at time i in the camera coordinate system; Let z be the z-axis coordinate of the t-th road boundary pixel in the panoramic image at time i in the camera coordinate system. Let be the angle between the t-th road boundary pixel of the panoramic image at time i and the positive z-axis. Let be the angle between the t-th road boundary pixel of the panoramic image at time i and the positive y-axis.

[0024] Step 3.3: The t-th road boundary pixel of the panoramic image at time i. The panoramic image is transformed from its spherical coordinate system to its pixel coordinate system using cylindrical projection transformation. The transformation formula is as follows:

[0025]

[0026] In the formula, Let be the t-th road boundary pixel of the panoramic image at time i. The x-coordinate in the pixel coordinate system of the panoramic image Let be the t-th road boundary pixel of the panoramic image at time i. In the pixel coordinate system of the panoramic image, W is the length of the panoramic image and H is the width.

[0027] Step 3 further transforms multiple road boundary pixels in the road surface area image at each time moment to the world coordinate system, obtaining the 3D point cloud of the road boundary after coordinate transformation at each time moment, as follows:

[0028] The vehicle-mounted panoramic image at each moment is registered with the denoised 3D point cloud of the road boundary to obtain a point in the world coordinate system. The corresponding points obtained after transforming to the pixel coordinate system of the panoramic image By traversing multiple road boundary pixels in the road surface area image at each time step and finding their corresponding points in the world coordinate system, the 3D point cloud of the road boundary after coordinate transformation at each time step can be obtained.

[0029] Preferably, step 4 involves projecting the stitched 3D point cloud of the road boundary onto the XOY gridded pixel plane to obtain the projected road boundary image, as detailed below:

[0030]

[0031] In the formula, x min It is the minimum value of the bounding box of the stitched road boundary 3D point cloud on the x-axis, y-axis. min It is the minimum value of the bounding box of the stitched road boundary 3D point cloud on the y-axis, resolution represents the length of each grid cell, and row n This represents the nth point (x) in the stitched 3D point cloud of the road boundary. n ,y n ,z n The row number, col, of the XOY gridded pixel plane after projection. n Representing point (x) n ,y n ,z n The column number of the pixel in the XOY gridded pixel plane after projection;

[0032] Step 4 involves binarizing the projected road boundary image to obtain the road boundary feature map, as detailed below:

[0033] Within the XOY gridded pixel plane, for each grid cell, it is determined whether the projected boundary point cloud falls into it. If it does, the pixel value of this grid cell is set to 255; if not, the pixel value of this grid cell is set to 0, thus obtaining the road boundary feature map.

[0034] Step 4, which involves projecting the stitched, coordinate-transformed 3D point cloud of the road boundary onto the XOY gridded pixel plane to obtain the projected road boundary image, specifically involves:

[0035]

[0036] In the formula, Ax min It is the minimum value of the bounding box of the 3D point cloud of the road boundary after coordinate transformation, Ay. min It is the minimum bounding box of the 3D point cloud of the road boundary after coordinate transformation after stitching, along with resolution representing the length of each grid cell. n This represents the nth point (x) in the 3D point cloud of the road boundary after coordinate transformation. n y n , z n The row number (Acol) in the XOY gridded pixel plane after projection. n Representing point (x) n y n , zn The column number of the pixel in the XOY gridded pixel plane after projection;

[0037] Step 4 involves binarizing the projected road boundary image to obtain the initial contour image, specifically as follows:

[0038] Within the XOY gridded pixel plane, for each grid cell, it is determined whether there is a 3D point cloud of the road boundary that has undergone coordinate transformation falling into it. If so, the pixel value of this grid cell is set to 255; if not, the pixel value of this grid cell is set to 0, thus obtaining the initial contour image.

[0039] The initial contour image consists of N contour points, v n =(Crow) n Ccol n Crow is the nth contour point on the initial contour image. n Ccol represents the row number of the nth contour point in the XOY gridded pixel plane. n This indicates the column number of the nth contour point in the XOY gridded pixel plane;

[0040] Step 4 describes the construction of the Snake model, as follows;

[0041] The initial contour of the Snake model is taken from the initial contour image;

[0042] The energy function of the Snake model includes: external energy E ext and internal energy E int

[0043] External energy E in the Snake model ext Using gradient vector flow, let F be the edge potential field, then F = -E ext The gradient vector flow field is defined as: E ext =-[x(row) n ,col n ), Y(row) n ,col n )];

[0044] Where X(row) n ,col n Y(row) represents the component of the gradient vector flow field on the horizontal axis. n ,col n ) represents the component of the gradient vector flow field on the vertical axis.

[0045] The internal energy E of the Snake model int Defined as:

[0046]

[0047] In the formula, α represents the elastic weight parameter of the control curve, β represents the rigid weight parameter of the control curve, N represents the number of contour points in the initial contour image, and v n =(Crow) n Ccol n Crow is the nth contour point on the initial contour image. n Ccol represents the row number of the nth contour point in the XOY gridded pixel plane. n This indicates the column number of the nth contour point in the XOY gridded pixel plane.

[0048] The feature map of the Snake model uses a road boundary feature map.

[0049] Step 4 involves solving the Snake energy function model using the improved internal force matrix to obtain the point set of the road boundary. The specific process is as follows:

[0050] During the iterative convergence process, the internal forces are controlled by the internal force matrix, which is formulated as follows:

[0051]

[0052] The road boundary is a non-closed contour line. The curve f(s) obtained through iteration maintains continuity of the second derivative at the third point and the third-to-last point. Therefore, the modified N×N internal force matrix is:

[0053]

[0054] In the formula, p = β, q = -α - 4β, r = γ + 2α + 6β, and γ is the iteration step size;

[0055] After modifying the internal force matrix, the energy function is minimized by iterative approximation using difference approximation.

[0056] Preferably, step 5 involves vectorizing the road boundary point set to obtain a road boundary polyline. The specific process is as follows:

[0057] Calculate the distance between the nth boundary point in the road boundary point set and the vehicle position at each time step, and then select the vehicle position Traj closest to the nth boundary point from all vehicle positions at all time steps. i Then the time of the nth boundary point is Traj i The corresponding time i;

[0058] Add the road boundary points at time i to the set to obtain the set C of all road boundary points at time i. i The specific definition is as follows:

[0059] C i={P ni |||P ni -Traj i ||2=min{||P n -Traj i ||2},n∈[1,N]};

[0060] Among them, Traj i Let P represent the vehicle position at time i. n Let P represent the point cloud of the nth road boundary. ni Let i represent the point cloud of the nth road boundary and the time of that point cloud.

[0061] For time i, based on the vehicle positions Traj of the two adjacent time times... i and Traj i+1 Calculate the vehicle's forward direction For set C i Each point P in ni Calculate its position Traj relative to the vehicle position at time i. i The resulting vector Then calculate the cross product. If S ni <0, then P ni The left-hand road boundary point; conversely, if S... ni If P > 0, then P ni The right road boundary points are used to obtain the left road boundary point set and the right road boundary point set;

[0062] With Traj i The origin is the direction of vehicle movement. Y′ i The axis, with the vertical direction being X′ i Establish a local coordinate system along the axis;

[0063] Based on the vehicle's forward direction vector The rotation angle between the local coordinate system and the world coordinate system is calculated as follows:

[0064] For set C i All road boundary points P ni =(x ni y ni The coordinates P′ of the boundary point in the transformed local coordinate system are obtained using the coordinate transformation formula. ni =(x′) ni y′ ni The details are as follows:

[0065]

[0066] In the formula, For Traj i Let x be the rotation angle of the local coordinate system relative to the world coordinate system at the origin. ni Let P be the road boundary point. ni The x-coordinate in the world coordinate system, y ni Let P be the road boundary point. ni The ordinate in the world coordinate system, x′ ni P′ is the boundary point in the local coordinate system. ni x-coordinate, y′ ni P′ is the boundary point in the local coordinate system. ni The ordinate, X i Let Traj be the vehicle position at time i. i The horizontal coordinate in the world coordinate system, Y i Let Traj be the vehicle position at time i. i The vertical coordinate in the world coordinate system;

[0067] For the left and right boundary point sets at time i, respectively, according to X′ in the local coordinate system i The sizes on the axes are sorted to construct the topological relationship of the boundaries, and the road boundary polylines are obtained by connecting them sequentially.

[0068] In summary, this invention can integrate vehicle-mounted imagery and laser point clouds to extract and reconstruct road boundaries in 3D. The results obtained can be displayed from both global and local perspectives. The road boundary extraction method proposed in this invention exhibits strong robustness to different road shapes and point densities. Even in scenarios with severe road boundary wear or occlusion, it can still obtain a complete and accurately positioned vectorized model, which can be applied to the production of high-precision maps for autonomous driving and the updating of basic geographic information data. Attached Figure Description

[0069] Figure 1 : Flowchart of the method according to an embodiment of the present invention;

[0070] Figure 2 : A schematic diagram of shrinkage distance clustering according to an embodiment of the present invention;

[0071] Figure 3 : A schematic diagram of the point cloud and image registration process according to an embodiment of the present invention;

[0072] Figure 4 : A schematic diagram of world coordinates to camera coordinates according to an embodiment of the present invention;

[0073] Figure 5 : A schematic diagram of the construction of the local coordinate system in an embodiment of the present invention. Detailed Implementation

[0074] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0075] In specific implementation, the method proposed in the technical solution of this invention can be automatically executed by those skilled in the art using computer software technology. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of this invention and computer equipment including the computer program running the corresponding computer program, should also be within the protection scope of this invention.

[0076] This invention proposes a method for road boundary extraction and vectorization that integrates vehicle-mounted laser scanning point clouds and vehicle-mounted images. The method includes data acquisition; road boundary point cloud extraction; image boundary extraction; point cloud and image result-level fusion; and polyline simplification and vectorization.

[0077] The following is combined with Figures 1 to 5 The technical solution of the method in this embodiment of the invention is a method for road boundary extraction and vectorization that integrates vehicle-mounted images and point clouds, as detailed below:

[0078] This invention proposes a method for road boundary extraction and vectorization that integrates vehicle-mounted imagery and point clouds. The implementation process is as follows: Figure 1 As shown. First, the vehicle-mounted mobile laser scanning system acquires road 3D point clouds, vehicle positions, vehicle panoramic images, and pose information of the panoramic camera at multiple times in real time. Then, the point cloud data is segmented into supervoxels, and boundary point clouds are extracted using shrinking distance clustering. The DBSCAN algorithm is then used to remove noise from the boundary points. Next, a neural network model based on a hierarchical multi-scale attention mechanism is used for semantic segmentation of the image. Based on this, a region growing algorithm is used to extract boundary pixels. The image and point cloud are then registered using intrinsic and extrinsic parameters, and the true 3D spatial coordinates of the road boundary in the image are obtained through pixel back-projection. Then, a boundary feature map is constructed using the boundary point cloud, and the image extraction results are used as the initial contour to solve a non-closed Snake model to obtain an ordered set of points for the road boundary. Finally, the point set is vectorized, and polyline simplification is performed based on the Douglas poker method. The specific implementation process includes the following steps:

[0079] A method for extracting road boundaries by fusing point cloud and panoramic imagery includes the following steps:

[0080] Step 1: The mobile laser scanning system uses a 3D laser scanner to collect road 3D point clouds at multiple times in real time, uses a GNSS positioning module to collect vehicle positions at multiple times in real time, uses a panoramic camera to collect vehicle panoramic images at multiple times in real time, and uses an inertial measurement unit to collect roll angle, yaw angle, and pitch angle at multiple times when the panoramic camera is shooting.

[0081] The scanning system described in this embodiment is a RIEGLVUX-1 laser scanner;

[0082] The model of the GNSS positioning module and inertial measurement unit is i90 IMU-RTK GNSS.

[0083] The model of the panoramic camera is: 30MP HDR 360° panoramic camera;

[0084] Step 2: Perform super-voxel segmentation on the 3D point cloud of the road at each time step, extract the 3D point cloud of the road boundary at each time step using the Euclidean clustering method based on shrinkage distance, and denoise the 3D point cloud of the road boundary at each time step to obtain the denoised 3D point cloud of the road boundary at each time step.

[0085] The acquisition of the denoised road boundary 3D point cloud is as follows:

[0086] First, the point cloud is projected onto a fixed-size 2D grid. The maximum elevation difference for each grid is calculated, and an elevation difference threshold is set. Grids with elevation differences greater than the predefined threshold are marked as "uneven grids" (UG), while other grids are marked as "smooth grids" (SG). Candidate points are then reassigned to supervoxels based on the minimum weighted distance. The weighted distance between each candidate point cloud and its neighboring SG and UG is obtained by weighting the point density difference, elevation difference, and horizontal distance, W. xy w z and w ds D respectively xy D z and D ds Weights of plane Euclidean distance difference, elevation difference, and local density difference.

[0087] In this embodiment, w xy Defined as 0.25, w z Defined as 0.15, w d Defined as 0.6.

[0088] Then, as Figure 2 As shown, calculate the contraction distance at point p. n and p mThe formula for calculating the contraction distance between them is as follows:

[0089]

[0090] In the formula, p n and p m These are two points in adjacent supervoxels, d(p n p m ) is p n and p m The Euclidean distance between them; Is it far from p n The nearest trajectory point; express The direction vector; d c (p n p m ) is p n and p m The contraction distance, A vector representing two points in adjacent supervoxels;

[0091] Discrete point cloud data is organized into KD trees to construct topological relationships, and road boundary point clouds are obtained using a clustering algorithm based on shrinkage distance.

[0092] Finally, the DBSCAN algorithm is used to cluster the point cloud of road boundaries to remove noise;

[0093] In this embodiment, the neighborhood radius is defined as 0.1m and the neighborhood density threshold is 15.

[0094] Step 3: Semantically segment the vehicle panoramic image at each time step using a hierarchical multi-scale attention mechanism neural network model to obtain the road surface area image at each time step. Further, use the region growing algorithm to extract multiple road boundary pixels in the road surface area image at each time step. Register the vehicle panoramic image at each time step with the denoised road boundary 3D point cloud to obtain the relationship model between pixels and point cloud. Further, transform multiple road boundary pixels in the road surface area image at each time step to the world coordinate system to obtain the coordinate transformed road boundary 3D point cloud at each time step.

[0095] Step 3 involves registering the vehicle-mounted panoramic image at each moment with the denoised 3D point cloud of the road boundary to obtain the correspondence between the pixels and the point cloud. The process is as follows: Figure 3 As shown, the specific process is as follows:

[0096] Step 3.1: Map the vehicle-mounted 3D laser point cloud coordinate system corresponding to the denoised road boundary 3D point cloud to the panoramic image coordinate system using the roll angle, yaw angle, and pitch angle of the panoramic camera at each moment. For the vehicle-mounted panoramic image at the i-th moment.

[0097] Calculate the rotation matrix R at time i. i The details are as follows:

[0098]

[0099] In the formula, γ i Let β be the roll angle during the panoramic image capture at time i. i Let α be the yaw angle during the panoramic image capture at time i. i Let R be the pitch angle during the panoramic image capture at time i. i Let be the rotation matrix at time i;

[0100] For the panoramic image at time i, the t-th point in the denoised 3D point cloud of the road boundary at time i in the world coordinate system Transformed into the t-th road boundary pixel of the panoramic image at time i in the camera coordinate system The specific formula is as follows:

[0101]

[0102]

[0103]

[0104] In the formula, Let x be the x-axis coordinate of the t-th point in the denoised 3D point cloud of the road boundary at time i in the world coordinate system; Let y be the y-coordinate of the t-th point in the denoised 3D point cloud of the road boundary at time i in the world coordinate system; Let t be the z-axis coordinate of the denoised 3D point cloud of the road boundary at time i in the world coordinate system; Let x be the x-coordinate of the t-th road boundary pixel in the panoramic image at time i in the camera coordinate system; Let y be the y-coordinate of the t-th road boundary pixel in the panoramic image at time i in the camera coordinate system; Let T be the z-axis coordinate of the t-th road boundary pixel in the panoramic image at time i in the camera coordinate system; i Let be the translation vector between the camera coordinate system and the world coordinate system for the panoramic image at time i.

[0105] Step 3.2: Based on the spherical imaging model, the t-th road boundary pixel of the panoramic image at time i is... When mapping to the pixel spherical coordinate system of the panoramic image, the transformation formula is as follows:

[0106]

[0107] In the formula, Let x be the x-coordinate of the t-th road boundary pixel in the panoramic image at time i in the camera coordinate system; Let y be the y-coordinate of the t-th road boundary pixel in the panoramic image at time i in the camera coordinate system; Let z be the z-axis coordinate of the t-th road boundary pixel in the panoramic image at time i in the camera coordinate system. Let be the angle between the t-th road boundary pixel of the panoramic image at time i and the positive z-axis. Let be the angle between the t-th road boundary pixel of the panoramic image at time i and the positive y-axis.

[0108] Step 3.3: The t-th road boundary pixel of the panoramic image at time i. Transform the panoramic image from its spherical coordinate system to its pixel coordinate system using cylindrical projection transformation, such as... Figure 4 The conversion formula is as follows:

[0109]

[0110] In the formula, Let be the t-th road boundary pixel of the panoramic image at time i. The x-coordinate in the pixel coordinate system of the panoramic image Let be the t-th road boundary pixel of the panoramic image at time i. The y-coordinate in the pixel coordinate system of the panoramic image.

[0111] In this embodiment, W = 1024 is the length of the panoramic image, and H = 768 is the width.

[0112] Step 3 further transforms multiple road boundary pixels in the road surface area image at each time moment to the world coordinate system, obtaining the 3D point cloud of the road boundary after coordinate transformation at each time moment, as follows:

[0113] The vehicle-mounted panoramic image at each moment is registered with the denoised 3D point cloud of the road boundary to obtain a point in the world coordinate system. The corresponding points obtained after transforming to the pixel coordinate system of the panoramic image By traversing multiple road boundary pixels in the road surface area image at each time step and finding their corresponding points in the world coordinate system, the 3D point cloud of the road boundary after coordinate transformation at each time step can be obtained.

[0114] Step 4: Stitch together the denoised 3D point clouds of the road boundary from multiple time points to obtain a stitched 3D point cloud of the road boundary. Calculate the bounding box of the stitched 3D point cloud of the road boundary. Construct an XOY gridded pixel plane based on the bounding box of the stitched 3D point cloud of the road boundary. Project the stitched 3D point cloud of the road boundary onto the XOY gridded pixel plane to obtain a projected road boundary image. Binarize the projected road boundary image to obtain a road boundary feature map. Alternatively, stitch together the coordinate-transformed 3D point clouds of the road boundary from multiple time points by setting a region of interest to obtain a coordinate-transformed 3D point cloud of the road boundary. Calculate the bounding box of the coordinate-transformed 3D point cloud of the road boundary. Construct an XOY gridded pixel plane based on the bounding box of the coordinate-transformed 3D point cloud of the road boundary. Project the stitched coordinate-transformed 3D point cloud of the road boundary onto the XOY gridded pixel plane to obtain a projected road boundary image. Binarize the projected road boundary image to obtain an initial contour image. Construct a Snake model and solve the Snake model using an improved internal force matrix to obtain a set of road boundary points.

[0115] Step 4 involves projecting the stitched 3D point cloud of the road boundary onto the XOY gridded pixel plane to obtain the projected road boundary image, as detailed below:

[0116]

[0117] In the formula, x min It is the minimum value of the bounding box of the stitched road boundary 3D point cloud on the x-axis, y min It is the minimum value of the bounding box of the stitched road boundary 3D point cloud on the y-axis, resolution represents the length of each grid cell, and row n This represents the nth point (x) in the stitched 3D point cloud of the road boundary. n y n , z n The row number (col) in the XOY gridded pixel plane after projection. n Representing point (x) n y n , z n The column number of the pixel in the XOY gridded pixel plane after projection;

[0118] In this embodiment, resolution = 0.2;

[0119] Step 4 involves binarizing the projected road boundary image to obtain the road boundary feature map, as detailed below:

[0120] Within the XOY gridded pixel plane, for each grid cell, it is determined whether the projected boundary point cloud falls into it. If it does, the pixel value of this grid cell is set to 255; if not, the pixel value of this grid cell is set to 0, thus obtaining the road boundary feature map.

[0121] Step 4, which involves projecting the stitched, coordinate-transformed 3D point cloud of the road boundary onto the XOY gridded pixel plane to obtain the projected road boundary image, specifically involves:

[0122]

[0123] In the formula, Ax min It is the minimum value of the bounding box of the 3D point cloud of the road boundary after coordinate transformation, Ay. min It is the minimum bounding box of the 3D point cloud of the road boundary after coordinate transformation after stitching, along with resolution representing the length of each grid cell. n This represents the nth point (x) in the 3D point cloud of the road boundary after coordinate transformation. n y n , z n The row number (Acol) in the XOY gridded pixel plane after projection. n Representing point (x) n y n , z n The column number of the pixel in the XOY gridded pixel plane after projection;

[0124] Step 4 involves binarizing the projected road boundary image to obtain the initial contour image, specifically as follows:

[0125] Within the XOY gridded pixel plane, for each grid cell, it is determined whether there is a 3D point cloud of the road boundary that has undergone coordinate transformation falling into it. If so, the pixel value of this grid cell is set to 255; if not, the pixel value of this grid cell is set to 0, thus obtaining the initial contour image.

[0126] The initial contour image consists of N contour points, v n =(Crow) n Ccol n Crow is the nth contour point on the initial contour image. n Ccol represents the row number of the nth contour point in the XOY gridded pixel plane. n This indicates the column number of the nth contour point in the XOY gridded pixel plane;

[0127] Step 4 describes the construction of the Snake model, as follows;

[0128] The initial contour of the Snake model is taken from the initial contour image;

[0129] The energy function of the Snake model includes: external energy E ext and internal energy E int ;

[0130] External energy E in the Snake model ext Using gradient vector flow, let F be the edge potential field, then F = -E ext The gradient vector flow field is defined as: E ext =-[X(row) n ,col n ), Y(row) n ,col n )];

[0131] Where X(row) n ,col n Y(row) represents the component of the gradient vector flow field on the horizontal axis. n ,col n ) represents the component of the gradient vector flow field on the vertical axis.

[0132] The internal energy E of the Snake model int Defined as:

[0133]

[0134] In the formula, α represents the elastic weight parameter of the control curve, β represents the rigid weight parameter of the control curve, N represents the number of contour points in the initial contour image, and v n =(Crow) n Ccol n Crow is the nth contour point on the initial contour image. n Ccol represents the row number of the nth contour point in the XOY gridded pixel plane. n This indicates the column number of the nth contour point in the XOY gridded pixel plane.

[0135] The feature map of the Snake model uses a road boundary feature map.

[0136] Step 4 involves solving the Snake energy function model using the improved internal force matrix to obtain the point set of the road boundary. The specific process is as follows:

[0137] During the iterative convergence process, the internal forces are controlled by the internal force matrix, which is formulated as follows:

[0138]

[0139] The road boundary is a non-closed contour line. The curve obtained through iteration maintains the continuity of the second derivative at the third point and the third-to-last point. Therefore, the modified N×N internal force matrix is:

[0140]

[0141] In the formula, p = β, q = -α - 4β, r = γ + 2α + 6β, and γ is the iteration step size;

[0142] In this embodiment, α = 0.05, β = 0.95, and γ = 0.95;

[0143] After modifying the internal force matrix, the energy function is minimized by iterative approximation using difference approximation.

[0144] Step 5: Vectorize the road boundary point set to obtain the road boundary polyline, and further simplify the road boundary polyline using the Douglas poker method to obtain the vector line road boundary;

[0145] Step 5 involves vectorizing the road boundary point set to obtain the road boundary polyline. The specific process is as follows:

[0146] Calculate the distance between the nth boundary point in the road boundary point set and the vehicle position at each time step, and then select the vehicle position Traj closest to the nth boundary point from all vehicle positions at all time steps. i Then the time of the nth boundary point is Traj i The corresponding time i;

[0147] Add the road boundary points at time i to the set to obtain the set C of all road boundary points at time i. i The specific definition is as follows:

[0148] C i ={P ni |||P ni -Traj i || 2 =min{||P n -Traj i || 2}, n∈[1, N]};

[0149] Among them, Traj i Let P represent the vehicle position at time i. n Let P represent the point cloud of the nth road boundary. ni Let i represent the point cloud of the nth road boundary and the time of that point cloud.

[0150] For time i, based on the vehicle positions Traj of the two adjacent time times... i and Traji+1 Calculate the vehicle's forward direction For set C i Each point P in ni Calculate its position Traj relative to the vehicle position at time i. i The resulting vector Then calculate the cross product. If S ni <0, then P ni The left-hand road boundary point; conversely, if S... ni >0, then P ni The right road boundary points are used to obtain the left road boundary point set and the right road boundary point set;

[0151] With Traj i The origin is the direction of vehicle movement. Y′ i The axis, with the vertical direction being X′ i Establish a local coordinate system for the axes, such as Figure 5 As shown;

[0152] Based on the vehicle's forward direction vector The rotation angle between the local coordinate system and the world coordinate system is calculated as follows:

[0153] For set C i All road boundary points P ni =(x ni yn i The coordinates P′ of the boundary point in the transformed local coordinate system are obtained using the coordinate transformation formula. ni =(x′) ni y′ ni The details are as follows:

[0154]

[0155] In the formula, For Traj i Let x be the rotation angle of the local coordinate system relative to the world coordinate system at the origin. ni Let P be the road boundary point. ni The x-coordinate in the world coordinate system, y ni Let P be the road boundary point. ni The ordinate in the world coordinate system, x′ ni P′ is the boundary point in the local coordinate system. ni x-coordinate, y′ ni P′ is the boundary point in the local coordinate system. ni The ordinate, X i Let Traj be the vehicle position at time i. i The horizontal coordinate in the world coordinate system, Yi Let Traj be the vehicle position at time i. i The vertical coordinate in the world coordinate system;

[0156] For the left and right boundary point sets at time i, respectively, according to X′ in the local coordinate system i The sizes on the axes are sorted to construct the topological relationship of the boundaries, and the road boundary polylines are obtained by connecting them sequentially.

[0157] In summary, this invention can integrate vehicle-mounted images and laser point clouds to extract and reconstruct road boundaries in 3D. The results obtained from the embodiments of this invention can be displayed in the figure from both global and local perspectives. The road boundary extraction method proposed in this invention has strong robustness to different road shapes and point densities. It can still obtain a complete and accurately positioned vectorized model even in scenarios where the road boundary is severely worn or occluded, and it can also achieve good results in areas with sparse point clouds, unclear boundaries, and incomplete data.

[0158] It should be understood that any parts not described in detail in this specification belong to the prior art.

[0159] Although this document uses terms such as laser scanning system, GNSS positioning module, panoramic camera, and inertial measurement unit frequently, the possibility of using other terms is not excluded. These terms are used merely for the convenience of describing the essence of the invention, and interpreting them as any additional limitation would contradict the spirit of the invention.

[0160] It should be understood that the above description of the preferred embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A method for road boundary extraction and vectorization that integrates vehicle-mounted imagery and point clouds, characterized in that, Includes the following steps: Step 1: The mobile laser scanning system uses a 3D laser scanner to collect road 3D point clouds at multiple times in real time, uses a GNSS positioning module to collect vehicle positions at multiple times in real time, uses a panoramic camera to collect vehicle panoramic images at multiple times in real time, and uses an inertial measurement unit to collect roll angle, yaw angle, and pitch angle at multiple times when the panoramic camera is shooting. Step 2: Perform super-voxel segmentation on the 3D point cloud of the road at each time step, extract the 3D point cloud of the road boundary at each time step using the Euclidean clustering method based on shrinkage distance, and denoise the 3D point cloud of the road boundary at each time step to obtain the denoised 3D point cloud of the road boundary at each time step. Step 3: Semantically segment the vehicle panoramic image at each time step using a hierarchical multi-scale attention mechanism neural network model to obtain the road surface area image at each time step. Further, use the region growing algorithm to extract multiple road boundary pixels in the road surface area image at each time step. Register the vehicle panoramic image at each time step with the denoised road boundary 3D point cloud to obtain the relationship model between pixels and point cloud. Further, transform multiple road boundary pixels in the road surface area image at each time step to the world coordinate system to obtain the coordinate transformed road boundary 3D point cloud at each time step. Step 4: Stitch together the denoised 3D point clouds of the road boundary from multiple time points to obtain a stitched 3D point cloud of the road boundary. Calculate the bounding box of the stitched 3D point cloud of the road boundary. Construct an XOY gridded pixel plane based on the bounding box of the stitched 3D point cloud of the road boundary. Project the stitched 3D point cloud of the road boundary onto the XOY gridded pixel plane to obtain a projected road boundary image. Binarize the projected road boundary image to obtain a road boundary feature map. Alternatively, stitch together the coordinate-transformed 3D point clouds of the road boundary from multiple time points by setting a region of interest to obtain a coordinate-transformed 3D point cloud of the road boundary. Calculate the bounding box of the coordinate-transformed 3D point cloud of the road boundary. Construct an XOY gridded pixel plane based on the bounding box of the coordinate-transformed 3D point cloud of the road boundary. Project the stitched coordinate-transformed 3D point cloud of the road boundary onto the XOY gridded pixel plane to obtain a projected road boundary image. Binarize the projected road boundary image to obtain an initial contour image. Construct a Snake model and solve the Snake model using an improved internal force matrix to obtain a set of road boundary points. Step 5: Vectorize the road boundary point set to obtain the road boundary polyline, and further simplify the road boundary polyline using the Douglas poker method to obtain the vector line road boundary.

2. The method for road boundary extraction and vectorization by fusing vehicle-mounted images and point clouds according to claim 1, characterized in that: Step 3 involves registering the vehicle-mounted panoramic image at each moment with the denoised 3D point cloud of the road boundary to obtain the correspondence between the pixels and the point cloud. The specific process is as follows: Step 3.1: Map the vehicle-mounted 3D laser point cloud coordinate system corresponding to the denoised road boundary 3D point cloud to the panoramic image coordinate system using the roll angle, yaw angle, and pitch angle of the panoramic camera at each moment. For the vehicle-mounted panoramic image at the i-th moment. Calculate the rotation matrix at time i. The details are as follows: In the formula, Let be the roll angle during the panoramic image capture at time i. Let be the yaw angle during the panoramic image capture at time i. Let be the pitch angle during the panoramic image capture at time i. Let be the rotation matrix at time i; For the panoramic image at time i, the t-th point in the denoised 3D point cloud of the road boundary at time i in the world coordinate system Transformed into the t-th road boundary pixel of the panoramic image at time i in the camera coordinate system The specific formula is as follows: In the formula, Let x be the x-axis coordinate of the t-th point in the denoised 3D point cloud of the road boundary at time i in the world coordinate system; Let y be the y-coordinate of the t-th point in the denoised 3D point cloud of the road boundary at time i in the world coordinate system; Let t be the z-axis coordinate of the denoised 3D point cloud of the road boundary at time i in the world coordinate system; Let x be the x-coordinate of the t-th road boundary pixel in the panoramic image at time i in the camera coordinate system; Let y be the y-coordinate of the t-th road boundary pixel in the panoramic image at time i in the camera coordinate system; Let z be the z-axis coordinate of the t-th road boundary pixel in the panoramic image at time i in the camera coordinate system. Let be the translation vector between the camera coordinate system and the world coordinate system for the panoramic image at time i. Step 3.2: Based on the spherical imaging model, the t-th road boundary pixel of the panoramic image at time i is... When mapping to the pixel spherical coordinate system of the panoramic image, the transformation formula is as follows: In the formula, Let x be the x-coordinate of the t-th road boundary pixel in the panoramic image at time i in the camera coordinate system; Let y be the y-coordinate of the t-th road boundary pixel in the panoramic image at time i in the camera coordinate system; Let z be the z-axis coordinate of the t-th road boundary pixel in the panoramic image at time i in the camera coordinate system. Let be the angle between the t-th road boundary pixel of the panoramic image at time i and the positive z-axis. Let be the angle between the t-th road boundary pixel of the panoramic image at time i and the positive y-axis. Step 3.3: The t-th road boundary pixel of the panoramic image at time i. The panoramic image is transformed from its spherical coordinate system to its pixel coordinate system using cylindrical projection transformation. The transformation formula is as follows: In the formula, Let be the t-th road boundary pixel of the panoramic image at time i. The x-coordinate in the pixel coordinate system of the panoramic image Let be the t-th road boundary pixel of the panoramic image at time i. In the pixel coordinate system of the panoramic image, W is the length of the panoramic image and H is the width.

3. The method for road boundary extraction and vectorization by fusing vehicle-mounted images and point clouds according to claim 1, characterized in that: Step 3 further transforms multiple road boundary pixels in the road surface area image at each time moment to the world coordinate system, obtaining the 3D point cloud of the road boundary after coordinate transformation at each time moment, as follows: The vehicle-mounted panoramic image at each moment is registered with the denoised 3D point cloud of the road boundary to obtain a point in the world coordinate system. The corresponding points obtained after transforming to the pixel coordinate system of the panoramic image ; By traversing multiple road boundary pixels in the road surface area image at each time step and finding their corresponding points in the world coordinate system, the 3D point cloud of the road boundary after coordinate transformation at each time step can be obtained.

4. The method for road boundary extraction and vectorization by fusing vehicle-mounted images and point clouds according to claim 1, characterized in that: Step 4 involves projecting the stitched 3D point cloud of the road boundary onto the XOY gridded pixel plane to obtain the projected road boundary image, as detailed below: In the formula, xmin is the minimum value of the bounding box of the stitched 3D point cloud of the road boundary on the x-axis, ymin is the minimum value of the bounding box of the stitched 3D point cloud of the road boundary on the y-axis, and resolution represents the length of each grid cell. This represents the row number of the nth point (xn, yn, zn) in the stitched 3D point cloud of the road boundary, projected onto the XOY gridded pixel plane. This indicates the column number of the point (xn, yn, zn) in the XOY gridded pixel plane after projection; Step 4 involves binarizing the projected road boundary image to obtain the road boundary feature map, as detailed below: Within the XOY gridded pixel plane, for each grid cell, it is determined whether the projected boundary point cloud falls into it. If it does, the pixel value of this grid cell is set to 255; otherwise, the pixel value of this grid cell is set to 0, thus obtaining the road boundary feature map.

5. The method for road boundary extraction and vectorization by fusing vehicle-mounted images and point clouds according to claim 1, characterized in that: Step 4, which involves projecting the stitched, coordinate-transformed 3D point cloud of the road boundary onto the XOY gridded pixel plane to obtain the projected road boundary image, specifically involves: In the formula, It is the minimum value of the bounding box on the x-axis of the 3D point cloud of the road boundary after coordinate transformation. It represents the minimum bounding box of the 3D point cloud of the road boundary after coordinate transformation, along the y-axis. `resolution` represents the length of each grid cell. This represents the row number of the nth point (xn, yn, zn) in the 3D point cloud of the road boundary after coordinate transformation, projected onto the XOY gridded pixel plane. This indicates the column number of the point (xn, yn, zn) in the XOY gridded pixel plane after projection; Step 4 involves binarizing the projected road boundary image to obtain the initial contour image, specifically as follows: Within the XOY gridded pixel plane, for each grid cell, it is determined whether there is a 3D point cloud of the road boundary that has undergone coordinate transformation falling into it. If so, the pixel value of this grid cell is set to 255; if not, the pixel value of this grid cell is set to 0, thus obtaining the initial contour image. The initial contour image consists of N contour points. For the nth contour point on the initial contour image, This indicates the row number of the nth contour point in the XOY gridded pixel plane. This indicates the column number of the nth contour point in the XOY gridded pixel plane.

6. The method for road boundary extraction and vectorization by fusing vehicle-mounted images and point clouds according to claim 1, characterized in that: Step 4 describes the construction of the Snake model, as follows; The initial contour of the Snake model is taken from an initial contour image; The energy function of the Snake model includes: external energy Eext and internal energy Eint; In the Snake model, the external energy Eext is represented by a gradient vector flow. Let F be the edge potential field, then F = -Eext. The gradient vector flow field is defined as: Eext ; in, This represents the components of the gradient vector flow field on the horizontal axis. Components on the vertical axis; The internal energy Eint of the Snake model is defined as: In the formula, The elastic weighting parameter represents the control curve. This represents the rigid weighting parameter of the control curve, and N represents the number of contour points in the initial contour image. For the nth contour point on the initial contour image, This indicates the row number of the nth contour point in the XOY gridded pixel plane. This indicates the column number of the nth contour point in the XOY gridded pixel plane; The feature map of the Snake model uses the road boundary feature map.

7. The method for road boundary extraction and vectorization by fusing vehicle-mounted images and point clouds according to claim 1, characterized in that: Step 4 involves solving the Snake energy function model using the improved internal force matrix to obtain the point set of the road boundary. The specific process is as follows: During the iterative convergence process, the internal forces are controlled by the internal force matrix, which is formulated as follows: The road boundary is a non-closed contour line, and the curve is obtained through iteration. If the second derivative remains continuous at the third point and the third-to-last point, then the modified result is obtained. The internal force matrix is: In the formula, p = β, q = -α - 4β, r = γ + 2α + 6β, and γ is the iteration step size; After modifying the internal force matrix, the energy function is minimized by iteratively applying the difference approximation.

8. The method for road boundary extraction and vectorization by fusing vehicle-mounted images and point clouds according to claim 1, characterized in that: Step 5 involves vectorizing the road boundary point set to obtain the road boundary polyline. The specific process is as follows: Calculate the distance between the nth boundary point in the road boundary point set and the vehicle position at each time step, and then select the vehicle position closest to the nth boundary point from all vehicle positions at all time steps. Then the time of the nth boundary point is The corresponding time i; Add the road boundary points at time i to the set to obtain the set of all road boundary points at time i. The specific definition is as follows: ； in, Let i represent the vehicle position at time i. This represents the point cloud of the nth road boundary. Let i represent the point cloud of the nth road boundary and the time of that point cloud be i; For time i, based on the vehicle positions of the two adjacent time points... and Calculate the vehicle's forward direction For each point in set Ci Calculate its position relative to the vehicle at time i. The resulting vector Then calculate the cross product. ,like 0, then The left boundary point of the road, and vice versa. 0, then The right road boundary points are used to obtain the left road boundary point set and the right road boundary point set; by The origin is the direction of vehicle movement. for The axis, perpendicular to the direction is Establish a local coordinate system along the axis; Based on the vehicle's forward direction vector The rotation angle between the local coordinate system and the world coordinate system is calculated as follows: ; For all road boundary points in set Ci The coordinates of the boundary points in the transformed local coordinate system are obtained using the coordinate transformation formula. The details are as follows: In the formula, For Let be the rotation angle of the local coordinate system relative to the world coordinate system at the origin. Road boundary point The horizontal coordinate in the world coordinate system Road boundary point The vertical coordinate in the world coordinate system Boundary points within the local coordinate system x-coordinate Boundary points within the local coordinate system The ordinate, Let i be the vehicle position at time i. The horizontal coordinate in the world coordinate system Let i be the vehicle position at time i. The vertical coordinate in the world coordinate system; For the left and right boundary point sets at time i respectively, in the local coordinate system The sizes on the axes are sorted to construct the topological relationship of the boundaries, and the road boundary polylines are obtained by connecting them sequentially.

Images