Curb detection methods, systems and vehicles

Abstract

This invention relates to the field of intelligent technology, providing a curb detection method, system, and vehicle. The method includes: extracting candidate curbs from an image of a road environment; projecting point cloud data of the road environment onto a pixel coordinate system to obtain a first image; determining a target region in the first image, the target region containing a region corresponding to the candidate curb; acquiring point cloud data corresponding to the target region from the point cloud data of the road environment as first point cloud data; and detecting curbs in the road environment based on the first point cloud data. By using candidate curbs in the image of the road environment as prior information, determining a target region in the first image containing a region corresponding to the candidate curb, and acquiring point cloud data corresponding to the target region from the point cloud data of the road environment as the first point cloud data, the invention ensures that curb information exists in the first point cloud data, significantly reducing environmental interference. The method then detects curbs in the road environment based on the first point cloud data, thereby achieving simple and accurate curb detection.

Description

Technical Field

[0001] This invention relates to the field of intelligent technology, and in particular to a curb detection method, system, and vehicle. Background Technology

[0002] Many roads have curbs to separate the road surface from the shoulder, thus distinguishing sidewalks and green spaces, and playing a role in ensuring the safety of pedestrians and vehicles and keeping the road edges neat.

[0003] Currently, with the development of autonomous driving technology, curb detection is of great significance. Taking road construction machinery as an example, road construction environments are harsh and have a high risk factor. Unmanned construction is the future development direction of road machinery, and curb detection is crucial for unmanned construction. Key technologies such as automatic curb alignment, path planning, and seamless paving all rely on accurate curb detection. However, current curb detection methods are either not accurate enough or difficult to implement. Summary of the Invention

[0004] This invention provides a curb detection method, system, and vehicle to solve or improve the problems of low accuracy or high implementation difficulty in curb detection in the prior art.

[0005] This invention provides a curb detection method, comprising:

[0006] Extract candidate curbs from images of the road environment;

[0007] The point cloud data of the road environment is projected onto the pixel coordinate system to obtain the first image;

[0008] Determine a target region in the first image, the target region comprising the region corresponding to the candidate road edge;

[0009] Obtain the point cloud data corresponding to the target area from the point cloud data of the road environment, and use it as the first point cloud data;

[0010] Based on the first point cloud data, the curb in the road environment is detected.

[0011] According to a curb detection method provided by the present invention, the step of detecting curbs in the road environment based on the first point cloud data includes:

[0012] The first point cloud data is transformed using a first transformation matrix to obtain the second point cloud data; wherein, the first transformation matrix is ​​used to transform the calibrated road surface plane in the spatial coordinate system where the point cloud data is located to coincide with the horizontal plane formed by the horizontal coordinate axis in the spatial coordinate system;

[0013] Extract the current road surface plane from the second point cloud data;

[0014] Determine a first rotation matrix to rotate the current road surface plane to be parallel to the horizontal plane;

[0015] The second point cloud data is rotated using the first rotation matrix to obtain the third point cloud data;

[0016] The roadside in the road environment is detected based on the third point cloud data.

[0017] According to a curb detection method provided by the present invention, the step of extracting the current road surface plane from the second point cloud data includes:

[0018] Plane fitting is performed on the second point cloud data to obtain multiple first candidate planes;

[0019] From a plurality of first candidate planes, select the first candidate plane whose angle between the normal vector and the vertical coordinate axis in the spatial coordinate system satisfies the first angle range, and use it as the second candidate plane;

[0020] A target point is generated based on the second point cloud data. On the horizontal coordinate axis, the coordinate value of the target point is the average value of the coordinate values ​​of the second point cloud data. On the vertical coordinate axis, the coordinate value of the target point is higher than the coordinate value of the second point cloud data.

[0021] The second candidate plane that is furthest from the target point is taken as the current road surface plane.

[0022] According to a curb detection method provided by the present invention, the step of detecting curbs in the road environment based on the third point cloud data includes:

[0023] From the third point cloud data, obtain the fourth point cloud data that meets the road surface height range and the fifth point cloud data that meets the roadside height range, wherein the roadside height range is the height range of the side of the roadside that is perpendicular to the road surface.

[0024] Based on the fourth point cloud data, the first target plane is obtained;

[0025] Based on the fifth point cloud data, the second target plane is obtained;

[0026] Based on the intersection of the first target plane and the second target plane, the curb in the road environment is obtained.

[0027] According to a curb detection method provided by the present invention, obtaining a first target plane based on the fourth point cloud data includes:

[0028] Plane fitting is performed on the fourth point cloud data to obtain multiple third candidate planes;

[0029] From a plurality of third candidate planes, the third candidate plane whose angle between the normal vector and the vertical coordinate axis in the spatial coordinate system satisfies the second angle range is selected as the first target plane;

[0030] The step of obtaining the second target plane based on the fifth point cloud data includes:

[0031] Plane fitting is performed on the fifth point cloud data to obtain multiple fourth candidate planes;

[0032] From the plurality of fourth candidate planes, the fourth candidate plane whose angle between the normal vector and the vertical coordinate axis satisfies the third angle range is selected as the second target plane.

[0033] According to a curb detection method provided by the present invention, the first transformation matrix is ​​obtained in the following manner:

[0034] Using the pitch angle and roll angle required to transform the calibrated road surface plane to coincide with the horizontal plane as variables, the optimal solutions for the pitch angle and roll angle are solved based on the particle swarm optimization algorithm;

[0035] The first transformation matrix is ​​obtained based on the optimal solutions for the pitch angle and the roll angle.

[0036] According to a curb detection method provided by the present invention, the step of extracting candidate curbs from an image of a road environment includes:

[0037] Based on the image of the road environment, a second image including the road surface area and the non-road surface area is obtained;

[0038] In the row direction of the second image, the second image is divided into two sub-images to obtain two regions of interest;

[0039] Edge detection is performed in each region of interest, and the candidate curbs are extracted from the edge detection results.

[0040] According to a curb detection method provided by the present invention, the image of the road environment is acquired by an image acquisition device, and the step of projecting the point cloud data of the road environment onto a pixel coordinate system includes:

[0041] Using the second transformation matrix and the intrinsic parameters of the image acquisition device, the point cloud data of the road environment is projected onto the pixel coordinate system. The second transformation matrix is ​​the transformation matrix between the coordinate system of the image acquisition device and the spatial coordinate system where the point cloud data is located. The second transformation matrix is ​​obtained by solving point cloud data samples and image samples. The two sides of the image samples contain calibration plates.

[0042] The present invention also provides a curb detection system, comprising:

[0043] Image acquisition device, used to acquire images of the road environment;

[0044] LiDAR, used to collect point cloud data of the road environment;

[0045] The controller is connected to the image acquisition device and the lidar, respectively, and the controller is used to execute the curb detection method as described above.

[0046] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the curb detection methods described above.

[0047] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the curb detection method as described above.

[0048] The present invention also provides a vehicle, including a vehicle body, wherein the vehicle body is provided with a curb detection system as described in any of the above, or includes an electronic device as described in any of the above, or includes a computer-readable storage medium as described in any of the above.

[0049] The curb detection method provided by this invention can extract candidate curbs from images of the road environment, project the point cloud data of the road environment onto a pixel coordinate system to obtain a first image, and use the candidate curbs in the image of the road environment as prior information to determine a target region in the first image that contains the region corresponding to the candidate curbs. The point cloud data corresponding to the target region in the point cloud data of the road environment is obtained as the first point cloud data, thereby ensuring that curb information exists in the first point cloud data, greatly reducing environmental interference. Based on the first point cloud data, curbs in the road environment can be detected, thereby achieving simple and accurate curb detection. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0051] Figure 1 This is one of the flowcharts of the curb detection method provided by the present invention;

[0052] Figure 2 This is a schematic diagram of the structure of the fusion sensor provided by the present invention;

[0053] Figure 3 This is one of the schematic diagrams of the target region in the first image provided by the present invention;

[0054] Figure 4 This is a second schematic diagram of the target region in the first image provided by the present invention;

[0055] Figure 5 This is the second flowchart of the curb detection method provided by the present invention;

[0056] Figure 6 This is a schematic diagram of the first and second regions of interest provided by the present invention;

[0057] Figure 7 This is the third flowchart of the curb detection method provided by the present invention;

[0058] Figure 8 This is the fourth flowchart of the curb detection method provided by the present invention;

[0059] Figure 9 This is the fifth flowchart of the curb detection method provided by the present invention;

[0060] Figure 10 This is a schematic diagram of the third point cloud data provided by the present invention;

[0061] Figure 11 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0063] There are various methods for curb detection in related technologies. For example, curb detection methods based on Global Positioning System (GPS) and Real-time Kinematic (RTK) rely on GPS and RTK point-based positioning to create maps. However, due to point-based errors and edge fitting errors, the detection accuracy is only around 5cm, which cannot meet the centimeter-level requirements of mechanical curb-adhesion operations. Another example is curb detection based on monocular depth estimation. The accuracy of curb detection information obtained from monocular depth recovery depends on the differences between the calibration environment and the actual measurement environment. Uneven ground or changes in camera angle can affect the detection accuracy, failing to meet the accuracy requirements for curb-adhesion positioning. Yet another example is curb detection based on binocular vision-generated point clouds. In this method, the accuracy is insufficient when the camera is far from the curb. For example, the roadside detection method based on lidar can extract roadside points based on the geometric features of the roadside. The methods for extracting roadside points based on the geometric features of the roadside can be existing methods such as region growing, threshold determination, point cloud rasterization and ground segmentation. However, these methods all require dense point clouds to achieve high-precision roadside detection. Generating dense point clouds has high requirements for sensor hardware and vehicle speed, making it difficult to implement.

[0064] Therefore, the present invention provides a curb detection method, which is described below in conjunction with... Figures 1 to 10 A detailed introduction will be provided.

[0065] This embodiment provides a curb detection method, such as Figure 1 As shown, the curb detection method may include:

[0066] Step 101: Extract candidate curbs from the road environment image.

[0067] Step 102: Project the point cloud data of the road environment onto the pixel coordinate system to obtain the first image.

[0068] Step 103: Determine the target region in the first image, wherein the target region includes the region corresponding to the candidate road edge.

[0069] Step 104: Obtain the point cloud data corresponding to the target area from the point cloud data of the road environment, and use it as the first point cloud data.

[0070] Step 105: Based on the first point cloud data, detect the curb in the road environment.

[0071] The curb detection method provided in this embodiment can be applied to vehicles. This method can be executed by the vehicle's overall controller or by a separately installed on-board terminal. The vehicle can be road machinery such as a road roller, grader, or paver, or it can be a car.

[0072] In practical applications, image acquisition devices can be used to capture images of the road environment. These devices can be cameras or other terminal devices with image acquisition capabilities. LiDAR can be used to acquire point cloud data of the road environment. The image acquisition device and LiDAR have overlapping perception areas of the road environment. In implementation, the image acquisition device and LiDAR can be mounted together on the vehicle body. Thus, the image acquisition device and LiDAR form a fusion sensor combining vision and LiDAR. The image acquisition device and LiDAR can be mounted on the vehicle body using a bracket. Figure 2 As shown, the bracket includes a base 201 and a mounting plate 202 disposed on the base 201. An image acquisition device 203 and a lidar 204 are mounted on the mounting plate 202, arranged side-by-side along the height of the vehicle body. The image acquisition device 203 can be positioned above or below the lidar 204, maximizing the overlap of their sensing areas. For example, the fusion sensor can be fixed on the vehicle's central axis for easy detection of road edges on both sides of the vehicle. Furthermore, since the vehicle control coordinate system and the fusion sensor's coordinate system share the same coordinate axes in the longitudinal direction of the vehicle, switching to the vehicle control coordinate system is more convenient when applying the road edge detection results; only a coordinate translation on the left-right coordinate axis of the vehicle is required.

[0073] An image is composed of pixels, and the coordinates of these pixels can be expressed using a pixel coordinate system. The pixel coordinate system is a Cartesian coordinate system with pixels as the unit. Pixels are arranged in rows and columns in an image; the direction of a row of pixels is the row direction, and the direction of a column of pixels is the column direction. The road environment image described above is an image expressed using a pixel coordinate system.

[0074] Both road environment images and road environment point cloud data contain curb information. In this embodiment, curbs in the road environment image can be extracted as candidate curbs through semantic segmentation. The road environment point cloud data is then projected onto a pixel coordinate system to obtain a first image. The region in this first image corresponding to the candidate curbs in the road environment image also contains curb information. Based on this, a target region in the first image can be determined. This target region contains the region corresponding to the candidate curbs, thus ensuring that the target region contains curb information. Based on this, the point cloud data corresponding to the target region in the road environment point cloud data is obtained, thereby extracting local point cloud data of the road environment as the first point cloud data. This ensures that the extracted local point cloud data also contains curb information. Therefore, curbs in the road environment can be detected based on the first point cloud data. In this way, using the candidate curbs provided by the road environment image as prior information for the LiDAR ensures that the first point cloud data used for curb detection contains curb information, significantly reducing environmental interference (such as smoke, vibration, etc.), thereby achieving simple and accurate curb detection.

[0075] like Figure 3 As shown, the target region can be region 301 in the first image corresponding to the candidate road edge, such as... Figure 4 As shown, the target area can also be an expanded area 302 obtained based on the area 301 in the first image that corresponds to the candidate road edge.

[0076] There are several ways to acquire expansion areas.

[0077] For example, the methods for obtaining the expanded region may include: Figure 4 As shown, based on the region in the first image corresponding to the candidate road edge, the region is expanded along the row direction of the first image to at least one side of the region corresponding to the candidate road edge, resulting in an expanded region. Thus, expanding the region corresponding to the candidate road edge along the row direction of the first image allows the target region to contain more road edge information. When expanding one side of the region corresponding to the candidate road edge along the row direction of the first image, the number of expanded pixels is within a set limit. The value of the set limit can be set according to actual needs; for example, the set limit can be 100. If there are no pixels to expand on one side of the region corresponding to the candidate road edge along the row direction of the first image, expansion is not required.

[0078] Point cloud data is unordered, and rule-based filtering cannot accurately extract the point cloud data corresponding to road edges. In this embodiment, candidate road edges can be extracted from the image of the road environment. The point cloud data of the road environment is projected onto the pixel coordinate system to obtain a first image. Using the candidate road edges in the image of the road environment as prior information, a target region corresponding to the candidate road edges is determined in the first image. The point cloud data corresponding to the target region in the point cloud data of the road environment is then obtained as the first point cloud data. This ensures that road edge information exists in the first point cloud data, significantly reducing environmental interference. Based on the first point cloud data, road edges in the road environment can be detected, thereby achieving simple and accurate road edge detection.

[0079] In an exemplary embodiment, the step of extracting candidate curbs from an image of the road environment, such as... Figure 5 As shown, its specific implementation methods may include:

[0080] Step 501: Based on the image of the road environment, obtain a second image including the road surface area and the non-road surface area.

[0081] Specifically, an image of the road environment can be input into a pre-trained semantic segmentation model to obtain a second image output by the semantic segmentation model. The semantic segmentation model can be used to perform semantic segmentation on the input road environment image, obtaining distinguishable road surface regions and non-road surface regions, and outputting a second image including both road surface and non-road surface regions. Training the semantic segmentation model can include training an initial model using image samples of the road environment and corresponding sample labels. For example, the semantic segmentation model can adopt the MODNet network structure. Semantic segmentation technology can accurately distinguish between road surface and non-road surface regions in a road environment image. In a road environment image, the area containing the road surface is the road surface region, and the area outside the road surface region is the non-road surface region.

[0082] Alternatively, the pixel value ranges for pixels corresponding to the road surface area and pixels corresponding to non-road surface areas can be set. Based on these ranges, the road surface area and non-road surface area in the road environment image can be distinguished to obtain a second image that includes both road surface and non-road surface areas. Directly distinguishing between road surface and non-road surface areas using the pixel value ranges of their corresponding pixels is simple to implement.

[0083] The second image can be a binary image, where the pixel values ​​of the pixels in the road surface area are the first pixel values, and the pixel values ​​of the pixels in the non-road surface areas are the second pixel values. For example, the first pixel value is 1, and the second pixel value is 0.

[0084] Step 502: Divide the second image into two sub-images along the row direction to obtain two regions of interest.

[0085] Since curbs are generally located on the left and right sides of a road, the second image can be segmented into two sub-images along its row direction. For example, segmentation can be performed at the midpoint of the row direction to obtain two sub-images. The left sub-image can contain information about the curb on the left side of the road, and the right sub-image can contain information about the curb on the right side of the road. These two sub-images can be considered as two regions of interest, such as... Figure 6 As shown, it includes a first region of interest and a second region of interest.

[0086] Step 503: Perform edge detection in each region of interest and extract the candidate curb from the edge detection results.

[0087] In this step, candidate road edges are extracted in each region of interest. First, edge detection is performed in each region of interest to obtain the edge lines between the road surface area and the non-road surface area in the second image. The straight line L1 is extracted from the edge detection results as a candidate road edge.

[0088] For example, in each region of interest, edge detection can be performed using the Canny operator, and then the Hough transform can be used to extract straight lines from the edge detection results as candidate curbs.

[0089] Among them, the Canny operator is an edge detection method that seeks the optimal trade-off between noise resistance and accurate localization, and can accurately extract the edge lines between road surface areas and non-road surface areas in the second image. The Hough transform can detect curves whose shapes can be described by functional relationships, such as straight lines, circles, parabolas, and ellipses. The Hough transform can accurately extract straight lines as candidate road edges. Specifically, the Hough transform can be used to obtain the equations of the straight lines, i.e., the equations of the candidate road edges.

[0090] In this embodiment, by distinguishing between road surface areas and non-road surface areas in the image of the road environment, a second image including road surface areas and non-road surface areas is obtained. Then, the second image is divided into two sub-images in the row direction as two regions of interest. Each region of interest may contain road edge information on one side of the road. Based on this, edge detection is performed on the two regions of interest respectively to obtain the edge lines of the road surface area and the non-road surface area, thereby accurately extracting the candidate road edges on each side of the road.

[0091] In an exemplary embodiment, the image of the road environment is acquired by an image acquisition device, and the step of projecting the point cloud data of the road environment onto a pixel coordinate system includes:

[0092] Using the second transformation matrix and the intrinsic parameters of the image acquisition device, the point cloud data of the road environment is projected onto the pixel coordinate system. The second transformation matrix is ​​the transformation matrix between the coordinate system of the image acquisition device and the spatial coordinate system where the point cloud data is located. The second transformation matrix is ​​obtained by solving point cloud data samples and image samples. The two sides of the image samples contain calibration plates.

[0093] Here, the spatial coordinate system of the point cloud data is the coordinate system of the point cloud data acquisition device. When using a lidar to acquire point cloud data, the spatial coordinate system of the point cloud data is the lidar's coordinate system.

[0094] The image acquisition device can be a camera. Based on this, the point cloud data of the road environment can be projected onto the pixel coordinate system using camera extrinsic and intrinsic parameters. The camera extrinsic parameter is the second transformation matrix. In this case, the second transformation matrix is ​​a transformation matrix between the camera coordinate system and the spatial coordinate system where the point cloud data is located. For example, projecting the point cloud data of the road environment onto the pixel coordinate system includes: using the second transformation matrix to transform the point cloud data of the road environment into the camera coordinate system, and then using the camera intrinsic parameters to transform the point cloud data of the road environment in the camera coordinate system into the pixel coordinate system. For camera imaging, generally, camera coordinate system, image coordinate system, and pixel coordinate system are involved. The camera coordinate system is a coordinate system centered on the camera, while the image coordinate system describes the imaging of a real object at the camera's focal length. A relationship can be established between the camera coordinate system and the pixel coordinate system. The origins of the image coordinate system and the pixel coordinate system are different. The origin of the image coordinate system is the intersection of the camera's optical axis and the imaging plane, while the origin of the pixel coordinate system is generally at the upper left corner of the image. When converting point cloud data of the road environment in the camera coordinate system to the pixel coordinate system, it is necessary to first convert the point cloud data of the road environment in the camera coordinate system to the image coordinate system through perspective projection, and then convert it to the pixel coordinate system through affine transformation.

[0095] The expressions for the second transformation matrix and camera intrinsic parameters are shown in the following examples:

[0096]

[0097]

[0098]

[0099]

[0100] Among them, (X) c Y c Z c (X) represents the coordinates in the camera coordinate system; l Yl Z l (u,v) represents the coordinates of the spatial coordinate system; (u,v) represents the coordinates of the pixel in the pixel coordinate system; R represents the second rotation matrix, which is a 3*3 matrix containing the element R. 11 R 12 ...,R 33 ; t represents the translation matrix, containing elements t1, t2, t3; T′ represents the second transformation matrix, composed of R and t; f represents the camera focal length; dX and dY represent the physical size of the pixel; θ represents the angle of the affine transformation; K represents the camera intrinsic parameters; c x and c y This represents the translation amount of an affine transformation.

[0101] In practical applications, after obtaining the fused sensor data, in environments with many linear features (i.e., many straight lines), while maintaining a static state, image samples can be acquired using an image acquisition device, and point cloud data samples can be acquired using a LiDAR. Then, the second transformation matrix projecting the point cloud data onto the pixel coordinate system is solved using the point cloud data samples and the image samples. The process of solving the second transformation matrix includes:

[0102] Linear features are extracted from point cloud data samples and image samples to obtain an objective function. This objective function takes the second transformation matrix as a variable and aims to minimize the distance between all linear features in the point cloud data samples after projection onto the pixel coordinate system and the linear features in the nearest image sample. The second transformation matrix is ​​solved through the objective function.

[0103] The smaller the distance between the linear features in a point cloud data sample and the linear features in the nearest image sample after projection onto the pixel coordinate system, the higher the degree of overlap between the two.

[0104] The second transformation matrix can project the point cloud data of the road environment onto the pixel coordinate system, thereby achieving the fusion of the point cloud data and the image of the road environment.

[0105] In addition, when collecting image samples, calibration plates can be set at the lower left and lower right corners of the sensing area. The calibration plate is a flat plate with a fixed-spacing pattern array, which can increase linear features. In this way, the two sides of the image sample can contain the calibration plate, thereby further enhancing the fusion accuracy of the lower right and lower left corners. The road edge is generally located at the lower left and lower right corners, which helps to improve the accuracy of road edge detection.

[0106] During curb detection, after acquiring point cloud data of the road environment, a second transformation matrix can be used to project the point cloud data onto a pixel coordinate system to obtain a first image. The pixel values ​​of the pixels in the first image contain indices of the point cloud data, which can be used to determine the corresponding point cloud data. Some pixels in the first image may not have corresponding projected point cloud data; in that case, their pixel values ​​will not contain indices of the point cloud data.

[0107] In the first image, the target region is obtained based on the straight line equation of L1. Pixels within a set number of distances to the left and right of L1 can be extracted in the row direction of the first image to obtain the target region. Then, the point cloud data corresponding to the target region is extracted as the first point cloud data.

[0108] In this embodiment, by calibrating the second transformation matrix using a calibration board, the fusion of image and point cloud data in the calibration board area can be made more accurate. The second transformation matrix is ​​used to project the point cloud data of the road environment onto the pixel coordinate system, which improves the fusion effect of the point cloud data and image of the road environment and makes it more accurate.

[0109] In step 105 above, based on the first point cloud data, the road edge in the road environment is detected. There are several ways to achieve this. In one implementation, the target area is the area corresponding to the candidate road edge; the first point cloud data can be directly fitted with a straight line as the detected road edge. This allows for simple and fast road edge detection. In another implementation, such as... Figure 7 As shown, its specific implementation methods may include:

[0110] Step 701: Transform the first point cloud data using the first transformation matrix to obtain the second point cloud data; wherein, the first transformation matrix is ​​used to transform the calibrated road surface plane in the spatial coordinate system where the point cloud data is located to coincide with the horizontal plane formed by the horizontal coordinate axis in the spatial coordinate system.

[0111] Here, the spatial coordinate system of the point cloud data is the OXYZ coordinate system. The horizontal coordinate axes include the X and Y axes, which form the XOY plane, i.e., the horizontal plane formed by the horizontal coordinate axes. The vertical coordinate axis includes the Z axis. The road surface plane is the plane on which the road surface is located. In implementation, the road surface plane needs to be calibrated in advance. A calibration environment with a flat road surface can be selected in advance, point cloud data can be collected, and the road surface point cloud data can be extracted from the collected point cloud data. Based on the road surface point cloud data, the calibrated road surface plane is obtained by fitting, and the first transformation matrix is ​​obtained so that the calibrated road surface plane coincides with the horizontal plane formed by the horizontal coordinate axes in the spatial coordinate system of the point cloud data.

[0112] For example, the first transformation matrix is ​​obtained in the following way:

[0113] Using the pitch angle and roll angle required to transform the calibrated road surface plane to coincide with the horizontal plane as variables, the optimal solutions for the pitch angle and roll angle are obtained based on the Particle Swarm Optimization (PSO) algorithm; based on the optimal solutions for the pitch angle and roll angle, the first transformation matrix is ​​obtained.

[0114]

[0115]

[0116]

[0117] R′=R x R y R z (8)

[0118] Where α represents the pitch angle, and the corresponding rotation matrix R y β represents the roll angle, corresponding to the rotation matrix R. x γ represents the yaw angle, corresponding to the rotation matrix R. z R′ represents the third rotation matrix.

[0119] The pitch angle is the rotation angle relative to the Y-axis, the roll angle is the rotation angle relative to the X-axis, and the yaw angle is the rotation angle relative to the Z-axis. For the calibration of the road surface plane, since it is necessary to transform the road surface plane to coincide with the XOY plane formed by the X-axis and Y-axis, only the rotation angles of the X-axis and Y-axis are needed. Based on this, the pitch angle and roll angle can be solved, and the yaw angle can be zero.

[0120] The third rotation matrix is ​​used to make the horizontal plane formed by the calibration road surface plane and the horizontal coordinate axis in the spatial coordinate system of the point cloud data parallel. Further, after rotating the first point cloud data using the third rotation matrix, a Z-axis translation is performed so that the calibration road surface plane coincides with the horizontal plane formed by the horizontal coordinate axis in the spatial coordinate system of the point cloud data. The Z-axis translation value is the distance between the rotated calibration road surface plane and the origin of the spatial coordinate system. Therefore, the aforementioned first transformation matrix includes the third rotation matrix and the Z-axis translation value.

[0121] Specifically, when solving for the optimal pitch and roll angles based on PSO, a particle swarm can be initialized first. Each particle in the swarm represents a solution for the pitch and roll angles. Based on the initialized particle swarm, the optimal solutions for the pitch and roll angles are sought. The process of seeking the optimal solutions can refer to the relevant techniques of the PSO algorithm, which will not be elaborated here. Finally, the optimal solutions for the pitch and roll angles are represented in matrix form to obtain the first transformation matrix. This first transformation matrix is ​​stored in a configuration file. Thus, during road surface detection, the pre-stored first transformation matrix can be directly obtained from the configuration file.

[0122] In this step, the first point cloud data is transformed using the first transformation matrix. This transformation not only transforms the road surface in the first point cloud data, but also transforms the first point cloud data as a whole along with the transformation of the road surface, resulting in the second point cloud data. This achieves the initial calibration of the road surface.

[0123] Step 702: Extract the current road surface plane from the second point cloud data.

[0124] Because the actual environment differs from the calibration environment, and the road surface changes in real time, this step extracts the current road surface plane from the second point cloud data. The current road surface plane is represented by its equation.

[0125] Step 703: Determine the first rotation matrix that rotates the current road surface plane to be parallel to the horizontal plane.

[0126] Specifically, the rotation angle required to rotate the current road surface plane to be parallel to the horizontal plane formed by the horizontal coordinate axis in the spatial coordinate system where the point cloud data is located can be calculated. This rotation angle can be the angle between the normal vector of the current road surface plane and the vertical coordinate axis in the spatial coordinate system. Representing this rotation angle in matrix form yields the first rotation matrix.

[0127] An example of the first rotation matrix is ​​as follows:

[0128]

[0129] Where δ represents the rotation angle; I represents the identity matrix; and n represents the rotation axis, which is a vector with three components: the X-axis, the Y-axis, and the Z-axis. x , ny, nz; R″ denotes the first rotation matrix.

[0130] The normal vector of a plane is a non-zero vector perpendicular to the plane, and it is an important vector for determining the plane's position. Therefore, the normal vector of the current road surface plane is a non-zero vector perpendicular to the current road surface plane.

[0131] Step 704: Rotate the second point cloud data using the first rotation matrix to obtain the third point cloud data.

[0132] In step 701, the initial calibration of the road surface plane is achieved using the first transformation matrix. Due to the real-time changes in the road surface plane, the initial calibration achieves approximate parallelism. This means that the current road surface plane may still not be parallel to the horizontal plane formed by the horizontal coordinate axis in the spatial coordinate system where the point cloud data resides. Therefore, in this step, the second point cloud data is further rotated using the aforementioned first rotation matrix, making the current road surface plane parallel to the horizontal plane formed by the horizontal coordinate axis in the spatial coordinate system where the point cloud data resides. This achieves real-time calibration of the current road surface plane, automatically calibrating it in real time, eliminating errors caused by differences between the actual environment and the calibration environment, and improving the accuracy of curb detection. As the second point cloud data is rotated, the height of the point cloud data in the vertical coordinate system changes.

[0133] Step 705: Detect the curb in the road environment based on the third point cloud data.

[0134] In this embodiment, since the road surface plane in the third point cloud data is calibrated in real time and remains parallel to the horizontal plane formed by the horizontal coordinate axis in the spatial coordinate system where the point cloud data is located, it can adapt to the changing environment of the road surface, and the road edge detection is more accurate based on this.

[0135] In an exemplary embodiment, the step of extracting the current road surface plane from the second point cloud data is as follows: Figure 8 As shown, its specific implementation methods may include:

[0136] Step 801: Perform plane fitting on the second point cloud data to obtain multiple first candidate planes.

[0137] Specifically, the second point cloud data can be fitted with planes using the Random Sample Consensus (RANSAC) algorithm to obtain multiple planes, which can then be used as multiple first candidate planes. RANSAC is an iterative method for estimating mathematical model parameters using observed data points. Plane fitting can be achieved based on RANSAC; for details, please refer to relevant technical implementations, which will not be elaborated here.

[0138] Step 802: From the multiple first candidate planes, select the first candidate plane whose angle between the normal vector and the vertical coordinate axis in the spatial coordinate system satisfies the first angle range, and use it as the second candidate plane.

[0139] Here, the first included angle range can be 0 ± a degrees, where a can be 5. Since the vertical coordinate axis is perpendicular to the horizontal plane formed by the horizontal coordinate axis in the spatial coordinate system where the point cloud data is located, when the angle between the normal vector of the first candidate plane and the vertical coordinate axis (e.g., the Z-axis) satisfies the first included angle range, the first candidate plane is approximately parallel to the horizontal plane formed by the horizontal coordinate axis in the spatial coordinate system where the point cloud data is located. Thus, in this step, a second candidate plane that is approximately parallel to the horizontal plane formed by the horizontal coordinate axis in the spatial coordinate system where the point cloud data is located can be selected.

[0140] Step 803: Generate a target point based on the second point cloud data. On the horizontal coordinate axis, the coordinate value of the target point is the average value of the coordinate values ​​of the second point cloud data. On the vertical coordinate axis, the coordinate value of the target point is higher than the coordinate value of the second point cloud data.

[0141] The target point here is a reference point used to select the current road surface.

[0142] In the spatial coordinate system of the point cloud data, the coordinates of the target point include the coordinates of the horizontal axis and the vertical axis. The horizontal coordinate of the target point can be the average of the horizontal coordinates of the second point cloud data. The vertical coordinate of the target point can also be the average of the vertical coordinates of the second point cloud data. If the spatial coordinate system is OXYZ, then the X-axis coordinate of the target point P is the average of the X-axis coordinates (Xave) of the second point cloud data, and the Y-axis coordinate is the average of the Y-axis coordinates (Yave) of the second point cloud data. On the vertical axis, the target point can be higher than the second point cloud data; the coordinates of the target point can be set to be higher than the coordinates of the second point cloud data. Thus, the target point is directly above the second point cloud data. For example, the vertical coordinates of the target point can be set to the height of the fused sensor above the ground, for example, 1m. The target point P is (Xave, Yave, 1).

[0143] Step 804: Select the second candidate plane that is furthest from the target point as the current road surface plane.

[0144] In this step, the current road surface is selected as the second candidate plane furthest from the target point. Since the target point is directly above the second point cloud data and relatively far from it, it is less susceptible to interference from non-parallel second candidate planes. Non-parallel second candidate planes create intersections, leading to mislocalization of the current road surface, thus improving the accuracy of the extracted road surface. Furthermore, selecting the second candidate plane furthest from the target point removes interference from other parallel second candidate planes, ensuring stable extraction of the current road surface.

[0145] In this embodiment, multiple first candidate planes are obtained by performing plane fitting on the second point cloud data. Then, from the multiple first candidate planes, the first candidate plane whose angle between the normal vector and the vertical coordinate axis in the spatial coordinate system satisfies the first angle range is selected, thereby obtaining a second candidate plane that is approximately parallel to the horizontal plane formed by the horizontal coordinate axis in the spatial coordinate system where the point cloud data is located. Furthermore, a target point is generated directly above the second point cloud data. The second candidate plane that is farthest from the target point is taken as the current road surface plane. This not only removes the interference of the second candidate planes not being parallel but also removes the interference of other second candidate planes that are parallel to the road surface plane, thereby accurately obtaining the current road surface plane.

[0146] In an exemplary embodiment, the detection of road edges in the road environment based on the third point cloud data, such as... Figure 9 As shown, its specific implementation methods may include:

[0147] Step 901: From the third point cloud data, obtain the fourth point cloud data that meets the road surface height range and the fifth point cloud data that meets the roadside height range, wherein the roadside height range is the height range of the side of the roadside perpendicular to the road surface.

[0148] The curb side refers to the side of the curb perpendicular to the road surface. In the third point cloud data set, some point cloud data corresponds to the road surface; see [link to relevant documentation]. Figure 10 The left side shows point cloud data at lower elevations. Some of the point cloud data corresponds to the side of the curb perpendicular to the road surface. See [link / reference]. Figure 10 The right side shows point cloud data with higher elevations, etc. In this step, a fourth point cloud data that meets the road surface height range and a fifth point cloud data that meets the roadside elevation range are obtained from the third point cloud data. In this way, direct filtering is achieved, filtering out point cloud data that is irrelevant to the road surface and roadside, resulting in cleaner point cloud data for sparse point cloud data.

[0149] Step 902: Based on the fourth point cloud data, obtain the first target plane.

[0150] Specifically, obtaining the first target plane based on the fourth point cloud data includes:

[0151] Plane fitting is performed on the fourth point cloud data to obtain multiple third candidate planes;

[0152] From a plurality of third candidate planes, the third candidate plane whose angle between the normal vector and the vertical coordinate axis in the spatial coordinate system satisfies the second angle range is selected as the first target plane.

[0153] In practice, the fourth point cloud data can be fitted with a plane based on RANSAC to obtain multiple third candidate planes.

[0154] The second included angle range here can be 0 ± b degrees. This range can be the same as or different from the first included angle range. For example, b can be 5. Since the vertical coordinate axis is perpendicular to the horizontal plane formed by the horizontal coordinate axis in the spatial coordinate system where the point cloud data is located, when the angle between the normal vector of the third candidate plane and the vertical coordinate axis (e.g., the Z-axis) satisfies the second included angle range, the third candidate plane is approximately parallel to the horizontal plane formed by the horizontal coordinate axis in the spatial coordinate system where the point cloud data is located. Then, from multiple third candidate planes, the third candidate plane whose angle between the normal vector and the vertical axis in the spatial coordinate system satisfies the second included angle range is selected, thus obtaining a third candidate plane that is approximately parallel to the horizontal plane formed by the horizontal axis in the spatial coordinate system where the point cloud data is located, which serves as the first target plane. This allows for the filtering out of noise planes. Due to the pass-through filtering, a first target plane is generally obtained. This first target plane can represent the road surface.

[0155] Step 903: Based on the fifth point cloud data, obtain the second target plane.

[0156] Specifically, obtaining the second target plane based on the fifth point cloud data includes:

[0157] Plane fitting is performed on the fifth point cloud data to obtain multiple fourth candidate planes;

[0158] From the plurality of fourth candidate planes, the fourth candidate plane whose angle between the normal vector and the vertical coordinate axis satisfies the third angle range is selected as the second target plane.

[0159] In practice, the fifth point cloud data can be fitted with a plane based on RANSAC to obtain multiple fourth candidate planes.

[0160] The third included angle range here can be 90 ± c degrees, where c can be 5. Since the vertical coordinate axis is perpendicular to the horizontal plane formed by the horizontal coordinate axis in the spatial coordinate system where the point cloud data is located, when the angle between the normal vector of the fourth candidate plane and the vertical coordinate axis (e.g., the Z-axis) satisfies the third included angle range, the fourth candidate plane is approximately perpendicular to the horizontal plane formed by the horizontal coordinate axis in the spatial coordinate system where the point cloud data is located. Then, from multiple fourth candidate planes, the fourth candidate plane whose angle between the normal vector and the vertical coordinate axis in the spatial coordinate system satisfies the third included angle range is selected, thus obtaining a fourth candidate plane that is approximately perpendicular to the horizontal plane formed by the horizontal coordinate axis in the spatial coordinate system where the point cloud data is located, as the second target plane. In this way, noise planes can be filtered out. And because of the pass-through filtering, a second target plane can generally be obtained. The second target plane can represent the side of the curb.

[0161] Step 904: Based on the intersection of the first target plane and the second target plane, obtain the curb in the road environment.

[0162] Since the first target plane represents the road surface and the second target plane represents the side of the curb, the intersection of the first and second target planes can represent the curb. Therefore, in this step, the intersection of the first and second target planes can be solved based on the equations of the first and second target planes, and this intersection serves as the detected curb of the road environment. The solved intersection is a three-dimensional straight line. In this way, curb detection is achieved through the intersection of surfaces.

[0163] In this embodiment, fourth point cloud data satisfying the road surface height range and fifth point cloud data satisfying the roadside side height range can be obtained from the third point cloud data. This achieves direct-pass filtering, ensuring the purity of the point cloud data when fitting the plane under sparse point cloud conditions. This allows for accurate extraction of point cloud data from the road surface and roadside sides. Furthermore, a first target plane representing the road surface and a second target plane representing the roadside sides are obtained. The intersection of the first and second target planes is used to detect the roadside, which is stable and accurate, and reduces the impact of inherent accuracy differences in the lidar on the results. Moreover, because the amount of point cloud data required for plane fitting is extremely small, it is applicable to sparse point cloud data. Furthermore, plane fitting is based on all information, effectively filtering the point cloud data output, thus improving data stability.

[0164] It should be noted that the first transformation matrix, the second transformation matrix, and the camera intrinsic parameters can all be pre-stored in the configuration file. When curb detection is initiated, the configuration file and the semantic segmentation model can be read.

[0165] The present invention provides a curb detection system, which can be referred to in correspondence with the curb detection method described above.

[0166] This embodiment provides a curb detection system, including:

[0167] Image acquisition device, used to acquire images of the road environment;

[0168] LiDAR, used to collect point cloud data of the road environment;

[0169] The controller is connected to the image acquisition device and the lidar, respectively, and the controller is used to execute the curb detection method provided in any of the above embodiments.

[0170] In this embodiment, the specific implementation of the curb detection method can be referred to the above embodiments, and will not be repeated here.

[0171] Figure 11 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 11 As shown, the electronic device may include: a processor 1110, a communications interface 1120, a memory 1130, and a communication bus 1140, wherein the processor 1110, the communications interface 1120, and the memory 1130 communicate with each other via the communication bus 1140. The processor 1110 can call logical instructions in the memory 1130 to execute an edge detection method, which includes:

[0172] Extract candidate curbs from images of the road environment;

[0173] The point cloud data of the road environment is projected onto the pixel coordinate system to obtain the first image;

[0174] Determine a target region in the first image, the target region comprising the region corresponding to the candidate road edge;

[0175] Obtain the point cloud data corresponding to the target area from the point cloud data of the road environment, and use it as the first point cloud data;

[0176] Based on the first point cloud data, the curb in the road environment is detected.

[0177] Furthermore, the logical instructions in the aforementioned memory 1130 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0178] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the curb detection method provided by the above methods, the method comprising:

[0179] Extract candidate curbs from images of the road environment;

[0180] The point cloud data of the road environment is projected onto the pixel coordinate system to obtain the first image;

[0181] Determine a target region in the first image, the target region comprising the region corresponding to the candidate road edge;

[0182] Obtain the point cloud data corresponding to the target area from the point cloud data of the road environment, and use it as the first point cloud data;

[0183] Based on the first point cloud data, the curb in the road environment is detected.

[0184] In another aspect, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the aforementioned curb detection methods, the method comprising:

[0185] Extract candidate curbs from images of the road environment;

[0186] The point cloud data of the road environment is projected onto the pixel coordinate system to obtain the first image;

[0187] Determine a target region in the first image, the target region comprising the region corresponding to the candidate road edge;

[0188] Obtain the point cloud data corresponding to the target area from the point cloud data of the road environment, and use it as the first point cloud data;

[0189] Based on the first point cloud data, the curb in the road environment is detected.

[0190] The present invention also provides a vehicle, including a vehicle body, wherein the vehicle body is provided with a curb detection system as provided in any of the above embodiments, or for executing road electronic equipment provided in any of the above embodiments, or includes a computer program product provided in any of the above embodiments, or includes a computer-readable storage medium provided in any of the above embodiments.

[0191] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0192] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0193] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A curb detection method characterized by, include: Extract candidate curbs from images of the road environment; The point cloud data of the road environment is projected onto the pixel coordinate system to obtain the first image; A target region in the first image is determined. The target region is obtained by expanding the region corresponding to the candidate road edge in the row direction of the first image by pixels based on the region in the first image. Obtain the point cloud data corresponding to the target area from the point cloud data of the road environment, and use it as the first point cloud data; Based on the first point cloud data, detect the curb in the road environment; The step of detecting road edges in the road environment based on the first point cloud data includes: The first point cloud data is transformed using a first transformation matrix to obtain the second point cloud data; wherein, the first transformation matrix is ​​used to transform the calibrated road surface plane in the spatial coordinate system where the point cloud data is located to coincide with the horizontal plane formed by the horizontal coordinate axis in the spatial coordinate system; Extract the current road surface plane from the second point cloud data; Determine a first rotation matrix to rotate the current road surface plane to be parallel to the horizontal plane; The second point cloud data is rotated using the first rotation matrix to obtain the third point cloud data; From the third point cloud data, obtain the fourth point cloud data that meets the road surface height range and the fifth point cloud data that meets the roadside height range, wherein the roadside height range is the height range of the side of the roadside that is perpendicular to the road surface. Based on the fourth point cloud data, the first target plane is obtained; Based on the fifth point cloud data, the second target plane is obtained; Based on the intersection of the first target plane and the second target plane, the curb in the road environment is obtained.

2. The curb detection method according to claim 1, characterized by, Extracting the current road surface plane from the second point cloud data includes: Plane fitting is performed on the second point cloud data to obtain multiple first candidate planes; From a plurality of first candidate planes, select the first candidate plane whose angle between the normal vector and the vertical coordinate axis in the spatial coordinate system satisfies the first angle range, and use it as the second candidate plane; A target point is generated based on the second point cloud data. On the horizontal coordinate axis, the coordinate value of the target point is the average value of the coordinate values ​​of the second point cloud data. On the vertical coordinate axis, the coordinate value of the target point is higher than the coordinate value of the second point cloud data. The second candidate plane that is furthest from the target point is taken as the current road surface plane.

3. The curb detection method according to claim 1, characterized in that, The step of obtaining the first target plane based on the fourth point cloud data includes: Plane fitting is performed on the fourth point cloud data to obtain multiple third candidate planes; From a plurality of third candidate planes, the third candidate plane whose angle between the normal vector and the vertical coordinate axis in the spatial coordinate system satisfies the second angle range is selected as the first target plane; The step of obtaining the second target plane based on the fifth point cloud data includes: Plane fitting is performed on the fifth point cloud data to obtain multiple fourth candidate planes; From the plurality of fourth candidate planes, the fourth candidate plane whose angle between the normal vector and the vertical coordinate axis satisfies the third angle range is selected as the second target plane.

4. The curb detection method according to claim 1, characterized in that, The first transformation matrix was obtained in the following way: Using the pitch angle and roll angle required to transform the calibrated road surface plane to coincide with the horizontal plane as variables, the optimal solutions for the pitch angle and roll angle are solved based on the particle swarm optimization algorithm; The first transformation matrix is ​​obtained based on the optimal solutions for the pitch angle and the roll angle.

5. The curb detection method according to any one of claims 1 to 4, characterized in that, Extracting candidate curbs from images of the road environment includes: Based on the image of the road environment, a second image including the road surface area and the non-road surface area is obtained; In the row direction of the second image, the second image is divided into two sub-images to obtain two regions of interest; Edge detection is performed in each region of interest, and the candidate curbs are extracted from the edge detection results.

6. The curb detection method according to any one of claims 1 to 4, characterized in that, The image of the road environment is acquired by an image acquisition device, and the step of projecting the point cloud data of the road environment onto the pixel coordinate system includes: Using the second transformation matrix and the intrinsic parameters of the image acquisition device, the point cloud data of the road environment is projected onto the pixel coordinate system. The second transformation matrix is ​​the transformation matrix between the coordinate system of the image acquisition device and the spatial coordinate system where the point cloud data is located. The second transformation matrix is ​​obtained by solving point cloud data samples and image samples. The two sides of the image sample contain calibration plates.

7. A curb detection system, characterized in that, include: Image acquisition device, used to acquire images of the road environment; LiDAR, used to collect point cloud data of the road environment; The controller is connected to the image acquisition device and the lidar respectively, and the controller is used to execute the curb detection method as described in any one of claims 1 to 6.

8. A vehicle, comprising a vehicle body, characterized in that, The vehicle body is equipped with the curb detection system as described in claim 7.

Images