Method and device for determining road surface unevenness risk area and storage medium

Abstract

The invention discloses a method and device for determining a road surface unevenness risk area and a storage medium, and belongs to the technical field of road surface evenness detection.The method comprises the steps that initial point cloud data of a detected road surface are acquired; based on a predetermined calibration relation between a point cloud coordinate system and a three-dimensional detection coordinate system, the initial point cloud data is converted into the three-dimensional detection coordinate system to obtain target point cloud data, the transverse axis and the longitudinal axis of the three-dimensional detection coordinate system are parallel to the detected road surface, and the vertical axis of the three-dimensional detection coordinate system is perpendicular to the detected road surface; determining an elevation map of the detected pavement according to the target point cloud data; and determining an uneven risk area of the detected pavement according to the elevation map. According to the detection method based on the point cloud, the point cloud is converted into the elevation map to determine the uneven risky road surface, and the detection method is not affected by environmental factors such as illumination and is high in accuracy.

Description

Technical Field

[0001] The present application relates to the technical field of road surface flatness detection, and in particular to a method, device and storage medium for determining a risk area of ​​road surface unevenness. Background Art

[0002] With the rapid development of automatic assisted driving technology, the detection of uneven areas such as potholes / bumps on the road has become an important part of ensuring driving safety and comfort. Such uneven areas not only affect the driving stability of the vehicle, but may also cause accidents and vehicle damage. In this regard, most of the current existing technologies use 2D visual detection methods to detect raised obstacles on the ground and unsafe areas such as potholes, but such methods require huge samples, are not robust to new scenes, and the accuracy of the detection results is affected by environmental factors such as light. Therefore, the existing technology has the problem of low accuracy. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a method, device and machine-readable storage medium for determining risk areas of uneven road surfaces, so as to solve the problem of low accuracy in the prior art.

[0004] In order to achieve the above-mentioned purpose, a first aspect of an embodiment of the present application provides a method for determining a risk area for uneven road surface, the method comprising:

[0005] Obtaining initial point cloud data of the road surface being inspected;

[0006] Based on the predetermined calibration relationship between the point cloud coordinate system and the three-dimensional detection coordinate system, the initial point cloud data is converted into the three-dimensional detection coordinate system to obtain the target point cloud data, wherein the horizontal axis and the vertical axis of the three-dimensional detection coordinate system are parallel to the road surface being detected, and the vertical axis of the three-dimensional detection coordinate system is perpendicular to the road surface being detected;

[0007] Determine the elevation map of the road surface being inspected based on the target point cloud data;

[0008] Determine the uneven risk areas of the inspected road surface based on the elevation map.

[0009] In an embodiment of the present application, the elevation map includes a plurality of pixel grids that are gridded according to a preset resolution; determining the uneven risk area of ​​the detected road surface according to the elevation map includes: determining a target gradient corresponding to each pixel grid according to a target point cloud in each pixel grid; determining a target pixel grid set according to the target gradient of each pixel grid and a preset gradient threshold, wherein the target gradient corresponding to the target pixel grid in the target pixel grid set is greater than the preset gradient threshold, or there is no target point cloud in the target pixel grid;

[0010] Clustering the target pixel grids in the pixel grid set to obtain corresponding region clusters;

[0011] The local area of ​​the detected road surface corresponding to the area cluster is determined as the uneven risk area.

[0012] In an embodiment of the present application, a target gradient corresponding to each pixel grid is determined according to a target point cloud within each pixel grid, including: based on a first preset gradient convolution kernel, determining a first gradient of the pixel grid in the horizontal coordinate direction according to the three-dimensional coordinates of the target point cloud within the pixel grid; based on a second preset gradient convolution kernel, determining a second gradient of the pixel grid in the vertical coordinate direction according to the three-dimensional coordinates of the target point cloud within the pixel grid; and fusing the first gradient and the second gradient to obtain a target gradient corresponding to the pixel grid.

[0013] In an embodiment of the present application, the method also includes: for any uneven risk area, according to the three-dimensional coordinates of the target point cloud corresponding to the uneven risk area, based on the predetermined calibration relationship between the camera coordinate system and the three-dimensional detection coordinate system, determining the three-dimensional bounding box of the uneven risk area in the camera coordinate system; and displaying the three-dimensional bounding box in the two-dimensional image of the detected road surface.

[0014] In an embodiment of the present application, according to the three-dimensional coordinates of the target point cloud corresponding to the uneven risk area, based on the predetermined calibration relationship between the camera coordinate system and the three-dimensional detection coordinate system, a three-dimensional bounding box of the uneven risk area in the camera coordinate system is determined, including: according to the three-dimensional coordinates of the target point cloud corresponding to the uneven risk area, determining the center point and dimension lengths of the uneven risk area in the three-dimensional detection coordinate system, the dimensions including the horizontal axis dimension length, the longitudinal axis dimension length and the vertical axis dimension length; based on the calibration relationship between the camera coordinate system and the three-dimensional detection coordinate system, determining the three-dimensional bounding box of the uneven risk area in the camera coordinate system according to the center point and the dimension lengths.

[0015] In an embodiment of the present application, based on the calibration relationship between the camera coordinate system and the three-dimensional detection coordinate system, a three-dimensional bounding box of the uneven risk area in the camera coordinate system is determined according to the center point and the dimension length, including: determining a three-dimensional coordinate set of multiple vertices of the three-dimensional box according to the center point and the dimension length; based on the calibration relationship between the camera coordinate system and the three-dimensional detection coordinate system, according to the three-dimensional coordinate set, converting the multiple vertices to the camera coordinate system to obtain a three-dimensional bounding box of the uneven risk area in the camera coordinate system.

[0016] In an embodiment of the present application, displaying a three-dimensional bounding box in a two-dimensional image of the detected road surface includes: projecting the three-dimensional bounding box to a pixel coordinate system in the two-dimensional image according to a camera projection transformation, so that the three-dimensional bounding box is displayed in the two-dimensional image.

[0017] A second aspect of an embodiment of the present application provides a processor configured to execute the above-mentioned method for determining a risk area for uneven road surface.

[0018] A third aspect of an embodiment of the present application provides a device for determining a risk area for uneven road surface, the device comprising:

[0019] A point cloud acquisition module is used to obtain initial point cloud data of the road surface being inspected;

[0020] A coordinate conversion module, for converting the initial point cloud data into the three-dimensional detection coordinate system based on the predetermined calibration relationship between the point cloud coordinate system and the three-dimensional detection coordinate system to obtain target point cloud data, wherein the horizontal axis and the vertical axis of the three-dimensional detection coordinate system are parallel to the road surface being detected, and the vertical axis of the three-dimensional detection coordinate system is perpendicular to the road surface being detected;

[0021] An elevation map determination module, used to determine the elevation map of the road being inspected based on the target point cloud data;

[0022] The detection result determination module is used to determine the uneven risk area of ​​the detected road surface based on the elevation map.

[0023] A fourth aspect of an embodiment of the present application provides a machine-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the above-mentioned method for determining the risk area of ​​uneven road surface is implemented.

[0024] The above technical solution obtains the initial point cloud data of the road to be inspected, and then converts the initial point cloud data into a three-dimensional detection coordinate system based on the predetermined calibration relationship between the point cloud coordinate system and the three-dimensional detection coordinate system to obtain the target point cloud data. The horizontal and vertical axes of the three-dimensional detection coordinate system are parallel to the road to be inspected, and the vertical axis of the three-dimensional detection coordinate system is perpendicular to the road to be inspected. Then, the elevation map of the road to be inspected is determined based on the target point cloud data, and finally, the uneven risk area of ​​the road to be inspected is determined based on the elevation map. The present application is based on a point cloud-based detection method, which converts point clouds into elevation maps to determine uneven risky roads. This detection method is not affected by environmental factors such as light and has high accuracy.

[0025] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:

[0027] Figure 1 A schematic flow chart of a method for determining a risk area for uneven road surface provided in an embodiment of the present application;

[0028] Figure 2A schematic diagram of a sensor detection range provided in a specific embodiment of the present application;

[0029] Figure 3 A schematic diagram of a vehicle detection coordinate system provided in a specific embodiment of the present application;

[0030] Figure 4 A schematic diagram of an elevation coordinate system for a bird's-eye view of a detection area provided in an embodiment of the present application;

[0031] Figure 5 A schematic diagram of a bird's-eye view vehicle detection coordinate system and an elevation map coordinate system provided in a specific embodiment of the present application;

[0032] Figure 6 A schematic diagram of the structure of a device for determining risk areas of uneven road surfaces provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0034] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back...), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0035] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0036] Figure 1 A flow chart of a method for determining a risk area of ​​uneven road surface provided in an embodiment of the present application. Figure 1As shown, an embodiment of the present application provides a method for determining a risk area for uneven road surface, which is described by taking the method applied to a processor as an example. The method may include the following steps.

[0037] Step S101, obtaining initial point cloud data of the road surface to be detected.

[0038] Specifically, the initial point cloud data of the inspected road surface can be acquired using a point cloud acquisition device, such as a laser scanner, radar, or stereo camera. The number of point cloud acquisition devices in the embodiment of the present application may be one or more. It can be understood that after the point cloud acquisition device is installed, the area size of the inspected road surface is also limited accordingly, and each point on the road surface of the area of ​​this specific size has a point cloud falling. Compared with the traditional method of using 2D visual detection, the present application implements the collection of point cloud data of the monitored road surface, which is not affected by environmental factors such as light and has higher accuracy.

[0039] Step S102, based on the predetermined calibration relationship between the point cloud coordinate system and the three-dimensional detection coordinate system, the initial point cloud data is converted into the three-dimensional detection coordinate system to obtain the target point cloud data, the horizontal axis and the vertical axis of the three-dimensional detection coordinate system are parallel to the road surface being detected, and the vertical axis of the three-dimensional detection coordinate system is perpendicular to the road surface being detected.

[0040] It can be understood that since the point cloud coordinate system where the initial point cloud data is located is determined by the installation position of the point cloud acquisition device, for the convenience of calculation, the initial point cloud data can be converted into a three-dimensional detection coordinate system for subsequent analysis and processing. Specifically, the initial point cloud data can be converted into a three-dimensional detection coordinate system based on a predetermined calibration relationship between the point cloud coordinate system and the three-dimensional detection coordinate system using mathematical transformations (such as rotation and translation). With the flat area of ​​the road being inspected as a reference, the three-dimensional detection coordinate system is usually set with the horizontal and vertical axes parallel to the road being inspected, and the vertical axis perpendicular to the road being inspected, to facilitate subsequent elevation analysis and unevenness detection.

[0041] Step S103, determining the elevation map of the detected road surface according to the target point cloud data.

[0042] It can be understood that an elevation map is a graph that represents the change in road height, which can intuitively reflect the ups and downs of an area. Specifically, by interpolating or fitting the target point cloud data, a continuous elevation surface can be obtained. Then, this elevation surface is represented in a graphical manner to obtain an elevation map. In one example, each point in the elevation map is associated with an elevation value, and the elevation value can be determined according to the vertical coordinate of the point cloud coordinate of the point cloud corresponding to each point.

[0043] Step S104, determining the uneven risk area of ​​the detected road surface according to the elevation map.

[0044] Specifically, by analyzing the elevation map based on the elevation value, areas with large height changes can be identified, which are areas where there may be unevenness risks. In one example, each point in the elevation map is associated with an elevation value, and the elevation value can be determined according to the vertical coordinate of the point cloud coordinates of the point cloud corresponding to each point. Based on the elevation values ​​of each point in the elevation map, the uneven risk areas in the elevation map can be analyzed by setting thresholds, including sunken risk areas and convex risk areas. For example, a first height threshold for detecting sunken risk areas and a second height threshold for detecting convex risk areas are set, and the area formed by each point with an elevation value less than the first height threshold is determined as a sunken risk area, and the area formed by each point with an elevation value greater than the second height threshold is determined as a convex risk area. The sunken risk area and the convex risk area are combined to obtain the uneven risk area of ​​the detected road surface. In another example, a cluster analysis method can be used to determine the risk area of ​​the detected road surface based on the elevation map.

[0045] The above technical solution obtains the initial point cloud data of the road to be inspected, and then converts the initial point cloud data into a three-dimensional detection coordinate system based on the predetermined calibration relationship between the point cloud coordinate system and the three-dimensional detection coordinate system to obtain the target point cloud data. The horizontal and vertical axes of the three-dimensional detection coordinate system are parallel to the road to be inspected, and the vertical axis of the three-dimensional detection coordinate system is perpendicular to the road to be inspected. Then, the elevation map of the road to be inspected is determined based on the target point cloud data, and finally, the uneven risk area of ​​the road to be inspected is determined based on the elevation map. The present application is based on a point cloud-based detection method, which converts point clouds into elevation maps to determine uneven risky roads. This detection method is not affected by environmental factors such as light and has high accuracy.

[0046] In an embodiment of the present application, the elevation map includes a plurality of pixel grids that are gridded according to a preset resolution; determining the uneven risk area of ​​the detected road surface based on the elevation map may include: determining a target gradient corresponding to each pixel grid based on a target point cloud in each pixel grid; determining a target pixel grid set based on the target gradient of each pixel grid and a preset gradient threshold, wherein the target gradient corresponding to the target pixel grid in the target pixel grid set is greater than the preset gradient threshold, or the target point cloud does not exist in the target pixel grid; clustering the target pixel grids in the pixel grid set to obtain corresponding area clusters; and determining a local area of ​​the detected road surface corresponding to the area cluster as an uneven risk area.

[0047] It can be understood that in order to improve the accuracy of the detection results, the area corresponding to the detected road surface can be gridded, the resolution can be set based on the size of the area of ​​the detected road surface, and the elevation map can be gridded according to the preset resolution to obtain a gridded elevation map that coincides with the area of ​​the detected road surface. The elevation map includes multiple pixel grids that are gridded according to the preset resolution. The size of each pixel grid is the same. Each pixel grid represents a specific area on the detected road surface and contains point cloud information of the point cloud falling within the area. It can be understood that the resolution of the grid determines the fineness of the elevation map. The higher the resolution, the smaller the area represented by each pixel grid, and the more detailed the information contained in the elevation map. Therefore, the resolution can be set according to the actual situation.

[0048] Further, the target gradient corresponding to each pixel grid can be determined according to the target point cloud in each pixel grid. The target point cloud is the point cloud data after conversion to the three-dimensional detection coordinate system. The target gradient refers to the height change rate of the point cloud in the pixel grid, which can be used to reflect the slope or unevenness of the road surface area corresponding to the pixel grid in different directions. It can be understood that in order to identify the road surface area that may have the risk of unevenness, the embodiment of the present application sets a preset gradient threshold, which is a safety threshold for the height of the road surface bulge. The preset gradient threshold is determined according to the actual situation of the road surface and the detection requirements. Then, based on the preset gradient threshold, the pixel grids in the elevation map are screened, and those pixel grids whose target gradient is greater than the preset gradient threshold, or those pixel grids whose data is missing or abnormal due to the deep road depression, so that there is no target point cloud inside, are screened out to form a target pixel grid set. At this time, the road surface areas corresponding to the target pixel grids in the target pixel grid set all have depressions or bulges, and exceed the safety value.

[0049] Next, in order to obtain a complete uneven risk area, the pixel grids in the target pixel grid set can be clustered, and adjacent target pixel grids can be clustered together to obtain one or more regional clusters. Each regional cluster corresponds to an uneven risk area of ​​the detected road surface, and there is a certain distance between the regional clusters. In one example, the Euclidean clustering algorithm can be used for clustering.

[0050] In this way, accurate detection of uneven risk areas on the detected road surface is achieved through the steps of gridding of elevation maps, determination of target gradients, screening of target pixel grid sets, clustering processing, and determination of uneven risk areas.

[0051] In an embodiment of the present application, determining the target gradient corresponding to each pixel grid according to the target point cloud within each pixel grid may include: based on a first preset gradient convolution kernel, determining the first gradient of the pixel grid in the horizontal coordinate direction according to the three-dimensional coordinates of the target point cloud within the pixel grid; based on a second preset gradient convolution kernel, determining the second gradient of the pixel grid in the vertical coordinate direction according to the three-dimensional coordinates of the target point cloud within the pixel grid; and fusing the first gradient and the second gradient to obtain the target gradient corresponding to the pixel grid.

[0052] It can be understood that the first preset gradient convolution kernel is used to calculate the gradient of the pixel grid in the horizontal coordinate direction, and the second preset gradient convolution kernel is used to calculate the gradient of the pixel grid in the vertical coordinate direction. The size and weight of the convolution kernel can be determined according to the requirements of road surface unevenness detection. Specifically, for each pixel grid, in order to determine the target gradient corresponding to the pixel grid, the first preset gradient convolution kernel is first applied to the three-dimensional coordinate data of the target point cloud in the pixel grid, that is, the first preset gradient convolution kernel is slid on the point cloud data, and the gradient value at each position is calculated to obtain the first gradient, which reflects the height change rate between adjacent points in the pixel grid in the horizontal coordinate direction. Similarly, the second preset gradient convolution kernel is applied to the three-dimensional coordinate data of the target point cloud in the pixel grid, and the gradient value at each position is calculated to obtain the second gradient, which reflects the height change rate between adjacent points in the vertical coordinate direction. After obtaining the first gradient and the second gradient, they need to be fused to obtain the target gradient corresponding to the pixel grid. In an example, the fusion method can be a simple weighted summation, taking the maximum value, taking the average value, etc., depending on the detection requirements and data characteristics. Through fusion processing, a target gradient that comprehensively reflects the height change rate of the pixel grid in the horizontal and vertical directions can be obtained.

[0053] In this way, the embodiment of the present application introduces the first preset gradient convolution kernel and the second preset gradient convolution kernel to calculate the gradient of the pixel grid in the horizontal and vertical directions respectively, and fuses the two gradients, thereby realizing the accurate calculation of the target gradient corresponding to the pixel grid. This process provides important data support for the subsequent clustering processing and the determination of uneven risk areas.

[0054] In an embodiment of the present application, the method may further include: for any uneven risk area, according to the three-dimensional coordinates of the target point cloud corresponding to the uneven risk area, based on a predetermined calibration relationship between the camera coordinate system and the three-dimensional detection coordinate system, determining a three-dimensional bounding box of the uneven risk area in the camera coordinate system; and displaying the three-dimensional bounding box in the two-dimensional image of the detected road surface.

[0055] It can be understood that in some application scenarios, it is necessary to display the uneven risk area in three dimensions. For example, in the field of autonomous driving, it is necessary to determine the three-dimensional bounding box corresponding to the uneven risk area and determine its position in the two-dimensional image of the detected road surface so that the system can perform obstacle avoidance processing. Specifically, for any uneven risk area detected, the three-dimensional coordinates of the target point cloud corresponding to the uneven risk area are obtained, and based on the predetermined calibration relationship between the camera coordinate system and the three-dimensional detection coordinate system, the three-dimensional bounding box of the uneven risk area in the camera coordinate system is calculated. The calibration relationship between the camera coordinate system and the three-dimensional detection coordinate system is obtained through a series of complex calibration processes, which can ensure the accurate conversion from the three-dimensional detection coordinate system to the camera coordinate system, which will not be elaborated here.

[0056] Furthermore, in order to display the three-dimensional bounding box in the two-dimensional image, the three-dimensional bounding box can be displayed in the two-dimensional image of the road being detected based on the camera intrinsic parameters. In one example, the three-dimensional bounding box can be projected onto the two-dimensional image plane through camera projection transformation to obtain a two-dimensional bounding box corresponding to the three-dimensional bounding box, and the two-dimensional bounding box can accurately reflect the position and size of the uneven risk area in the two-dimensional image, and then the two-dimensional bounding box is superimposed on the two-dimensional image of the road being detected, so as to intuitively display the position of the uneven risk area in the road being detected.

[0057] Thus, the embodiments of the present application provide intuitive and accurate information about uneven risk areas through precise spatial positioning and visualization technology. This is of great significance to the fields of road maintenance, autonomous driving, video surveillance, etc., because it helps to timely discover and deal with potential road problems, thereby ensuring driving safety and smooth roads.

[0058] In an embodiment of the present application, according to the three-dimensional coordinates of the target point cloud corresponding to the uneven risk area, based on the predetermined calibration relationship between the camera coordinate system and the three-dimensional detection coordinate system, determining the three-dimensional bounding box of the uneven risk area in the camera coordinate system may include: determining the center point and dimension lengths of the uneven risk area in the three-dimensional detection coordinate system according to the three-dimensional coordinates of the target point cloud corresponding to the uneven risk area, the dimensions including the horizontal axis dimension length, the longitudinal axis dimension length and the vertical axis dimension length; based on the calibration relationship between the camera coordinate system and the three-dimensional detection coordinate system, determining the three-dimensional bounding box of the uneven risk area in the camera coordinate system according to the center point and the dimension lengths.

[0059] It can be understood that in order to further provide accurate spatial positioning information in the detection of uneven risk areas, the three-dimensional bounding box of the uneven risk area in the camera coordinate system can be determined. Among them, the camera coordinate system refers to the coordinate system corresponding to the camera used to collect images of the detected area, which is determined by the installation position of the camera. Specifically, according to the three-dimensional coordinate data of the target point cloud corresponding to the uneven risk area, the maximum and minimum values ​​of the area in each coordinate axis direction are determined, and the center point of the uneven risk area in the three-dimensional detection coordinate system can be obtained, and the dimensional length of the area on the horizontal axis, longitudinal axis and vertical axis of the three-dimensional detection coordinate system can be calculated.

[0060] Furthermore, based on the predetermined calibration relationship between the camera coordinate system and the three-dimensional detection coordinate system, the center point and dimension length of the uneven risk area in the three-dimensional detection coordinate system are converted to the camera coordinate system. Finally, a three-dimensional bounding box is determined in the camera coordinate system based on the converted center point and dimension length. This bounding box accurately defines the position and size of the uneven risk area in the camera field of view.

[0061] In this way, a more intuitive and accurate spatial positioning information can be provided, providing important spatial positioning information for subsequent road repair, maintenance, and related machine vision applications.

[0062] In an embodiment of the present application, based on the calibration relationship between the camera coordinate system and the three-dimensional detection coordinate system, a three-dimensional bounding box of the uneven risk area in the camera coordinate system is determined according to the center point and the dimension length, which may include: determining a three-dimensional coordinate set of multiple vertices of the three-dimensional box according to the center point and the dimension length; based on the calibration relationship between the camera coordinate system and the three-dimensional detection coordinate system, according to the three-dimensional coordinate set, converting the multiple vertices to the camera coordinate system to obtain a three-dimensional bounding box of the uneven risk area in the camera coordinate system.

[0063] Specifically, after determining the center point of the uneven risk area in the three-dimensional detection coordinate system and its dimension lengths on the horizontal axis, the vertical axis and the vertical axis, multiple vertices of a three-dimensional box surrounding the uneven risk area can be determined. In three-dimensional space, a rectangular box (or cube) consists of 8 vertices, which can be determined by the center point, the dimension length and the orientation of the box. Then, the three-dimensional coordinates of these vertices are combined to form a three-dimensional coordinate set, which contains all vertex information of the three-dimensional box in the three-dimensional detection coordinate system. Further, based on the predetermined calibration relationship between the camera coordinate system and the three-dimensional detection coordinate system, coordinate transformation is performed. For each vertex in the three-dimensional coordinate set, the three-dimensional coordinates of each vertex are multiplied by the rotation matrix in the calibration relationship between the camera coordinate system and the three-dimensional detection coordinate system, and the translation vector is added to obtain the new coordinates of the vertex in the camera coordinate system. Finally, all the converted vertices are combined to obtain the three-dimensional bounding box of the uneven risk area in the camera coordinate system. The three-dimensional bounding box can accurately define the position and size of the uneven risk area in the camera field of view.

[0064] In this way, by determining the three-dimensional coordinate set of multiple vertices of the three-dimensional box and converting these vertices to the camera coordinate system based on the calibration relationship between the camera coordinate system and the three-dimensional detection coordinate system, the three-dimensional bounding box of the uneven risk area in the camera coordinate system can be accurately obtained, which provides important spatial positioning information for subsequent road repair, maintenance and related machine vision applications.

[0065] In an embodiment of the present application, displaying a three-dimensional bounding box in a two-dimensional image of the detected road surface may include: projecting the three-dimensional bounding box to a pixel coordinate system in the two-dimensional image according to a camera projection transformation, so that the three-dimensional bounding box is displayed in the two-dimensional image.

[0066] In order to display the three-dimensional bounding box in the two-dimensional image, the three-dimensional bounding box can be displayed in the two-dimensional image of the road being inspected based on the camera intrinsic parameters. In one example, the three-dimensional bounding box can be projected onto the two-dimensional image plane through camera projection transformation to obtain a two-dimensional bounding box corresponding to the three-dimensional bounding box, which can accurately reflect the position and size of the uneven risk area in the two-dimensional image. Then, the two-dimensional bounding box is superimposed on the two-dimensional image of the road being inspected using image processing technology to intuitively display the position of the uneven risk area in the road being inspected.

[0067] It can be understood that the embodiments of the present application can have broad application prospects in the fields of road surface detection, autonomous driving, video surveillance, etc. It allows observers to quickly identify potential uneven road areas and take corresponding measures to ensure driving safety or perform road maintenance.

[0068] In a specific embodiment of the present application, the application scenario of the method for determining the risk area of ​​uneven road surface is described as an unmanned vehicle, and a point cloud sensor is used to collect point cloud data of the detection area. Figure 2 This is a schematic diagram of the sensor detection range provided in a specific embodiment of the present application. Figure 2 As shown, in this specific embodiment, the range detected by the point cloud sensor when it is installed is as shown in the figure, so that point clouds fall on the ground range in the detection area.

[0069] There are four coordinate systems involved in this specific embodiment. The first coordinate system is the vehicle detection coordinate system (i.e., the three-dimensional detection coordinate system in the above embodiment), denoted as detec ; The second coordinate system is the point cloud coordinate system of the point cloud sensor, denoted as O sensor ; The third is the elevation map coordinate system of the bird's-eye view of the detection area, denoted by O bev ; The fourth coordinate system is the camera coordinate system, denoted by O camera .

[0070] Figure 3 A schematic diagram of a vehicle detection coordinate system provided in a specific embodiment of the present application. Figure 3 As shown, O detec The coordinate system is a 3D coordinate system, and its posture is related to the direction of vehicle travel. Its y-axis is in the same direction as the direction of travel, the z-axis is vertically upward, and the x-axis is perpendicular to the y-axis to the right from a bird's-eye view. The origin is the projection position of the front center of the vehicle's travel direction on the ground.

[0071] O sensor The coordinate system is a 3D coordinate system and is only related to the location where the sensor is installed.

[0072] Figure 4 A schematic diagram of an elevation coordinate system for a bird's-eye view of a detection area provided in an embodiment of the present application. Figure 4 As shown, O bev The coordinate system is a 2D coordinate system from a bird's eye view, which is related to the detection area.

[0073] O camera It is the camera coordinate system, which is only related to the installation position of the camera.

[0074] It can be understood that after the vehicle structure design is completed, O detec With O sensor The external parameters (external parameters of the vehicle detection coordinate system and the point cloud coordinate system) are recorded as follows: and Among them, sensor Any point in the coordinate system, multiply on the left It can be obtained that the point is at O detecCoordinate values ​​in the coordinate system. And, calibrate the camera coordinate system O camera With the point cloud detection coordinate system O sensor The external parameters of . Its rotation and translation matrix is and Among them, sensor Any point in the coordinate system, multiply on the left It can be obtained that the point is at O camera The coordinate value in the coordinate system. Then the camera coordinate system O camera With O detec External reference:

[0075]

[0076] Among them, O detec Any point in the coordinate system, multiply on the left It can be obtained that the point is at O camera The coordinate value in the coordinate system. The above calibration can refer to the existing relevant calibration methods, which will not be described here.

[0077] Further, Figure 4 Take the detection area shown in the figure as an example. The detection area is a rectangle. First, define the detection range in the forward direction, set the horizontal detection range to [-L, L], and set the vertical detection range to [q, W]. Among them, q is the longitudinal blind area of ​​the point cloud sensor, that is, the point cloud sensor may not be able to scan the distance area with a vertical coordinate of 0 to q.

[0078] Rasterize the point cloud data and set the resolution to α. Then the length of the grid unit in the horizontal range is The grid cell length of the vertical range is Both are rounded up. The value is rounded up to M, The rounded-up value is N.

[0079] Get real-time point cloud data, denoted as P sensor , the point cloud data converted to the vehicle detection coordinate system is recorded as P detec For P detec Point cloud data of the X / Y direction is filtered, and the point cloud data with the X direction in the range of [-L, L] and the Y direction in the range of [q, W] are retained. The corresponding set is recorded as P clip . clip Project the point cloud data to generate an elevation map. Figure 5 A schematic diagram of a bird's-eye view vehicle detection coordinate system and an elevation map coordinate system provided in a specific embodiment of the present application. Figure 5As shown, the two-dimensional coordinate system X′Y′ is the elevation map coordinate system, the three-dimensional detection coordinate system XYZ is the vehicle detection coordinate system, the area in the figure is the rasterized detection area, and the point is the point cloud falling in the area. The image resolution of the elevation map is M*N. For P clip Any point p in clip (x,y), the pixel coordinates in the elevation map satisfy:

[0080] pixel x = int[(x+L) / α];

[0081]

[0082] Furthermore, the average value of the height of each pixel grid in the elevation map is calculated, that is, the mean value of the z coordinates of all points in the pixel grid in the point cloud coordinates, which is recorded as the pixel value corresponding to the pixel grid. At this point, the pixel values ​​of all pixel grids in the elevation map are obtained, which is recorded as image high . Create a new logo image mark , the same size as the elevation map M*N. It can be understood that in this specific embodiment, detec When the xy plane of the coordinate system coincides with the ground and there are no obstacles or potholes in the detection area, the point clouds all fall on the ground plane, and the z value of each point is equal to or close to 0. If there are raised obstacles, some point clouds that should have fallen on the ground may be blocked by the raised obstacles, leaving gaps behind them, and the raised part of the point cloud will form a height difference with the surrounding point clouds; if there is a gentle "pit" with a small slope, there will be no obstruction or partial point cloud loss; if there is a deep "pit" with a large slope difference, it will cause partial point cloud loss. Therefore, the image can be determined based on the point cloud distribution in each pixel grid in the elevation map. mark The pixel value of each corresponding pixel grid in the elevation map. If there is no point cloud point information in the pixel grid in the elevation map, record image mark The pixel value of the pixel grid at the corresponding position in is 1; if there is point cloud data, record image mark The pixel value of the corresponding pixel in the image is 0. mark The road surface area corresponding to the pixel grid with a pixel value of 1 has a depression and can be determined as a risk area.

[0083] Furthermore, for the above elevation map, the gradients in the x- and y-directions can be calculated in sequence using the convolution kernel according to the point cloud coordinates of the point cloud in each pixel grid in the elevation map, and the “boundary copy” filling rule can be used at the edges.

[0084] Among them, the gradient convolution kernel in the x direction is:

[0085]

[0086] The gradient convolution kernel in the y direction is:

[0087]

[0088] The gradient image obtained by convolution in the x direction is called image grad_x , the gradient image obtained by convolution in the y direction is called image grad_y The square root of the corresponding position of the two gradient images is taken to obtain the fused gradient image, which is recorded as image grad .

[0089] image grad 、image grad_x With image grad_y The image resolution is M*N. grad Anywhere:

[0090]

[0091] Where 0≤m <M,0≤n<N。

[0092] Then, the gradient threshold γ is set according to experiments or experience. grad If the pixel value in is greater than γ, it means that the slope is large and it is a dangerous point. mark The pixel value at the corresponding position in is 1.

[0093] Next, for image mark The pixel grids with pixel values ​​of 1 in the image are clustered in an Euclidean manner. The distance unit of the clustering is the pixel grid dimension of the image, and the unsafe area cluster set C is obtained. dangerous {c1,c2,…,c n}. For the unsafe area cluster set C dangerous {c1,c2,…,c n}, calculate the range of pixel dimensions corresponding to each cluster, and calculate the maximum value of each cluster in the x / y pixel grid dimension. For example, for any cluster c n , its maximum and minimum values ​​in the x direction are: max , x min ; The maximum and minimum values ​​in the y direction are: y max ,y min , get the center point of the unsafe area Dimensions in the xy direction They are:

[0094]

[0095] Then, its center position information and dimension information are projected into the real vehicle detection coordinate system.

[0096] The transformation rules are as follows:

[0097]

[0098] At the same time, traverse to find the interval range calculated by each cluster in P clip The maximum and minimum values ​​of the height of the corresponding point cloud are recorded as and Its z-axis center point is:

[0099]

[0100] Its z-axis length is:

[0101]

[0102] So far, for any unsafe region c n , and get its detec The center point of the coordinate system And the lengths of the three dimensions:

[0103] Furthermore, the detection results are visualized on the image. In this specific embodiment, camera dedistortion is not considered. If a distorted camera is used, please refer to the prior art to add an image dedistortion process. In addition, this specific embodiment does not consider the detection results projected beyond the image field of view. If it is beyond the range, it can be removed. After the above processing, C dangerous {c1,c2,…,c n Any unsafe area c in n In P detec The 3D minimum bounding box information is shown in the figure, with the unsafe area c n For example, c n The corresponding set of 8 vertices of the three-dimensional bounding box is B n {b n1 , b n2 , b n3 , b n4 , b n5 , b n6 , b n7 , b n8}. Further, we can use The rotation matrix transforms the 8 vertices to the camera coordinate system to obtain the 3D projection vertex set B in the camera coordinate system. n ` {b n1 ` , bn2 ` , b n3 ` , b n4 ` , b nS ` , b n6 ` , b n7 ` , b n8 `}, where for any vertex transformation: Finally, combined with the camera projection transformation, the eight unsafe area vertices in the camera coordinate system are projected to the pixel coordinate system, and the 3D bounding box is drawn in the 2D image to realize the visualization of the three-dimensional bounding box of the unsafe area.

[0104] An embodiment of the present application also provides a processor configured to execute the method for determining a risk area for uneven road surface in the above-mentioned implementation.

[0105] Figure 6 The present invention also provides a device 600 for determining a risk area for uneven road surface, and the device 600 includes:

[0106] The point cloud acquisition module 610 is used to acquire initial point cloud data of the detected road surface.

[0107] The coordinate conversion module 620 is used to convert the initial point cloud data into the three-dimensional detection coordinate system based on the calibration relationship between the predetermined point cloud coordinate system and the three-dimensional detection coordinate system to obtain the target point cloud data. The horizontal axis and the vertical axis of the three-dimensional detection coordinate system are parallel to the road surface being detected, and the vertical axis of the three-dimensional detection coordinate system is perpendicular to the road surface being detected.

[0108] The elevation map determination module 630 is used to determine the elevation map of the detected road surface according to the target point cloud data.

[0109] The detection result determination module 640 is used to determine the uneven risk area of ​​the detected road surface according to the elevation map.

[0110] The above-mentioned device 600 for determining the uneven road surface risk area obtains the initial point cloud data of the road surface to be detected, and then converts the initial point cloud data into a three-dimensional detection coordinate system based on the calibration relationship between the predetermined point cloud coordinate system and the three-dimensional detection coordinate system to obtain the target point cloud data. The horizontal axis and the vertical axis of the three-dimensional detection coordinate system are parallel to the road surface to be detected, and the vertical axis of the three-dimensional detection coordinate system is perpendicular to the road surface to be detected. Then, the elevation map of the road surface to be detected is determined according to the target point cloud data, and finally the uneven risk area of ​​the road surface to be detected is determined according to the elevation map. The detection method based on the point cloud of the present application converts the point cloud into an elevation map to determine the uneven risk road surface. The detection method is not affected by environmental factors such as light and has high accuracy.

[0111] In one embodiment, the elevation map includes a plurality of pixel grids that are gridded according to a preset resolution; the detection result determination module 640 is further used to: determine a target gradient corresponding to each pixel grid based on a target point cloud in each pixel grid; determine a target pixel grid set based on the target gradient of each pixel grid and a preset gradient threshold, wherein the target gradient corresponding to the target pixel grid in the target pixel grid set is greater than the preset gradient threshold, or the target point cloud does not exist in the target pixel grid; cluster the target pixel grids in the pixel grid set to obtain corresponding regional clusters; and determine a local area of ​​the detected road surface corresponding to the regional cluster as an uneven risk area.

[0112] In one embodiment, the detection result determination module 640 is also used to: determine the first gradient of the pixel grid in the horizontal coordinate direction according to the three-dimensional coordinates of the target point cloud in the pixel grid based on a first preset gradient convolution kernel; determine the second gradient of the pixel grid in the vertical coordinate direction according to the three-dimensional coordinates of the target point cloud in the pixel grid based on a second preset gradient convolution kernel; and fuse the first gradient and the second gradient to obtain the target gradient corresponding to the pixel grid.

[0113] In one embodiment, the detection result determination module 640 is also used to: for any uneven risk area, determine the three-dimensional bounding box of the uneven risk area in the camera coordinate system according to the three-dimensional coordinates of the target point cloud corresponding to the uneven risk area, based on the predetermined calibration relationship between the camera coordinate system and the three-dimensional detection coordinate system; and display the three-dimensional bounding box in the two-dimensional image of the detected road surface.

[0114] In one embodiment, the detection result determination module 640 is also used to: determine the center point and dimension length of the uneven risk area in the three-dimensional detection coordinate system according to the three-dimensional coordinates of the target point cloud corresponding to the uneven risk area, the dimensions including the horizontal axis dimension length, the longitudinal axis dimension length and the vertical axis dimension length; based on the calibration relationship between the camera coordinate system and the three-dimensional detection coordinate system, determine the three-dimensional bounding box of the uneven risk area in the camera coordinate system according to the center point and the dimension length.

[0115] In one embodiment, the detection result determination module 640 is also used to: determine a three-dimensional coordinate set of multiple vertices of the three-dimensional box according to the center point and the dimension length; based on the calibration relationship between the camera coordinate system and the three-dimensional detection coordinate system, according to the three-dimensional coordinate set, convert the multiple vertices to the camera coordinate system to obtain a three-dimensional bounding box of the uneven risk area in the camera coordinate system.

[0116] In one embodiment, the detection result determination module 640 is further used to: project the three-dimensional bounding box to a pixel coordinate system in the two-dimensional image according to the camera projection transformation, so that the three-dimensional bounding box is displayed in the two-dimensional image.

[0117] An embodiment of the present application also provides a machine-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the method for determining the risk area of ​​uneven road surface in the above-mentioned embodiment is implemented.

[0118] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0119] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0120] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0122] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0123] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0124] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0125] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0126] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for determining a risk area of ​​road unevenness, characterized in that: The method comprises: Obtaining initial point cloud data of the road surface being inspected; Based on a predetermined calibration relationship between a point cloud coordinate system and a three-dimensional detection coordinate system, the initial point cloud data is converted into the three-dimensional detection coordinate system to obtain target point cloud data, wherein the horizontal axis and the vertical axis of the three-dimensional detection coordinate system are parallel to the road surface being detected, and the vertical axis of the three-dimensional detection coordinate system is perpendicular to the road surface being detected; Determine an elevation map of the detected road surface according to the target point cloud data; The uneven risk area of ​​the detected road surface is determined according to the elevation map.

2. The method according to claim 1, characterized in that The elevation map includes a plurality of pixel grids that are gridded according to a preset resolution; Determining the uneven risk area of ​​the detected road surface according to the elevation map includes: Determine the target gradient corresponding to each pixel grid according to the target point cloud in each pixel grid; Determine a target pixel grid set according to the target gradient of each pixel grid and a preset gradient threshold, wherein the target gradient corresponding to the target pixel grid in the target pixel grid set is greater than the preset gradient threshold, or there is no target point cloud in the target pixel grid; Performing clustering processing on target pixel grids in the pixel grid set to obtain corresponding region clusters; A local area of ​​the detected road surface corresponding to the area cluster is determined as an uneven risk area.

3. The method according to claim 2, characterized in that Determining the target gradient corresponding to each pixel grid according to the target point cloud in each pixel grid includes: Based on a first preset gradient convolution kernel, determining a first gradient of the pixel grid in a horizontal coordinate direction according to the three-dimensional coordinates of the target point cloud in the pixel grid; Based on a second preset gradient convolution kernel, determining a second gradient of the pixel grid in the ordinate direction according to the three-dimensional coordinates of the target point cloud in the pixel grid; The first gradient and the second gradient are fused to obtain a target gradient corresponding to the pixel grid.

4. The method according to claim 2, characterized in that: The method further comprises: For any uneven risk area, according to the three-dimensional coordinates of the target point cloud corresponding to the uneven risk area, based on a predetermined calibration relationship between the camera coordinate system and the three-dimensional detection coordinate system, determine a three-dimensional bounding box of the uneven risk area in the camera coordinate system; The three-dimensional bounding box is displayed in the two-dimensional image of the detected road surface.

5. The method according to claim 4, characterized in that The determining, according to the three-dimensional coordinates of the target point cloud corresponding to the uneven risk area and based on a predetermined calibration relationship between the camera coordinate system and the three-dimensional detection coordinate system, of a three-dimensional bounding box of the uneven risk area in the camera coordinate system includes: According to the three-dimensional coordinates of the target point cloud corresponding to the uneven risk area, determine the center point and dimension length of the uneven risk area in the three-dimensional detection coordinate system, where the dimension includes the horizontal axis dimension length, the longitudinal axis dimension length and the vertical axis dimension length; Based on the calibration relationship between the camera coordinate system and the three-dimensional detection coordinate system, a three-dimensional bounding box of the uneven risk area in the camera coordinate system is determined according to the center point and the dimension length.

6. The method according to claim 5, characterized in that The step of determining a three-dimensional bounding box of the uneven risk area in the camera coordinate system based on the calibration relationship between the camera coordinate system and the three-dimensional detection coordinate system and according to the center point and the dimension length includes: Determine a three-dimensional coordinate set of a plurality of vertices of the three-dimensional frame according to the center point and the dimension length; Based on the calibration relationship between the camera coordinate system and the three-dimensional detection coordinate system, the multiple vertices are converted to the camera coordinate system according to the three-dimensional coordinate set to obtain a three-dimensional bounding box of the uneven risk area in the camera coordinate system.

7. The method according to claim 4, characterized in that The displaying of the three-dimensional bounding box in the two-dimensional image of the detected road surface comprises: According to the camera projection transformation, the three-dimensional bounding box is projected to the pixel coordinate system in the two-dimensional image, so that the three-dimensional bounding box is displayed in the two-dimensional image.

8. A processor, characterized in that: The method is configured to execute the method for determining a road surface irregularity risk area according to any one of claims 1 to 7.

9. A device for determining risk areas of road surface unevenness, characterized in that: The device comprises: A point cloud acquisition module is used to obtain initial point cloud data of the road surface being inspected; A coordinate conversion module, for converting the initial point cloud data into the three-dimensional detection coordinate system based on a predetermined calibration relationship between the point cloud coordinate system and the three-dimensional detection coordinate system to obtain target point cloud data, wherein the horizontal axis and the vertical axis of the three-dimensional detection coordinate system are parallel to the detected road surface, and the vertical axis of the three-dimensional detection coordinate system is perpendicular to the detected road surface; An elevation map determination module, used to determine the elevation map of the detected road surface according to the target point cloud data; The detection result determination module is used to determine the uneven risk area of ​​the detected road surface according to the elevation map.

10. A machine-readable storage medium storing a program or an instruction, characterized in that: When the program or the instruction is executed by a processor, the method for determining a road surface unevenness risk area according to any one of claims 1 to 7 is implemented.

Images