Road pit slot detection method fusing laser point cloud and image

Abstract

The invention discloses a laser point cloud and image fused road pit slot detection method, which comprises the following steps of: S1, acquiring point cloud and image data by using a mobile measurement device acquisition vehicle, and processing to obtain point cloud and image data of a road; and S2, on the basis of the point cloud and the image data, calculating the coordinates, the area and the depth of the pit slot according to a central projection principle. According to the method, extraction of pit point coordinate system information of the edge of the pit slot and attributes such as the position, the area and the depth of the pit slot is realized.

Description

TECHNICAL FIELD

[0001] The application belongs to a kind of road pit detection method of fusing laser point cloud and image. BACKGROUND

[0002] The road surface is seriously damaged due to long-term rolling of heavy trucks, which easily produces pits, causing great hidden dangers to the safe driving of vehicles. The existence of these pits, especially on the highway, will cause great damage to the vehicles running at high speed. Since the highway bears the task of transportation, the manual detection method has low operation efficiency and high risk coefficient, so it is urgent to improve the traditional detection method with new technology to improve the detection efficiency and quality. SUMMARY

[0003] To solve the above problems existing in the prior art, the application provides a kind of road pit detection method of fusing laser point cloud and image, realizes the extraction of pit point coordinate system information of pit edge and pit position, area, pit depth and other attributes.

[0004] The technical scheme to achieve the above purpose is: A kind of road pit detection method of fusing laser point cloud and image, comprising: Step S1, using mobile measuring device to collect vehicle to collect point cloud, image data, and obtaining road point cloud and image data after processing; Step S2, according to the principle of central projection, calculate the coordinates, area and depth of the pit based on the point cloud and image data.

[0005] Preferably, in step S1, the mobile measuring device comprises: 1 panoramic camera for collecting road panoramic image; 3 scanners for collecting road point cloud data; 2 GNSS (Global Navigation Satellite System) receivers for positioning and orientation; 1 GPS (Global Positioning System) clock for time synchronization; 1 IMU (Inertial Measurement Unit) for attitude determination; 2 angle sensors for mileage measurement; 1 set of power supply system for supporting the operation of mobile measuring device.

[0006] Preferably, the step S2 comprises: Step S21, find out the image containing similar pit from all image data and unify to image file; Step S22, determine the starting image and the number of points of each image; Step S23, extracting pixel coordinates, extracting corresponding coordinates on each image and saving to the pit pixel file; Step S24, setting pixel threshold, reading the saved image file, pit pixel file, and camera exterior orientation element file, locating to the corresponding image according to the image serial number in the pit pixel file, extracting the camera exterior orientation element, calculating the coordinates, area and depth of the pit point according to the detection principle of the pit groove.

[0007] Preferably, in step S22, the number of points taken from each image includes four corner points and a plurality of pit groove edge points.

[0008] Preferably, in step S24, the camera exterior orientation element file includes position data collected by two GNSS receivers and attitude data collected by one IMU.

[0009] Preferably, in step S24, the collineation equation is established by using the principle of central projection, and the interior orientation elements of the panoramic camera are known, including: the principal distance from the center of projection to the image , the image principal point , the coordinates in the frame coordinate system ; Taking the central projection as the basic principle, first determine the four points on the image used to establish the plane equation, and select the pixel points near the edge of the pit groove when selecting; Further calculate the object coordinates of the four corner points in the point cloud coordinate system, position each image to the corresponding point cloud block according to the registration method of the point cloud and the image, and project the point cloud onto the image through the collineation equation; Set the pixel threshold to find the point cloud projection falling within the pixel threshold range of the four plane points, take the average of the actual object coordinates of each as the object coordinates of the four points, and use them to establish the plane equation, and solve the four parameters 、 、 and by the least square method; According to the determined collineation equation and plane equation, the relationship between the image and the point cloud is determined, the pixel coordinates of the pixel points around the pit groove are obtained, and the object coordinates of the corresponding points around the pit groove are calculated, that is: ; In the formula, is the image point coordinate, 、 、 、 、 、 、 、 and All are elements of a rotation matrix. , , These are the normal vector components of the plane. This is the distance parameter from the plane to the origin. For panoramic camera photography center coordinates The coordinates are in the object space. Based on the calculated object coordinates around the pit, the average value is used to obtain the coordinates of the center point of the pit, and the pit is located. The object coordinates of the pit point show the shape of the pit, which is regarded as a polygon. The area of ​​the pit is calculated using the polygon area formula to determine the size of the pit. Finally, using MATLAB's... The function (in MATLAB, this function is used to check if a point is inside the boundary of a polygon) determines the point cloud that falls into the pit.

[0010] Preferably, in step S24, MATLAB is used. The function determines which point clouds fall into the pit, including: set up Here are the coordinates of the point to be judged. Given the vertex coordinates of the pit polygon, we get: ; In the formula, It is a logical value with two values: 1 and 0. 1 indicates that it is inside the polygon, and 0 indicates that it is outside the polygon. use The function evaluates the point cloud that falls within the plane defined by the four corner points to determine whether it falls within the pit. If it falls into the pit, it is retained to form a new point cloud, and the maximum value of the height of the point cloud above the road surface is calculated; The results are compared with the pit depth threshold. If the result is greater than the threshold, it is considered a pit; if the result is less than the threshold, it is considered not a pit. If no point cloud falls into the pit, it is impossible to determine whether it is a pit or not.

[0011] Compared with the prior art, the beneficial effects of the present invention are: the present invention uses a vehicle-mounted mobile measuring device as the data acquisition equipment, which has less impact on traffic and high detection efficiency; the present invention uses a point cloud and image fusion algorithm to make up for the deficiencies of point cloud and image respectively, and realizes the extraction of relatively complete information such as pothole location, shape, size, area and depth, thereby improving the efficiency and quality of road pothole detection. Attached Figure Description

[0012] The accompanying drawings are included to provide a further understanding of the application, and are incorporated in and constitute a part of the specification, illustrate embodiments of the application, and are used to explain the present application, but do not limit the application. In the drawings: Figure 1 is a flow chart of a road pit detection method fusing laser point cloud and image of the application; Figure 2 is a specific flow chart of calculating the coordinates, area and depth of the pit according to the central projection principle based on the point cloud and image data in the application; Figure 3 is a schematic diagram of establishing a collinear equation to solve the object coordinates of a certain pixel point using the central projection principle in the application; Figure 4 is a schematic diagram of establishing a plane equation of four points around the pit in the application; Figure 5 is a schematic diagram of establishing a plane of four points in the application; Figure 6 is a schematic diagram of the projection of the point cloud on the image in the application; Figure 7 is a schematic diagram of calculating the object coordinates of the pit according to the collinear equation and the plane equation in the application; Figure 8 is a schematic diagram of the pit point coordinate information of the edge of the pit in the embodiment of the application; Figure 9 is a schematic diagram of the pit attribute information in the embodiment of the application; Figure 10 is a schematic diagram of the extracted pit display in the embodiment of the application. DETAILED DESCRIPTION

[0013] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0014] Large pits on the road cause great safety hazards to driving safety. The uninterrupted traffic flow brings great safety hazards to traditional manual detection and patrol, and the efficiency is very low and has certain safety risks, and at the same time, it brings certain interference to traffic operation, and it is difficult to give quantitative indicators of the pit. Therefore, the following method is provided to improve the detection efficiency and quality.

[0015] As shown in Figure 1 , a road pit detection method fusing laser point cloud and image, comprising: Step S1, collecting point cloud and image data by the mobile measuring device, and obtaining the point cloud and image data of the road through processing.

[0016] In the embodiment, the mobile measuring device comprises: 1 panoramic camera for collecting panoramic image of the road; 3 scanners for collecting point cloud data of the road; 2 GNSS receivers for positioning and orientation; 1 GPS clock for time synchronization; 1 IMU for attitude determination; 2 angle sensors for distance measurement; 1 power supply system for supporting the operation of the mobile measuring device.

[0017] In the embodiment, the mobile measuring device collects point cloud and image data, and obtains the point cloud and image data of the road through processing. If the density of the point cloud is large enough, the size, range and depth of the pit can be easily distinguished from the point cloud. However, due to the high speed of the vehicle-mounted mobile system, the density of the point cloud in the driving direction is small, which is not enough to distinguish the pit. Therefore, the registration method of the vehicle-mounted laser point cloud and the panoramic image is adopted. Starting from the image with good visual effect, the corresponding pit information in the point cloud is found according to the relationship between the image and the point cloud determined by registration.

[0018] Step S2, calculating the coordinates (i.e. pixel coordinates ), area and depth of the pit according to the central projection principle based on the point cloud and image data.

[0019] As shown in Figure 2 , step S2 comprises: Step S21, finding the image containing similar pit from all image data and unifying to the image file.

[0020] Step S22, determining the starting image and the number of points of each image.

[0021] In the embodiment, the number of points of each image comprises four corner points and a plurality of pit edge points, which can be defined by itself.

[0022] Step S23, extracting pixel coordinates, extracting corresponding coordinates on each image, and saving to the pit pixel file.

[0023] Step S24, setting pixel threshold, reading the saved image file, pit pixel file and lens exterior orientation element file, positioning to the corresponding image according to the image serial number in the pit pixel file, extracting the exterior orientation element of the lens, calculating the coordinates, area and depth of the pit point according to the detection principle of the pit.

[0024] In this embodiment, the lens's external orientation element file includes position data collected by two GNSS receivers and attitude data collected by one IMU.

[0025] like Figure 3 As shown, collinearity equations are established using the central projection principle, given: Interior orientation elements of a panoramic camera, center of view The principal distance to the image Like the main point Coordinates in the frame coordinate system ; Panoramic Camera Photography Center coordinates And the transformation relationship between point cloud coordinate systems. , , , These represent the rotation angles around the X, Y, and Z axes, respectively. The coordinates of the pixels surrounding the pit in the image are unknown, and the unknown coordinates to be solved are the object-space coordinates of the pit's perimeter, the coordinates of the pit's center, and the area of ​​the pit. These are represented by a plane equation established by four points around the pit, such as... Figure 4 As shown, establishing this plane equation is equivalent to giving the collinear equation a constraint condition, and the three equations can be used to solve for the three unknowns; Based on the principle of central projection, four points on the image are first determined to establish the plane equation. When selecting these points, pixels closest to the edge of the pit are chosen, such as... Figure 5 As shown, this ensures that the established plane is very close to the surface of the pit, thus guaranteeing the reliability of the constraints. The aforementioned plane establishment needs to be done within the point cloud coordinate system. This is necessary to serve as a constraint condition for the collinearity equation. Therefore, after determining the four corner control points on the image, it is necessary to further calculate the object coordinates of these four corner points in the point cloud coordinate system. Based on the point cloud and image registration method, each image is located to its corresponding point cloud block. Furthermore, the point cloud can be projected onto the image using the collinearity equation, as shown below. Figure 6 As shown; Because the pixel coordinates of the point cloud projection are not necessarily exactly equal to the four corner points selected beforehand, a pixel threshold needs to be defined in advance. Point cloud projections falling within the pixel threshold range of the four plane points are found, and the average of their actual object-space coordinates is taken as the object-space coordinates of the four points. Plane equations are then established using these averages, and the least squares method is used to calculate... , , and Four parameters; Based on the determined collinearity equation and plane equation, the bidirectional transformation relationship between the image and the point cloud is determined, the pixel coordinates of the pixels around the pit are obtained, and the corresponding object coordinates of the points around the pit are calculated. Figure 7 As shown, that is: ; In the formula, For image point coordinates, , , , , , , , and All are elements of a rotation matrix. , , These are the normal vector components of the plane. This is the distance parameter from the plane to the origin. For panoramic camera photography center coordinates The coordinates are in the object space. Based on the calculated object coordinates around the pit, the average value is used to obtain the coordinates of the center point of the pit, and the pit is located. The object coordinates of the pit point show the shape of the pit, which is regarded as a polygon. The area of ​​the pit is calculated using the polygon area formula to determine the size of the pit. Finally, using MATLAB's... The function determines which point cloud points fall into the pit.

[0026] In the example, MATLAB is used The function determines which point clouds fall into the pit, including: set up Here are the coordinates of the point to be judged. Given the vertex coordinates of the pit polygon, we get: ; In the formula, It is a logical value with two values: 1 and 0. 1 indicates that it is inside the polygon, and 0 indicates that it is outside the polygon. use The function evaluates the point cloud that falls within the plane defined by the four corner points to determine whether it falls within the pit. If it falls into the pit, it is retained to form a new point cloud, and the maximum value of the height of the point cloud above the road surface is calculated; The results are compared with the pit depth threshold. If the result is greater than the threshold, it is considered a pit; if the result is less than the threshold, it is considered not a pit. If no point cloud falls into the pit, it is impossible to determine whether it is a pit or not.

[0027] Example of pit extraction: Data was collected from a 10-kilometer stretch of the G15 Shenhai Expressway (Yexie section), and the coordinates of pothole edges were extracted based on the principles of this invention. Figure 8 As shown, information such as the location, area, and depth of the pit is as follows: Figure 9 As shown, there are three values ​​indicating whether it is a pit: 0, 1, and 2. 0 means it is not a pit, 1 means it is a pit, and 2 means that no point cloud falls into the pit, so it is impossible to determine whether it is a pit.

[0028] Unfold the calculated coordinates of the pit points at the edge of the pit in point cloud software to obtain, for example: Figure 10 The pit diagram shown shows that the shape and size of the pits extracted by the software match the actual situation, and according to... Figure 9 The sixth pit attribute indicates that this pit is relatively shallow, suggesting it was filled, and the final result also determines that it is not a pit. Figure 10 The images of the pits and grooves in the image match, indicating that the pit and groove extraction is effective.

[0029] After extracting potholes from the highway surface, the extracted potholes can be displayed on a map for relevant personnel to view online. Based on pothole information collected from approximately 10 kilometers of the Yexie section of the G15 Shenhai Expressway, WebGIS technology was used to publish the pothole information to the internet.

[0030] Finally, it should be noted that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting road potholes by fusing laser point clouds and imagery, characterized in that, include: Step S1: Use a mobile measurement device vehicle to collect point cloud and image data, and process them to obtain point cloud and image data of the road. Step S2: Calculate the coordinates, area, and depth of the pit based on the point cloud and image data according to the central projection principle.

2. The method for detecting road potholes by fusing laser point clouds and images according to claim 1, characterized in that, In step S1, the moving measuring device includes: One panoramic camera is used to capture panoramic images of the road. Three scanners were used to collect road point cloud data; Two GNSS receivers are used for positioning and orientation. One GPS clock for time synchronization; One IMU is used for attitude determination; Two angle sensors are used for mileage measurement; One power supply system is provided to support the operation of the mobile measuring device.

3. The method for detecting road potholes by fusing laser point clouds and images according to claim 1, characterized in that, Step S2 includes: Step S21: Find images containing similar pits from all image data and unify them into an image file; Step S22: Determine the starting image and the number of points to be sampled for each image; Step S23: Extract pixel coordinates. Extract the corresponding coordinates from each image and save them to the pit pixel file. Step S24: Set pixel threshold, read the saved image file, pit pixel file, and lens exterior orientation element file, locate the corresponding image according to the image sequence number in the pit pixel file, extract the lens exterior orientation elements, and calculate the coordinates, area, and depth of the pit according to the pit detection principle.

4. The method for detecting road potholes by fusing laser point clouds and images according to claim 3, characterized in that, In step S22, the number of points taken for each image includes four corner points and multiple pit edge points.

5. The method for detecting road potholes by fusing laser point clouds and images according to claim 3, characterized in that, In step S24, the lens's external orientation element file includes position data collected by two GNSS receivers and attitude data collected by one IMU.

6. The method for detecting road potholes by fusing laser point clouds and images according to claim 3, characterized in that, In step S24, collinearity equations are established using the central projection principle. The interior orientation elements of the panoramic camera are known, including the camera center. The principal distance to the image Like the main point Coordinates in the frame coordinate system ; Based on the principle of central projection, four points on the image are first determined to establish the plane equation. When selecting, the pixel points close to the edge of the pit are chosen. The object coordinates of these four corner points in the point cloud coordinate system are further calculated. Based on the registration method between the point cloud and the image, each image is located to the corresponding point cloud block. The point cloud can be projected onto the image through the collinearity equation. Set a pixel threshold, find the point cloud projections that fall within the pixel threshold range of four plane points, take the average of their actual object-space coordinates as the object-space coordinates of the four points, and use them to establish the plane equations. Then, use the least squares method to calculate... , , and Four parameters; Based on the determined collinearity equation and plane equation, the bidirectional transformation relationship between the image and the point cloud is determined, the pixel coordinates of the pixels around the pit are obtained, and the corresponding object coordinates of the points around the pit are calculated in reverse. ； In the formula, For image point coordinates, , , , , , , , and All are elements of a rotation matrix. , , These are the normal vector components of the plane. This is the distance parameter from the plane to the origin. For panoramic camera photography center coordinates The coordinates are in the object space. Based on the calculated object coordinates around the pit, the average value is used to obtain the coordinates of the center point of the pit, and the pit is located. The object coordinates of the pit point show the shape of the pit, which is regarded as a polygon. The area of ​​the pit is calculated using the polygon area formula to determine the size of the pit. Finally, using MATLAB... The function determines which point cloud points fall into the pit.

7. The method for detecting road potholes by fusing laser point clouds and images according to claim 6, characterized in that, In step S24, MATLAB is used. The function determines which point clouds fall into the pit, including: set up Here are the coordinates of the point to be judged. Given the vertex coordinates of the pit polygon, we get: ； In the formula, It is a logical value with two values: 1 and 0. 1 indicates that it is inside the polygon, and 0 indicates that it is outside the polygon. use The function evaluates the point cloud that falls within the plane defined by the four corner points to determine whether it falls within the pit. If it falls into the pit, it is retained to form a new point cloud, and the maximum value of the height of the point cloud above the road surface is calculated; The results are compared with the pit depth threshold. If the result is greater than the threshold, it is considered a pit; if the result is less than the threshold, it is considered not a pit. If no point cloud falls into the pit, it is impossible to determine whether it is a pit or not.

Images