Road corrugated plate integrity detection method based on vehicle-mounted depth camera
The vehicle-mounted depth camera takes visible light and depth images of the corrugated plate area, performs distortion correction and image segmentation, and eliminates error results in combination with the depth map, projecting the corrugated plate point cloud on the ground, solving the accuracy and coverage of corrugated plate detection in the existing technology The scope problem realizes high-precision detection of the integrity of the corrugated plate.
Patent Information
- Application Number
- CN202411084510.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-11-12
AI Technical Summary
The existing corrugated plate integrity detection algorithm cannot specifically obtain the height of the corrugated plate, and it is difficult to detect the corrugated plate at the turn. It relies on naked eye observation or image recognition, and the detection accuracy and coverage range are limited.
Using the detection method based on the vehicle-mounted depth camera, distortion correction, void filling, image segmentation and point cloud fitting are performed by taking visible light images and depth maps of the corrugated plate area. Combined with the depth map, error results are eliminated, and point clouds are placed in the corrugated plate. Ground projection to judge its integrity.
It realizes high-precision detection of the integrity of the corrugated plate, without the naked eye observation, and can detect the corrugated plate at the turn, with high accuracy and no excessive additional calculations, and a wider coverage.
Smart Images

Figure CN118941545A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of intelligent management and maintenance of transportation infrastructure, and in particular to a method for detecting integrity of highway corrugated plates based on a vehicle-mounted depth camera. Background Art
[0002] Waveform board integrity detection is one of the core research contents of intelligent management and maintenance of transportation infrastructure. The input is various sensor data including waveform boards. By judging the information of waveform boards in these sensor data, the integrity of waveform boards is detected automatically or manually. Waveform board integrity detection has a wide range of applications in many fields, such as highway operation management, autonomous driving and other industries that need to ensure the integrity of transportation infrastructure and traffic safety.
[0003] Most of the existing corrugated plate integrity detection algorithms are based on naked eye observation or can only identify deformation defects of continuous straight corrugated plates through images, but cannot specifically obtain the height of the corrugated plates or detect the corrugated plates at the corners. Summary of the invention
[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a road corrugated plate integrity detection method based on a vehicle-mounted depth camera.
[0005] The purpose of the present invention is achieved through the following technical solution: A method for detecting the integrity of a highway corrugated plate based on a vehicle-mounted depth camera, comprising the following steps:
[0006] S1: Capture the visible light image and depth map of the corrugated plate area through the depth camera mounted on the vehicle;
[0007] S2: perform distortion correction on the visible light image and depth image, and fill holes in the depth image;
[0008] S3: Perform image segmentation on the visible light image to obtain pixel coordinates of the waveform plate area;
[0009] S4: Eliminate erroneous results by combining the depth map;
[0010] S5: Fit the ground according to the point cloud of the depth map, project the point cloud of the waveform plate on the ground, and judge the integrity of the waveform plate according to the continuity between the projection points.
[0011] Preferably, in step S1, the depth camera is horizontally mounted on the vehicle and perpendicular to the traveling direction of the vehicle.
[0012] Preferably, in step S2, the formula for performing distortion correction on the visible light image and the depth image is:
[0013]
[0014]
[0015]
[0016]
[0017]
[0018] x d =f x x nd +c x ;
[0019] y d =f y y nd +c y ;
[0020] Among them, f x is the focal length in the x direction, f y is the focal length in the y direction, c x is the coordinate of the center point in the x direction, c y is the coordinate of the center point in the y direction, k1, k2, k3, p1 and p2 are the five distortion parameters of the pinhole model, x is the horizontal coordinate of the pixel after dedistortion, y is the vertical coordinate of the pixel after dedistortion, x d is the horizontal coordinate of the corresponding pixel on the original image, y d is the corresponding pixel ordinate on the original image,
[0021] According to the formula, we can get the corresponding point x on the original image. d and d The pixel value of the current point is used as the pixel value of the current point.
[0022] Preferably, in step S2, the depth image is traversed by rows and columns respectively, and holes in the depth image are filled.
[0023] Preferably, in step S3, the visible light image is segmented by YOLOv8-seg.
[0024] Preferably, step S4 further includes the following steps:
[0025] S41: Mapping the depth image of the corrugated plate area in step S3 into three-dimensional points,
[0026] Suppose the coordinates of one point of the depth image segmented into the waveform plate are (x o ,y o ), then the three-dimensional coordinates of the current point relative to the camera are (X o ,Y o ,Z o )for:
[0027] Z o =z o ;
[0028]
[0029]
[0030] Among them, z o is the real distance from the camera;
[0031] S42: Perform plane fitting on all the corrugated plate point clouds obtained in step S41, and calculate the distance from the point to the fitting plane. The fitted plane equation is ax+by+cz+d=0, then d i =|aX i +bY i +cZ i +d|, where d i For any point P in the point cloud data i (X i ,Y i ,Z i ) to the plane;
[0032] Then the fitting target is min||AB||, where
[0033]
[0034]
[0035] After performing SVD decomposition on A, its minimum eigenvalue is obtained as the required plane equation.
[0036] Preferably, in step S5, the ground is fitted according to the depth map point cloud, the corrugated plate point cloud 85 cm above the ground is projected on the ground, and the point cloud 75 cm from the ground is selected as the ground point cloud. Then any point P in the point cloud data i (X i ,Y i ,Z i ) on the ground (X p ,Y p ,Z p )for:
[0037] d i =|aX i +bY i +cZ i +d|;
[0038]
[0039]
[0040]
[0041] The actual ground is obtained through step S4. When the distance from the ground exceeds 83-87 cm, it is determined that the point cloud of the complete corrugated plate is projected onto the ground. The integrity of the corrugated plate is determined based on the continuity between the projection points.
[0042] The present invention has the following advantages: the present invention uses a depth camera to shoot a visible light image and a depth map of a waveform plate area, performs distortion correction on the visible light image and the depth image, and fills holes in the depth image, then performs image segmentation on the visible light image, obtains the pixel coordinates of the waveform plate area, eliminates erroneous results in combination with the depth map, projects the waveform plate point cloud on the ground, and finally judges the integrity of the waveform plate by the continuity between the projection points, thereby eliminating the need to observe the waveform plate with the naked eye, and requiring no excessive additional calculations, with high accuracy, and at the same time, the waveform plate at the corner can be detected. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 It is a structural schematic diagram of the process of the highway corrugated plate integrity detection method;
[0044] Figure 2 It is a structural schematic diagram of a visible light image;
[0045] Figure 3 It is a structural diagram of a depth image;
[0046] Figure 4 This is a schematic diagram of the structure of the depth image after hole filling;
[0047] Figure 5 Schematic diagram of the structure of the segmented waveform plate pixel area. DETAILED DESCRIPTION
[0048] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.
[0049] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0050] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.
[0051] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0052] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, or the positions or positional relationships in which the product of the invention is usually placed when in use, or the positions or positional relationships commonly understood by those skilled in the art, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.
[0053] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0054] In this embodiment, if Figure 1 to Figure 5 As shown, a method for detecting the integrity of a highway corrugated plate based on a vehicle-mounted depth camera comprises the following steps:
[0055] S1: Capture a visible light image and a depth map of the corrugated plate area by a depth camera mounted on the vehicle. Preferably, the depth camera is mounted horizontally on the vehicle and perpendicular to the direction of travel of the vehicle.
[0056] S2: Perform distortion correction on the visible light image and depth image, and fill holes in the depth image;
[0057] S3: Perform image segmentation on the visible light image to obtain pixel coordinates of the waveform plate area; preferably, perform image segmentation on the visible light image through YOLOv8-seg.
[0058] S4: Eliminate erroneous results by combining the depth map;
[0059] S5: Fit the ground according to the point cloud of the depth map, project the point cloud of the waveform plate on the ground, and judge the integrity of the waveform plate according to the continuity between the projection points. The visible light image and depth map of the waveform plate area are captured by the depth camera, and the visible light image and the depth image are distorted and the depth image is filled with holes. Then the visible light image is segmented to obtain the pixel coordinates of the waveform plate area, and the wrong results are eliminated in combination with the depth map. The point cloud of the waveform plate is projected on the ground, and finally the integrity of the waveform plate is judged by the continuity between the projection points, so that the waveform plate does not need to be observed by the naked eye, and there is no need for too much additional calculation, and the accuracy is high. That is to say, the present invention directly uses the depth camera to obtain the point cloud, and the accuracy is higher than that of the conventional binocular camera or other image processing methods. In addition, the point cloud is obtained on the hardware, and excessive calculations are avoided. At the same time, the waveform plate at the corner can be detected. Specifically, because it does not rely on conventional straight line detection, but through the continuity of the projection, the waveform plate at the corner can be detected.
[0060] Furthermore, in step S2, the formula for performing distortion correction on the visible light image and the depth image is:
[0061]
[0062]
[0063]
[0064]
[0065]
[0066] x d =f x x nd +c x ;
[0067] y d =f y y nd +c y ;
[0068] Among them, f x is the focal length in the x direction, f y is the focal length in the y direction, c x is the coordinate of the center point in the x direction, c y is the coordinate of the center point in the y direction, k1, k2, k3, p1 and p2 are the five distortion parameters of the pinhole model, x is the horizontal coordinate of the pixel after dedistortion, y is the vertical coordinate of the pixel after dedistortion, x d is the horizontal coordinate of the corresponding pixel on the original image, y d is the corresponding pixel ordinate on the original image,
[0069] According to the formula, we can get the corresponding point x on the original image. d and d The pixel value of the current point is taken as the pixel value of the current point. Specifically, the camera’s internal parameter is f x 、f y 、c x and c y And the five distortion parameters are all known.
[0070] Further, if Figure 4 As shown, in step S2, the depth image is traversed by rows and columns respectively, and the holes in the depth image are filled. Specifically, when the pixel value of the current point is less than 0.001, the average depth value of the 5*5 area is taken and given to the current point.
[0071] In this embodiment, step S4 further includes the following steps:
[0072] S41: Mapping the depth image of the corrugated plate area in step S3 into three-dimensional points,
[0073] Suppose the coordinates of one point of the depth image segmented into the waveform plate are (x o ,y o ), then the three-dimensional coordinates of the current point relative to the camera are (X o ,Y o ,Z o )for:
[0074] Z o =z o ;
[0075]
[0076]
[0077] Among them, z o is the real distance from the camera;
[0078] S42: Perform plane fitting on all the corrugated plate point clouds obtained in step S41, and calculate the distance from the point to the fitting plane. The fitted plane equation is ax+by+cz+d=0, then d i =|aX i +bY i +cZ i +d|, where d i For any point P in the point cloud data i (X i ,Y i ,Z i) to the plane; specifically, when the distance from the point to the fitting plane exceeds 0.2 meters, it is determined to be an error point and is excluded.
[0079] Then the fitting target is min||AB||, where
[0080]
[0081]
[0082] After performing SVD decomposition on A, its minimum eigenvalue is obtained as the required plane equation.
[0083] Furthermore, in step S5, the ground is fitted according to the depth map point cloud, the corrugated plate point cloud 85 cm above the ground is projected on the ground, and the point cloud 75 cm from the ground is selected as the ground point cloud. Then any point P in the point cloud data i (X i ,Y i ,Z i ) on the ground (X p ,Y p ,Z p )for:
[0084] d i =|aX i +bY i +cZ i +d|;
[0085]
[0086]
[0087]
[0088] The actual ground is obtained through step S4. When the distance from the ground exceeds 83-87 cm, the point cloud of the corrugated board is identified as complete and projected to the ground. The integrity of the corrugated board is determined based on the continuity between the projected points. Specifically, since the camera is placed horizontally, the installation height can be obtained in advance, and then the point cloud with a distance of 75 cm from the camera to the ground is selected as the ground point cloud.
[0089] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for detecting the integrity of a highway corrugated plate based on a vehicle-mounted depth camera, characterized in that: The following steps are involved: S1: Capture the visible light image and depth map of the corrugated plate area through the depth camera mounted on the vehicle; S2: Perform distortion correction on the visible light image and depth image, and fill holes in the depth image; S3: performing image segmentation on the visible light image to obtain pixel coordinates of the waveform plate area; S4: Eliminate erroneous results by combining the depth map; S5: Fit the ground according to the point cloud of the depth map, project the point cloud of the waveform plate on the ground, and judge the integrity of the waveform plate according to the continuity between the projection points.
2. The method for detecting integrity of a highway corrugated plate based on a vehicle-mounted depth camera according to claim 1, characterized in that: In step S1, the depth camera is horizontally mounted on the vehicle and perpendicular to the moving direction of the vehicle.
3. The method for detecting the integrity of a highway corrugated plate based on a vehicle-mounted depth camera according to claim 2, characterized in that: In step S2, the formula for performing distortion correction on the visible light image and the depth image is: x d =f x x nd +c x ; y d =f y y nd +c y ; Among them, f x is the focal length in the x direction, f y is the focal length in the y direction, c x is the coordinate of the center point in the x direction, c y is the coordinate of the center point in the y direction, k1, k2, k3, p1 and p2 are the five distortion parameters of the pinhole model, x is the horizontal coordinate of the pixel after dedistortion, y is the vertical coordinate of the pixel after dedistortion, x d is the horizontal coordinate of the corresponding pixel on the original image, y d is the corresponding pixel ordinate on the original image, According to the formula, we can get the corresponding point x on the original image. d and d The pixel value of the current point is used as the pixel value of the current point.
4. The method for detecting the integrity of a highway corrugated plate based on a vehicle-mounted depth camera according to claim 3 is characterized in that: In the step S2, the depth image is traversed by rows and columns respectively, and holes in the depth image are filled.
5. The method for detecting integrity of a highway corrugated plate based on a vehicle-mounted depth camera according to claim 4 is characterized in that: In step S3, the visible light image is segmented by YOLOv8-seg.
6. The method for detecting integrity of a highway corrugated plate based on a vehicle-mounted depth camera according to claim 5, characterized in that: The step S4 further includes the following steps: S41: Mapping the depth image of the corrugated plate area in step S3 into three-dimensional points, Suppose the coordinates of one point of the depth image segmented into the waveform plate are (x o ,y o ), then the three-dimensional coordinates of the current point relative to the camera are (X o , Y o , Z o )for: WITH o =from o ; Among them, z o is the real distance from the camera; S42: Perform plane fitting on all the corrugated plate point clouds obtained in step S41, and calculate the distance from the point to the fitting plane. The fitted plane equation is ax+by+cz+d=0, then d i =|aX i +bY i +cZ i +d|, where d i For any point P in the point cloud data i (X i , Y i , Z i ) to the plane; Then the fitting target is min||AB||, where After performing SVD decomposition on A, its minimum eigenvalue is obtained as the required plane equation.
7. The method for detecting integrity of a highway corrugated plate based on a vehicle-mounted depth camera according to claim 6, characterized in that: In step S5, the ground is fitted according to the depth map point cloud, the corrugated plate point cloud 85 cm above the ground is projected on the ground, and the point cloud 75 cm from the ground is selected as the ground point cloud. Then any point P in the point cloud data i (X i , Y i , Z i ) on the ground (X p , Y p , Z p )for: d i =|aX i +bY i +cZ i +d|; The actual ground is obtained through step S4. When the distance from the ground exceeds 83-87 cm, it is considered to be a complete corrugated plate point cloud and projected onto the ground. The integrity of the corrugated plate is determined based on the continuity between the projection points.
Citation Information
Patent Citations
Method and device for extracting guardrails in road point clouds
CN111783648A
Guardrail detection method and device, electronic equipment and storage medium
CN116863447A
Expressway corrugated plate deformation detection method based on image recognition
CN117455874A
Method, device and system for extracting road waveform guardrail
CN117893991A
Method, apparatus, and computer program for providing logistics management solution using 3D geofences based on lidar point clouds
WO2024025021A1
Cited By
Road corrugated plate integrity detection method based on vehicle-mounted three-dimensional camera
CN120747098A
