Road disease detection vehicle-mounted video pavement panoramic image reconstruction method based on driving speed

Abstract

The invention discloses a road disease detection vehicle-mounted video pavement panoramic image reconstruction method based on a driving speed. The sampling rate difference problem of the GPS and the video is solved through equal-time interpolation, and frame-by-frame space-time matching of the GPS and the video is achieved; correcting perspective distortion caused by the pitch angle and the installation height of the camera through a camera projection model, and generating an orthographic projection image; and finally, continuous frame splicing without feature point dependence is completed based on the vehicle speed increment. Experimental results show that the method can accurately recover the real spatial form of a lane, effectively eliminate perspective distortion caused by a camera depression angle in an original video, and significantly suppress an edge sawtooth effect in a splicing process, and the finally generated panoramic image can clearly and completely present pavement texture and pavement disease distribution, and has a wide application prospect. Powerful support is provided for analysis, positioning and treatment after road surface layer disease detection.

Description

Technical Field

[0001] This invention relates to the field of road inspection technology, and specifically to a method for reconstructing panoramic road surface images from vehicle-mounted video for road defect detection based on driving speed. Background Technology

[0002] In road surface defect detection scenarios (such as cracks, potholes, and spalling), vehicle-mounted high-definition cameras are the core data acquisition devices. In practical applications, these cameras are typically fixed at a predetermined position on the rear of the inspection vehicle, acquiring road surface images at a frame rate of 20-40 frames per second and a resolution of 720P or higher. This ensures that a single frame of video clearly covers a 3-5 meter range of the road surface, fully capturing the morphology, size, and other characteristic information of the defects. To achieve visualization of the road surface condition across the entire road section, road surface area extraction and stitching processing is required for continuous video frames, ultimately reconstructing a panoramic road image to provide fundamental data support for defect detection.

[0003] In existing technologies, to address the issue that "vehicle-mounted videos only contain relative time information and lack spatial location correlation," vehicle-mounted high-definition cameras and Global Positioning System (GPS) devices are often used to collect data synchronously, relying on the spatial coordinates output by GPS to achieve spatiotemporal correlation of video frames. However, there is a fundamental difference in the sampling rates of the two. For example, the GPS data acquisition frequency is usually 1 time / second, while the video acquisition frequency is 25 frames / second. This means that 25 frames of video per second can only match one set of GPS coordinates, making it impossible to directly establish a one-to-one correspondence between "frames" and "coordinates."

[0004] On the other hand, due to the installation tilt angle and height of the vehicle-mounted camera, the road surface image in the original video frame exhibits significant perspective distortion—manifesting as a "nearer is wider, farther is narrower" visual effect. Simultaneously, panoramic image reconstruction requires stitching together consecutive video frames, but road surface features (such as road markings and manhole covers) are sparsely distributed and easily repeated, while texture features (such as asphalt particles and cement patterns) have low recognizability. This leads to traditional feature point matching algorithms (such as SIFT and SURF) easily experiencing "matching errors" or "missed matches" in this scenario. Furthermore, the large volume of data collected daily by the vehicle-mounted video system results in insufficient computational efficiency for the matching algorithms, ultimately leading to poor stitching accuracy and low processing efficiency, failing to meet the timeliness and high-precision requirements of road detection.

[0005] In summary, existing technologies for reconstructing panoramic road images from vehicle-mounted video still face three major technical bottlenecks: first, the sampling rate mismatch between high-definition video and GPS leads to insufficient spatiotemporal correlation accuracy; second, camera installation parameters cause perspective distortion in road images, affecting the stitching effect; and third, the scarcity of road features and the massive amount of video data result in low accuracy and poor efficiency of frame stitching algorithms. Therefore, there is an urgent need to propose a technical solution for reconstructing panoramic road images from vehicle-mounted video that can simultaneously address the above problems. Summary of the Invention

[0006] The purpose of this invention is to provide a method for reconstructing panoramic road surface images from vehicle-mounted video based on vehicle speed for detecting road defects. This method achieves precise spatiotemporal matching of GPS data and high frame-rate video, eliminates perspective distortion in road surface images, generates orthophoto images, and uses vehicle speed increments to achieve continuous frame stitching without feature point dependence, efficiently generating high-quality panoramic road surface images. Experimental results show that the vehicle speed-based video frame extraction method can accurately restore the true spatial morphology of the lanes; after perspective correction, the perspective distortion caused by the camera's downward angle in the original video is effectively eliminated, and the edge jaggedness effect that easily occurs during stitching is also significantly suppressed. The final generated panoramic image can clearly and completely present the road surface texture and the distribution of road defects.

[0007] The objective of this invention is achieved as follows: A method for reconstructing panoramic road surface images from vehicle-mounted video for road defect detection based on vehicle speed, characterized by the following steps: Step S1: Acquire in-vehicle high-definition video and synchronously collected GPS data, and perform isochronous linear interpolation on the GPS data so that the interpolated GPS data corresponds one-to-one with each frame of video in the time dimension. Step S2: Define three types of coordinate systems: world coordinate system, camera coordinate system, and image pixel coordinate system, as the reference for the camera projection model; Step S3: Based on the coordinate system defined in step S2, and combined with the intrinsic and extrinsic parameters of the vehicle-mounted high-definition camera, establish a camera projection model from the ground point in the world coordinate system to the image pixel coordinate system. Step S4: Select a road area and determine the world coordinates of the four corner points of the road area based on the horizontal field of view. Step S5: Using the camera projection model constructed in step S3, calculate the corresponding coordinates of the four corner points of the road area in step S4 in the image pixel coordinate system; based on the mapping relationship between the rectangular features of the road area in the world coordinate system and the trapezoidal features in the image, solve the perspective transformation matrix H to realize the perspective distortion correction from the trapezoidal region to the rectangular region in the image. Step S6: Extract the driving speed corresponding to each video frame based on the GPS data processed in step S1, and calculate the road increment recorded in each video frame in combination with the video frame rate. Step S7: Using the camera projection model from step S3, convert the road increment obtained in step S6 into a pixel increment in the image; use the perspective transformation matrix H obtained in step S5 to perform perspective correction on the image region corresponding to the pixel increment. Step S8: For each frame of the corrected image from step S7, directly stitch them together based on pixel increments without relying on road surface feature point matching, and finally generate a complete panoramic image of the road surface.

[0008] In step S1, the GPS data includes latitude and longitude, vehicle speed, and time information, and each frame of the vehicle video has a timestamp. The synchronous acquisition relationship between the GPS data and the vehicle video is confirmed based on the time information. The isochronous linear interpolation is performed on the latitude and longitude, vehicle speed, and time information in the GPS data to achieve accurate alignment between the interpolated data and the time dimension of each frame of video.

[0009] In step S2, the world coordinate system is a vehicle rear coordinate system, with the ground point directly below the camera as the origin, the Xw axis pointing to the left side of the vehicle, the Yw axis pointing to the rear of the vehicle, and the Zw axis pointing vertically upward; the camera coordinate system has the camera optical center as the origin, the Xc axis is in the same direction as the Xw axis of the world coordinate system, the Yc axis is perpendicular to the optical axis pointing to the road surface, and the Zc axis points along the optical axis towards the rear of the vehicle; the origin of the image pixel coordinate system is located at the upper left corner of the image, u represents the horizontal position of the pixel, and v represents the vertical position of the pixel.

[0010] In step S3, the camera projection model includes an intrinsic parameter matrix K, an extrinsic parameter matrix R, and a translation vector t; the camera intrinsic parameters include the lens focal length f, the optical center (c... x c y The height h in the camera's external parameters is obtained based on the camera's own parameters; the actual measurement of the height h is achieved by measuring with a tape measure, with an accuracy of ±1cm; the pitch angle θ in the external parameters is measured by a tilt sensor, with an accuracy of ±0.5°.

[0011] In step S4, the nearest distance d of the road area n and the distance d f The settings are adapted to the camera's field of view and the road increments generated by the vehicle's speed, generally d n ≤1m,d f ≤5m; the left and right distances of the road area are calculated based on the camera's field of view.

[0012] In step S5, the process of solving the perspective transformation matrix H is as follows: Based on the rectangular coordinates of the four corner points of the road area determined in step S4 in the world coordinate system (which have known rectangular geometric features), the corresponding coordinates of the four corner points in the image pixel coordinate system (which present trapezoidal geometric features) are calculated through the projection model in step S3; using the mapping relationship between the geometric constraints of the rectangle in the world coordinate system, such as right angles and parallel opposite sides, and the geometric features of the trapezoid in the image, the perspective transformation matrix H is calculated through the homography matrix solving algorithm. This matrix H is used to convert the trapezoidal distortion area in the image caused by the camera's tilt angle into a rectangular area consistent with the geometric shape of the road area in the world coordinate system, thereby achieving accurate elimination of perspective distortion.

[0013] In step S6, the driving speed vel (unit: meters / second) corresponding to each video frame is extracted based on the GPS data processed in step S1. Combined with the video frame rate fps, the road increment Δd meters / frame recorded in each video frame is calculated. The calculation formula is Δd=vel / fps.

[0014] In step S7, the road increment Δd obtained in step S6 is converted into a pixel increment in the image using the camera projection model in step S3; perspective correction is performed on the image region corresponding to the pixel increment using the perspective transformation matrix H obtained in step S5.

[0015] In step S8, the corrected images from each frame in step S7 are directly stitched together based on pixel increments, without relying on road surface feature point matching, ultimately generating a complete panoramic image of the road surface.

[0016] The beneficial effects of this invention are as follows: This invention achieves precise spatiotemporal matching of GPS data and video frames frame by frame through isochronous interpolation, providing a reliable spatiotemporal reference for subsequent reconstruction; based on the camera projection model and perspective transformation matrix, it effectively eliminates perspective distortion of road images caused by the camera's tilt angle; by using vehicle speed increments to calculate pixel increments, it achieves continuous frame stitching without feature point dependence, avoiding stitching errors caused by missing or mismatched feature points; the generated panoramic image can clearly and completely present the true spatial morphology and disease distribution of the road surface, providing high-precision data support for road inspection, maintenance, and management. Attached Figure Description

[0017] Figure 1 This is a flowchart of the present invention; Figure 2 The diagrams show the world coordinate system and camera coordinate system in this invention; (a) a schematic diagram of the camera coordinate system, (b) a three-dimensional schematic diagram of the world coordinate system and camera coordinate system, and (c) a two-dimensional schematic diagram of the world coordinate system and camera coordinate system. Figure 3 The following is a schematic diagram of the road image extraction, correction and stitching process in this invention: (a) Schematic diagram of the road area in the world coordinate system, (b) Schematic diagram of the road area in the image pixel coordinate system, (c) Schematic diagram of video frame image extraction, correction and stitching. Figure 4 The original video frame images containing different road surface features in this invention are: (a) an original video frame image containing only lane lines; (b) an original video frame image containing zebra crossings; and (c) an original video frame image containing expansion joints. Figure 5 The images shown are the stitched panoramic images in this invention; (a) stitched panoramic images without perspective correction; (b) stitched panoramic images after perspective correction. Detailed Implementation

[0018] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0019] like Figure 1 As shown, a method for reconstructing panoramic road images based on vehicle speed via in-vehicle video includes the following steps: Step S1: In order to compensate for the difference between high-definition video and GPS sampling rate, linear interpolation is performed on the longitude, latitude and velocity in GPS data so that the interpolated GPS data can accurately correspond to each video frame.

[0020] For the i-th video frame (counting from 0), its timestamp is: t = i / fps, (1) FPS represents the video frame rate. We determine two adjacent GPS sampling points located on either side of this frame: k = floor(t) = floor(i / fps), (2) Where k represents the index of the GPS sampling point acquired at the k-th second, indicating that the i-th video frame lies between the k-th and (k+1)-th GPS points, and the function floor(·) represents rounding down. The interpolation ratio r of the i-th video frame is defined as: r(i) = t – k = i / fps– k. (3) The geographic reference coordinates (longitude, latitude) and velocity of the i-th video frame are obtained through linear interpolation: lon(i) = lon(k) + r(i) * [lon(k+1) – lon(k)], (4) lat(i) = lat(k) + r(i) * [lat(k+1) – lat(k)], (5) vel(i) = vel(k) + r(i) * [vel(k+1) – vel(k)], (6)

[0021] Step S2, as follows Figure 2 As shown, coordinate system definition: Three types of coordinate systems are constructed as the reference for the camera projection model, and each coordinate system satisfies the right-hand rule: World coordinate system (vehicle rear coordinate system): The origin is the ground point directly below the camera, the Xw axis points to the left side of the vehicle, the Yw axis points to the rear of the vehicle, and the Zw axis points vertically upward; Camera coordinate system: The origin is the optical center of the camera, the Xc axis is aligned with the Xw axis of the world coordinate system (to ensure no rotational deviation in the horizontal direction of the road surface), the Yc axis is perpendicular to the optical axis and points towards the road surface, and the Zc axis points towards the rear of the vehicle along the optical axis; Image pixel coordinate system: The origin is located at the top left corner of the image, u is the horizontal position of the pixel, and v is the vertical position of the pixel.

[0022] Step S3: The ground point (X, Y, Z=0) is projected onto the image pixel (u, v) in the following manner: (7) (8) (9) The translation vector t represents the offset of the camera position (0, 0, h) after transformation from the world coordinate system to the camera coordinate system: (10) The model is calibrated once each time a vehicle is assembled and remains valid throughout the measurement period.

[0023] Step S4: First, set the proximity d. n and long distance d f (Along the Y-axis of the world coordinate system), determine the left and right boundaries at close range based on the horizontal field of view (calculated through intrinsic parameters), and obtain the world coordinates of the four corner points of the road area. Figure 3 a): Point C in the upper left corner: (-d n ·tan(α / 2), d f , 0), point D in the upper right corner: (d n ·tan(α / 2), d f , 0), point B in the lower right corner: (d n ·tan(α / 2), d n , 0), point A in the lower left: (-d n ·tan(α / 2), d n , 0), where α = 2arctan(W / 2f) is the horizontal field of view angle, and W is the image width.

[0024] Step S5: Calculate the pixel coordinates of the four corner points of the road area in the image using the camera projection model (Formula (7)). Assume they are: top left: (uC, vC), top right: (uD, vD), bottom right: (uB, vB), bottom left: (uA, vA). Due to perspective distortion, the area is trapezoidal ( Figure 3 (b) Perspective correction is required. Assume the coordinates of the four corner points of the corrected target rectangle are defined as: top left: (0, 0), top right: (Wt, 0), bottom right: (Wt, Ht), bottom left: (0, Ht), where Wt equals the camera image width, and Ht is the image height extracted based on the image increment. The perspective transformation matrix H is calculated using the least squares method: (11) in The image coordinates of the trapezoid vertices. These are the coordinates of the vertices of the rectangle.

[0025] Step S6, calculating the road increment for each video frame, involves extracting the instantaneous vehicle speed vel (unit: meters per second) corresponding to each video frame from the GPS data output in step S1, obtaining the frame rate fps (unit: frames per second) of the vehicle video, calculating the time interval Δt = 1 / fps (unit: seconds) between adjacent video frames, and calculating the physical road increment Δd (unit: meters) for each video frame using the formula Δd = vel × Δt. This road increment directly reflects the actual distance traveled by the vehicle within the shooting interval between two adjacent video frames, providing a physical scale benchmark for subsequently converting real-world road travel distances into pixel increments in the image.

[0026] Step S7, converting the road increment Δd obtained in S6 into an image pixel increment and performing perspective correction, is as follows: Based on the camera projection model constructed in step S3, the physical road increment Δd (distance along the vehicle's driving direction) is converted into the corresponding pixel increment Δp (unit: pixels) in the image pixel coordinate system through inverse projection calculation from the world coordinate system to the image pixel coordinate system; then, using the perspective transformation matrix H obtained in step S5, the image region corresponding to the pixel increment Δp (i.e., the newly added road image region in each frame of video) is subjected to perspective transformation, so that the perspective distortion of the region is eliminated, and a corrected image region with the same scale as the world coordinate system is obtained.

[0027] Step S8, the process of directly stitching the corrected images of each frame, is as follows: taking the first corrected image as the reference, the subsequent corrected images of each frame are translated and aligned along the vehicle's driving direction (consistent with the pixel increment direction) based on the pixel increment Δp obtained in step S7. Figure 3 c) The entire stitching process does not involve the detection, extraction or matching of road feature points, but only relies on pixel increments to achieve position alignment, ultimately generating a geometrically continuous and textured complete panoramic image of the road surface.

[0028] Simulation experiment: The effects of the present invention can be further illustrated by the following specific example: 1. The research data was selected from data collected on real roads. To effectively capture details of road surface defects (such as cracks ≥2mm wide and potholes ≥0.1 m² in area) and adapt to the coverage of panoramic stitching, a camera resolution of 1280×720 was chosen to ensure that each frame clearly captures road texture and minor defect features. The frame rate was fixed at 25 frames / second to strike a balance between real-time acquisition efficiency and data storage costs, avoiding image blurring at high speeds due to a low frame rate or data redundancy due to a high frame rate. A 6mm fixed-focus lens was rigidly mounted on the centerline of the roof rack, 3.32 m above the ground, tilted downwards at 50°. At this height and angle, a single frame image can cover a road surface strip 3.5–4.5 m wide. The vehicle speed was maintained at approximately 30 km / h to strike a balance between detection efficiency and image clarity.

[0029] 2. Experimental Results Raw road video images captured by the vehicle's high-definition camera, such as Figure 4 As shown, the video first undergoes speed-based frame extraction processing (matching frame intervals according to driving speed to ensure the continuity of road surface information between frames), then is processed by a perspective correction algorithm to finally obtain a seamlessly stitched panoramic image of the road surface. Figure 5 The results show that the video frame extraction method based on vehicle speed can accurately restore the real spatial shape of the lane; after perspective correction, the perspective distortion caused by the camera's downward angle in the original video is effectively eliminated, and the edge jaggedness effect that easily occurs during the stitching process is also reduced. Figure 5 a) was also significantly suppressed. The final panoramic image was able to clearly and completely present the road surface texture and the distribution of surface defects. Figure 5 b).

Claims

1. A method for reconstructing panoramic road surface images from vehicle-mounted video based on driving speed for road defect detection, characterized in that, Includes the following steps: Step S1: Acquire in-vehicle high-definition video and synchronously collected GPS data, and perform isochronous linear interpolation on the GPS data so that the interpolated GPS data corresponds one-to-one with each frame of video in the time dimension. Step S2: Define three types of coordinate systems: world coordinate system, camera coordinate system, and image pixel coordinate system, as the reference for the camera projection model; Step S3: Based on the coordinate system defined in step S2, and combined with the intrinsic and extrinsic parameters of the vehicle-mounted high-definition camera, establish a camera projection model from the ground point in the world coordinate system to the image pixel coordinate system. Step S4: Select a road area and determine the world coordinates of the four corner points of the road area based on the horizontal field of view. Step S5: Using the camera projection model constructed in step S3, calculate the corresponding coordinates of the four corner points of the road area in step S4 in the image pixel coordinate system; based on the mapping relationship between the rectangular features of the road area in the world coordinate system and the trapezoidal features in the image, solve the perspective transformation matrix H to realize the perspective distortion correction from the trapezoidal region to the rectangular region in the image. Step S6: Extract the driving speed corresponding to each video frame based on the GPS data processed in step S1, and calculate the road increment recorded in each video frame in combination with the video frame rate. Step S7: Using the camera projection model from step S3, convert the road increment obtained in step S6 into pixel increments in the image. Using the perspective transformation matrix H obtained in step S5, perspective correction is performed on the image region corresponding to the pixel increment. Step S8: For each frame of the corrected image from step S7, directly stitch them together based on pixel increments without relying on road surface feature point matching, and finally generate a complete panoramic image of the road surface.

2. The method for reconstructing panoramic road surface images based on vehicle speed for road defect detection according to claim 1, characterized in that, In step S1, the GPS data includes latitude and longitude, vehicle speed, and time information, and each frame of the vehicle video has a timestamp. The synchronous acquisition relationship between the GPS data and the vehicle video is confirmed based on the time information. The isochronous linear interpolation is performed on the latitude and longitude, vehicle speed, and time information in the GPS data to achieve accurate alignment between the interpolated data and the time dimension of each frame of video.

3. The method for reconstructing panoramic road surface images based on vehicle speed for road defect detection according to claim 1, characterized in that, In step S2, the world coordinate system is a vehicle rear coordinate system, with the ground point directly below the camera as the origin, the Xw axis pointing to the left side of the vehicle, the Yw axis pointing to the rear of the vehicle, and the Zw axis pointing vertically upward; the camera coordinate system has the camera optical center as the origin, the Xc axis is in the same direction as the Xw axis of the world coordinate system, the Yc axis is perpendicular to the optical axis pointing to the road surface, and the Zc axis points along the optical axis towards the rear of the vehicle; the origin of the image pixel coordinate system is located at the upper left corner of the image, u represents the horizontal position of the pixel, and v represents the vertical position of the pixel.

4. The method for reconstructing panoramic road surface images based on vehicle speed for road defect detection according to claim 1, characterized in that, In step S3, the camera projection model includes an intrinsic parameter matrix K, an extrinsic parameter matrix R, and a translation vector t; the camera intrinsic parameters include the lens focal length f and the optical center (cx, cy), which are obtained based on the camera's own parameters; the height h in the camera extrinsic parameters is measured by measuring tape with an accuracy of ±1cm, and the pitch angle θ in the extrinsic parameters is measured by tilt sensor with an accuracy of ±0.5°.

5. The method for reconstructing panoramic road surface images based on vehicle speed for road defect detection according to claim 1, characterized in that, In step S4, the nearest distance d of the road area n The df setting adapts to the camera's field of view and the image road increments generated by the vehicle speed at a distance from the camera. n ≤1m, df≤5m; the left and right distances of the road area are calculated based on the camera's field of view.

6. The method for reconstructing panoramic road surface images based on vehicle speed for road defect detection according to claim 1, characterized in that, In step S5, the process of solving the perspective transformation matrix H is as follows: Based on the rectangular coordinates of the four corner points of the road area determined in step S4 in the world coordinate system, the corresponding coordinates of the four corner points in the image pixel coordinate system are calculated through the projection model in step S4; using the mapping relationship between the right angle and parallel opposite sides of the rectangle in the world coordinate system and the geometric features of the trapezoid in the image, the perspective transformation matrix H is calculated through the homography matrix solving algorithm. This matrix H is used to convert the trapezoidal distortion area in the image caused by the camera's tilt angle into a rectangular area consistent with the geometric shape of the road area in the world coordinate system, thereby achieving accurate elimination of perspective distortion.

7. The method for reconstructing panoramic road surface images based on vehicle speed for road defect detection according to claim 1, characterized in that, In step S6, the driving speed vel corresponding to each video frame is extracted based on the GPS data processed in step S1. Combined with the video frame rate fps, the road increment Δd meters / frame recorded in each video frame is calculated. The calculation formula is Δd=vel / fps.

8. The method for reconstructing panoramic road surface images based on vehicle speed for road defect detection according to claim 1, characterized in that, In step S7, the road increment Δd obtained in step S6 is converted into a pixel increment in the image using the camera projection model in step S3; perspective correction is performed on the image region corresponding to the pixel increment using the perspective transformation matrix H obtained in step S5.

9. The method for reconstructing panoramic road surface images based on vehicle speed for road defect detection according to claim 1, characterized in that, In step S8, the corrected images from each frame in step S7 are directly stitched together based on pixel increments, without relying on road surface feature point matching, ultimately generating a complete panoramic image of the road surface.

Images