A binocular vision road defect detection method considering the influence of vehicle dynamics

Abstract

The present invention relates to a binocular vision road surface defect detection method considering the influence of vehicle dynamics, and belongs to the field of road surface defect detection. First, a binocular camera is used to take real-time images of the road surface in front of the vehicle, and a defect detection algorithm is used to identify road surface defects in the left-eye RGB image, and the road surface defect type is output; considering the changes in the binocular camera external parameters caused by the pitch, roll and yaw of the vehicle platform due to road surface excitation during vehicle driving, the external parameters of the binocular camera are corrected by estimating the position change of the vehicle platform, and the absolute position of the road surface defect is determined by the real position of the vehicle platform, combined with the relative position of the corrected road surface defect relative to the vehicle platform, so as to achieve accurate positioning of the road surface defect.

Description

Technical Field

[0001] The invention relates to the field of road surface defect detection, and in particular to a binocular vision road surface defect detection method taking into account the influence of vehicle dynamics. Background Art

[0002] After a period of use, the road surface is prone to horizontal cracks, longitudinal cracks, cracks, bulges, potholes and other road surface defects due to the impact of wheels and natural environmental factors. Poor road condition will lead to high road maintenance costs, poor vehicle ride comfort, and economy, long vehicle travel time, road congestion, and increased vehicle maintenance costs. In this case, road defect detection and positioning technology is quite necessary.

[0003] Traditional road surface defect detection uses manual methods, which are prone to false detection or missed detection, and are inefficient and costly. The road surface defect detection method based on vehicle dynamics response (such as CN103793570A, CN112590796A, CN205819110U) has a detection range fixed on the wheel trajectory, a limited detection range, and few identifiable defect types, which is prone to missed detection. The road surface defect detection method based on non-contact sensors (such as CN204594502U, CN113870263A) uses a defect detection algorithm based on a deep neural network to detect road surface defects. This method has a good detection effect on road surface defects, but the traditional road surface defect detection algorithm based on non-contact sensors does not take into account the road surface defect position measurement error caused by the change of sensor external parameters caused by the pitch, roll and yaw of the vehicle body due to road surface excitation during the driving process of the vehicle platform, and cannot determine the true position of the road surface defect. Summary of the invention

[0004] The purpose of the present invention is to provide a binocular vision road surface defect detection method taking into account the influence of vehicle dynamics, so as to achieve accurate positioning of road surface defects.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A binocular vision road surface defect detection method considering the influence of vehicle dynamics, comprising:

[0007] The vehicle platform and the left camera of the binocular camera mounted on the vehicle platform are jointly calibrated to determine the transformation matrix between the original coordinate system of the left camera and the original coordinate system of the vehicle platform;

[0008] Use a binocular camera to shoot the road ahead of the vehicle in real time to obtain left-eye RGB images and right-eye RGB images;

[0009] Use defect detection algorithm to identify road surface defects in the left RGB image, and output the road surface defect type and the center position of the road surface defect identification frame;

[0010] Stereo matching of the left RGB image and the right RGB image to obtain a depth map based on the left camera;

[0011] The center position of the road surface defect recognition frame is used as the position of the road surface defect, and the relative position of the road surface defect relative to the real-time coordinate system of the left camera is determined by using a parallax ranging method in combination with the depth map;

[0012] Obtaining real-time posture changes of the vehicle platform, and using the real-time posture changes to correct the external parameters of the binocular camera;

[0013] Determine the relative position of the road surface defect relative to the original coordinate system of the left camera according to the corrected external parameters and the relative position of the road surface defect relative to the real-time coordinate system of the left camera;

[0014] Determine the relative position of the road surface defect relative to the original coordinate system of the vehicle platform according to the conversion matrix and the relative position of the road surface defect relative to the original coordinate system of the left camera;

[0015] The absolute position of the road surface defect is determined based on the real-time absolute position of the vehicle platform and the relative position of the road surface defect with respect to the original coordinate system of the vehicle platform.

[0016] Optionally, the defect detection algorithm is a deep neural network-based defect detection algorithm trained according to the road surface defect type that needs to be identified.

[0017] Optionally, the stereo matching of the left-eye RGB image and the right-eye RGB image to obtain a depth map based on the left-eye camera specifically includes:

[0018] A semi-global matching algorithm is used to perform stereo matching on the left RGB image and the right RGB image to obtain a disparity map based on the left camera.

[0019] According to the disparity map based on the left camera, using the formula Get the initial depth map; where depth is the depth, baseline is the baseline length of the stereo camera, f is the focal length of the stereo camera, and disp is the disparity;

[0020] The initial depth map is filled with holes using mean filtering to obtain a depth map based on the left camera.

[0021] Optionally, the center position of the road surface defect identification frame is used as the position of the road surface defect, and the relative position of the road surface defect relative to the real-time coordinate system of the left camera is determined by using a parallax ranging method in combination with the depth map, specifically including:

[0022] Performing filtering and smoothing on the depth map;

[0023] The center position of the road surface defect recognition frame is used as the position of the road surface defect, and the depth information of the position of the road surface defect in the depth map after filtering and smoothing is determined;

[0024] According to the position information of the road surface defect in the pixel coordinate system and the depth information of the road surface defect in the filtered and smoothed depth map, the relative position of the road surface defect relative to the real-time coordinate system of the left camera is calculated using the parallax ranging method.

[0025] Optionally, the acquiring of the real-time posture change of the vehicle platform and using the real-time posture change to correct the external parameters of the binocular camera specifically includes:

[0026] The inertial measurement unit is used to measure the angular velocity data of the vehicle platform caused by the body vibration caused by road excitation during driving;

[0027] According to the angular velocity data, using the formula Calculate the real-time roll angle, pitch angle and yaw angle of the vehicle platform; where, and They represent the instantaneous angular velocity of the vehicle platform around the X, Y and Z axes of the vehicle's original coordinate system at time t, Δt is the sampling time of the inertial measurement unit, and They represent the roll angle, pitch angle and yaw angle of the vehicle platform at time t, respectively. and They represent the roll angle, pitch angle and yaw angle of the vehicle platform at time t+1 respectively;

[0028] According to the real-time roll angle, pitch angle and yaw angle of the vehicle platform, according to the formula Calculate the external parameter correction rotation matrix; where R represents the external parameter correction rotation matrix, α, β, γ represent the flip angle, pitch angle and yaw angle respectively, S θ = sinθ, C θ =cosθ, θ is α, β or γ.

[0029] Optionally, the formula for determining the relative position of the road surface defect with respect to the original coordinate system of the left camera is:

[0030]

[0031] Where [XYZ] c is the relative position of the road surface defect relative to the real-time coordinate system of the left camera, [XYZ] R is the relative position of the road surface defect relative to the original coordinate system of the left camera.

[0032] Optionally, the real-time absolute position of the vehicle platform is obtained by:

[0033] Through the GPS receiver, the carrier phase difference technology is used to calculate the real-time absolute position of the vehicle platform.

[0034] A binocular vision road defect detection system considering the influence of vehicle dynamics, comprising:

[0035] A calibration module is used to jointly calibrate the vehicle platform and the left camera of the binocular camera mounted on the vehicle platform, and determine the conversion matrix between the original coordinate system of the left camera and the original coordinate system of the vehicle platform;

[0036] The shooting module is used to use a binocular camera to shoot the road surface in front of the vehicle in real time to obtain a left-eye RGB image and a right-eye RGB image;

[0037] A defect detection module is used to identify road surface defects in the left RGB image using a defect detection algorithm, and output the road surface defect type and the center position of the road surface defect identification frame;

[0038] A stereo matching module is used to stereo match the left RGB image and the right RGB image to obtain a depth map based on the left camera;

[0039] A parallax ranging module is used to use the center position of the road surface defect recognition frame as the position of the road surface defect, and in combination with the depth map, use a parallax ranging method to determine the relative position of the road surface defect relative to the real-time coordinate system of the left camera;

[0040] An external parameter correction module is used to obtain the real-time posture change of the vehicle platform and use the real-time posture change to correct the external parameters of the binocular camera;

[0041] A position correction module is used to determine the relative position of the road surface defect relative to the original coordinate system of the left camera according to the corrected external parameters and the relative position of the road surface defect relative to the real-time coordinate system of the left camera;

[0042] A conversion module, used to determine the relative position of the road surface defect relative to the original coordinate system of the vehicle platform according to the conversion matrix and the relative position of the road surface defect relative to the original coordinate system of the left camera;

[0043] The absolute position determination module is used to determine the absolute position of the road surface defect according to the real-time absolute position of the vehicle platform and the relative position of the road surface defect with respect to the original coordinate system of the vehicle platform.

[0044] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0045] The present invention discloses a binocular vision road surface defect detection method considering the influence of vehicle dynamics. Firstly, a binocular camera is used to capture a road surface image in front of a vehicle in real time, a defect detection algorithm is used to identify road surface defects in a left-eye RGB image, and the road surface defect type is output. Taking into account the changes in the binocular camera extrinsic parameters caused by the pitch, roll and yaw of a vehicle platform due to road surface excitation during vehicle driving, the extrinsic parameters of the binocular camera are corrected by estimating the position change of the vehicle platform, and the absolute position of the road surface defect is determined by the real position of the vehicle platform combined with the relative position of the corrected road surface defect relative to the vehicle platform, so as to achieve accurate positioning of the road surface defect. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 A flow chart of a binocular vision road surface defect detection method considering the influence of vehicle dynamics provided by an embodiment of the present invention;

[0048] Figure 2 Schematic diagram of a binocular vision road surface defect detection method considering the influence of vehicle dynamics provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] The purpose of the present invention is to provide a binocular vision road surface defect detection method taking into account the influence of vehicle dynamics, so as to achieve accurate positioning of road surface defects.

[0051] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] The detection and location of road surface defects can help the vehicle platform accurately perceive the road conditions ahead, provide a basis for the active control of the vehicle suspension and driving path planning, and timely discover road surface defects to assist the road surface management department in formulating road surface maintenance plans. It is of great significance for the development of active suspension, reducing the risk of vehicle damage, and reducing road surface maintenance costs.

[0053] In order to accurately identify road defects and determine their precise locations, the present invention proposes a binocular vision road defect detection method considering the influence of vehicle dynamics. Based on a binocular camera and considering the influence of vehicle dynamics, the extrinsic parameters of the binocular camera are changed due to the pitch, roll and yaw of the vehicle platform caused by road excitation during vehicle driving. By estimating the flip angle θ of the vehicle platform, the method can detect road defects accurately and accurately. R , pitch angle θ P and yaw angle θ Y Correct the external parameters of the binocular camera, obtain more accurate point cloud data of road defects based on the depth image, determine the relative position of the road defect relative to the vehicle platform, and then determine the absolute position of the road defect.

[0054] The embodiment of the present invention provides a binocular vision road surface defect detection method considering the influence of vehicle dynamics, which mainly consists of four parts: road surface defect recognition, camera posture calculation, platform positioning and road surface defect positioning. Specifically, it includes the following steps:

[0055] Step S1, jointly calibrate the vehicle platform and the left camera of the binocular camera mounted on the vehicle platform to determine the transformation matrix between the original coordinate system of the left camera and the original coordinate system of the vehicle platform.

[0056] Corresponding coordinate systems are established for the vehicle platform and the left camera of the binocular camera mounted on the vehicle platform, the inertial measurement unit, and the GPS receiver, and joint calibration is performed to determine the conversion relationship between different coordinate systems.

[0057] The binocular camera is calibrated to determine its intrinsic and extrinsic parameters.

[0058] Step S2, using a binocular camera to shoot the road surface in front of the vehicle in real time to obtain a left-eye RGB image and a right-eye RGB image.

[0059] When the vehicle is driving, the binocular camera captures the road conditions in front of the vehicle, and extracts the original RGB image of the left camera (left RGB image) and the original RGB image of the right camera (right RGB image).

[0060] Step S3, using a defect detection algorithm to identify road surface defects in the left RGB image, and outputting the road surface defect type and the center position of the road surface defect identification frame.

[0061] The defect detection algorithm based on deep neural network is trained in advance according to the road surface defect type that needs to be identified. The defect detection algorithm identifies road surface defects in the original RGB image of the left camera, and outputs the type of road surface defect, the center point coordinates of the road surface defect identification frame, and the width and height data of the identification frame.

[0062] Step S4: stereo matching the left-eye RGB image and the right-eye RGB image to obtain a depth map based on the left-eye camera.

[0063] Reference Figure 2 According to the original RGB images of the left and right cameras of the binocular camera, the semi-global block matching algorithm (SGBM) is used for stereo matching, and the disparity map based on the left camera is calculated. The depth map is obtained by using the geometric relationship of the binocular cameras (the geometric relationship refers to the focal length and baseline length of the binocular cameras). Then, the mean filter is used to fill the holes in the depth map to obtain the depth map based on the left camera.

[0064]

[0065] Where depth is the depth, baseline is the baseline length of the stereo camera, f is the focal length of the stereo camera, and disp is the disparity.

[0066] Steps S1 to S4 constitute Figure 2 The road surface defect identification part in .

[0067] Step S5, taking the center position of the road surface defect identification frame as the position of the road surface defect, and combining the depth map, using the parallax ranging method to determine the relative position of the road surface defect with respect to the real-time coordinate system of the left camera.

[0068] The depth map output in step S4 is filtered and smoothed, and the center position of the road surface defect identification frame is used as the position of the defect. The relative position of the road surface defect relative to the real-time coordinate system of the binocular camera is calculated using the depth information of the center point of the road surface defect and the position information in the pixel coordinate system and the parallax ranging method.

[0069] Step S6, obtaining the real-time posture change of the vehicle platform, and using the real-time posture change to correct the external parameters of the binocular camera.

[0070] Since the binocular camera is fixed to the vehicle platform, the flip angle θ of the left camera real-time coordinate system relative to the left camera original coordinate system is R , pitch angle θ P and yaw angle θ Y Approximately equal to the flip angle θ of the vehicle platform real-time coordinate system relative to the vehicle platform original coordinate system R , pitch angle θ P and yaw angle θ Y The inertial measurement unit is used to measure the angular velocity data of the vehicle platform caused by the body vibration caused by road excitation during driving, and the real-time flip angle θ of the vehicle platform is obtained by integrating it. R , pitch angle θ P and yaw angle θY Angular acceleration is sensitive to short-term rapid motion, so the sampling frequency is high. The sampling frequency of the inertial measurement unit is higher than the image acquisition frequency of the binocular camera. In order to make the image collected by the camera at a certain moment correspond to the angle change of the coordinate system at that time, the binocular camera and the inertial measurement unit are time-stamped and aligned. The angular velocity data is integrated to obtain the flip angle θ of the vehicle platform real-time coordinate system relative to the vehicle platform original coordinate system. R , pitch angle θ P and yaw angle θ Y .

[0071]

[0072]

[0073]

[0074] In the formula, and They represent the instantaneous angular velocity of the vehicle platform around the XYZ axes of the vehicle's original coordinate system at time t, and Δt is the sampling time of the inertial measurement unit.

[0075] Using the measured flip angle θ R , pitch angle θ P and yaw angle θ Y Calculate the external parameter correction rotation matrix R, where [α, β, γ] represent the flip angle θ R , pitch angle θ P and yaw angle θ Y , S θ = sinθ, C θ =cosθ.

[0076]

[0077] Step S6 corresponds to Figure 2 The camera pose calculation part in .

[0078] Step S7, determining the relative position of the road surface defect relative to the original coordinate system of the left camera according to the corrected external parameters and the relative position of the road surface defect relative to the real-time coordinate system of the left camera.

[0079] The relative position of the road surface defect relative to the original coordinate system of the left camera is calculated using the external parameter correction rotation matrix R, [XYZ] c is the relative position of the road surface defect relative to the real-time coordinate system of the left camera, [XYZ] R is the relative position of the road surface defect relative to the original coordinate system of the left camera.

[0080]

[0081] Step S8, determining the relative position of the road surface defect relative to the original coordinate system of the vehicle platform according to the transformation matrix and the relative position of the road surface defect relative to the original coordinate system of the left camera.

[0082] The relative position of the road surface defect with respect to the original coordinate system of the vehicle platform is calculated through the coordinate system transformation matrix of the left camera original coordinate system and the vehicle platform original coordinate system.

[0083] Step S9, determining the absolute position of the road surface defect according to the real-time absolute position of the vehicle platform and the relative position of the road surface defect with respect to the original coordinate system of the vehicle platform.

[0084] The high-precision position of the vehicle platform is solved in real time through the carrier phase differential technology (RTK), and the absolute position of the vehicle platform is obtained by using the conversion matrix between the original coordinate system of the vehicle platform and the coordinate system of the GPS receiver. Figure 2 The platform positioning section in .

[0085] The absolute position of the road surface defect is calculated using the obtained real position of the vehicle platform and the relative position of the road surface defect relative to the original coordinate system of the vehicle platform obtained in step S8.

[0086] Step S5, step S7 to step S9 constitute Figure 2 The pavement defect location part in.

[0087] The present invention considers the problem of inaccurate positioning of road defects caused by vehicle dynamic response caused by road excitation during vehicle driving, uses the original RGB image of the left camera of the binocular camera to identify road defects, and obtains a depth image through stereo matching calculation of the binocular camera. The posture change of the vehicle platform during driving is obtained in real time through the inertial measurement unit (IMU), the external parameters of the binocular camera are corrected, and the point cloud data of the road defect is generated through the depth image, and the relative position of the road defect relative to the vehicle platform is calculated. Combined with the absolute position of the vehicle platform determined by GPS, the absolute position of the road defect is determined.

[0088] The road surface defect recognition part includes two-dimensional road surface defect recognition and road surface depth image extraction. The road surface in front of the vehicle is photographed by a binocular camera, and the road surface defect recognition algorithm is used to recognize road surface defects on the original RGB image taken by the left camera of the binocular camera. The road surface defect type (such as bulges, potholes and cracks), the center point coordinates of the road surface defect recognition frame in the pixel coordinate system, and the width and height data of the recognition frame are output. At the same time, the semi-global matching algorithm is used to perform stereo matching on the images taken by the binocular camera to generate a disparity map based on the left camera, and the depth map is obtained using the internal parameters of the binocular camera. The depth map is then filled with holes in the depth map using the mean filter to obtain a depth map based on the left camera.

[0089] The camera pose calculation part uses an inertial measurement unit to measure the changes in body acceleration and angular acceleration caused by the dynamic response of the vehicle platform caused by road excitation during driving. The absolute position of the vehicle platform is obtained by GPS in the platform positioning part, so the camera pose calculation part only needs to calculate the real-time pose information of the binocular camera, and the flip angle θ of the vehicle platform is obtained by integrating the three-dimensional angular acceleration data of the center of mass of the vehicle platform measured by the inertial measurement unit. R , pitch angle θ P and yaw angle θ Y , through the posture change of the vehicle platform, the real-time coordinate system of the left camera is updated, and the external parameters of the binocular camera are updated.

[0090] The platform positioning part uses carrier phase difference technology to solve the high-precision position of the vehicle platform in real time, and outputs it to the road surface defect positioning part as the absolute position of the vehicle platform to calculate the absolute position of the road surface defect.

[0091] The pavement defect positioning part calculates the absolute position of the pavement defect using the data obtained from the pavement defect recognition part, the camera posture calculation part, and the platform positioning part. The absolute position of the center point of the pavement defect recognition frame is taken as the absolute position of the pavement defect. The pavement point cloud data is calculated using the depth map output by the pavement defect recognition part, and the pavement point cloud data is smoothed and filtered to extract the position of the center point of the defect recognition frame as the relative position of the pavement defect relative to the binocular camera. The pavement defect point cloud data is converted into coordinates using the corrected binocular camera extrinsics calculated by the camera posture calculation part to correct the relative position of the pavement defect based on the left camera. The absolute position of the pavement defect is calculated using the absolute position of the vehicle platform output by the platform positioning part combined with the corrected relative position of the pavement defect.

[0092] This method takes into account the changes in binocular camera extrinsic parameters caused by vehicle body vibration during normal driving. By measuring the angular change of the real-time coordinates of the vehicle platform relative to the original coordinates of the vehicle platform, the binocular camera extrinsic parameters are corrected, and the relative position of the road surface defect relative to the vehicle platform is calculated. The absolute position of the road surface defect is determined in combination with the absolute position of the vehicle platform, which effectively improves the accuracy of road surface defect positioning.

[0093] The present invention also provides a binocular vision road surface defect detection system considering the influence of vehicle dynamics, comprising:

[0094] A calibration module is used to jointly calibrate the vehicle platform and the left camera of the binocular camera mounted on the vehicle platform, and determine the conversion matrix between the original coordinate system of the left camera and the original coordinate system of the vehicle platform;

[0095] The shooting module is used to use a binocular camera to shoot the road surface in front of the vehicle in real time to obtain a left-eye RGB image and a right-eye RGB image;

[0096] A defect detection module is used to identify road surface defects in the left RGB image using a defect detection algorithm, and output the road surface defect type and the center position of the road surface defect identification frame;

[0097] A stereo matching module is used to stereo match the left RGB image and the right RGB image to obtain a depth map based on the left camera;

[0098] A parallax ranging module is used to use the center position of the road surface defect recognition frame as the position of the road surface defect, and in combination with the depth map, use a parallax ranging method to determine the relative position of the road surface defect relative to the real-time coordinate system of the left camera;

[0099] An external parameter correction module is used to obtain the real-time posture change of the vehicle platform and use the real-time posture change to correct the external parameters of the binocular camera;

[0100] A position correction module is used to determine the relative position of the road surface defect relative to the original coordinate system of the left camera according to the corrected external parameters and the relative position of the road surface defect relative to the real-time coordinate system of the left camera;

[0101] A conversion module, used to determine the relative position of the road surface defect relative to the original coordinate system of the vehicle platform according to the conversion matrix and the relative position of the road surface defect relative to the original coordinate system of the left camera;

[0102] The absolute position determination module is used to determine the absolute position of the road surface defect according to the real-time absolute position of the vehicle platform and the relative position of the road surface defect with respect to the original coordinate system of the vehicle platform.

[0103] The binocular vision road surface defect detection system considering the influence of vehicle dynamics provided in the embodiment of the present invention has similar working principles and beneficial effects as the binocular vision road surface defect detection method considering the influence of vehicle dynamics described in the above embodiment, so they will not be described in detail here. For specific contents, please refer to the introduction of the above method embodiment.

[0104] Terminology explanation:

[0105] Geodetic coordinate system: The geodetic coordinate system is fixed to the ground, with the origin O base is the center of mass of the vehicle platform at the initial position, the x-axis direction is defined as the forward direction of the vehicle platform, the y-axis direction is defined as the left direction of the vehicle platform, and the z-axis direction is defined as vertically upward.

[0106] Vehicle fixed coordinate system: The vehicle fixed coordinate system is fixed to the vehicle body, with the origin O odomIt is the center of mass of the vehicle platform. The direction of the initial coordinate axis coincides with the direction of the earth coordinate system, which mainly reflects the real-time posture changes of the vehicle platform.

[0107] Left camera geodetic coordinate system: The left camera geodetic coordinate system is fixed to the ground, with the origin O cam The optical center of the left camera at the initial position is taken as the coordinate origin, and the direction of the coordinate axis is the same as that of the geodetic coordinate system.

[0108] Left camera real-time coordinate system: The left camera real-time coordinate system is fixed to the left camera, with the origin O camodom It is the optical center of the left camera. The direction of the initial coordinate system is the same as that of the earth coordinate system, which mainly reflects the real-time posture change of the left camera.

[0109] Pixel coordinate system: A coordinate system with pixels as the origin, established with the upper left corner of the image. Absolute position: The position of the target in the earth coordinate system.

[0110] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0111] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A binocular vision road defect detection method considering the influence of vehicle dynamics, characterized in that: include: The vehicle platform and the left camera of the binocular camera mounted on the vehicle platform are jointly calibrated to determine the transformation matrix between the original coordinate system of the left camera and the original coordinate system of the vehicle platform; Use a binocular camera to shoot the road ahead of the vehicle in real time to obtain left-eye RGB images and right-eye RGB images; Use defect detection algorithm to identify road surface defects in the left RGB image, and output the road surface defect type and the center position of the road surface defect identification frame; Stereo matching of the left RGB image and the right RGB image to obtain a depth map based on the left camera; The center position of the road surface defect recognition frame is used as the position of the road surface defect, and the relative position of the road surface defect relative to the real-time coordinate system of the left camera is determined by using a parallax ranging method in combination with the depth map; Obtaining real-time posture changes of the vehicle platform, and using the real-time posture changes to correct the external parameters of the binocular camera; Determine the relative position of the road surface defect relative to the original coordinate system of the left camera according to the corrected external parameters and the relative position of the road surface defect relative to the real-time coordinate system of the left camera; Determine the relative position of the road surface defect relative to the original coordinate system of the vehicle platform according to the conversion matrix and the relative position of the road surface defect relative to the original coordinate system of the left camera; The absolute position of the road surface defect is determined based on the real-time absolute position of the vehicle platform and the relative position of the road surface defect with respect to the original coordinate system of the vehicle platform.

2. The binocular vision road surface defect detection method considering the influence of vehicle dynamics according to claim 1 is characterized in that: The defect detection algorithm is a defect detection algorithm based on a deep neural network that is trained according to the road surface defect type that needs to be identified.

3. The binocular vision road surface defect detection method considering the influence of vehicle dynamics according to claim 1 is characterized in that: The stereo matching of the left-eye RGB image and the right-eye RGB image to obtain a depth map based on the left-eye camera specifically includes: A semi-global matching algorithm is used to perform stereo matching on the left RGB image and the right RGB image to obtain a disparity map based on the left camera. According to the disparity map based on the left camera, using the formula Get the initial depth map; where depth is the depth, baseline is the baseline length of the stereo camera, f is the focal length of the stereo camera, and disp is the disparity; The initial depth map is filled with holes using mean filtering to obtain a depth map based on the left camera.

4. The binocular vision road surface defect detection method considering the influence of vehicle dynamics according to claim 1 is characterized in that: The method of using the center position of the road surface defect recognition frame as the position of the road surface defect and combining the depth map to determine the relative position of the road surface defect relative to the real-time coordinate system of the left camera using a parallax ranging method specifically includes: Performing filtering and smoothing on the depth map; The center position of the road surface defect recognition frame is used as the position of the road surface defect, and the depth information of the position of the road surface defect in the depth map after filtering and smoothing is determined; According to the position information of the road surface defect in the pixel coordinate system and the depth information of the road surface defect in the filtered and smoothed depth map, the relative position of the road surface defect relative to the real-time coordinate system of the left camera is calculated using the parallax ranging method.

5. The binocular vision road surface defect detection method considering the influence of vehicle dynamics according to claim 1 is characterized in that: The obtaining of the real-time posture change of the vehicle platform and using the real-time posture change to correct the external parameters of the binocular camera specifically includes: The inertial measurement unit is used to measure the angular velocity data of the vehicle platform caused by the body vibration caused by road excitation during driving; According to the angular velocity data, using the formula Calculate the real-time roll angle, pitch angle and yaw angle of the vehicle platform; where, and They represent the instantaneous angular velocity of the vehicle platform around the X, Y and Z axes of the vehicle's original coordinate system at time t, Δt is the sampling time of the inertial measurement unit, and They represent the roll angle, pitch angle and yaw angle of the vehicle platform at time t, respectively. and They represent the roll angle, pitch angle and yaw angle of the vehicle platform at time t+1 respectively; According to the real-time roll angle, pitch angle and yaw angle of the vehicle platform, according to the formula Calculate the external parameter correction rotation matrix; where R represents the external parameter correction rotation matrix, [α, β, γ] represents the flip angle, pitch angle and yaw angle respectively, S θ = sinθ, C θ =cosθ, θ is α, β or γ.

6. The binocular vision road surface defect detection method considering the influence of vehicle dynamics according to claim 5 is characterized in that: The formula for determining the relative position of the road surface defect relative to the original coordinate system of the left camera is: Where [XYZ] c is the relative position of the road surface defect relative to the real-time coordinate system of the left camera, [XYZ] R is the relative position of the road surface defect relative to the original coordinate system of the left camera.

7. The binocular vision road surface defect detection method considering the influence of vehicle dynamics according to claim 1 is characterized in that: The real-time absolute position of the vehicle platform is obtained as follows: Through the GPS receiver, the carrier phase difference technology is used to calculate the real-time absolute position of the vehicle platform.

8. A binocular vision road defect detection system considering the influence of vehicle dynamics, characterized in that: include: A calibration module is used to jointly calibrate the vehicle platform and the left camera of the binocular camera mounted on the vehicle platform, and determine the conversion matrix between the original coordinate system of the left camera and the original coordinate system of the vehicle platform; The shooting module is used to use a binocular camera to shoot the road surface in front of the vehicle in real time to obtain a left-eye RGB image and a right-eye RGB image; A defect detection module is used to identify road surface defects in the left RGB image using a defect detection algorithm, and output the road surface defect type and the center position of the road surface defect identification frame; A stereo matching module is used to stereo match the left RGB image and the right RGB image to obtain a depth map based on the left camera; A parallax ranging module is used to use the center position of the road surface defect recognition frame as the position of the road surface defect, and in combination with the depth map, use a parallax ranging method to determine the relative position of the road surface defect relative to the real-time coordinate system of the left camera; An external parameter correction module is used to obtain the real-time posture change of the vehicle platform and use the real-time posture change to correct the external parameters of the binocular camera; A position correction module is used to determine the relative position of the road surface defect relative to the original coordinate system of the left camera according to the corrected external parameters and the relative position of the road surface defect relative to the real-time coordinate system of the left camera; A conversion module, used to determine the relative position of the road surface defect relative to the original coordinate system of the vehicle platform according to the conversion matrix and the relative position of the road surface defect relative to the original coordinate system of the left camera; The absolute position determination module is used to determine the absolute position of the road surface defect according to the real-time absolute position of the vehicle platform and the relative position of the road surface defect with respect to the original coordinate system of the vehicle platform.

Images