Visual compensation system based on fusion of depth vision camera and laser radar

Abstract

The invention discloses a vision compensation system based on fusion of a depth vision camera and a laser radar, and relates to the technical field of aided driving, the vision compensation system comprises a data acquisition module used for acquiring vehicle data including point cloud data and image data, and outputting the vehicle data; the data processing module is used for acquiring vehicle data, processing the point cloud data and acquiring point cloud attitude data; processing the road condition image data, performing registration judgment on the road condition image data and the road condition attitude data, and calculating an offset coefficient through perspective superposition when the point cloud attitude data is not registered with the road condition attitude data; meanwhile, road surface image data are obtained, a pit coefficient is calculated through machine recognition, and the offset coefficient and the pit coefficient are output; the bumping coefficient acquisition module is used for calculating a bumping coefficient according to the acquired pothole coefficient and the offset coefficient; based on a large amount of training data in a real scene, the machine learning model can continuously optimize the generalization ability of itself.

Description

Technical Field

[0001] The present invention relates to the field of assisted driving technology, and in particular to a vision compensation system based on the fusion of a depth vision camera and a laser radar. Background Art

[0002] With the development of science and technology, autonomous driving technology has gradually become an indispensable part of the modern transportation system. In order to ensure that autonomous vehicles can operate safely and efficiently in various complex environments, it is particularly important to accurately obtain and continuously update the vehicle's posture information (i.e., position and direction).

[0003] In addition, a Chinese patent with patent application number "CN202311835273.3" discloses a vehicle control method, device, equipment and storage medium under a bumpy road section; by obtaining vehicle status data during vehicle driving; if the vehicle status data meets the vehicle bumpy state condition, it is confirmed that the current vehicle is in a bumpy state; the position information of the vehicle in the bumpy state is obtained, the position is marked as a bumpy road section and uploaded to a high-precision map in the cloud; when it is detected that the vehicle will enter the bumpy road section for the second time, the vehicle chassis system is adjusted; this application can identify the bumps and potholes on the road ahead in advance, and at the same time, give an early warning to the driver, and adaptively adjust the chassis height in advance, so that the owner can understand the road conditions ahead in advance, and protect the life and property safety of the vehicle and the owner in a timely and effective manner; although the risk of the vehicle chassis scratching the road surface can be reduced, this method is only applicable to vehicles with a retractable chassis, and at the same time, the collision around the vehicle that may occur due to the bumpy road surface during the automatic driving process is not fully considered.

[0004] In recent years, the field of autonomous driving has made significant progress, but it still faces many challenges. On the one hand, although the existing global positioning system (GPS) can provide vehicles with approximate location information, in urban canyons, tunnels or country roads, the rugged road surface is full of uncertainty, causing the vehicle to bump, causing its initial positioning point to constantly change, which brings many uncertainties to the path planning of autonomous driving. On the other hand, although the inertial measurement unit (IMU) can make up for the shortcomings when the GPS signal is missing, due to the existence of drift, long-term reliance on IMU will also accumulate errors.

[0005] Therefore, how to combine data from multiple sensors to build a robust solution that provides sufficient safety redundant distance for autonomous vehicles has become one of the current research focuses. In addition, autonomous vehicles need to constantly adapt to changes in the surrounding environment, including but not limited to factors such as road conditions and traffic flow. Especially when driving on unstructured roads, such as country roads or construction areas, there is a lack of clear lane lines and landmarks, which places higher demands on vehicle navigation. Traditional methods based on a single sensor for recognition are often difficult to work in this case, because existing sensors may not be able to capture the latest road conditions in a timely and effective manner. Therefore, it is particularly important to explore a method that does not rely entirely on a single sensor to provide only a safety distance for bumpy roads. Summary of the invention

[0006] 1. Technical issues to be resolved

[0007] In view of the shortcomings of the prior art, the present invention provides a vision compensation system based on the fusion of a depth vision camera and a lidar to solve the problem that the rough road surface is full of uncertainty, causing the vehicle to shake, thereby causing its initial positioning point to constantly change. During the autonomous driving process, the bumpy road surface may lead to possible vehicle collisions, and bring many uncertain factors to the autonomous driving path planning.

[0008] (II) Technical solution

[0009] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0010] A visual compensation system based on the fusion of a deep vision camera and a laser radar, the compensation system:

[0011] Data acquisition module: used to collect vehicle data, including point cloud data and image data, and output the vehicle data;

[0012] Data processing module: used to obtain vehicle data, process point cloud data, and obtain point cloud posture data; at the same time, process road image data and perform registration judgment with road posture data. When point cloud posture data and road posture data are not registered, calculate the offset coefficient through perspective superposition; at the same time, obtain road image data, calculate pothole coefficient by machine recognition, and output the offset coefficient and pothole coefficient;

[0013] The bump coefficient acquisition module obtains the pothole coefficient and the offset coefficient and calculates the bump coefficient.

[0014] Furthermore, the process of acquiring point cloud data and image data is as follows:

[0015] Set a trigger mechanism with the acquisition frame rate of g per second to control the acquisition frequency of the lidar, and map the original distance measurement value output by the lidar from the lidar coordinate system to the vehicle coordinate system to convert it into 3D point cloud data;

[0016] The image data includes road surface image data and road condition image data; and is used to capture road surface images and road condition images.

[0017] Furthermore, the process of acquiring point cloud posture data is as follows:

[0018] S101: After calibration, a transformation matrix is ​​obtained, the matrix including rotation and translation information, which is used to map points in the laser radar coordinate system to the vehicle coordinate system, and obtain the vehicle coordinate system transformation matrix T;

[0019] S102: Using laser radar to collect point cloud data set P LiDAR ; By multiplying the transformation matrix T, we can obtain its corresponding position P in the vehicle coordinate system Vehicle；

[0020] S103: Obtain the rotation matrix R1 and translation vector t1 of the initial point before the vehicle bumps, as well as the rotation matrix R2 and translation vector t2 of the initial point with uncertainty offset, iteratively find the closest point pair, and minimize the distance error between them. The formula is:

[0021] Formula 1:

[0022] Formula 2:

[0023] Where pi and pj are the corresponding points selected from the corresponding position PVehicle respectively; N is the maximum value range; i is the current corresponding point, and its value is i=1,2,3...N; j is the current corresponding point, and its value is j=1,2,3...N, and j≠i; Formula 1 and Formula 2 are combined to calculate the value of the rotation matrix R2 and the translation vector t2.

[0024] Furthermore, the process of processing the road condition image data is as follows:

[0025] The collected road condition image data is preprocessed, including noise removal, distortion correction, contrast enhancement and wide dynamic range processing.

[0026] Furthermore, the process of acquiring the road condition posture data is as follows:

[0027] The geometric relationship between the two frames decomposes the essential matrix into a rotation matrix R2' and a translation vector t2'.

[0028] Further, registration judgment:

[0029] The registration of the rotation matrix and translation vector in the point cloud posture data and the road condition posture data is judged by:

[0030] When the rotation matrix and translation vector in the point cloud posture data are equal to the rotation matrix and translation vector in the road condition posture data, it is determined to be aligned and the current distance is maintained;

[0031] When the rotation matrix and translation vector in the point cloud posture data are not equal to the rotation matrix and translation vector of the road posture data employee, it is judged as misalignment and the offset coefficient is calculated.

[0032] Furthermore, the calculation process of the offset coefficient is:

[0033] Get the corresponding position P respectively Vehicle The corresponding point p selected from i and p j , the point cloud posture data rotation matrix R2 and translation vector t2, as well as the rotation matrix R2' and translation vector t2', use the objective function to measure the difference between the transformed posture data and the actual image posture data, and minimize the sum of square errors between matching feature point pairs as the objective function. The formula is:

[0034]

[0035] Where E is the offset coefficient; t1 and t1' are the translation vector of the point cloud posture data and the translation vector of the road posture data of the initial point before the vehicle bumps, respectively; R1 and R1' are the rotation matrices of the point cloud posture data and the road posture data before the vehicle bumps, respectively.

[0036] Furthermore, the road surface image data processing process is:

[0037] S301: Collect road images and video data for annotation, and record the rotation matrix and road condition posture data during vehicle driving;

[0038] S302: Using computer vision technology to extract features from images or video streams, using gray-level co-occurrence matrices to describe the spatial relationship between pixels in the image; using deep learning models to automatically learn and extract complex feature representations;

[0039] S303: Identifying potholes on the road according to the trained classifier;

[0040] S304: Quantify the pothole area of ​​a specific road surface or location using comprehensive indicators, obtain the geometric characteristics of the pothole using machine algorithms, and generate a pothole coefficient in combination with vibration data.

[0041] Further, the calculation of geometric features:

[0042] By considering the geometric characteristics of the pothole, the area c is preliminarily estimated according to the formula q , the formula is:

[0043] C q =w d ·D+w a ·q

[0044] Where D is the maximum depth of the pothole, q is the maximum width of the pothole, wd and wa are given weight factors, reflecting the importance of different dimensions;

[0045] The speed factor Fv(v) is introduced to dynamically adjust the coefficient and calculate the pothole coefficient. The formula is:

[0046]

[0047] Where ka is the pothole coefficient; Fv(v) is the speed factor; A z is the vibration frequency; f is the average number of vehicle bumps.

[0048] Furthermore, the process of obtaining the turbulence coefficient is as follows:

[0049] After obtaining the pothole coefficient of the real-time road surface, the bump coefficient of the vehicle on the current road surface is calculated based on the offset coefficient calculated based on the point cloud posture data and the road condition posture data. The formula is:

[0050]

[0051] In the formula, B i is the turbulence coefficient.

[0052] (III) Beneficial effects

[0053] The present invention provides a vision compensation system based on the fusion of a depth vision camera and a laser radar, which has the following beneficial effects:

[0054] By introducing the method of feature point matching and geometric relationship calculation, the positioning accuracy of autonomous driving vehicles in complex environments can be significantly improved; this method uses visual sensors to capture feature points in continuous frame images, and through an efficient and accurate matching algorithm, it ensures stable performance even in unstructured roads or severe weather conditions; in addition, the rotation and translation parameters obtained by decomposing the essential matrix can accurately describe the relative motion between the two frames, thereby achieving high-precision updates of the global posture; this vision-based positioning solution not only makes up for the shortcomings of traditional positioning methods, but also enhances the overall robustness and adaptability of the system.

[0055] By using advanced computer vision and deep learning algorithms, the system can accurately identify potholes in complex road environments. Combining data collection methods with multiple sensors (such as depth vision cameras and lidars) not only enriches the input information, but also enhances the system's adaptability to various environmental conditions. By fusing data from different sources to compensate for the limitations of a single sensor, it ensures continuous and reliable pothole detection. In addition, based on a large amount of training data in real scenarios, the machine learning model can continuously optimize its generalization ability and better deal with unknown situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a schematic diagram of the process of the present invention;

[0057] Figure 2 It is a schematic diagram of the overall steps of the present invention;

[0058] Figure 3 This is a diagram of the road surface image data processing annotation of the present invention. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] Example

[0061] See also Figures 1 to 3 This embodiment provides a vision compensation system based on the fusion of a depth vision camera and a laser radar, comprising the following steps:

[0062] Data acquisition module: used to collect vehicle data, including point cloud data and image data, and output the vehicle data;

[0063] Point cloud data:

[0064] The laser radar sensor located on the roof emits laser pulses, measures the time it takes for the pulses to reflect back from the surface of an object to determine the distance, and scans the road conditions in real time; sets a trigger mechanism with a frame rate of g per second to control the acquisition frequency of the laser radar (it is necessary to ensure that the data acquisition is synchronized with sensors that use time as a reference system, including but not limited to depth vision cameras), and maps the original distance measurement value output by the laser radar from the laser radar coordinate system to the vehicle coordinate system and converts it into three-dimensional point cloud data;

[0065] LiDAR coordinate system: The local coordinate system of the LiDAR itself, usually with the installation location of the LiDAR as the origin. Its axis is determined according to the specific design of the device, usually based on the right-handed Cartesian coordinate system;

[0066] Vehicle coordinate system: A coordinate system centered on the vehicle, usually defined as the center of mass of the vehicle or a fixed point (such as the center of the front axle) as the origin, with the X-axis pointing in the direction of the vehicle's forward movement, the Y-axis pointing to the left side of the vehicle, and the Z-axis pointing vertically upward;

[0067] The laser radar coordinate system is mapped to the vehicle coordinate system:

[0068] In the process of mapping the laser radar coordinate system to the vehicle coordinate system, external calibration is performed through the static calibration plate method. By determining the precise position and direction of the laser radar relative to the vehicle coordinate system, a standard calibration plate of known size and position is placed at multiple locations around the vehicle. The standard calibration plate is scanned by the laser radar, and the mapping relationship between the laser radar and the vehicle coordinate system is calculated;

[0069] It should be noted that: when quickly scanning and responding to the current environmental state, the laser radar scanning data in the vehicle's own coordinate system can be used for processing. Even if the vehicle's posture changes (for example, due to bumps), the vehicle's own deflection can be processed relatively accurately by updating the vehicle's posture information in a timely manner.

[0070] Image Data:

[0071] The image data includes road surface image data and road condition image data; the surrounding environment is perceived through a camera group, and the camera group includes a main depth vision camera and an auxiliary depth vision camera, which are used to capture road surface images and road condition images respectively;

[0072] Main Depth Vision:

[0073] By installing it above the front windshield in the car to obtain road condition image data, it can cover most areas of the road ahead, including lane lines, traffic signs, pedestrians and other vehicles. Since it is installed inside the car, it is not affected by rain, ensuring image clarity. And because the position is relatively fixed, it is easy to perform initial calibration and subsequent maintenance.

[0074] Auxiliary depth vision camera:

[0075] It is installed near the bumper to obtain road image data, which is used to obtain road information in low-speed environments. It faces the ground directly and provides a height perspective that is closest to the actual road surface.

[0076] Data processing module: used to obtain vehicle data, process point cloud data, and obtain point cloud posture data; at the same time, process road image data and perform registration judgment with road posture data. When point cloud posture data and road posture data are not registered, calculate the offset coefficient through perspective superposition; at the same time, obtain road image data, calculate pothole coefficient by machine recognition, and output the offset coefficient and pothole coefficient;

[0077] The process of acquiring point cloud posture data is as follows:

[0078] S101: After calibration, a 4x4 transformation matrix is ​​obtained. The matrix includes rotation and translation information and is used to map points in the laser radar coordinate system to the vehicle coordinate system. The transformation matrix is ​​in the form of:

[0079]

[0080] Where T is the vehicle coordinate system transformation matrix; R is a 3x3 rotation matrix, which represents the rotation of the lidar coordinate system relative to the vehicle coordinate system; t is a 3x1 translation vector, which represents the position offset of the lidar coordinate system origin relative to the vehicle coordinate system origin;

[0081] S102: Using the point cloud data set collected by the laser radar, the formula is:

[0082] P LiDAR =[x,y,z,1] T

[0083] Where P LiDAR is a point cloud data set; x, y, and z are the three-dimensional coordinates of the point cloud data collected by the laser radar in the laser radar local coordinate system; by multiplying the transformation matrix T, the corresponding position P in the vehicle coordinate system is obtained Vehicle , the formula is:

[0084]

[0085] Where Rx is the rotation matrix R and the point cloud dataset P LiDAR The product of ty and tz are the components of the translation vector t in the y and z directions respectively;

[0086] S103: Obtain the rotation matrix R1 and translation vector t1 of the initial point before the vehicle bumps, the rotation matrix R2 and translation vector t2 of the initial point with uncertainty offset when the bump occurs, use the ICP algorithm or other point cloud registration methods to iteratively find the closest point pair and minimize the distance error between them, the formula is:

[0087] Formula 1:

[0088] Formula 2:

[0089] Where, pi and pj are the corresponding points selected from the corresponding position PVehicle; N is the maximum value range; i is the current corresponding point, and the value is i=1,2,3...N; j is the current corresponding point, and the value is j=1,2,3...N, and j≠i; Formula 1 and Formula 2 are combined to calculate the value of the rotation matrix R2 and the translation vector t2;

[0090] Example:

[0091] Suppose there are two sets of point cloud data, the point cloud data before the bump and the point cloud data after the bump:

[0092] Point cloud data point before bumping: p1 = (x1, y1, z1), p 2＝ (x2,y2,z2),...,p N =(xN,yN,zN);

[0093] Point cloud data after bumping: p1=(x1',y1',z1'),p2=(x2',y2',z2'),...,p N =(xN',yN',zN');

[0094] Use the ICP algorithm to align the point cloud data before and after the bump to obtain the rotation matrix R and translation vector t:

[0095] Rotation matrix R:

[0096]

[0097] Translation vector t:

[0098]

[0099] Thus, the rotation matrix and translation vector are obtained, and the rotation matrix R and translation vector t are calculated, and R and t are calculated by the above formula; the posture change of the vehicle is obtained through R and t;

[0100] And the rotation angle (pitch angle, roll angle and yaw angle) of the vehicle is obtained according to the rotation matrix R; for example, the pitch angle, roll angle φ, yaw angle ψ, the formula is:

[0101]

[0102] The following results are obtained through the ICP algorithm, the rotation matrix R:

[0103]

[0104] Translation vector t:

[0105]

[0106] The vehicle's direction of travel is estimated by analyzing the changes in road features in the point cloud data; the vehicle's direction of travel is determined by identifying objects or lane lines on both sides of the road; the vehicle's speed change is estimated by comparing the point cloud data at different time points, and the vehicle's acceleration is obtained; and the speed change is estimated by comparing the point cloud data at different time points, and the acceleration is obtained; the speed is estimated by calculating the change in the vehicle's position in the two sets of point cloud data before and after, and the acceleration is obtained;

[0107] The process of road condition image data processing is as follows:

[0108] Preprocessing the collected road image data includes noise removal, distortion correction, contrast enhancement and wide dynamic range processing;

[0109] Noise removal: Apply filters to reduce noise in road image data and improve the quality of subsequent feature extraction;

[0110] Correcting distortion: Using the camera intrinsic matrix to correct radial and tangential distortion in the road image data, ensuring that straight lines appear as straight lines in the road image data, thus simplifying geometric analysis;

[0111] Contrast enhancement: Histogram equalization method to improve the contrast of road image data, especially in low light conditions;

[0112] Wide dynamic range processing: If there are strong brightness changes in the scene, such as from a tunnel to sunlight, the road image data needs to be properly exposed and compensated;

[0113] The process of acquiring road condition posture data is as follows:

[0114] Generate a descriptor for each feature point to facilitate the subsequent matching process; use FLANN to find the most similar feature point pair between the previous and next frames, calculate the geometric relationship between the two frames based on the matched feature point pairs, decompose the essential matrix into a rotation matrix R2' and a translation vector t2', representing the relative motion from the previous frame to the current frame, and add each calculated relative motion to the previous posture to update the global posture of the vehicle;

[0115] Registration judgment:

[0116] By calculating whether the rotation matrix and translation vector of the point cloud posture data are equal to the rotation matrix and translation vector of the road posture data after the same time, that is, whether the values ​​of the rotation matrix R2' and the translation vector t2' are within the error threshold with the rotation matrix R2 and the translation vector t2. The error threshold of the rotation matrix is ​​C R , the error threshold of the translation vector is C t , judged as:

[0117] When the rotation matrix and translation vector of the point cloud posture data are within the error threshold with the rotation matrix and translation vector of the road posture data, it is determined to be aligned and the current distance is maintained;

[0118] When the rotation matrix and translation vector of the point cloud posture data are not within the error threshold with the rotation matrix and translation vector of the road posture data, it is judged as misalignment and the offset coefficient is calculated;

[0119] The calculation process of the offset coefficient is:

[0120] Get the corresponding position P respectively Vehicle The corresponding point p selected from i and p j , the point cloud posture data rotation matrix R2 and translation vector t2, as well as the rotation matrix R2' and translation vector t2', use the objective function to measure the difference between the transformed posture data and the actual image posture data for clustering, so that the sum of square errors between matching feature point pairs is minimized as the objective function, the formula is:

[0121]

[0122] Where, E is the offset coefficient, which represents the offset of the point cloud posture data and the road posture data relative to the initial posture after perspective superposition, and is used to estimate the deviation of vehicle posture estimation; t1 and t1' are the translation vector of the point cloud posture data of the initial point before the vehicle bumps and the translation vector of the road posture data; R1 and R1' are the rotation matrix of the point cloud posture data of the initial point before the vehicle bumps and the rotation matrix of the road posture data;

[0123] The road surface image data processing process is as follows:

[0124] By collecting a large amount of road image and video data, including pothole pictures under different conditions, recording the road image data collected by the auxiliary depth vision camera during vehicle driving, pre-processing the road image data, extracting features, and training the machine learning model to calculate the pothole coefficient of the road;

[0125] S301: Collect a large amount of road image and video data, including pothole images under different conditions, mark the pothole positions in these images or videos, manually or with the assistance of some automated tools, and record the rotation matrix and road posture data of the vehicle during driving to capture the vibration caused by uneven road surface;

[0126] S302: extract features from images or video streams using computer vision technology, use gray-level co-occurrence matrix to describe the spatial relationship between pixels in the image, and thus reflect the roughness of the surface; use deep learning models to automatically learn and extract complex feature representations;

[0127] Example:

[0128] In the same road image data, the probability distribution of two pixels having a specific gray value pair under a given distance and angle; when the angle is 0° and the distance is 1, a two-dimensional probability matrix F(v,m) is constructed, where v and m represent the gray values ​​of two adjacent pixels respectively;

[0129] S303: training a classifier or regression model based on the collected data to predict whether the road surface contains potholes and their severity; using a supervised learning method, in which labels are manually annotated or automatically generated through sensor data;

[0130] Learn useful feature representations from images without manually designing features; consists of multiple convolutional layers, each of which contains several filters to extract features at different levels; and combines data from different sensors (such as road images, accelerometers, gyroscopes, etc.) to build more robust feature representations; for example, combining image features with time series features of sensor signals can more accurately identify road conditions;

[0131] S304: using a comprehensive index to quantify the pothole area of ​​a specific road surface or location, using a machine algorithm to obtain the geometric characteristics of the potholes, and combining the vibration data and the pothole frequency to generate a pothole coefficient;

[0132] Calculation of geometric features:

[0133] By considering the geometric characteristics of the pothole, the area c is preliminarily estimated according to the formula q , the formula is:

[0134] C q =w d ·D+w a ·q

[0135] Where D is the maximum depth of the pothole, q is the maximum width of the pothole, wd and wa are given weight factors, reflecting the importance of different dimensions;

[0136] At the same time, the pothole coefficient will change with the change of vehicle speed v. When passing through potholes at high speed, the bumpy feeling will increase. The speed factor Fv(v) is introduced to dynamically adjust the coefficient and calculate the pothole coefficient. The formula is:

[0137]

[0138] Where ka is the pothole coefficient; Fv(v) is the speed factor, which represents a linear function or a more complex nonlinear function, depending on the specific road bump state; A z is the vibration frequency; f is the average number of vehicle bumps;

[0139] The bump coefficient acquisition module obtains the pothole coefficient and the offset coefficient and calculates the bump coefficient;

[0140] After obtaining the pothole coefficient of the real-time road surface, the bump coefficient of the vehicle on the current road surface is calculated based on the offset coefficient calculated based on the point cloud posture data and the road condition posture data. The formula is:

[0141]

[0142] In the formula, B i is the turbulence coefficient;

[0143] By considering the geometric characteristics of the current potholes on the road and combining the deviation of the vehicle when passing through the potholes, the safe distance between the two vehicles is guaranteed, and there is enough reaction time and space, and the distance that the front vehicle and the rear vehicle should maintain; for autonomous driving vehicles, an appropriate safe distance can provide enough time for braking or other risk avoidance operations, thereby avoiding collision accidents; on normal roads, the safe distance can be set according to factors such as speed; on bumpy roads, since the vehicle stability is affected, an additional safe distance is needed to deal with possible emergencies. The above method can not only effectively improve the driving experience under complex road conditions, but also further enhance the reliability and intelligence of the autonomous driving system.

[0144] In the application, the several formulas involved are all calculated by taking their numerical values ​​after removing the dimensions, and the formulas are established by collecting a large amount of data and performing software simulation to obtain a formula for the most recent actual situation. Some coefficients or weights in the formulas are set by technical personnel in this field according to actual conditions, so they will not be elaborated here.

[0145] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. A person of ordinary skill in the art may appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution.

[0146] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0147] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A visual compensation system based on the fusion of a depth vision camera and a laser radar, characterized in that: The compensation system: Data acquisition module: used to collect vehicle data, including point cloud data and image data, and output the vehicle data; Data processing module: used to obtain vehicle data, process point cloud data, and obtain point cloud posture data; at the same time, process road image data and perform registration judgment with road posture data. When point cloud posture data and road posture data are not registered, calculate the offset coefficient through perspective superposition; at the same time, obtain road image data, calculate pothole coefficient by machine recognition, and output the offset coefficient and pothole coefficient; The bump coefficient acquisition module obtains the pothole coefficient and the offset coefficient and calculates the bump coefficient.

2. The visual compensation system based on the fusion of a depth vision camera and a laser radar according to claim 1, characterized in that: The process of acquiring point cloud data and image data is as follows: Set a trigger mechanism with the acquisition frame rate of g per second to control the acquisition frequency of the lidar, and map the original distance measurement value output by the lidar from the lidar coordinate system to the vehicle coordinate system to convert it into 3D point cloud data; The image data includes road surface image data and road condition image data; and is used to capture road surface images and road condition images.

3. The visual compensation system based on the fusion of a depth vision camera and a laser radar according to claim 2, characterized in that: The process of acquiring point cloud posture data is as follows: S101: After calibration, a transformation matrix is ​​obtained, the matrix including rotation and translation information, which is used to map points in the laser radar coordinate system to the vehicle coordinate system, and obtain the vehicle coordinate system transformation matrix T; S102: Using laser radar to collect point cloud data set P LiDAR ; By multiplying the transformation matrix T, we can obtain its corresponding position P in the vehicle coordinate system Vehicle； S103: Obtain the rotation matrix R1 and translation vector t1 of the initial point before the vehicle bumps, as well as the rotation matrix R2 and translation vector t2 of the initial point with uncertainty offset, iteratively find the closest point pair, and minimize the distance error between them. The formula is: Where pi and pj are the corresponding points selected from the corresponding position PVehicle respectively; N is the maximum value range; i is the current corresponding point, and its value is i=1,2,3...N; j is the current corresponding point, and its value is j=1,2,3...N, and j≠i; Formula 1 and Formula 2 are combined to calculate the value of the rotation matrix R2 and the translation vector t2.

4. The visual compensation system based on the fusion of a depth vision camera and a laser radar according to claim 3, characterized in that: The process of road condition image data processing is as follows: The collected road condition image data is preprocessed, including noise removal, distortion correction, contrast enhancement and wide dynamic range processing.

5. The visual compensation system based on the fusion of a depth vision camera and a laser radar according to claim 4, characterized in that: The process of acquiring road condition posture data is as follows: The geometric relationship between the two frames decomposes the essential matrix into a rotation matrix R2' and a translation vector t2'.

6. The visual compensation system based on the fusion of a depth vision camera and a laser radar according to claim 5, characterized in that: Registration judgment: The registration of the rotation matrix and translation vector in the point cloud posture data and the road condition posture data is judged by: When the rotation matrix and translation vector in the point cloud posture data are equal to the rotation matrix and translation vector in the road condition posture data, it is determined to be aligned and the current distance is maintained; When the rotation matrix and translation vector in the point cloud posture data are not equal to the rotation matrix and translation vector of the road posture data employee, it is judged as misalignment and the offset coefficient is calculated.

7. The visual compensation system based on the fusion of a depth vision camera and a laser radar according to claim 6, characterized in that: The calculation process of the offset coefficient is: Get the corresponding position P respectively Vehicle The corresponding point p selected from i and p j , the point cloud posture data rotation matrix R2 and translation vector t2, as well as the rotation matrix R2' and translation vector t2', use the objective function to measure the difference between the transformed posture data and the actual image posture data, and minimize the sum of square errors between matching feature point pairs as the objective function. The formula is: Where E is the offset coefficient; t1 and t1' are the translation vector of the point cloud posture data and the translation vector of the road posture data of the initial point before the vehicle bumps, respectively; R1 and R1' are the rotation matrices of the point cloud posture data and the road posture data before the vehicle bumps, respectively.

8. The visual compensation system based on the fusion of a depth vision camera and a laser radar according to claim 1, characterized in that: The road surface image data processing process is as follows: S301: Collect road images and video data for annotation, and record the rotation matrix and road condition posture data during vehicle driving; S302: extracting features from an image or video stream using computer vision technology, and using a gray-level co-occurrence matrix to describe the spatial relationship between pixels in the image; Use deep learning models to automatically learn and extract complex feature representations; S303: Identifying potholes on the road according to the trained classifier; S304: Quantify the pothole area of ​​a specific road surface or location using comprehensive indicators, obtain the geometric characteristics of the pothole using machine algorithms, and generate a pothole coefficient in combination with vibration data.

9. The visual compensation system based on the fusion of a depth vision camera and a laser radar according to claim 8, characterized in that: Calculation of geometric features: By considering the geometric characteristics of the pothole, the area c is preliminarily estimated according to the formula q , the formula is: C q =w d ·D+w a ·q Where D is the maximum depth of the pothole, q is the maximum width of the pothole, wd and wa are given weight factors respectively; The speed factor Fv(v) is introduced to dynamically adjust the coefficient and calculate the pothole coefficient. The formula is: Where ka is the pothole coefficient; Fv(v) is the speed factor; A z is the vibration frequency; f is the average number of vehicle bumps.

10. The visual compensation system based on the fusion of a depth vision camera and a laser radar according to claim 9, characterized in that: The process of obtaining the turbulence coefficient is: After obtaining the pothole coefficient of the real-time road surface, the bump coefficient of the vehicle on the current road surface is calculated based on the offset coefficient calculated based on the point cloud posture data and the road condition posture data. The formula is: In the formula, B i is the turbulence coefficient.

Images