A road condition recognition and reminder system based on intelligent vehicle steer-by-wire system

Abstract

The present invention discloses a road condition recognition and reminder system based on an intelligent automobile wire-controlled steering system, comprising a road condition information acquisition module, a road condition recognition module, a driver reminder control module, and a driver reminder execution module. The system recognizes road conditions respectively under different light intensities and working conditions through a combination of a binocular camera, a laser radar, and a series of vehicle sensors, and fuses the recognition results to obtain the road condition information ahead of the vehicle; and provides the driver with the road condition information ahead in advance through multiple methods such as animation display, steering wheel vibration, voice reminder, and signal light flashing, so that the driver can make correct driving decisions, avoid traffic accidents, and protect the driver's life and property.

Description

Technical field:

[0001] The present invention belongs to the field of intelligent vehicle identification, and relates to environmental perception and driver reminder technology, and specifically to a road condition identification and reminder system based on an intelligent vehicle wire-controlled steering system. Background technology:

[0002] As the number of cars continues to grow, the number of drivers is also increasing, but the driving skills of different drivers are uneven, and the number of traffic accidents caused by drivers is increasing day by day. In this context, the birth of smart cars is to make up for the defects of drivers, assist or replace drivers in driving, and improve the driving safety of vehicles. The degree of intelligence of smart cars depends on the ability of perception, planning, decision-making, and automatic execution. The basis of all vehicle intelligence is perception. As the eyes of smart cars, perception algorithms are the only information source that can pick up and interpret actual road conditions, detect and identify potential dangers. Perception algorithms use sensor technology to interpret the local road conditions where smart cars are located and make decisions on this basis so that smart cars can complete driving tasks. At present, these local road condition information mainly refers to lanes and their boundaries, obstacles on the road surface, pedestrians, etc. Among them, the only thing missing is the detection, identification and avoidance of whether there are abnormal road bumps and abnormal road depressions in the road state itself. Abnormal road depressions and abnormal road bumps not only bring huge financial expenditures and a large number of traffic accidents to various countries, but also pose a serious threat to the lives of drivers and passengers. In recent years, with the increasing number of vehicle users, drivers and passengers have died in traffic accidents caused by abnormal road depressions and abnormal road protrusions both at home and abroad. It can be seen that it is increasingly urgent to use advanced technologies and methods to detect and avoid abnormal road depressions and abnormal road protrusions. In addition, as common components of the road surface, shoulders and speed bumps are usually not taken seriously in intelligent car perception research. However, when a vehicle passes through shoulders and speed bumps at too high a speed, it has a huge destructive effect on the vehicle structure, so it is necessary to pay attention to shoulders and speed bumps.

[0003] The driver makes driving decisions and then performs driving operations when fully informed of the road conditions ahead. If the driver does not have sufficient information about the road conditions ahead and misses abnormal road depressions, abnormal road bulges, speed bumps, shoulders, or pedestrians, non-motor vehicles, etc., it may lead to the driver's wrong driving operations or failure to slow down and take evasive measures in time. Therefore, it is necessary to provide the driver with reminder information about the vehicle conditions ahead to ensure that the driver fully understands the road conditions ahead in a timely manner, avoid traffic accidents, and preserve the driver's life and property safety.

[0004] The steer-by-wire system is considered to be a necessary assembly for realizing autonomous driving of smart cars. It eliminates the mechanical connection between the steering wheel and the steering wheel, realizes the decoupling between the steering wheel operation and the steering wheel rotation, and provides a great design space for the transmission ratio and steering wheel motor control. The control of the steering wheel road sense motor is one of the main research contents of steer-by-wire. The current research focuses on the design of road sense feedback torque, which is to provide the driver with road sense feedback torque. The problem with such feedback torque is that there is no warning function, and the driver cannot know the road conditions ahead, because the road sense feedback torque is the feedback of the road conditions that the wheel is currently passing through, rather than the road conditions of the road ahead of the vehicle. Only by letting the driver know the road conditions ahead in advance can the driver's operation be further adjusted. Therefore, it is very meaningful to be able to predict the road conditions ahead for the driver. Based on the steer-by-wire technology, the steering wheel road sense motor can be conveniently used to provide the driver with road condition information ahead. Summary of the invention:

[0005] In order to solve the above technical problems, the present invention provides a road condition recognition and reminder system based on the intelligent automobile wire-controlled steering system, which respectively recognizes the road conditions under different light intensities and working conditions through a combination of binocular cameras, laser radars, and a series of vehicle sensors, and fuses the recognition results to obtain the road condition information in front of the vehicle. And it provides the driver with the road condition information in advance through animation display, steering wheel vibration, voice reminder, and signal light flashing, so that the driver can make correct driving decisions, avoid traffic accidents, and protect the driver's life and property.

[0006] In order to achieve the above object, the present invention is implemented according to the following technical solutions:

[0007] A road condition recognition and reminder system based on an intelligent automobile wire-controlled steering system comprises a road condition information collection module, a road condition recognition module, a driver reminder control module, and a driver reminder execution module; the road condition recognition module is connected to the road condition information collection module and the driver reminder control module respectively; the driver reminder control module is also connected to the driver reminder execution module;

[0008] The road condition information collection module described in the technical solution includes two binocular cameras, a long-range laser radar, 17 vehicle sensors, and an optical sensor. The 17 vehicle sensors include: four suspension displacement sensors located at the left front, left rear, right front, and right rear of the vehicle, a front axle height sensor, a rear axle height sensor, an inertial measurement unit, a yaw angle sensor, a three-axis acceleration sensor, four body acceleration sensors, and four tire pressure detection sensors; the binocular camera and the long-range laser radar are used to collect images and three-dimensional point clouds of the road ahead, respectively; the suspension displacement sensor is used to collect vertical displacement signals of the suspension, respectively; the front axle and rear axle height sensors are used to collect ground clearance signals of the front axle and the rear axle, respectively; The inertial measurement unit is used to collect the pitch angle displacement and the roll angle displacement of the vehicle body; the yaw angle sensor is used to collect the yaw angular velocity signal of the vehicle; the three-axis acceleration sensor is used to collect the lateral acceleration, longitudinal acceleration and vertical acceleration signals of the vehicle; the four body acceleration sensors are respectively installed on the left front, right front, left rear and right rear body at a distance of 5 cm from the top of the shock absorber, and are respectively used to collect the vertical vibration acceleration signals of the left front, right front, left rear and right rear body; the tire pressure detection sensor is respectively used to collect the tire pressure signals of the left front, right front, left rear and right rear tires; the light sensor is used to collect the light intensity signal of the surrounding environment of the vehicle for comparison with the light intensity threshold; the light intensity threshold is calculated according to the following expression:

[0009]

[0010] Where, L uth represents the light intensity threshold, k pr Indicates the binocular camera efficiency impact coefficient, and its value range is 0 to 1; k ex It represents the radar effectiveness influence coefficient, and its value range is from 0 to 1; L uth0 Indicates the initial value of light intensity, which can be in the range of 0.001 lux to 0.25 lux;

[0011] The two binocular cameras described in the technical solution have been calibrated before installation and are installed in the upper left and upper right corners of the windshield respectively. The height from the ground of the installation position is hc, and the horizontal distance between the two binocular cameras is Lc. hc and Lc are related to the size parameters of the front windshield of the vehicle and are determined by the vehicle manufacturer; the visual distance of the two binocular cameras is 250 meters, and the left and right viewing angle range is 120 degrees; the binocular cameras can rotate up and down, and images of different angles can be captured by rotating up and down. The downward shooting angle of the binocular camera is recorded as a negative angle, and the upward shooting angle is recorded as a positive angle. The inclination angle of the two binocular cameras is recorded as Ac, and the binocular cameras The head rotation angle range is from -30 degrees to +60 degrees, that is, it can rotate up to 30 degrees downward and up to 60 degrees upward. The two binocular cameras shoot one frame of image every 1 millisecond. The first frame shoots images in the angle range of -30 degrees to 0 degrees, the second frame shoots images in the angle range of -10 degrees to +20 degrees, the third frame shoots images in the angle range of +10 degrees to +40 degrees, the fourth frame shoots images in the angle range of +30 degrees to +60 degrees, the fifth frame shoots images in the angle range of +10 degrees to +40 degrees, and the sixth frame shoots images in the angle range of -10 degrees to +20 degrees. Shooting the above six frames of images in sequence is a shooting cycle , the camera reciprocates up and down to capture images in the order of the above shooting cycles; the long-range laser radar is installed in the middle of the front bumper, and the ground clearance of the installation position is hr, which is related to the size parameters of the front bumper of the vehicle and is determined by the vehicle manufacturer; it can be rotated up and down to scan and obtain point cloud data of environments in different angle ranges. The downward scanning angle of the long-range laser radar is recorded as a negative angle, and the upward scanning angle is recorded as a positive angle. The inclination angle of the long-range laser radar is recorded as Ar. The up and down rotation angle range of the long-range laser radar is from negative 20 degrees to positive 10 degrees, that is, it can rotate up to 20 degrees downward and up to 10 degrees upward. The detection distance is 3 00 meters; the order of the long-range laser radar scanning the environment is: first scan once when the tilt angle is minus 20 degrees, then scan once at 0 degrees, then scan once at 10 degrees, then scan once again when it returns to 0 degrees, and finally rotate back to the minus 20 degree angle position, and scan up and down in this order; each scan of the long-range laser radar generates three-dimensional point cloud data of the area in front of the vehicle, and each point cloud data has a position and distance label: x, y, z, d; x, y, z, d represent the horizontal axis coordinate, the vertical axis coordinate, the vertical axis coordinate, and the distance from the vehicle respectively; the road condition information acquisition module inputs all the collected information into the road condition recognition module;

[0012] The road condition recognition module described in the technical solution recognizes the road surface characteristics of the road ahead according to the input information, and recognizes whether there are pedestrians, bicycles, motorcycles, abnormal road depressions, abnormal road protrusions, speed bumps, lane lines, and shoulders on the road ahead, and inputs the recognition results into the driver reminder control module; the road condition recognition module has three recognition submodules, namely, an image recognition submodule, a three-dimensional point cloud recognition submodule, and a vehicle sensor signal recognition submodule. The specific recognition processes of each recognition submodule are described as follows:

[0013] The image recognition submodule performs road feature and target recognition based on the image acquired by the binocular camera. The specific recognition process of the image recognition submodule is as follows:

[0014] Step S1: Preprocessing each frame of the image captured by the left and right cameras respectively, first removing the lateral distortion and longitudinal distortion of the images acquired by the left and right cameras respectively to obtain corrected images, and then correcting the images of the left and right cameras to make the image planes of the left and right cameras parallel;

[0015] Step S2: Perform feature matching on the left and right camera images obtained in step S2 through a semi-global block matching algorithm, find corresponding points in the left and right images and solve the disparity, finally obtain the depth information of each pixel in the image, and form a depth map;

[0016] Step S3: Based on feature matching of the left and right images, the pixels of the left and right images are fused by using a wavelet transform method, so as to fuse the left and right images into one image and obtain a fused image;

[0017] Step S4: Input the fused image obtained in step S3 into a pre-trained single-point multi-box detector deep learning model to detect the target object category, which includes pedestrians, bicycles, motorcycles, abnormal road depressions, abnormal road protrusions, speed bumps, lane lines and shoulders; the output of the single-point multi-box detector deep learning model is the name and position of the target object category contained in the input image, and the regression box and name of the target object are marked in the image;

[0018] Step S5: searching the corresponding depth value in the depth map according to the position of the target object obtained in step S4, and obtaining the distance value between each target object and the vehicle;

[0019] Step S6: Record each pixel point of the fused image as a target point. Based on the intrinsic parameters of the left and right cameras, the focal length and the image coordinates of the target point in the left and right cameras, the three-dimensional spatial coordinates of the target point can be obtained through coordinate transformation, that is, the three-dimensional coordinates of each pixel point in the fused image can be obtained;

[0020] Step S7: By calculating the distance between the three-dimensional space coordinates of the pixels corresponding to the four vertices of the regression frame of the target object, the two side lengths of the regression frame can be obtained; if the target object category is an abnormal road depression, the horizontal side length and the vertical side length of the regression frame are respectively the long side and the wide side of the abnormal road depression, and the two points with the largest longitudinal coordinate difference in the pixel points of the abnormal road depression are found, and the longitudinal coordinate difference between the two points is used as the depression depth h of the abnormal road depression. ac ; If the target object category is an abnormal road surface bulge, the vertical side length of the regression frame is the height h of the abnormal road surface bulge tc ; If the target object category is one of the speed bumps and shoulders, the vertical length of the regression frame is the height of these target objects, respectively expressed as h jc 、h sc express;

[0021] Step S8: If an abnormal road depression is identified on the road ahead, the image is cropped according to the edge of the regression frame of the abnormal road depression, and the contour of the abnormal road depression is extracted by using the Gaussian Laplace edge detection operator on the cropped image containing the regression frame. The contour pixel points of the abnormal road depression are extracted, and the three-dimensional spatial coordinates of the contour pixel points of the abnormal road depression are obtained according to the three-dimensional spatial coordinates of each pixel point obtained in step S6. The area of ​​the rectangle with two adjacent horizontal pixel points on the contour and two longitudinal pixel points corresponding to the two points as vertices is accumulated to obtain the accurate area D of the abnormal road depression. ac ;

[0022] The 3D point cloud recognition submodule performs road feature and target recognition based on the acquired LiDAR 3D point cloud data. The specific recognition process of the 3D point cloud recognition submodule is as follows:

[0023] Step 1: Divide and filter the point cloud according to the z coordinate value of the 3D point cloud. The point cloud with a z coordinate value difference of less than 0.5 meters is regarded as the ground range and retained. The point cloud with the maximum z coordinate value greater than 2.5 meters is deleted, and the rest is defined as the target object point cloud and retained;

[0024] Step 2: Input the point cloud data retained in step 1 into the pre-trained PointNet++ deep learning classification model. The target objects that this model can identify include pedestrians, bicycles, motorcycles, abnormal road depressions, abnormal road bumps, speed bumps, lane lines and shoulders. The output of this model is the category and location of the target object contained in the input point cloud;

[0025] Step 3: Extract the geometric dimensions of the target object point cloud obtained in step 2 based on principal component analysis and projection method, and obtain the geometric dimensions of each target object; for abnormal road depression, obtain its depression area D ar and the depression depth h ar; For abnormal road surface bulges, obtain the bulge height h tr ; For speed bumps and shoulders, get their height values, respectively using h jr and h sr express;

[0026] The specific identification process of the vehicle sensor signal identification submodule is as follows:

[0027] The signal obtained by the vehicle sensor is input into the wavelet neural network model to identify the abnormal depression depth or abnormal bulge height of the road surface. The wavelet neural network model has been pre-trained using known road surface height data and corresponding vehicle sensor signals to establish a mapping relationship between the vehicle sensor signal and the road surface height. The sensor signals used here include the following 16 signals: vertical displacement of the left front suspension, vertical displacement of the right front suspension, vertical displacement of the left rear suspension, vertical displacement of the right rear suspension, ground clearance of the front axle, ground clearance of the rear axle, body pitch angle displacement, body roll angle displacement, yaw angular velocity signal, lateral acceleration, longitudinal acceleration and vertical acceleration, vertical vibration acceleration of the left front body, vertical vibration acceleration of the right front body, left front tire pressure, right front tire pressure; the network structure of the wavelet neural network model has 3 layers, including an input layer, a hidden layer and an output layer; the input layer has 16 neuron nodes corresponding to the above 16 sensor signals, and the input sequence x i , i=1,2,3,...,16; the hidden layer has 48 neuron nodes in total, and each node of the hidden layer is composed of a wavelet function, and the wavelet function expression is as follows:

[0028]

[0029] After the input sequence is processed by the wavelet function in the hidden layer, the output value of the hidden layer is obtained. The output value u of the jth neuron node in the hidden layer is j Is the following formula:

[0030]

[0031] Where a and b are the scaling factor and translation factor of the wavelet function, n is the number of input layer nodes, which is 16, and m is the number of hidden layer nodes, which is 48.

[0032] The output layer has only one neuron node, and the output of this node is as follows:

[0033]

[0034] In the formula, w j is the weight coefficient of the jth hidden layer output; the output of the output layer represents the height value h of the road surface x, in meters, with two significant figures after the decimal point, with positive and negative values. If it is a positive number, it means that the road surface is convex, but it cannot directly determine whether the convexity is an abnormal convexity of the road surface, and it cannot be ruled out that it is a speed bump or a shoulder; if it is a negative number, it means that the depression is considered to be an abnormal depression of the road surface, and the height of the road surface convexity h ts and the abnormal depression depth h as Calculated by the following formula:

[0035]

[0036] There are two choices for the input signal of the wavelet neural network model, which input is used according to the recognition mode. The first input is to set the vertical displacement of the left rear suspension, the vertical displacement of the right rear suspension, and the ground clearance of the rear axle to 0, and the vertical displacement of the left front suspension, the vertical displacement of the right front suspension, the ground clearance of the front axle, the body pitch angle displacement, the body roll angle displacement, the yaw angular velocity signal, the lateral acceleration, the longitudinal acceleration and the vertical acceleration, the vertical vibration acceleration of the left front body, the vertical vibration acceleration of the right front body, The left front tire pressure and the right front tire pressure are both input normally; the second input is to set the vertical displacement of the left front suspension, the vertical displacement of the right front suspension, and the ground clearance of the front axle to 0, while the vertical displacement of the left rear suspension, the vertical displacement of the right rear suspension, the ground clearance of the rear axle, the body pitch angle displacement, the body roll angle displacement, the yaw angular velocity signal, the lateral acceleration, the longitudinal acceleration and the vertical acceleration, the vertical vibration acceleration of the left rear body, the vertical vibration acceleration of the right rear body, the left rear tire pressure, and the right rear tire pressure are all input normally;

[0037] The beneficial effect of using the above 16 sensor signals as the input of the wavelet neural network model to identify the road surface is that the selected 16 sensor signals fully cover the response signals of the suspension, axle, body, and tire, covering three types of signals: displacement, velocity, and acceleration, and contain comprehensive information, which is conducive to accurate identification of the road surface;

[0038] The beneficial effect of using wavelet neural network to identify the road surface is that the 16 sensor signals affect each other and there is a certain coupling. Compared with the general neural network, the wavelet neural network has stronger learning ability, higher accuracy, faster convergence speed, and is suitable for processing complex multi-input signals. It is conducive to quickly identifying the road surface. In addition, the relationship between the 16 sensor signals and the road surface is not linear, and the wavelet neural network can fit this nonlinear relationship well.

[0039] The road condition recognition module described in the technical solution also includes a data storage device, which stores a historical anomaly map; the historical anomaly map contains the depth, area, and position of abnormal road surface depressions and the height and position information of abnormal road surface protrusions on the road historically recognized by the road condition recognition module, which is used to calibrate the current recognition result;

[0040] The road condition recognition module described in the technical solution has three recognition modes: the first recognition mode, the second recognition mode and the third recognition mode;

[0041] The driver reminder control module described in the technical solution generates reminder information to be provided to the driver based on the recognition result input by the road condition recognition module, the reminder information includes a prompt animation, steering wheel vibration information, voice reminder information and signal light flashing information, and sends a control signal of the corresponding reminder information to the driver reminder control module;

[0042] The driver reminder execution module described in the technical solution includes a vehicle central control display screen, a horn, a signal light, a steering wheel of a wire-controlled steer system, and a road feel simulation motor assembly. The horn is installed in the driver's seat headrest, and the signal light is installed at the geometric center of the steering wheel. It gives corresponding reminders to the driver based on the relevant control signals given by the driver reminder control module, including playing demonstration animations through the vehicle central control display screen, playing prompt voices through the horn, flashing signal lights, displaying colors, and realizing steering wheel vibration reminders through high-frequency vibrations of the road feel simulation motor.

[0043] The three recognition modes and mode selection conditions of the road condition recognition module described in the technical solution are as follows:

[0044] (1) The first recognition mode:

[0045] When the vehicle is moving forward and the light intensity is greater than the light intensity threshold, the first recognition mode is selected;

[0046] In the first recognition mode, the image recognition submodule, the three-dimensional point cloud recognition submodule and the vehicle sensor signal recognition submodule are called to work. The input of the wavelet neural network model in the vehicle sensor signal recognition submodule selects the first input signal to calculate the road surface bump height h ts1 and the abnormal depression depth h as1 , and h ts1 、h as1Output to the driver reminder control module; the recognition results obtained by the image recognition submodule and the three-dimensional point cloud recognition submodule respectively include: the height value of abnormal road surface bulge, the depth value of abnormal road surface depression, the area value of abnormal road surface depression, the height value of speed bump, the height value of shoulder, and the position of pedestrians, bicycles, motorcycles, abnormal road surface depression, abnormal road surface bulge, speed bump, lane line, shoulder on the road surface ahead and the distance from the vehicle;

[0047] Multiply the recognition result of the image recognition submodule by the weight coefficient W c , the recognition result of the 3D point cloud recognition submodule is multiplied by the weight coefficient W r , add them together to get the final result after fusion, and output the fusion result to the driver reminder control module, where the abnormal road surface convex height value, abnormal road surface concave depth value, abnormal road surface concave area value, speed bump height value, and shoulder height value obtained after fusion are respectively expressed as h t1 、h a1 , D a1 、h j1 、h s1 express;

[0048] W c , W r are the weight coefficients of the image recognition submodule and the 3D point cloud recognition submodule, respectively, and are calculated by the following formula:

[0049]

[0050] In the formula, k v is the vehicle speed influence factor, the value is 0.0003, the unit is h / km; V x is the longitudinal speed, in km / h, which can be obtained by the speed sensor; k s is the steering wheel angular velocity influence factor, the value is 0.00001, the unit is s / deg; is the angular velocity of the steering wheel, which can be obtained by the steering wheel angle sensor, and the unit is deg / s;

[0051] (2) Second recognition mode

[0052] When the vehicle is moving forward and the light intensity is less than the light intensity threshold, the second recognition mode is selected;

[0053] In the second recognition mode, the three-dimensional point cloud recognition submodule and the vehicle sensor signal recognition submodule are called to work, wherein the recognition results obtained by the three-dimensional point cloud recognition submodule include: the height value of abnormal road surface bulge, the depth value of abnormal road surface depression, the area value of abnormal road surface depression, the height value of speed bump, the height value of shoulder, and the position of pedestrians, bicycles, motorcycles, abnormal road surface depression, abnormal road surface bulge, speed bump, lane line, shoulder on the road surface ahead and the distance from the vehicle, and these recognition results are output to the driver reminder control module;

[0054] The input of the wavelet neural network model in the vehicle sensor signal recognition submodule selects the first input signal and calculates the road surface bump height h ts1 and the abnormal depression depth h as1 , and h ts1 and h as1 Output to the driver reminder control module;

[0055] (3) The third recognition mode

[0056] The third recognition mode is selected when the vehicle is driving in reverse;

[0057] In the third recognition mode, the road condition recognition module only calls the vehicle sensor signal recognition submodule to work. The input of the wavelet neural network model in the vehicle sensor signal recognition submodule selects the second input signal to calculate the road surface bump height h. ts2 and the abnormal depression depth h as2 , and h ts2 、h as2 Output to the driver reminder control unit module; since the abnormal road depression area cannot be known through the third recognition mode, the abnormal road depression area is defined as a fixed value of 0.05 in the third recognition mode and output to the driver reminder control module for related calculations.

[0058] The driver reminder control module described in the technical solution generates control signals for four reminder modes, including a schematic animation of the vehicle's central control display screen, the frequency of the steering wheel vibration reminder, the content of the voice reminder, and the frequency and color of the signal light flashing reminder;

[0059] When it is identified that there is a bulge or depression on the road ahead, or a pedestrian, bicycle or motorcycle appears, the driver reminder control module will generate a schematic animation signal and send it to the driver reminder execution module to remind the driver of the depression or bulge on the road ahead, and remind the driver to pay attention to the pedestrian or non-motor vehicle appearing in front; the schematic animation includes the following information: the height, position, and distance of the abnormal bulge of the road surface, or the depth, shape, position, and distance of the abnormal depression of the road surface, and the distance between the pedestrian, bicycle or motorcycle in front and the vehicle;

[0060] The logic of the driver reminder control submodule controlling the steering wheel vibration reminder described in the technical solution is as follows:

[0061] When an abnormal bump appears on the road ahead and the bump height is greater than the bump height threshold, the steering wheel starts vibrating to remind you, and the vibration frequency is F t Hz; vibration lasts t1 seconds, the preset value of t1 is 3, which can be modified by the driver; the frequency of steering wheel vibration reminder F is determined according to the following formula t :

[0062] F t =K d (C1V x +K rt h t )F0

[0063] In the formula, K d is the driver preference factor, dimensionless, with an initial value of 1, which can be modified by the driver within the range of 0.2 to 1.0; C1 is the speed correction coefficient, in h / km, and the specific value is determined by the user; K rt is the bump road correction coefficient, the specific value is determined by the user; h t represents the height of abnormal bulge on the road surface; F0 is the initial reference frequency, and its value can be selected in the range of 500 Hz to 3000 Hz;

[0064] When an abnormal depression appears on the road ahead, with a depth greater than the depression depth threshold and a depression area greater than the depression area threshold, the steering wheel starts vibrating to remind you, with a vibration frequency of F a Hz; vibration lasts t2 seconds, the preset value of t2 is 3, which can be modified by the driver; the frequency of steering wheel vibration reminder F is determined according to the following formula a :

[0065] F a =K d (C1V x +K ra h a D a )F0

[0066] In the formula, K ra is the correction coefficient for sunken road surface, and the specific value is determined by the user; h a Represents the depth of abnormal depression in the road surface, D a is the area of ​​the depression where the road surface is abnormally concave;

[0067] The logic of the voice reminder content played by the driver reminder control module described in the technical solution is as follows:

[0068] When an abnormal bulge with a height greater than the bulge height threshold appears on the road ahead, the voice reminder content is: "Please note that there is an abnormal bulge with a height of h1 meter on the road ahead d1 meters away, please avoid it in time", the content is played once, the voice reminder volume is Sv1 decibel, d1 and h1 are the distance and height of the abnormal bulge from the vehicle respectively; when an abnormal depression with a depth greater than the depression depth threshold and a depression area greater than the depression area threshold appears on the road ahead, the voice reminder content is: "Please note that there is an abnormal depression with a height of h2 meters and a depression area of ​​D2 square meters on the road ahead d2 meters away, please avoid it in time", the content is played once, the voice reminder volume is Sv1 decibel, d2, h2, D2 are the distance between the abnormal depression and the vehicle, the depression depth and the depression area respectively; the preset value of Sv1 is 25 decibels, and the driver is allowed to set an integer value in the range of 20 decibels to 70 decibels to replace the preset value; the bulge height threshold, the depression area threshold, and the depression depth threshold are all set by the driver according to needs;

[0069] The logic of the driver reminder control module controlling the signal light described in the technical solution is as follows:

[0070] When an abnormal convexity with a convexity height greater than the convexity height threshold appears on the road ahead, the signal light turns red and starts flashing at a frequency of F1 Hz. After the vehicle passes the abnormal convexity, the signal light goes out; when an abnormal concavity with a concavity depth greater than the concavity depth threshold and a concavity area greater than the concavity area threshold appears on the road ahead, the signal light turns red and starts flashing at a frequency of F2 Hz. After the vehicle passes the abnormal concavity, the signal light goes out; the preset values ​​of F1 and F2 are 1 Hz and 2 Hz respectively, and the values ​​can also be modified by the driver independently;

[0071] The specific process of the road condition recognition module in the technical solution using the historical abnormal map to calibrate the current recognition result is as follows:

[0072] Download the information of abnormal road depression and abnormal road protrusion in the historical abnormal map;

[0073] Compare the location of each abnormal road depression or abnormal road protrusion on the road in history with the location of the abnormal road depression or abnormal road protrusion identified by the three identification submodules;

[0074] If there is an abnormal road depression or abnormal road bulge at the same location in history, it is considered that the recognition accuracy of the current recognition submodule at this location is 100%, and the recognition result does not need to be changed;

[0075] If no abnormal depression or bulge in the road surface is found at the same location in the past, for safety reasons, it is also considered that the recognition accuracy of the current recognition submodule at this location is 100%, and the recognition result does not need to be changed;

[0076] If an abnormal road depression or abnormal road bulge is recorded at a certain location in history, and the recognition submodule does not currently recognize the existence of an abnormal road depression or abnormal road bulge at that location, the recognition submodule will re-run for detection. If the second recognition result is still non-existent, it is considered that there is indeed no abnormal road depression or abnormal road bulge at that location; if the second recognition result is existing, it is considered that the first detection is a false detection, and the second detection result shall prevail. It is considered that the current recognition accuracy of the recognition submodule at that location is 100%, and the recognition result does not need to be changed. At the same time, the coefficients n of false detection of the binocular camera and the laser radar within one week are respectively counted. 1c 、n 1r And the total number of detections n 2c 、n 2r , calculate the false detection rate of the binocular camera Pwc = n 1c / n 2c 、LiDAR false detection rate Pwr = n 1r / n 2r , thereby adjusting the binocular camera performance influence coefficient k pr and radar effectiveness influence coefficient k ex , the expression of the binocular camera efficiency influence coefficient is as follows:

[0077]

[0078] In the formula, K pr0 It is the initial performance factor of the binocular camera, which can be in the range of 0.3 to 0.95. The specific value is determined by the user.

[0079] The expression of the radar effectiveness influence coefficient is as follows:

[0080]

[0081] In the formula, K ex0 It is the initial effectiveness factor of the radar and can be in the range of 0.2 to 0.98. The specific value is determined by the user.

[0082] The beneficial effects of the present invention are:

[0083] 1. The present invention recognizes road conditions respectively under different light intensities and working conditions through a combination of binocular cameras, laser radars, and a series of vehicle sensors, and fuses the recognition results. The obtained road condition information ahead of the vehicle is more accurate and has wider adaptability to driving environments and driving conditions.

[0084] 2. The present invention adopts multiple methods such as animation display, steering wheel vibration, voice reminder, and signal light flashing to provide the driver with the road condition information ahead in advance, thereby improving the driver's understanding of the road condition ahead.

[0085] 3. The steering wheel vibration, voice reminder, and signal light flashing reminder methods described in the present invention fully consider the driver's preferences and wishes. The specific vibration intensity, voice reminder volume, and signal light flashing frequency are all adjustable to adapt to different drivers. Description of the drawings:

[0086] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0087] Figure 1 It is a block diagram of the system composition of the present invention;

[0088] Figure 2 This is a schematic diagram of the installation positions of the binocular camera and the long-range laser radar of the present invention;

[0089] Figure 3 A schematic diagram of a recognition mode selection process of a road condition recognition module of the present invention;

[0090] Figure 4 Schematic diagram of three recognition modes of the road condition recognition module of the present invention; Specific implementation method:

[0091] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0092] See also Figure 1 The road condition recognition and reminder system based on the intelligent automobile wire-controlled steering system of the present invention is characterized by comprising: a road condition information collection module, a road condition recognition module, a driver reminder control module, and a driver reminder execution module; the road condition recognition module is respectively connected to the road condition information collection module and the driver reminder control module; the driver reminder control module is also connected to the driver reminder execution module;

[0093] The road condition information acquisition module of the present invention includes two binocular cameras, a long-range laser radar, 17 vehicle sensors, and an optical sensor, wherein the vehicle sensors include four suspension displacement sensors, a front axle height sensor, a rear axle height sensor, an inertial measurement unit, a yaw angle sensor, a three-axis acceleration sensor, four body acceleration sensors, and four tire pressure detection sensors located at the left front, left rear, right front, and right rear of the vehicle, respectively; the binocular camera and the long-range laser radar are used to collect images and three-dimensional point clouds of the road ahead, respectively; the suspension displacement sensor is used to collect vertical displacement signals of the suspension, respectively; the front axle and rear axle height sensors are used to collect ground clearance signals of the front axle and the rear axle, respectively; the inertial measurement unit, a yaw angle sensor, a three-axis acceleration sensor, four body acceleration sensors, and four tire pressure detection sensors; the binocular camera and the long-range laser radar are used to collect images and three-dimensional point clouds of the road ahead, respectively; the suspension displacement sensor is used to collect vertical displacement signals of the suspension, respectively; the front axle and rear axle height sensors are used to collect ground clearance signals of the front axle and the rear axle, respectively; The measurement unit is used to collect the pitch angle displacement and the roll angle displacement of the vehicle body; the yaw angle sensor is used to collect the yaw angular velocity signal of the vehicle; the three-axis acceleration sensor is used to collect the lateral acceleration, longitudinal acceleration and vertical acceleration signals of the vehicle; the four body acceleration sensors are respectively installed on the left front, right front, left rear and right rear body 5 cm away from the top of the shock absorber, and are respectively used to collect the vertical vibration acceleration signals of the left front, right front, left rear and right rear body; the tire pressure detection sensor is respectively used to collect the tire pressure signals of the left front, right front, left rear and right rear tires; the light sensor is used to collect the light intensity signal of the surrounding environment of the vehicle for comparison with the light intensity threshold; the light intensity threshold is calculated by the following expression:

[0094]

[0095] Where, L uth represents the light intensity threshold, k pr Represents the binocular camera efficiency impact coefficient, and its value range is 0 to 1; k ex It represents the radar effectiveness influence coefficient, and its value range is from 0 to 1; L uth0 Indicates the initial value of light intensity, which can be in the range of 0.001 lux to 0.25 lux;

[0096] See also Figure 2 , the installation positions of the two binocular cameras and the laser radar described in the present invention are:

[0097] The two binocular cameras described in the present invention have been calibrated before installation. The left binocular camera 100 and the right binocular camera 200 are respectively installed at the upper left corner and the upper right corner of the windshield. The height from the ground of the installation position is hc, and the horizontal distance between the two binocular cameras is Lc. hc and Lc are related to the size parameters of the front windshield of the vehicle and are determined by the vehicle manufacturer; the visual distance of the two binocular cameras is 250 meters, and the left and right viewing angle range is 120 degrees; the binocular cameras can rotate up and down, and images of different angle ranges can be captured by rotating up and down. The downward shooting angle of the binocular camera is recorded as a negative angle, and the upward shooting angle is recorded as a positive angle. The tilt angle of the binocular camera is recorded as Ac, and the rotation angle range of the binocular camera is from -30 degrees to +60 degrees, that is, it can rotate up to 30 degrees downward and up to 60 degrees upward. The two binocular cameras shoot one frame of image every 1 millisecond. The first frame shoots images in the angle range of -30 degrees to 0 degrees, the second frame shoots images in the angle range of -10 degrees to +20 degrees, the third frame shoots images in the angle range of +10 degrees to +40 degrees, the fourth frame shoots images in the angle range of +30 degrees to +60 degrees, the fifth frame shoots images in the angle range of +10 degrees to +40 degrees, and the sixth frame shoots images in the angle range of -10 degrees to +20 degrees. Image; The six frames of images are taken in sequence as a shooting cycle, and the camera rotates up and down to shoot images in the order of the above shooting cycles; the long-range laser radar 300 is installed in the middle of the front bumper, and the height from the ground of the installation position is hr, which is related to the size parameters of the front bumper of the vehicle and is determined by the vehicle manufacturer; it can be rotated up and down to scan and obtain point cloud data of environments in different angle ranges. The downward scanning angle of the long-range laser radar is recorded as a negative angle, and the upward scanning angle is recorded as a positive angle. The inclination angle of the long-range laser radar is recorded as Ar, and the up and down rotation angle range of the long-range laser radar is from negative 20 degrees to positive 10 degrees, that is, the most downward It can rotate up to 20 degrees, and can rotate up to 10 degrees, with a detection distance of 300 meters. The order of scanning the environment by the long-range laser radar is: first scan once when the tilt angle is minus 20 degrees, then scan once at 0 degrees, then scan once at 10 degrees, then scan once again when it returns to 0 degrees, and finally rotate back to the minus 20 degree angle position, and scan up and down in this order. Each scan of the long-range laser radar generates three-dimensional point cloud data of the area in front of the vehicle, and each point cloud data has position and distance labels: x, y, z, d; x, y, z, d represent the horizontal axis coordinate, vertical axis coordinate, vertical axis coordinate, and distance from the vehicle, respectively.

[0098] See also Figure 3 The three recognition mode selection processes of the road condition recognition module of the present invention are specifically as follows:

[0099] When the vehicle is moving forward and the light intensity is greater than the light intensity threshold, the first recognition mode is selected; when the vehicle is moving forward and the light intensity is less than the light intensity threshold, the second recognition mode is selected; when the vehicle is moving backward, the third recognition mode is selected;

[0100] See also Figure 4 The three recognition modes of the road condition recognition module of the present invention are specifically:

[0101] In the first recognition mode, the image recognition submodule, the three-dimensional point cloud recognition submodule and the vehicle sensor signal recognition submodule are called to work. The input of the wavelet neural network model in the vehicle sensor signal recognition submodule selects the first input signal to calculate the road surface bump height h ts1 and the abnormal depression depth h as1 , and h ts1 、h as1 Output to the driver reminder control module; the recognition results obtained by the image recognition submodule and the three-dimensional point cloud recognition submodule respectively include: the height value of abnormal road surface bulge, the depth value of abnormal road surface depression, the area value of abnormal road surface depression, the height value of speed bump, the height value of shoulder, and the position of pedestrians, bicycles, motorcycles, abnormal road surface depression, abnormal road surface bulge, speed bump, lane line, shoulder on the road surface ahead and the distance from the vehicle;

[0102] Multiply the recognition result of the image recognition submodule by the weight coefficient W c , the recognition result of the 3D point cloud recognition submodule is multiplied by the weight coefficient W r , add them together to get the final result after fusion, and output the fusion result to the driver reminder control module, where the abnormal road surface convex height value, abnormal road surface concave depth value, abnormal road surface concave area value, speed bump height value, and shoulder height value obtained after fusion are respectively expressed as h t1 、h a1 , D a1 、h j1 、h s1 express;

[0103] W c , W r are the weight coefficients of the image recognition submodule and the 3D point cloud recognition submodule, respectively, and are calculated by the following formula:

[0104]

[0105] In the formula, k v is the vehicle speed influencing factor, the specific value is determined by the user, and the unit is h / km; V x is the longitudinal speed, in km / h, which can be obtained by the speed sensor; k sIt is the steering wheel angular velocity influence factor. The specific value is determined by the user. The unit is s / deg. is the angular velocity of the steering wheel, which can be obtained by the steering wheel angle sensor, and the unit is deg / s;

[0106] In the second recognition mode, the three-dimensional point cloud recognition submodule and the vehicle sensor signal recognition submodule are called to work, wherein the recognition results obtained by the three-dimensional point cloud recognition submodule include: the height value of abnormal road surface bulge, the depth value of abnormal road surface depression, the area value of abnormal road surface depression, the height value of speed bump, the height value of shoulder, and the position of pedestrians, bicycles, motorcycles, abnormal road surface depression, abnormal road surface bulge, speed bump, lane line, shoulder on the road surface ahead and the distance from the vehicle, and these recognition results are output to the driver reminder control module;

[0107] The input of the wavelet neural network model in the vehicle sensor signal recognition submodule selects the first input signal and calculates the road surface bump height h ts1 and the abnormal depression depth h as1 , and h ts1 and h as1 output to the driver reminder control module;

[0108] In the third recognition mode, the road condition recognition module only calls the vehicle sensor signal recognition submodule to work. The input of the wavelet neural network model in the vehicle sensor signal recognition submodule selects the second input signal to calculate the road surface bump height h. ts2 and the abnormal depression depth h as2 , and h ts2 and h as2 Output to the driver reminder control unit module; since the abnormal road depression area cannot be known through the third recognition mode, the abnormal road depression area is defined as a fixed value of 0.05 in the third recognition mode and output to the driver reminder control module for related calculations.

[0109] The above discussion is only a preferred embodiment of the present invention, which is for explanation and illustration, and is not intended to limit the present invention itself. The present invention is not limited to the specific embodiments disclosed herein, but is determined by the following claims. In addition, the records related to specific embodiments in the foregoing description cannot be interpreted as limiting the scope of the present invention or the definition of terms used in the claims. Various other different embodiments and various different variations of the disclosed embodiments are obvious to those skilled in the art. However, all these embodiments, changes and variations that do not deviate from the basic concept of the present invention are within the scope of the appended claims.

Claims

1. A road condition recognition and reminder system based on an intelligent vehicle wire-controlled steering system, characterized in that: It includes a road condition information collection module, a road condition recognition module, a driver reminder control module, and a driver reminder execution module; the road condition recognition module is respectively connected to the road condition information collection module and the driver reminder control module; the driver reminder control module is also connected to the driver reminder execution module; the road condition information collection module includes two binocular cameras, a long-range laser radar, 17 vehicle sensors, and an optical sensor, and the 17 vehicle sensors include: four suspension displacement sensors located at the left front, left rear, right front, and right rear of the vehicle, a front axle height sensor, a rear axle height sensor, an inertial measurement unit, a yaw angle sensor, a three-axis acceleration sensor, four body acceleration sensors, and four tire pressure detection sensors; the binocular camera and the long-range laser radar are respectively used to collect images and three-dimensional point clouds of the road ahead; the suspension displacement sensor is respectively used to collect the vertical and horizontal displacement of the suspension displacement signal; the front axle and rear axle height sensors are used to collect the ground clearance signal of the front axle and the ground clearance signal of the rear axle respectively; the inertial measurement unit is used to collect the pitch angle displacement and the roll angle displacement of the vehicle body; the yaw angle sensor is used to collect the yaw angular velocity signal of the vehicle; the three-axis acceleration sensor is used to collect the lateral acceleration, longitudinal acceleration and vertical acceleration signals of the vehicle; the four body acceleration sensors are respectively installed on the left front, right front, left rear and right rear body 5 cm away from the top of the shock absorber, and are respectively used to collect the vertical vibration acceleration signals of the left front, right front, left rear and right rear body; the four tire pressure detection sensors are respectively used to collect the tire pressure signals of the left front, right front, left rear and right rear tires; the light sensor is used to collect the light intensity signal of the surrounding environment of the vehicle for comparison with the light intensity threshold; the light intensity threshold is calculated according to the following expression: Where, L uth represents the light intensity threshold, k pr Indicates the binocular camera efficiency impact coefficient, and its value range is 0 to 1; k ex It represents the radar effectiveness influence coefficient, and its value range is from 0 to 1; L uth0 Indicates the initial value of light intensity, which can be in the range of 0.001 lux to 0.25 lux; The two binocular cameras have been calibrated before installation and are installed in the upper left and upper right corners of the windshield respectively. The height from the ground of the installation position is hc, and the horizontal distance between the two binocular cameras is Lc. hc and Lc are related to the size parameters of the vehicle's front windshield and are determined by the vehicle manufacturer; the visible distance of the two binocular cameras is 250 meters, and the left and right viewing angles are 120 degrees; the binocular cameras can rotate up and down, and images of different angles can be captured by rotating up and down. The downward shooting angle of the binocular camera is recorded as a negative angle, and the upward shooting angle is recorded as a positive angle. The tilt angle of the two binocular cameras is recorded as Ac, and the rotation angle of the binocular camera is recorded as Ac. The angle range is from -30 degrees to +60 degrees, that is, it can rotate up to 30 degrees downward and up to 60 degrees upward. The two binocular cameras shoot one frame of image every 1 millisecond. The first frame shoots images in the angle range of -30 degrees to 0 degrees, the second frame shoots images in the angle range of -10 degrees to +20 degrees, the third frame shoots images in the angle range of +10 degrees to +40 degrees, the fourth frame shoots images in the angle range of +30 degrees to +60 degrees, the fifth frame shoots images in the angle range of +10 degrees to +40 degrees, and the sixth frame shoots images in the angle range of -10 degrees to +20 degrees. Shooting the above six frames of images in sequence is a shooting cycle. The camera head reciprocates up and down to capture images in the order of the above shooting cycles; the long-range laser radar is installed in the middle of the front bumper, and the height from the ground of the installation position is hr, which is related to the size parameters of the front bumper of the vehicle and is determined by the vehicle manufacturer; it rotates up and down to scan and obtain point cloud data of environments in different angle ranges. The downward scanning angle of the long-range laser radar is recorded as a negative angle, and the upward scanning angle is recorded as a positive angle. The tilt angle of the long-range laser radar is recorded as Ar. The up and down rotation angle range of the long-range laser radar is from negative 20 degrees to positive 10 degrees, that is, it can rotate up to 20 degrees downward and up to 10 degrees upward, and the detection distance is 300 m; the order of the long-range laser radar scanning the environment is: first scan once when the tilt angle is minus 20 degrees, then scan once at 0 degrees, then scan once at 10 degrees, then scan once again when it returns to 0 degrees, and finally rotate back to the minus 20 degree angle position, and scan up and down in this order; each scan of the long-range laser radar generates three-dimensional point cloud data of the area in front of the vehicle, and each point cloud data has a position and distance label: x, y, z, d; x, y, z, d represent the horizontal axis coordinate, the vertical axis coordinate, the vertical axis coordinate, and the distance from the vehicle respectively; the road condition information collection module inputs all the collected information into the road condition recognition module; The road condition recognition module recognizes the road surface features of the road ahead according to the input information, and recognizes whether there are pedestrians, bicycles, motorcycles, abnormal road depressions, abnormal road protrusions, speed bumps, lane lines, and shoulders on the road ahead, and inputs the recognition results into the driver reminder control module; the road condition recognition module has three recognition submodules, namely, an image recognition submodule, a three-dimensional point cloud recognition submodule, and a vehicle sensor signal recognition submodule. The specific recognition processes of each recognition submodule are as follows: The image recognition submodule performs road feature and target recognition based on the image acquired by the binocular camera. The specific recognition process of the image recognition submodule is as follows: Step S1: Preprocessing each frame of the image captured by the left and right cameras respectively, first removing the lateral distortion and longitudinal distortion of the images acquired by the left and right cameras respectively to obtain corrected images, and then correcting the images of the left and right cameras to make the image planes of the left and right cameras parallel; Step S2: Perform feature matching on the left and right camera images obtained in step S2 through a semi-global block matching algorithm, find corresponding points in the left and right images and solve the disparity, finally obtain the depth information of each pixel in the image, and form a depth map; Step S3: Based on feature matching of the left and right images, the pixels of the left and right images are fused by using a wavelet transform method, so as to fuse the left and right images into one image and obtain a fused image; Step S4: Input the fused image obtained in step S3 into a pre-trained single-point multi-box detector deep learning model to detect the target object category, which includes pedestrians, bicycles, motorcycles, abnormal road depressions, abnormal road protrusions, speed bumps, lane lines and shoulders; the output of the single-point multi-box detector deep learning model is the name and position of the target object category contained in the input image, and the regression box and name of the target object are marked in the image; Step S5: searching the corresponding depth value in the depth map according to the position of the target object obtained in step S4, and obtaining the distance value between each target object and the vehicle; Step S6: Record each pixel point of the fused image as a target point. Based on the intrinsic parameters of the left and right cameras, the focal length and the image coordinates of the target point in the left and right cameras, the three-dimensional spatial coordinates of the target point can be obtained through coordinate transformation, that is, the three-dimensional coordinates of each pixel point in the fused image can be obtained; Step S7: By calculating the distance between the three-dimensional space coordinates of the pixels corresponding to the four vertices of the regression frame of the target object, the two side lengths of the regression frame can be obtained; if the target object category is an abnormal road depression, the horizontal side length and the vertical side length of the regression frame are respectively the long side and the wide side of the abnormal road depression, and the two points with the largest longitudinal coordinate difference in the pixel points of the abnormal road depression are found, and the longitudinal coordinate difference between the two points is used as the depression depth h of the abnormal road depression. ac ; If the target object category is an abnormal road surface bulge, the vertical side length of the regression frame is the height h of the abnormal road surface bulge tc ; If the target object category is one of the speed bumps and shoulders, the vertical length of the regression frame is the height of these target objects, respectively expressed as h jc 、h sc express; Step S8: If an abnormal road depression is identified on the road ahead, the image is cropped according to the edge of the regression frame of the abnormal road depression, and the contour of the abnormal road depression is extracted by using the Gaussian Laplace edge detection operator on the cropped image containing the regression frame. The contour pixel points of the abnormal road depression are extracted, and the three-dimensional spatial coordinates of the contour pixel points of the abnormal road depression are obtained according to the three-dimensional spatial coordinates of each pixel point obtained in step S6. The area of ​​the rectangle with two adjacent horizontal pixel points on the contour and two longitudinal pixel points corresponding to the two points as vertices is accumulated to obtain the accurate area D of the abnormal road depression. ac ; The 3D point cloud recognition submodule performs road feature and target recognition based on the acquired LiDAR 3D point cloud data. The specific recognition process of the 3D point cloud recognition submodule is as follows: Step 1: Divide and filter the point cloud according to the z coordinate value of the 3D point cloud. The point cloud with a z coordinate value difference of less than 0.5 meters is regarded as the ground range and retained. The point cloud with the maximum z coordinate value greater than 2.5 meters is deleted, and the rest is defined as the target object point cloud and retained; Step 2: Input the point cloud data retained in step 1 into the pre-trained PointNet++ deep learning classification model. The target objects that this model can identify include pedestrians, bicycles, motorcycles, abnormal road depressions, abnormal road bumps, speed bumps, lane lines and shoulders. The output of this model is the category and location of the target object contained in the input point cloud; Step 3: Extract the geometric dimensions of the target object point cloud obtained in step 2 based on principal component analysis and projection method, and obtain the geometric dimensions of each target object; for abnormal road depression, obtain its depression area D ar and the depression depth h ar ; For abnormal convexity of the road surface, obtain its convexity height h tr ; For speed bumps and shoulders, get their height values, respectively using h jr and h sr express; The specific identification process of the vehicle sensor signal identification submodule is as follows: The signal obtained by the vehicle sensor is input into the wavelet neural network model to identify the abnormal depression depth or abnormal bulge height of the road surface. The wavelet neural network model has been pre-trained using known road surface height data and corresponding vehicle sensor signals to establish a mapping relationship between the vehicle sensor signal and the road surface height. The sensor signals used here include the following 16 signals: vertical displacement of the left front suspension, vertical displacement of the right front suspension, vertical displacement of the left rear suspension, vertical displacement of the right rear suspension, ground clearance of the front axle, ground clearance of the rear axle, body pitch angle displacement, body roll angle displacement, yaw angular velocity signal, lateral acceleration, longitudinal acceleration and vertical acceleration, vertical vibration acceleration of the left front body, vertical vibration acceleration of the right front body, left front tire pressure, right front tire pressure; the network structure of the wavelet neural network model has 3 layers, including an input layer, a hidden layer and an output layer; the input layer has 16 neuron nodes, which correspond to the above 16 sensor signals respectively. The input sequence x i , i=1,2,3,...,16; the hidden layer has 48 neuron nodes in total, and each node of the hidden layer is composed of a wavelet function, and the wavelet function expression is as follows: After the input sequence is processed by the wavelet function in the hidden layer, the output value of the hidden layer is obtained. The output value u of the jth neuron node in the hidden layer is j Is the following formula: Where a and b are the scaling factor and translation factor of the wavelet function, n is the number of input layer nodes, which is 16, and m is the number of hidden layer nodes, which is 48. The output layer has only one neuron node, and the output of this node is as follows: In the formula, w j is the weight coefficient of the jth hidden layer output; the output of the output layer represents the height value h of the road surface x , in meters, with two significant figures after the decimal point, with positive and negative values. If it is a positive number, it means that the road surface is convex, but it cannot directly determine whether the convexity is an abnormal convexity of the road surface, and it cannot be ruled out that it is a speed bump or a shoulder; if it is a negative number, it means that the depression is considered to be an abnormal depression of the road surface, and the height of the road surface convexity h ts and the abnormal depression depth h as Calculated by the following formula: There are two choices for the input signal of the wavelet neural network model, which is determined by the recognition mode. The first input is to set the vertical displacement of the left rear suspension, the vertical displacement of the right rear suspension, and the ground clearance of the rear axle to 0, and the vertical displacement of the left front suspension, the vertical displacement of the right front suspension, the ground clearance of the front axle, the body pitch angle displacement, the body roll angle displacement, the yaw angular velocity signal, the lateral acceleration, the longitudinal acceleration and the vertical acceleration, the vertical vibration acceleration of the left front body, and the vertical vibration acceleration of the right front body. , the left front tire pressure, and the right front tire pressure are normally input; the second input is to set the vertical displacement of the left front suspension, the vertical displacement of the right front suspension, and the ground clearance of the front axle to 0, and the vertical displacement of the left rear suspension, the vertical displacement of the right rear suspension, the ground clearance of the rear axle, the body pitch angle displacement, the body roll angle displacement, the yaw angular velocity signal, the lateral acceleration, the longitudinal acceleration and the vertical acceleration, the vertical vibration acceleration of the left rear body, the vertical vibration acceleration of the right rear body, the left rear tire pressure, and the right rear tire pressure are normally input; The road condition recognition module further includes a data storage device, which stores a historical anomaly map; the historical anomaly map includes the depth, area, and position of abnormal road surface depressions and the height and position information of abnormal road surface protrusions on the road historically recognized by the road condition recognition module, and is used to calibrate the current recognition result; The road condition recognition module has three recognition modes: a first recognition mode, a second recognition mode and a third recognition mode; (1) The first recognition mode: When the vehicle is moving forward and the light intensity is greater than the light intensity threshold, the first recognition mode is selected; In the first recognition mode, the image recognition submodule, the three-dimensional point cloud recognition submodule and the vehicle sensor signal recognition submodule are called to work. The input of the wavelet neural network model in the vehicle sensor signal recognition submodule selects the first input signal to calculate the road surface bump height h ts1 and the abnormal depression depth h as1 , and h ts1 and h as1 Output to the driver reminder control module; the recognition results obtained by the image recognition submodule and the three-dimensional point cloud recognition submodule respectively include: the height value of abnormal road surface bulge, the depth value of abnormal road surface depression, the area value of abnormal road surface depression, the height value of speed bump, the height value of shoulder, and the position of pedestrians, bicycles, motorcycles, abnormal road surface depression, abnormal road surface bulge, speed bump, lane line, shoulder on the road surface ahead and the distance from the vehicle; Multiply the recognition result of the image recognition submodule by the weight coefficient W c , the recognition result of the 3D point cloud recognition submodule is multiplied by the weight coefficient W r , add them together to get the final result after fusion, and output the fusion result to the driver reminder control module, where the abnormal road surface convex height value, abnormal road surface concave depth value, abnormal road surface concave area value, speed bump height value, and shoulder height value obtained after fusion are respectively expressed as h t1 、h a1 , D a1 、h j1 and h s1 express; W c and W r are the weight coefficients of the image recognition submodule and the 3D point cloud recognition submodule, respectively, and are calculated by the following formula: In the formula, k v is the vehicle speed influencing factor, the specific value is determined by the user, and the unit is h / km; V x is the longitudinal speed, in km / h, which can be obtained by the speed sensor; k s It is the steering wheel angular velocity influence factor. The specific value is determined by the user. The unit is s / deg. is the angular velocity of the steering wheel, which can be obtained by the steering wheel angle sensor, and the unit is deg / s; (2) The second recognition mode: When the vehicle is moving forward and the light intensity is less than the light intensity threshold, the second recognition mode is selected; In the second recognition mode, the three-dimensional point cloud recognition submodule and the vehicle sensor signal recognition submodule are called to work, wherein the recognition results obtained by the three-dimensional point cloud recognition submodule include: the height value of abnormal road surface bulge, the depth value of abnormal road surface depression, the area value of abnormal road surface depression, the height value of speed bump, the height value of shoulder, as well as the positions of pedestrians, bicycles, motorcycles, abnormal road surface depression, abnormal road surface bulge, speed bump, lane line and shoulder on the road surface ahead and the distance between the shoulder and the vehicle, and these recognition results are output to the driver reminder control module; The input of the wavelet neural network model in the vehicle sensor signal recognition submodule selects the first input signal and calculates the road surface bump height h ts1 and the abnormal depression depth h as1 , and h ts1 and h as1 Output to the driver reminder control module; (3) The third recognition mode: The third recognition mode is selected when the vehicle is driving in reverse; In the third recognition mode, the road condition recognition module only calls the vehicle sensor signal recognition submodule to work. The input of the wavelet neural network model in the vehicle sensor signal recognition submodule selects the second input signal to calculate the road surface bump height h. ts2 and the abnormal depression depth h as2 , and h ts2 and h as2 Output to the driver reminder control unit module; since the abnormal road depression area cannot be known through the third recognition mode, the abnormal road depression area is defined as a fixed value of 0.05 in the third recognition mode and output to the driver reminder control module for related calculation; The driver reminder control module generates reminder information to be provided to the driver based on the recognition result input by the road condition recognition module, and the reminder information includes schematic animation, steering wheel vibration information, voice reminder information and signal light flashing information, and sends the control signal of the corresponding reminder information to the driver reminder control module; the driver reminder execution module includes a vehicle central control display screen, a horn, a signal light, a steering wheel of a wire-controlled steering system, and a road feel simulation motor assembly. The horn is installed in the driver's seat headrest, and the signal light is installed at the geometric center of the steering wheel. It reminds the driver accordingly according to the relevant control signal given by the driver reminder control module, including playing a demonstration animation through the vehicle central control display screen, playing a reminder voice through the horn, flashing the signal light, displaying color, and realizing a steering wheel vibration reminder through the high-frequency vibration of the road feel simulation motor.

2. A road condition recognition and reminder system based on an intelligent automobile steer-by-wire system according to claim 1, characterized in that: The driver reminder control module generates control signals for four reminder modes, including a schematic animation of the vehicle's central control display screen, the frequency of the steering wheel vibration reminder, the content of the voice reminder, and the frequency and color of the signal light flashing reminder; When it is identified that there is a bulge or depression on the road ahead, or a pedestrian, bicycle or motorcycle appears, the driver reminder control module will generate a schematic animation signal and send it to the driver reminder execution module to remind the driver of the depression or bulge on the road ahead, and remind the driver to pay attention to the pedestrian or non-motor vehicle appearing in front; the schematic animation includes the following information: the height, position, and distance of the abnormal bulge of the road surface, or the depth, shape, position, and distance of the abnormal depression of the road surface, and the distance between the pedestrian, bicycle or motorcycle in front and the vehicle; The logic of the driver reminder control module controlling the steering wheel vibration reminder is as follows: When an abnormal bump appears on the road ahead and the bump height is greater than the bump height threshold, the steering wheel starts vibrating to remind you, and the vibration frequency is F t Hz; vibration lasts t1 seconds, the preset value of t1 is 3, which can be modified by the driver; the frequency of steering wheel vibration reminder F is determined according to the following formula t : F t =K d (C1V x +K rt h t )F0 In the formula, K d is the driver preference factor, dimensionless, with an initial value of 1, which can be modified by the driver within the range of 0.2 to 1.0; C1 is the speed correction coefficient, in h / km, and the specific value is determined by the user; K rt is the bump road correction coefficient, the specific value is determined by the user; h t The height of the bump represents the abnormal bump in the road surface; F0 is the initial reference frequency, and its value can be selected in the range of 500 Hz to 3000 Hz; When an abnormal depression appears on the road ahead, with a depth greater than the depression depth threshold and a depression area greater than the depression area threshold, the steering wheel starts vibrating to remind you, with a vibration frequency of F a Hz; vibration lasts t2 seconds, the preset value of t2 is 3, which can be modified by the driver; the frequency of steering wheel vibration reminder F is determined according to the following formula a : F a =K d (C1V x +K ra h a D a )F0 In the formula, K ra is the correction coefficient for sunken road surface, and the specific value is determined by the user; h a Represents the depth of the abnormal depression in the road surface, D a is the area of ​​the depression where the road surface is abnormally depressed; The logic of the voice reminder content played by the driver reminder control module is as follows: When an abnormal road bulge with a height greater than the bulge height threshold appears on the road ahead, the voice reminder content is: "Please note that there is an abnormal road bulge with a height of h1 meters on the road ahead d1 meters away. Please avoid it in time." This content is played once, and the voice reminder volume is Sv1 decibels. d1 and h1 are the distance and height of the abnormal road bulge to the vehicle respectively. When an abnormal road bulge with a depth greater than the bulge depth threshold and a bulge area greater than the bulge area threshold appears on the road ahead, the voice reminder content is: "Please note that there is an abnormal road bulge with a height of h1 meters on the road ahead d2 meters away. Please avoid it in time." This content is played once, and the voice reminder volume is Sv1 decibels. d1 and h1 are the distance and height of the abnormal road bulge to the vehicle respectively. There is an abnormal road depression with a height of h2 meters and an area of ​​D2 square meters on the road surface. Please avoid it in time." This content is played once, and the voice reminder volume is Sv1 decibels. d2, h2 and D2 are the distance between the abnormal road depression and the vehicle, the depression depth and the depression area of ​​the abnormal road depression respectively; the preset value of Sv1 is 25 decibels, and the driver is allowed to set an integer value in the range of 20 decibels to 70 decibels to replace the preset value; the convex height threshold, depression area threshold and depression depth threshold are all set by the driver according to needs; The logic of the driver reminder control module controlling the signal light is as follows: When an abnormal road surface bulge with a height greater than the bulge height threshold appears on the road ahead, the signal light turns red and starts flashing at a frequency of F1 Hz. After the vehicle passes the abnormal road surface bulge, the signal light goes out; when an abnormal road surface depression with a depth greater than the depression depth threshold and a depression area greater than the depression area threshold appears on the road ahead, the signal light turns red and starts flashing at a frequency of F2 Hz. After the vehicle passes the abnormal road surface depression, the signal light goes out; the preset values ​​of F1 and F2 are 1 Hz and 2 Hz respectively, or the values ​​can be modified by the driver independently.

3. The road condition recognition and reminder system based on the intelligent automobile steer-by-wire system according to claim 1, characterized in that: The specific process of the road condition recognition module using the historical abnormal map to calibrate the current recognition result is as follows: Download the information of abnormal road depression and abnormal road protrusion in the historical abnormal map; Compare the location of each abnormal road depression or abnormal road protrusion on the road in history with the location of the abnormal road depression or abnormal road protrusion identified by the three identification submodules; If there is an abnormal road depression or abnormal road bulge at the same location in history, it is considered that the recognition accuracy of the current recognition submodule at this location is 100%, and the recognition result does not need to be changed; If no abnormal depression or bulge in the road surface is found at the same location in the past, for safety reasons, it is also considered that the recognition accuracy of the current recognition submodule at this location is 100%, and the recognition result does not need to be changed; If an abnormal road depression or abnormal road bulge is recorded at a certain location in history, and the recognition submodule does not currently recognize the existence of an abnormal road depression or abnormal road bulge at that location, the recognition submodule will re-run for detection. If the second recognition result is still non-existent, it is considered that there is indeed no abnormal road depression or abnormal road bulge at that location; if the second recognition result is existing, it is considered that the first detection is a false detection, and the second detection result shall prevail. It is considered that the current recognition accuracy of the recognition submodule at that location is 100%, and the recognition result does not need to be changed. At the same time, the coefficients n of false detection of the binocular camera and the laser radar within one week are respectively counted. 1c and n 1r And the total number of detections n 2c and n 2r , calculate the false detection rate of the binocular camera Pwc = n 1c / n 2c 、LiDAR false detection rate Pwr = n 1r / n 2r , thereby adjusting the binocular camera performance influence coefficient k pr and radar effectiveness influence coefficient k ex , the expression of the binocular camera efficiency influence coefficient is as follows: k pr =K pr0 e -pwc In the formula, K pr0 It is the initial performance factor of the binocular camera, which can be in the range of 0.3 to 0.

95. The specific value is determined by the user. The expression of the radar effectiveness influence coefficient is as follows: k ex =k ex0 e -pwr In the formula, K ex0 It is the initial effectiveness factor of the radar and can be in the range of 0.2 to 0.

98. The specific value is determined by the user.

Images