Vehicle, road collapse identification and early warning method and device

Abstract

The present application belongs to the technical field of road detection, and particularly relates to a vehicle, a road collapse identification and early warning method and device. The method comprises: acquiring a Z-direction vibration signal of the vehicle and image data of a road ahead of the vehicle, respectively judging whether there is an abnormal area affecting the vehicle passing based on the image data and the Z-direction vibration signal, if any of the judgment results is that there is an abnormal area affecting the vehicle passing, it is determined that there is a collapse risk of the road ahead of the vehicle, a collapse early warning of the road ahead is given, long-distance collapse prediction is realized, the situation that the road is about to collapse can be accurately identified, and when the collapse has been formed, whether there is an abnormal area can be judged through the vibration signal, whether the road has a collapse defect can be identified through the image, whether the defect affects the vehicle passing is judged, the collapse danger is identified as much as possible through comprehensive judgment, and the problem that the timeliness and the accuracy of the road collapse early warning are low is improved.

Description

Technical Field

[0001] This invention belongs to the field of road detection technology, specifically relating to a method and device for identifying and warning of vehicle and road collapse. Background Technology

[0002] Highways, urban roads, and bridges are the main carriers of urban transportation and daily travel. If a road surface collapses, it could cause a large number of vehicles to fall, resulting in serious casualties. Studies indicate that most road collapse accidents leading to vehicle falls and significant casualties occur at night or at dawn, when lighting conditions are poor. This is because drivers cannot clearly see or observe distant bridge or road collapses, leading to insufficient braking distance by the time they realize the danger and attempt emergency braking, resulting in catastrophic accidents.

[0003] Chinese patent application CN119434065A discloses a method, apparatus, and computer-readable storage medium for handling road subsidence. During vehicle operation, onboard sensing components acquire real-time sensor data of the area ahead via vehicle-mounted sensors. These components include at least one of onboard radar and an onboard camera. The radar is a forward-looking LiDAR, and the camera is a forward-looking onboard camera. Based on the sensor data, the area, depth, and location information of abnormal areas are determined. A subsidence confidence value is then determined based on the area and depth information. If the subsidence confidence value and location information meet the warning conditions, a subsidence warning is issued via at least one of audio, text, and light methods to alert the driver to the abnormal road conditions. While this method can detect road subsidence to some extent using LiDAR and cameras, the detection range of these devices is limited, and they may not accurately identify impending subsidence. This results in low timeliness and accuracy of road subsidence warnings, and the risk of accidents occurring when the vehicle is too close to the subsidence location, potentially causing personal injury and property damage. Summary of the Invention

[0004] The purpose of this invention is to provide a vehicle and road collapse identification and early warning method and device to solve the problems of low timeliness and accuracy of existing road collapse early warning systems.

[0005] To solve the above-mentioned technical problems, this invention provides a method for identifying and warning of road collapses, comprising: Acquire the vehicle's Z-axis vibration signal and image data of the road surface ahead. Z-axis refers to the direction perpendicular to the road surface. Determine whether there are any abnormal areas affecting vehicle traffic based on image data; The Z-axis vibration signal is used to determine whether there is an abnormal area that affects vehicle passage. The judgment criteria are: if the amplitude of the Z-axis vibration signal exceeds the set threshold and the Z-axis vibration signal is identified as an abnormal vibration signal when there is a risk of road collapse, then it is determined that there is an abnormal area that affects vehicle passage. If any judgment result indicates the existence of an abnormal area affecting vehicle passage, it is determined that there is a risk of road collapse ahead of the vehicle, and a road collapse warning is issued ahead.

[0006] Furthermore, the abnormal vibration signals include at least one of the following: the Z-axis vibration signal corresponding to the critical point of road collapse caused by an earthquake, the Z-axis vibration signal corresponding to road collapse caused by an earthquake, the Z-axis vibration signal corresponding to a bridge deck falling and colliding, the Z-axis vibration signal corresponding to a road collapse collision, and the Z-axis vibration signal corresponding to a vehicle entering a pothole and colliding.

[0007] Furthermore, the process of identifying whether the Z-axis vibration signal is an abnormal vibration signal when there is a risk of road collapse is to compare the characteristics of the Z-axis vibration signal with the characteristics of the abnormal vibration signal. If the comparison is successful, the Z-axis vibration signal is identified as an abnormal vibration signal.

[0008] Furthermore, the image data is obtained using at least one environmental perception sensor selected from lidar, millimeter-wave radar, and infrared camera.

[0009] Furthermore, a road collapse warning is issued to the outside world through at least one communication method, such as V2V communication, V2I communication, and mobile network communication, and / or by projecting collapse information through a display unit.

[0010] The beneficial effects of the above technical solution are as follows: This invention is an improved invention that combines vehicle Z-axis vibration signals and image data of the road surface ahead to comprehensively judge road collapse. Based on the image data, it determines whether there are abnormal areas on the road surface that affect vehicle passage. Based on the Z-axis vibration signals, it determines whether there are abnormal areas on the road surface ahead caused by earthquakes or about to collapse, whether vehicles ahead have fallen into a collapse, or whether the bridge surface ahead has collapsed. Regardless of which data is used to determine the existence of an abnormal area affecting vehicle passage, it confirms that there is a risk of road collapse ahead. Because vibration signals can determine whether the road surface is abnormal from a distance, it enables long-distance collapse prediction and can promptly identify scenarios where the road surface has not yet collapsed but is at a critical point and is about to collapse. In this case, the image data cannot identify the collapse, but the road surface may collapse immediately, improving the timeliness of collapse warnings. Furthermore, when a collapse has already formed (such as a bridge surface that has completely collapsed), and it is impossible to determine whether there are abnormal areas through vibration signals, the presence of collapse defects on the road surface can be identified through image analysis. Based on the defects, it can be determined whether they affect vehicle passage. Through comprehensive judgment, the collapse hazard can be identified as much as possible, improving the timeliness and accuracy of road collapse warnings.

[0011] To address the aforementioned technical problems, the present invention also provides a road collapse identification and early warning device, including a processor, which is used to identify and warn of road collapses according to the road collapse identification and early warning method described above.

[0012] To address the aforementioned technical problems, the present invention also provides a vehicle equipped with an environmental perception sensor, an acceleration sensor, and a processor. The environmental perception sensor unit is used to collect image data of the road surface ahead of the vehicle, and the acceleration sensor is used to collect the vehicle's Z-axis vibration signal. The processor is used to acquire the vehicle's Z-axis vibration signal and the image data of the road surface ahead, determine whether there is an abnormal area affecting vehicle passage based on the image data, and determine whether there is an abnormal area affecting vehicle passage based on the Z-axis vibration signal. If either determination result indicates the existence of an abnormal area affecting vehicle passage, it is determined that there is a risk of road collapse ahead of the vehicle, and a road collapse warning is issued. Specifically, if the amplitude of the Z-axis vibration signal exceeds a set threshold and the Z-axis vibration signal is identified as an abnormal vibration signal indicating a risk of road collapse, then an abnormal area affecting vehicle passage is determined to exist. Z-axis refers to the direction perpendicular to the road surface.

[0013] Furthermore, the abnormal vibration signals include at least one of the following: the Z-axis vibration signal corresponding to the critical point of road collapse caused by an earthquake, the Z-axis vibration signal corresponding to road collapse caused by an earthquake, the Z-axis vibration signal corresponding to a bridge deck falling and colliding, the Z-axis vibration signal corresponding to a road collapse collision, and the Z-axis vibration signal corresponding to a vehicle entering a pothole and colliding.

[0014] Furthermore, the environmental perception sensor is at least one of lidar, millimeter-wave radar, and infrared camera.

[0015] Furthermore, the vehicle's ECU or AEB brakes the vehicle based on the road collapse warning ahead, and sends the road collapse warning ahead to the outside world through at least one communication method such as V2V communication, V2I communication and mobile network communication, and / or sends the road collapse warning ahead by projecting collapse information through the display unit.

[0016] The beneficial effects of the above technical solution are as follows: This invention is an improved invention that combines vehicle Z-axis vibration signals and image data of the road surface ahead to comprehensively judge road collapse. Based on the image data, it determines whether there are abnormal areas on the road surface that affect vehicle passage. Based on the Z-axis vibration signals, it determines whether there are abnormal areas on the road surface ahead caused by earthquakes or about to collapse, whether vehicles ahead have fallen into a collapse, or whether the bridge surface ahead has collapsed. Regardless of which data is used to determine the existence of an abnormal area affecting vehicle passage, it confirms that there is a risk of road collapse ahead. Because vibration signals can determine whether the road surface is abnormal from a distance, it enables long-distance collapse prediction and can promptly identify scenarios where the road surface has not yet collapsed but is at a critical point and is about to collapse. In this case, the image data cannot identify the collapse, but the road surface may collapse immediately, improving the timeliness of collapse warnings. Furthermore, when a collapse has already formed (such as a bridge surface that has completely collapsed), and it is impossible to determine whether there are abnormal areas through vibration signals, the presence of collapse defects on the road surface can be identified through image analysis. Based on the defects, it can be determined whether they affect vehicle passage. Through comprehensive judgment, the collapse hazard can be identified as much as possible, improving the timeliness and accuracy of road collapse warnings. Attached Figure Description

[0017] Figure 1 This is a flowchart of the road collapse identification and early warning process according to an embodiment of the method of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the present invention clearer, the specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0019] This invention identifies and judges landslides based on the Z-axis vibration signal of a vehicle. It determines whether the vibration signal is an abnormal vibration signal indicating a risk of landslides ahead of the vehicle and issues a landslide warning ahead.

[0020] Method Implementation The present invention provides a method for identifying and warning of road collapses, the principle of which is as follows: Figure 1 As shown below, a detailed explanation will follow.

[0021] 1. Acquire the vehicle's Z-axis vibration signal and image data of the road surface ahead.

[0022] 1) Acquire image data of the road surface ahead of the vehicle.

[0023] While a vehicle's forward-facing camera can acquire rich road image information, it is affected by factors such as lighting, weather, and obstruction from debris, especially failing to recognize objects in low-light conditions at night. Furthermore, its typical detection range is around 15 meters, which is insufficient to meet the high-speed braking distance requirements (exceeding 40-50 meters).

[0024] Currently, mainstream LiDAR systems can detect images at distances exceeding 250 meters, with a maximum range of 500 meters, and are not limited by nighttime lighting conditions. LiDAR boasts high precision and high resolution. It measures distance and acquires shape information by emitting a laser beam towards a target object and receiving the reflected laser. It is particularly accurate in detecting potholes with complex depths and shapes, and can capture road surface details more clearly. When a road collapses, potholes and protruding objects appear on the surface. LiDAR can detect the height differences and shape changes between the potholes and the surrounding road surface, providing detailed 3D road surface information and creating high-precision contour maps containing the outlines of potholes, protruding objects, and collapsed road surfaces. This improves the ability to recognize the contours of potholes, protruding objects, and collapsed road surfaces.

[0025] Millimeter-wave radar performs relatively stably in adverse weather conditions, is not limited by nighttime light, and has a measurement range of approximately 200 meters. It senses the surrounding environment by emitting millimeter waves and receiving reflected signals, acquiring information such as target distance, speed, and angle, and using this information to construct environmental contours and object distribution. In the event of a road collapse, millimeter-wave radar can construct high-precision contour maps containing potholes, protrusions, foreign objects, and the outline of the collapsed road surface.

[0026] The detection range of an infrared camera depends on several factors, including camera resolution, lens focal length, infrared lamp power, environmental conditions, and the size and temperature of the observed object. Professional or high-end infrared cameras are equipped with more powerful infrared illumination and high-performance optical components, achieving detection ranges of several hundred meters. When a road or bridge surface collapses, there is a temperature difference between the collapsed area and the road surface. Infrared cameras can construct high-precision contour maps of potholes, protruding objects, and the outline of the collapsed road surface based on the temperature profile of the collapsed area, generating infrared image data.

[0027] Therefore, in order to accurately acquire image data even during periods of poor lighting, this invention employs at least one environmental perception sensor, including lidar, millimeter-wave radar, and infrared camera, to acquire image data of the road surface ahead of the vehicle. The image data is at least one of the aforementioned lidar data, millimeter-wave radar data, and infrared image data.

[0028] In a preferred embodiment, the environmental sensors include LiDAR, millimeter-wave radar, and an infrared camera, and the acquired image data includes LiDAR data, millimeter-wave radar data, and infrared image data. Since LiDAR's recognition accuracy decreases in inclement weather such as rain and snow, and infrared cameras' recognition accuracy decreases on roads with laid heating pipes, while millimeter-wave radar, although having lower resolution and limited ability to depict details, has recognition accuracy unaffected by inclement weather and road surface temperature, combining multiple technologies leverages their respective advantages to achieve more accurate and reliable identification of potholes and landslides.

[0029] 2) Obtain the Z-axis vibration signal of the vehicle.

[0030] Vehicles are typically equipped with acceleration sensors, such as gyroscopes, for obstacle recognition and vehicle collision detection. This invention utilizes the vehicle's existing acceleration sensors to acquire the vehicle's Z-axis vibration signal, and identifies whether road surface collapse or other abnormalities have occurred based on the Z-axis vibration signal perpendicular to the road surface.

[0031] 2. Determine whether there are abnormal areas affecting vehicle traffic based on image data, and determine whether there are abnormal areas affecting vehicle traffic based on Z-axis vibration signals.

[0032] 1) Determine whether there are abnormal areas affecting vehicle traffic based on image data.

[0033] In one embodiment, an AI-based image recognition method is used to identify LiDAR data, millimeter-wave radar data, and infrared image data, respectively, and output a judgment result. Specifically, it determines whether the road surface has contour defects for each type of image data. If all image data has no contour defects, the judgment result is that the road surface is smooth. If any image data identifies a contour defect, it further determines whether the contour defect affects vehicle passage. If all image data identifies contour defects that do not affect vehicle passage, the judgment result is that there is an abnormal area that does not affect vehicle passage; if any image data identifies a contour defect that affects vehicle passage, the judgment result is that there is an abnormal area that affects vehicle passage. Abnormal areas include: road collapse areas and road surface potholes.

[0034] Among them, AI image recognition methods are existing technologies, such as neural network-based image recognition methods, nonlinear dimensionality reduction image recognition methods, or pattern recognition methods. For example, when using neural networks for image recognition, the neural network model can be pre-trained based on the labeled image data, and the features of the image data to be recognized can be extracted using the pre-trained neural network model. The recognition result is then output based on the image features.

[0035] In another embodiment, since the lidar image is high-precision and high-resolution image data, it can clearly show the height difference and shape changes between the pothole and the surrounding road surface, providing detailed three-dimensional road surface information. It is more accurate for detecting potholes with complex depth and shape. Therefore, it is possible to directly determine whether there are contour defects on the corresponding road surface through the lidar data output by the lidar, and if contour defects exist, determine whether the contour defects affect vehicle passage.

[0036] 2) Determine whether there are abnormal areas affecting vehicle traffic based on the Z-axis vibration signal.

[0037] If the amplitude of the Z-axis vibration signal exceeds the set threshold and the Z-axis vibration signal is identified as an abnormal vibration signal when there is a risk of road collapse, then it is determined that there is an abnormal area affecting vehicle traffic; otherwise, the judgment result of the Z-axis vibration signal is that the road surface is flat.

[0038] The threshold is set as the amplitude of the Z-axis vibration signal when the vehicle is driving on a smooth road surface, such as 9.8 m / s. 2 If the vehicle's Z-axis vibration signal exceeds a set threshold, it indicates an abnormality in the road surface, causing a change in the Z-axis vibration signal. Road surface abnormalities may be caused by vibration anomalies resulting from landslides, earthquakes, collisions when vehicles enter potholes, or bridge collapses. Different causes of abnormal vibration signals have different characteristics. For example, landslides caused by earthquakes are typically caused by the transmission of seismic transverse and longitudinal waves to the ground, resulting in low-frequency vibration signals. Collision vibrations caused by bridge falls and collapses are broadband vibration signals. Therefore, this invention uses self-learning to acquire the characteristics of abnormal vibration signals when there is a risk of road collapse. The acquired characteristics of the vehicle's Z-axis vibration signal are compared with the characteristics of the abnormal vibration signal (including signal frequency and amplitude). If the comparison is successful, the Z-axis vibration signal is identified as an abnormal vibration signal. Abnormal vibration signals include at least one of the following: Z-axis vibration signal corresponding to the critical point of road collapse caused by earthquake, Z-axis vibration signal corresponding to road collapse caused by earthquake, Z-axis vibration signal corresponding to bridge deck falling collision, Z-axis vibration signal corresponding to road collapse collision, and Z-axis vibration signal corresponding to vehicle entering pothole collision.

[0039] 3. Control is based on the judgment results of the Z-axis vibration signal and the judgment results of the image data.

[0040] When both the Z-axis vibration signal and the image data indicate that the road surface is smooth, vehicles can pass normally.

[0041] When the Z-axis vibration signal indicates that the road surface is smooth and the image data indicates that there is an abnormal area that does not affect vehicle passage, the vehicle can be controlled to slow down or pass normally without slowing down.

[0042] If either the judgment result of the Z-axis vibration signal or the judgment result of the image data indicates the existence of an abnormal area affecting vehicle passage, then it is determined that there is a risk of road collapse ahead of the vehicle, and a road collapse warning is issued.

[0043] Methods for issuing road collapse warnings include: sending the warning to the vehicle's instrument panel display and alerting occupants; the vehicle's ECU or AEB controllers applying the warning to brake in advance to avoid the collapse. Alternatively, warnings can be issued via in-vehicle navigation apps or mobile navigation apps using at least one of vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), or mobile network communication. This allows for sharing of road collapse information between vehicles, infrastructure, and the mobile internet, providing real-time traffic flow guidance and road closure information. Furthermore, warnings can be issued by projecting collapse information onto a display unit, such as the vehicle's taillights or welcome lights. Projecting road collapse patterns and text onto these lights allows following vehicles to identify the collapse in advance, preventing chain-reaction collisions and guiding following vehicles to brake and avoid catastrophic accidents. This can prevent vehicles from being knocked into bridge or road collapse zones, thus mitigating and reducing the risk of serious injuries or fatalities.

[0044] Device Implementation The present invention provides a road collapse identification and early warning device, which includes a processor. The processor is used to identify and warn of road collapses according to the road collapse identification and early warning method described in the above-described method embodiments, which will not be repeated here.

[0045] Vehicle implementation method This invention discloses a vehicle equipped with an environmental perception sensor, an acceleration sensor, and a processor. The environmental perception sensor collects image data of the road surface ahead of the vehicle, the acceleration sensor collects the vehicle's Z-axis vibration signal, and the processor acquires the vehicle's Z-axis vibration signal and the image data of the road surface ahead. The vehicle then uses the road collapse identification and early warning method described in the above-described method embodiment to identify and warn of road collapses, which will not be elaborated further here. The environmental perception sensor is at least one of lidar, millimeter-wave radar, and infrared camera.

Claims

1. A method for identifying and issuing early warnings of road collapses, characterized in that, include: Acquire the vehicle's Z-axis vibration signal and image data of the road surface ahead. Z-axis refers to the direction perpendicular to the road surface. Determine whether there are any abnormal areas affecting vehicle traffic based on image data; The Z-axis vibration signal is used to determine whether there is an abnormal area that affects vehicle passage. The judgment criteria are: if the amplitude of the Z-axis vibration signal exceeds the set threshold and the Z-axis vibration signal is identified as an abnormal vibration signal when there is a risk of road collapse, then it is determined that there is an abnormal area that affects vehicle passage. If any judgment result indicates the existence of an abnormal area affecting vehicle passage, it is determined that there is a risk of road collapse ahead of the vehicle, and a road collapse warning is issued ahead.

2. The road collapse identification and early warning method according to claim 1, characterized in that, Abnormal vibration signals include at least one of the following: Z-axis vibration signal corresponding to the critical point of road collapse caused by earthquake, Z-axis vibration signal corresponding to road collapse caused by earthquake, Z-axis vibration signal corresponding to bridge deck falling collision, Z-axis vibration signal corresponding to road collapse collision, and Z-axis vibration signal corresponding to vehicle entering pothole collision.

3. The road collapse identification and early warning method according to claim 2, characterized in that, The process of identifying whether a Z-axis vibration signal is an abnormal vibration signal when there is a risk of road collapse is as follows: the characteristics of the Z-axis vibration signal are compared with the characteristics of abnormal vibration signals. If the comparison is successful, the Z-axis vibration signal is identified as an abnormal vibration signal.

4. The road collapse identification and early warning method according to claim 1, characterized in that, The image data is obtained using at least one of the following environmental perception sensors: lidar, millimeter-wave radar, and infrared camera.

5. The road collapse identification and early warning method according to claim 1, characterized in that, The warning of road collapse ahead is issued to the outside world through at least one communication method, such as V2V communication, V2I communication and mobile network communication, and / or the warning of road collapse is issued by projecting the collapse information through a display unit.

6. A road collapse identification and early warning device, comprising a processor, characterized in that, The processor is used to identify and warn of road collapses according to the road collapse identification and early warning method according to any one of claims 1-5.

7. A vehicle, wherein an environmental perception sensor, an acceleration sensor, and a processor are installed on the vehicle, the environmental perception sensor unit being used to collect image data of the road surface ahead of the vehicle, characterized in that, The accelerometer is used to collect the Z-axis vibration signal of the vehicle; the processor is used to acquire the Z-axis vibration signal of the vehicle and image data of the road surface ahead, and to determine whether there is an abnormal area affecting the passage of the vehicle based on the image data and the Z-axis vibration signal. If either judgment result indicates the existence of an abnormal area affecting the passage of the vehicle, it is determined that there is a risk of road collapse ahead of the vehicle, and a road collapse warning is issued. If the amplitude of the Z-axis vibration signal exceeds a set threshold and the Z-axis vibration signal is identified as an abnormal vibration signal indicating a risk of road collapse, then it is determined that there is an abnormal area affecting the passage of the vehicle. Z-axis refers to the direction perpendicular to the road surface.

8. The vehicle according to claim 7, characterized in that, Abnormal vibration signals include at least one of the following: Z-axis vibration signal corresponding to the critical point of road collapse caused by earthquake, Z-axis vibration signal corresponding to road collapse caused by earthquake, Z-axis vibration signal corresponding to bridge deck falling collision, Z-axis vibration signal corresponding to road collapse collision, and Z-axis vibration signal corresponding to vehicle entering pothole collision.

9. The vehicle according to claim 7, characterized in that, The environmental perception sensor is at least one of lidar, millimeter-wave radar, and infrared camera.

10. The vehicle according to claim 7, characterized in that, The vehicle's ECU or AEB brakes the vehicle based on the road collapse warning ahead, and sends the road collapse warning ahead to the outside world through at least one communication method such as V2V communication, V2I communication and mobile network communication, and / or sends the road collapse warning ahead by projecting the collapse information through the display unit.

Images