A road vehicle anti-collision warning method and system based on multi-modal data fusion

Abstract

The application discloses a kind of road vehicle anti-collision early warning method and system based on multi-modal data fusion, it is related to intelligent transportation technology field, obtain the first video stream in the preset range in the road side where rubber intelligent buffer bucket is located;First motion state of target vehicle driving towards buffer bucket is analyzed;Second video stream that can simultaneously photograph target vehicle and buffer bucket is obtained, and cross-view angle correlation matching is carried out with first video stream, the second motion state between target vehicle and buffer bucket is fused and analyzed;Third motion state of target vehicle towards buffer bucket is obtained;Real-time environmental information and historical accident information in buffer bucket warning area are fused first motion state, second motion state, third motion state, the probability that target vehicle impacts rubber intelligent buffer bucket is calculated, and target vehicle driving towards buffer bucket is classified early warning based on probability.This application deeply fuses multi-source multi-modal data, significantly improves early warning accuracy, timeliness and reliability.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation technology, specifically to a road vehicle collision avoidance and early warning method and system based on multimodal data fusion. Background Technology

[0002] Rubber intelligent buffer drums are an indispensable part of the transportation system. They are mainly installed on highways and urban roads where collisions between vehicles and fixed road structures are likely to occur, such as road curves, entrances and exits, toll booths, bridge guardrail ends, and tunnel entrances, serving as isolation warnings and collision prevention. Traditional rubber intelligent buffer drums reduce impact force through physical cushioning, effectively reducing the impact force in the event of an accidental vehicle collision and minimizing injuries to people and vehicles.

[0003] In recent years, intelligent solutions have emerged that incorporate accelerometers, deformation sensors, and NB-IoT or 5G communication modules into smart rubber buffer barrels. These solutions can send alarm information to the backend after a collision, enabling timely accident reporting. However, existing smart rubber buffer barrels still have the following shortcomings: First, limited sensing range. Existing smart rubber buffer barrels mainly rely on their integrated accelerometers and deformation sensors for sensing. These sensors can essentially only sense changes in their own state (i.e., changes in deformation and acceleration after an impact), but cannot sense the approach of external objects. The sensing range of the smart rubber buffer barrel is limited to the instant of impact, lacking the ability to detect and predict the trajectory of vehicles approaching from a distance, resulting in a significant lag in early warning. Second, a lack of effective linkage between roadside monitoring equipment and smart rubber buffer barrels. Roadside sensing equipment such as surveillance cameras and millimeter-wave radars are widely deployed on highways for traffic monitoring, violation capture, and accident evidence collection. These devices can collect real-time video streams and trajectory data of passing vehicles, but in the current technology, there is no effective linkage mechanism between this data and the rubber smart buffer tank, forming data silos and making it impossible to use the vehicle dynamic information captured by the monitoring equipment for the proactive early warning of the rubber smart buffer tank. Summary of the Invention

[0004] The purpose of this invention is to provide a road vehicle collision avoidance warning method and system based on multimodal data fusion. It deeply integrates the multimodal data of vehicles captured by multiple fixed monitoring devices along the road section and the camera module of the rubber intelligent buffer barrel itself with the multimodal data of the rubber intelligent buffer barrel. Combined with real-time environmental information and historical accident information, it realizes full-link trajectory tracking of vehicles from far to near and collision risk classification warning, effectively improving the accuracy, timeliness and reliability of the warning.

[0005] To achieve the above objectives, this application proposes the following solution: On the one hand, this application provides a road vehicle collision avoidance warning method based on multimodal data fusion, specifically including the following steps: S1. Obtain several segments of the first video stream from road monitoring equipment within a preset range on the side of the road where the rubber intelligent buffer barrel is located; S2. Identify the driving trajectory of the target vehicle heading towards the rubber smart buffer in several segments of the first video stream, analyze the driving trajectory, and obtain the first motion state of the target vehicle. S3. When the target vehicle enters the field of view of at least one fixed monitoring device that can simultaneously capture the target vehicle and the rubber intelligent buffer barrel, the second video stream of at least one fixed monitoring device is acquired, the target vehicle in the second video stream is identified, cross-view correlation matching is performed with the target vehicle in the first video stream, and the relative motion relationship between the target vehicle and the rubber intelligent buffer barrel is analyzed to obtain the second motion state. S4. When the target vehicle enters the warning area of ​​the rubber intelligent buffer barrel, the camera module on the rubber intelligent buffer barrel itself is triggered to collect the third video stream, extract the direct motion state of the target vehicle toward the rubber intelligent buffer barrel, and obtain the third motion state of the target vehicle. S5. Obtain real-time environmental information and historical accident information within the warning area of ​​the rubber intelligent buffer barrel; integrate the first motion state, second motion state, third motion state, real-time environmental information and historical accident information to calculate the probability of the target vehicle colliding with the rubber intelligent buffer barrel, and provide graded warnings for target vehicles heading towards the rubber intelligent buffer barrel based on the probability.

[0006] In some specific technical solutions, the first motion state includes the velocity change trend and the trend classification result determined based on the velocity change trend. The specific process of obtaining the first motion state is as follows: Several segments of the first video stream are arranged from farthest to closest according to their distance from the rubber smart buffer tank. Each segment of the first video stream is analyzed in the order of arrangement, and each vehicle appearing in the first video stream is treated as a target vehicle. The average speed of the target vehicle is obtained by analyzing the motion process of each target vehicle in each segment of the first video stream. The average speed of the target vehicle obtained from each first video stream is analyzed in order to obtain the speed change trend. The speed change trend is compared with the normal driving change trend to determine the trend classification result.

[0007] In some specific technical solutions, the specific process of fusion analysis is as follows: When there are fixed monitoring devices with multiple perspectives, the second video streams from multiple perspectives are time-stamped and synchronized. The relative motion parameters between the target vehicle and the rubber smart buffer barrel are extracted from each viewpoint. Compare the consistency of relative motion parameters measured from different viewpoints and calculate the standard deviation; When the standard deviation is less than the preset threshold, the weighted fusion is performed according to the reciprocal of the measurement accuracy of each viewpoint to obtain the second motion state between the target vehicle and the rubber intelligent buffer. The second motion state includes the fusion distance between the target vehicle and the rubber intelligent buffer, the distance change rate, and the angle between the driving direction and the direction of the rubber intelligent buffer. When the standard deviation is greater than the preset threshold, the weight of the view with large deviation is automatically reduced during the weighted fusion process, which is performed by allocating weights according to the reciprocal of the measurement accuracy of each view. The correlation verification results are obtained by correlating the trend classification results of the second motion state with those of the first motion state.

[0008] In some specific technical solutions, the specific process of correlation verification includes: The distance change rate and the included angle are correlated and verified with the trend classification results. When the trend classification result is continuous acceleration, the distance change rate is negative and the included angle is less than the preset included angle, the risk trend is determined to continue and the first correlation verification result is output. When the trend classification result is continuous acceleration, and the distance change rate approaches 0 from a negative value or becomes a positive value, it is determined that the driver has taken braking measures, and the second correlation verification result is output. When the trend classification result is stable driving, but the included angle decreases by more than the preset angle threshold within a unit of time, it is determined that there is an emergency and the third correlation verification result is output.

[0009] In some specific technical solutions, the third motion state includes the proximity value between the target vehicle and the rubber intelligent buffer, the target vehicle's front facing angle, front wheel steering angle, brake light status, current vehicle speed, and the driver's intention inference result based on the comprehensive inference of the front facing angle, front wheel steering angle, brake light status and speed change trend in the first motion state.

[0010] In some specific technical solutions, the inference process for determining the driver's intent is as follows: When the speed change trend is stable or decelerating, the brake lights are on, and the vehicle's heading angle deviates from the direction of the rubber intelligent buffer, the driver's intention is to actively avoid danger. When the speed change trend shows continuous acceleration, the brake lights are off, and the vehicle's front angle points towards the rubber smart buffer, the driver's intention is to conclude that no risk is perceived. When the front wheel steering angle changes abruptly, the brake lights are on, and the vehicle's posture is abnormal, the driver's intention is to avoid an emergency collision. When the angle between the target vehicle's driving direction and the direction of the rubber intelligent buffer tank deviates from the vehicle's heading angle and there are no signs of steering operation, it is inferred to be out of control.

[0011] In some specific technical solutions, the process of calculating the probability of a target vehicle colliding with a rubber smart buffer is as follows: The first motion state is used as the trend feature vector, the second motion state as the relative motion feature vector, and the third motion state as the direct state feature vector. Real-time environmental information and historical accident information are used as auxiliary feature vectors. A collision risk prediction model is constructed by concatenating trend feature vectors, relative motion feature vectors, direct state feature vectors, and auxiliary feature vectors into a comprehensive feature vector, which is then input into the collision risk prediction model to calculate the probability of the target vehicle colliding with the rubber smart buffer.

[0012] In some specific technical solutions, calculating the probability of a target vehicle colliding with a rubber smart buffer also includes the following steps: The trend classification results of the second motion state are correlated and verified with those of the first motion state to obtain the correlation verification results. Perform a consistency check between the third motion state and the second motion state; When calculating probabilities, the input weights of the trend feature vector and the relative motion feature vector are adjusted based on the consistency verification results, and the contribution of the relative motion feature vector in the collision risk prediction model is adjusted based on the correlation verification results.

[0013] In some specific technical solutions, the second motion state includes the fusion distance between the target vehicle and the rubber intelligent buffer. The specific process of adjusting the input weights of the trend feature vector and the relative motion feature vector based on the consistency verification result is as follows: The near distance value is compared with the fusion distance. When the difference between the near distance value and the fusion distance is less than or equal to the preset warning distance, the consistency verification result is that the two source measurements are consistent, and the input weights of the relative motion feature vector are maintained. When the difference between the near distance value and the fusion distance is greater than the preset warning distance, the consistency verification result indicates that there is a calibration error or sensor failure, and the input weight of the relative motion feature vector is reduced.

[0014] Secondly, this application provides a road vehicle collision avoidance warning system based on multimodal data fusion, comprising: The vehicle long-distance motion state analysis module is used to acquire several segments of first video streams from road monitoring equipment within a preset range on the side of the road where the rubber intelligent buffer barrel is located, identify the driving trajectory of the target vehicle heading towards the rubber intelligent buffer barrel in the several segments of first video streams, analyze the driving trajectory, and obtain the first motion state of the target vehicle. The video fusion analysis module is used to acquire a second video stream from at least one fixed monitoring device when the target vehicle enters the field of view of at least one fixed monitoring device that can simultaneously capture the target vehicle and the rubber intelligent buffer barrel, identify the target vehicle in the second video stream, perform cross-view correlation matching with the target vehicle in the first video stream, fuse and analyze the relative motion relationship between the target vehicle and the rubber intelligent buffer barrel, and obtain the second motion state. The vehicle close-range motion state analysis module is used to trigger the camera module on the rubber intelligent buffer barrel to collect a third video stream when the target vehicle enters the warning area of ​​the rubber intelligent buffer barrel, and extract the direct motion state of the target vehicle toward the rubber intelligent buffer barrel to obtain the third motion state of the target vehicle. The early warning analysis module is used to acquire real-time environmental information and historical accident information within the early warning area of ​​the rubber intelligent buffer barrel; it integrates the first motion state, the second motion state, the third motion state, real-time environmental information, and historical accident information to calculate the probability of a target vehicle colliding with the rubber intelligent buffer barrel, and provides graded early warnings for target vehicles heading towards the rubber intelligent buffer barrel based on the probability.

[0015] The advantages of this invention over the prior art are as follows: This application expands the sensing range of the rubber smart buffer barrel by incorporating multiple fixed monitoring devices along the road into its sensing range. By analyzing the average speed change trend of vehicles in the intervals between each monitoring device, the first motion state is obtained, and the vehicle's motion trend is predicted in advance, thus achieving long-distance behavior pattern recognition and providing time-dimensional behavior pattern information for subsequent association verification and probability calculation. When the vehicle moves close to the rubber smart buffer (i.e., the fixed monitoring equipment can simultaneously capture images of the target vehicle and the rubber smart buffer), the relative positional relationship between the vehicle and the rubber smart buffer under the changing trend of the first motion state is analyzed to obtain the second motion state, which provides a basis for subsequently predicting the probability of the vehicle driving towards the rubber smart buffer. When a vehicle moves into the warning area of ​​the rubber intelligent buffer, that is, when the target vehicle enters the monitoring range of the camera module built into the rubber intelligent buffer, the direct motion state of the target vehicle toward the rubber intelligent buffer is extracted and fused with the first motion state and the second motion state mentioned above for analysis and verification. Taking into account the real-time environmental information and historical accident information within the warning area of ​​the rubber intelligent buffer, the probability of the target vehicle colliding with the rubber intelligent buffer is comprehensively analyzed. Based on the probability, the target vehicle heading toward the rubber intelligent buffer is given a graded warning, which effectively improves the accuracy, timeliness and reliability of the warning. Attached Figure Description

[0016] Figure 1A flowchart of a road vehicle collision avoidance and early warning method based on multimodal data fusion is provided for an embodiment of the present invention; Figure 2 This is a schematic diagram of a rubber intelligent buffer bucket placement scenario provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the distribution of roadside monitoring equipment provided in an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0019] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0020] Example 1 The inner wall of the rubber smart buffer drum is equipped with active sensing sensors to detect changes in temperature, humidity, deformation, acceleration, etc. By collecting information such as temperature and humidity, it helps the backend identify road conditions such as slippery surfaces and icy surfaces, in order to determine whether to issue safety warnings. Each rubber smart buffer drum is used as a distributed edge computing node, with a built-in collision risk prediction model to analyze collision risks. The rubber smart buffer drum also has a communication module deployed inside. Since roadside monitoring equipment on the side of the road leading to the rubber smart buffer drum can monitor vehicles traveling in the direction of the rubber smart buffer drum's location within its communication range, the rubber smart buffer drum can communicate with the roadside monitoring equipment. Based on the geographical coordinates of the rubber smart buffer drum (obtained through the built-in GPS / BDS dual-mode positioning module), it determines which roadside monitoring equipment it needs to communicate with. The preset range can be determined based on the communication range of the rubber smart buffer drum (e.g., a radius of 500-1000 meters), or it can be an optimal range determined based on historical analysis.

[0021] Roadside monitoring equipment can be categorized based on whether its monitoring field of view includes the location of the rubber smart buffer drum: When the monitoring field of view can only capture vehicles on the road, but not the rubber smart buffer drum itself, this type of equipment is typically deployed at a location far from the rubber smart buffer drum or at an angle where it cannot be captured (e.g., at a turn), for long-distance vehicle perception. The following description of the monitoring equipment refers to road monitoring equipment. When the monitoring field of view can simultaneously capture both vehicles and the rubber smart buffer drum, it indicates that the monitoring equipment is typically deployed near the area where the rubber smart buffer drum is located, for mid-range multi-view fusion analysis. The following description of this type of monitoring equipment refers to fixed monitoring equipment. Simultaneously, the calibration parameters of each roadside monitoring equipment (installation location latitude and longitude, installation height, pitch angle, horizontal viewing angle, lens focal length, etc.) are obtained. Based on the above scenario deployment, the solution of this application is described in detail below: like Figure 1 As shown, this embodiment provides a road vehicle collision avoidance warning method based on multimodal data fusion, specifically including the following steps: S1. Obtain several segments of the first video stream from road monitoring equipment within a preset range on the side of the road where the rubber intelligent buffer barrel is located; S2. Identify the driving trajectory of the target vehicle heading towards the rubber smart buffer in several segments of the first video stream, analyze the driving trajectory, and obtain the first motion state of the target vehicle. The first state of motion includes the velocity change trend and the trend classification result determined based on the velocity change trend. The specific process of obtaining the first state of motion is as follows: S21. Arrange several segments of the first video stream from farthest to closest according to their distance from the rubber intelligent buffer barrel, and analyze each segment of the first video stream in the order of arrangement. Treat each vehicle appearing in the first video stream as a target vehicle. S22. Analyze the motion process of each target vehicle in each segment of the first video stream (the time of appearance and disappearance and the detection length of the corresponding road monitoring equipment) to obtain the average speed of the target vehicle. S23. Analyze the average speed of the target vehicle obtained from each first video stream in the order of arrangement to obtain the speed change trend. Compare the speed change trend with the normal driving change trend to determine the trend classification result.

[0022] The first video stream only captures vehicles and not the rubber smart buffer. Target detection and multi-target tracking are performed on each of the first video streams. Target detection uses the YOLOv8s model to identify various vehicle types such as sedans, SUVs, trucks, and buses. Multi-target tracking uses the ByteTrack algorithm, assigning a unique tracking ID to each vehicle and recording a sequence of trajectory points. The road monitoring devices corresponding to several segments of the first video stream are arranged from farthest to closest to the rubber smart buffer, forming a monitoring device sequence. For adjacent devices in the monitoring device sequence, the spatiotemporal consistency similarity, motion state evolution similarity, and appearance feature similarity of the target vehicle at the end of the previous device's field of view and the target vehicle at the beginning of the current device's field of view are extracted. A comprehensive similarity score is calculated. Since the speed, acceleration, and direction angle of the same target vehicle do not change abruptly when traveling between adjacent monitoring devices, the smaller the difference in motion state, the higher the similarity. Therefore, when the comprehensive similarity score is greater than a preset standard threshold, it is determined to be the same vehicle.

[0023] Based on the appearance and disappearance times of each target vehicle in each segment of the first video stream, and the corresponding road detection length covered by the road monitoring equipment, the average speed of the target vehicle passing through the coverage area of ​​the monitoring equipment is calculated. The average speeds of the same target vehicle across each road monitoring equipment segment are then arranged in chronological order (from farthest to closest) to the road monitoring equipment and the rubber smart buffer drum, forming a speed sequence. The speed variation between adjacent segments is calculated to obtain the speed variation trend. This speed variation trend is then compared with the normal driving variation trend. The normal driving variation trend refers to the statistical average of the average speed variations of a large number of vehicles passing through each road monitoring equipment segment under historical normal traffic flow conditions on that road segment, reflecting the typical driving behavior pattern of that road segment.

[0024] When the speed change trend of multiple consecutive road monitoring equipment intervals is significantly greater than the normal change trend, the trend classification result is continuous acceleration, indicating that the vehicle is continuously accelerating over a long distance, which may be speeding or out of control. When the speed change trend of multiple consecutive road monitoring equipment intervals is less than the normal change trend, the trend classification result is continuous deceleration, indicating that the vehicle is continuously decelerating over a long distance, which may be due to the detection of a risk ahead. When each speed change trend is within the tolerance range of the normal change trend, the trend classification result is stable driving.

[0025] S3. When the target vehicle enters the field of view of at least one fixed monitoring device that can simultaneously capture the target vehicle and the rubber intelligent buffer barrel, the second video stream of at least one fixed monitoring device is acquired, the target vehicle in the second video stream is identified, cross-view correlation matching is performed with the target vehicle in the first video stream, and the relative motion relationship between the target vehicle and the rubber intelligent buffer barrel is analyzed to obtain the second motion state. When the target vehicle is captured by the road monitoring device closest to the rubber intelligent buffer barrel, the time when the target vehicle appears in the fixed monitoring device can be estimated based on the distance between the road monitoring device and the nearest fixed monitoring device. Video segments starting from that time are then extracted from the second video stream for analysis and tracking of the target vehicle. The second motion state includes the fused distance between the target vehicle and the rubber intelligent buffer barrel, the rate of change of distance, and the angle between the vehicle's direction of travel and the direction of the rubber intelligent buffer barrel. The specific process of fusion analysis is as follows: S31. When there are fixed monitoring devices with multiple perspectives, timestamp synchronization is performed on the second video streams from multiple perspectives. Rubber intelligent buffer barrels are typically placed in accident-prone areas, where there are usually many fixed monitoring devices to observe traffic conditions from multiple perspectives. Therefore, they can generally capture images of target vehicles and rubber intelligent buffer barrels from multiple angles.

[0026] S32. Extract the relative motion parameters between the target vehicle and the rubber smart buffer barrel from each viewpoint; The relative motion parameters include the Euclidean distance between the target vehicle's position and the rubber intelligent buffer barrel's position, the distance change rate Δd, where Δd represents the rate of change of distance with respect to time, and Δd<0 indicates close proximity; and the angle α between the target vehicle's driving direction and the rubber intelligent buffer barrel's direction, where a smaller α indicates a more direct approach to the rubber intelligent buffer barrel. S33. Compare the consistency of the same relative motion parameter measured from different viewpoints and calculate the standard deviation; S34. When the standard deviation is less than the preset threshold, the relative motion parameters are weighted and fused according to the reciprocal of the measurement accuracy of each viewpoint to obtain the second motion state between the target vehicle and the rubber intelligent buffer. S35. When the standard deviation is greater than the preset threshold, during the weighted fusion process of allocating weights according to the reciprocal of the measurement accuracy of each viewpoint, the weight of the viewpoint with large deviation (i.e. the deviation between the standard deviation and the preset threshold, and sorted according to the size of the deviation) is automatically reduced. S36. Correlate the trend classification results of the second motion state with those of the first motion state to obtain the correlation verification results.

[0027] The specific process of association verification includes: S361. Correlate the distance change rate and the included angle with the trend classification result. When the trend classification result is continuous acceleration, and the distance change rate is negative and the included angle is less than the preset included angle (e.g., 30°), the risk trend is determined to continue, and the first correlation verification result is output, indicating that the risk of the target vehicle continues to increase from far to near. S362. When the trend classification result is continuous acceleration, and the distance change rate approaches 0 from a negative value or becomes a positive value, it is determined that the driver has taken braking measures, and the second correlation verification result is output, indicating that the driver may have discovered the risk and started braking during the approach process. S363. When the trend classification result is stable driving, and the included angle decreases by more than the preset angle threshold (e.g., 20°) within a unit of time, it is determined that there is a sudden situation. The third correlation verification result is output, indicating that the target vehicle may suddenly turn towards the rubber smart buffer, which may be due to obstacle avoidance or driver distraction.

[0028] S4. When the target vehicle enters the warning area of ​​the rubber intelligent buffer barrel, the camera module on the rubber intelligent buffer barrel itself is triggered to collect a third video stream, extract the direct motion state of the target vehicle toward the rubber intelligent buffer barrel, and obtain the third motion state of the target vehicle; the camera module on the rubber intelligent buffer barrel itself takes pictures of the target vehicle at very close range, performs target detection and tracking, and focuses on the front features of the target vehicle (front of the vehicle, front bumper, windshield, license plate, etc.). The third motion state includes the proximity value between the target vehicle and the rubber intelligent buffer (calculated using the calibration parameters of the rubber intelligent buffer's own camera module to determine the precise distance between the front edge of the vehicle and the rubber intelligent buffer), the target vehicle's heading angle, front wheel steering angle, brake light status, current vehicle speed (estimated by analyzing the scale and position change rates of the vehicle's front area between video frames), and the driver's intention inference result based on the comprehensive inference of the heading angle, front wheel steering angle, brake light status, and speed change trends in the first motion state. The heading angle refers to the angle between the axis of symmetry of the target vehicle's front contour in the field of view of the rubber intelligent buffer's own camera module and the direction of the rubber intelligent buffer, reflecting the vehicle's posture; the direction of the rubber intelligent buffer refers to the unit vector direction pointing from the location of the rubber intelligent buffer as the origin to the current position of the target vehicle. Specifically, the inference process for determining the driver's intent is as follows: S41. When the speed change trend is stable or continuously decelerating, the brake lights are on, and the vehicle's front angle deviates from the direction of the rubber intelligent buffer, the driver's intention is to actively avoid danger, indicating that the driver may have discovered the risk and actively avoided it. S42. When the speed change trend shows continuous acceleration, the brake lights are off, and the vehicle's front angle points towards the rubber smart buffer, the driver's intention inference is that they did not perceive the risk, indicating that the driver may not have noticed the danger ahead. S43. When the steering angle of the front wheel changes abruptly, the brake lights are on, and the vehicle body posture is abnormal, the driver's intention is inferred to be an emergency avoidance maneuver, indicating that the driver may be taking emergency avoidance measures. S44. When the angle between the target vehicle's driving direction and the direction of the rubber intelligent buffer tank deviates from the vehicle's heading angle and there are no signs of steering operation, it is inferred that the vehicle is out of control.

[0029] S5. Obtain real-time environmental information and historical accident information within the warning area of ​​the rubber intelligent buffer barrel; integrate the first motion state, second motion state, third motion state, real-time environmental information and historical accident information to calculate the probability of the target vehicle colliding with the rubber intelligent buffer barrel, and provide graded warnings for target vehicles heading towards the rubber intelligent buffer barrel based on the probability.

[0030] Real-time environmental information includes humidity data obtained through the built-in temperature and humidity sensor of the rubber smart buffer barrel, visibility data obtained from the meteorological service platform, and light intensity obtained through the built-in light sensor. This information is used to comprehensively infer road surface conditions (dry / wet / waterlogged / icy). The above real-time environmental information is weighted and fused to calculate the environmental risk impact factor. Historical accident information is retrieved from the traffic management platform's historical accident database, including historical accident records for this road segment (within a 500-meter radius before and after the location of the rubber smart buffer barrel), including the total number of accidents and accident cause classification statistics. Based on this, a historical accident risk factor is calculated. The environmental risk impact factor and the historical accident risk factor serve as auxiliary feature vectors. The specific process for calculating the probability of a target vehicle colliding with the rubber smart buffer barrel is as follows: S51. Construct a collision risk prediction model by concatenating the trend feature vector, relative motion feature vector, direct state feature vector, and auxiliary feature vector into a comprehensive feature vector, and inputting it into the collision risk prediction model to calculate the probability of the target vehicle colliding with the rubber smart buffer.

[0031] S52. The first motion state is used as the trend feature vector, the second motion state is used as the relative motion feature vector, the third motion state is used as the direct state feature vector, and real-time environmental information and historical accident information are used as auxiliary feature vectors. In this embodiment, the collision risk prediction model adopts an LSTM neural network model, including a three-layer LSTM structure with 36-dimensional input, 64-dimensional hidden layers, and a single-neuron Sigmoid activation function in the output layer. It calculates the probability P∈[0,1] of the target vehicle colliding with the rubber smart buffer and estimates the estimated time to collision (TTC). The model includes a trend feature vector (speed change trend + trend classification result, 12 dimensions), a relative motion feature vector (e.g., fused distance, distance change rate, angle, etc., 8 dimensions), a direct state feature vector (near distance value, vehicle heading angle, front wheel steering angle, brake light status, current vehicle speed, driver intention inference result, 10 dimensions), and an auxiliary feature vector (environmental risk impact factor, historical accident risk factor, 6 dimensions). These four types of feature vectors are concatenated into a 36-dimensional comprehensive feature vector, which is then input into the LSTM neural network model for calculation.

[0032] S53. Verify the correlation between the trend classification results of the second motion state and the first motion state to obtain the correlation verification results; S54. Perform a consistency check between the third motion state and the second motion state; When calculating probabilities, the input weights of the trend feature vector and the relative motion feature vector are adjusted based on the consistency verification results, and the contribution of the relative motion feature vector in the collision risk prediction model is adjusted based on the correlation verification results.

[0033] The near-range value is compared with the fused distance. When the difference between the near-range value and the fused distance is less than or equal to the preset warning distance (e.g., 2 meters), the consistency verification result is that the two source measurements are consistent, and the input weights of the relative motion feature vector are maintained. The near-range value is the distance between the leading edge of the target vehicle and the rubber intelligent buffer barrel, measured by the rubber intelligent buffer barrel's own camera module through extremely close-range visual measurement, with a measurement accuracy of centimeters. The fused distance is the distance between the target vehicle and the rubber intelligent buffer barrel, measured by multi-view weighted fusion based on at least one fixed monitoring device. When the two are inconsistent, the more accurate near-range value is used.

[0034] When the difference between the near distance value and the fusion distance is greater than the preset warning distance, the consistency verification result indicates that there is a calibration error or sensor failure. The input weight of the relative motion feature vector is reduced, while the input weight of the direct state feature vector is increased, with the near distance value in the third motion state as the standard.

[0035] Meanwhile, the correlation verification results are used as category-coded features to input into the collision risk prediction model to adjust the contribution of the relative motion feature vector within the model: the first correlation verification result increases the contribution and drives the probability up; the second correlation verification result decreases the contribution and drives the probability down; the third correlation verification result increases the contribution and shortens the early warning decision cycle.

[0036] Based on the numerical range of the impact probability P, different levels of early warning measures are triggered: When P exceeds the first preset threshold (e.g., 0.5) and the estimated time to collision (TTC) is approximately 5 seconds, a Level 1 warning (yellow) is issued: the LED strobe lights flash yellow and a directional voice announcement is activated. The 5-second warning lead time provides a complete operating time window and a 1-2 second safety margin based on the driver's perception-reaction-action time model (approximately 1.8~3.5 seconds in total).

[0037] When P exceeds the second preset threshold (e.g., 0.7) and the estimated time to collision (TTC) is 3 seconds, a second-level warning (orange) is issued: activating alternating red and yellow flashing, a high-decibel voice alarm, and pushing warning information to the vehicle's onboard terminal via the communication module. 3 seconds corresponds to a distance of approximately 100 meters from the rubber intelligent buffer at a vehicle speed of 120 km / h, which is less than the safe stopping sight distance on a highway. When the TTC is less than 3 seconds, the driver's own reaction may be insufficient. When P exceeds the third preset threshold (e.g., 0.9) and the estimated collision time (TTC) is approximately 1.5 seconds, a Level 3 warning (red) is issued: activating red flashing lights, a continuous alarm sound, mandatory warning, information board linkage, and push notification to the traffic management platform. The theoretical basis for the third preset threshold of 0.9 is the unavoidable collision threshold (TTC=1.5s)—at which point, regardless of the driver's reaction speed, a collision cannot be avoided, and the warning objective shifts from "assisting in risk avoidance" to "accident mitigation" and emergency response preparation. When the acceleration sensor or deformation sensor of the rubber intelligent buffer barrel detects a collision event, it uploads the sensor data at the moment of collision, key frames from multiple video streams, and fused trajectory data to the traffic management platform.

[0038] like Figure 2 As shown, taking a fork in the road at a highway intersection as an example, this section is a highway merging nose area. A merging island is set up at the intersection of the main road and the ramp, and a rubber intelligent buffer is deployed at the front end of the merging island (towards oncoming traffic). The vehicle trajectory at this location is complex, with multiple traffic flows intersecting, including vehicles going straight on the main road, vehicles changing lanes to the right to enter the ramp, and vehicles merging from the ramp into the main road. It is a typical accident-prone area, especially at night and in adverse weather conditions, when the risk of the rubber intelligent buffer at the front end of the merging island being hit is significantly increased.

[0039] like Figure 3 As shown, taking one road as an example, road monitoring devices D1 and D2 are installed 500 meters and 250 meters away from the rubber intelligent buffer barrel, respectively (they can only photograph vehicles on the road, not the rubber intelligent buffer barrel itself), for long-distance vehicle detection. In the diversion nose area (a location that can photograph both vehicles and the rubber intelligent buffer barrel), fixed monitoring equipment is deployed, capable of simultaneously photographing vehicles approaching the rubber intelligent buffer barrel and the rubber intelligent buffer barrel itself. In addition, the rubber intelligent buffer barrel itself is equipped with a camera module for direct detection at very close range.

[0040] The system acquires two first video streams captured by road monitoring devices D1 and D2 on the road, performs target detection (YOLOv8s model) and multi-target tracking (ByteTrack algorithm) respectively, identifies target vehicles heading towards the rubber smart buffer barrel, and analyzes the first motion state of the target vehicles. Based on the target vehicle in the video stream captured by device D2, the estimated time for the target vehicle to enter fixed monitoring device D3 is determined. Fixed monitoring device D3 then begins recording and generating a second video stream. When the target vehicle enters the field of view of fixed monitoring device D3 in the diversion nose area (at which point the vehicle is approximately 100 meters away from the rubber intelligent buffer), the second video stream is matched with the target vehicle in the first video stream. The relative motion parameters between the vehicle and the rubber intelligent buffer (Euclidean distance between the vehicle's position and the rubber intelligent buffer's position, rate of change of distance, and angle between the vehicle's direction of travel and the direction of the rubber intelligent buffer) are extracted from the second video stream, and the estimated collision time is calculated. Based on the video stream captured by the D2 device, the estimated time for the target vehicle to enter the warning range of the rubber intelligent buffer barrel is determined. When the target vehicle enters within 50 meters in front of the rubber intelligent buffer barrel, the third-level perception analysis is triggered. The camera module on the rubber intelligent buffer barrel itself is activated, acquiring a third video stream to capture the vehicle's state towards the rubber intelligent buffer barrel at extremely close range, calculating the third motion state. Finally, the first, second, and third motion states obtained from the above analysis are input into the collision risk prediction model for fusion analysis with current environmental information and historical accident information to obtain the probability of the target vehicle colliding with the rubber intelligent buffer barrel. Based on the probability, a graded warning is issued for the target vehicle heading towards the rubber intelligent buffer barrel.

[0041] Example 2 This embodiment provides a road vehicle collision avoidance warning system based on multimodal data fusion, implementing the method of Embodiment 1, including: The vehicle long-distance motion state analysis module is used to acquire several segments of first video streams from road monitoring equipment within a preset range on the side of the road where the rubber intelligent buffer barrel is located, identify the driving trajectory of the target vehicle heading towards the rubber intelligent buffer barrel in the several segments of first video streams, analyze the driving trajectory, and obtain the first motion state of the target vehicle. The video fusion analysis module is used to acquire a second video stream from at least one fixed monitoring device when the target vehicle enters the field of view of at least one fixed monitoring device that can simultaneously capture the target vehicle and the rubber intelligent buffer barrel, identify the target vehicle in the second video stream, perform cross-view correlation matching with the target vehicle in the first video stream, fuse and analyze the relative motion relationship between the target vehicle and the rubber intelligent buffer barrel, and obtain the second motion state. The vehicle close-range motion state analysis module is used to trigger the camera module on the rubber intelligent buffer barrel to collect a third video stream when the target vehicle enters the warning area of ​​the rubber intelligent buffer barrel, and extract the direct motion state of the target vehicle toward the rubber intelligent buffer barrel to obtain the third motion state of the target vehicle. The early warning analysis module is used to acquire real-time environmental information and historical accident information within the early warning area of ​​the rubber intelligent buffer barrel; it integrates the first motion state, the second motion state, the third motion state, real-time environmental information, and historical accident information to calculate the probability of a target vehicle colliding with the rubber intelligent buffer barrel, and provides graded early warnings for target vehicles heading towards the rubber intelligent buffer barrel based on the probability.

[0042] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Based on the technical essence of the present invention, any simple modifications, equivalent substitutions, and improvements made to the above embodiments within the spirit and principles of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A road vehicle collision avoidance and early warning method based on multimodal data fusion, characterized in that, Specifically, the following steps are included: S1. Obtain several segments of the first video stream from road monitoring equipment within a preset range on the side of the road where the rubber intelligent buffer barrel is located; S2. Identify the driving trajectory of the target vehicle heading towards the rubber smart buffer in several segments of the first video stream, analyze the driving trajectory, and obtain the first motion state of the target vehicle. S3. When the target vehicle enters the field of view of at least one fixed monitoring device that can simultaneously capture the target vehicle and the rubber intelligent buffer barrel, the second video stream of at least one fixed monitoring device is acquired, the target vehicle in the second video stream is identified, cross-view correlation matching is performed with the target vehicle in the first video stream, and the relative motion relationship between the target vehicle and the rubber intelligent buffer barrel is analyzed to obtain the second motion state. S4. When the target vehicle enters the warning area of ​​the rubber intelligent buffer barrel, the camera module on the rubber intelligent buffer barrel itself is triggered to collect the third video stream, extract the direct motion state of the target vehicle toward the rubber intelligent buffer barrel, and obtain the third motion state of the target vehicle. S5. Obtain real-time environmental information and historical accident information within the warning area of ​​the rubber intelligent buffer barrel; integrate the first motion state, second motion state, third motion state, real-time environmental information and historical accident information to calculate the probability of the target vehicle colliding with the rubber intelligent buffer barrel, and provide graded warnings for target vehicles heading towards the rubber intelligent buffer barrel based on the probability.

2. The road vehicle collision avoidance and early warning method based on multimodal data fusion according to claim 1, characterized in that, The first state of motion includes the velocity change trend and the trend classification result determined based on the velocity change trend. The specific process of obtaining the first state of motion is as follows: Several segments of the first video stream are arranged from farthest to closest according to their distance from the rubber smart buffer tank. Each segment of the first video stream is analyzed in the order of arrangement, and each vehicle appearing in the first video stream is treated as a target vehicle. The average speed of the target vehicle is obtained by analyzing the motion process of each target vehicle in each segment of the first video stream. The average speed of the target vehicle obtained from each first video stream is analyzed in order to obtain the speed change trend. The speed change trend is compared with the normal driving change trend to determine the trend classification result.

3. The road vehicle collision avoidance and early warning method based on multimodal data fusion according to claim 2, characterized in that, The specific process of fusion analysis is as follows: When there are fixed monitoring devices with multiple perspectives, the second video streams from multiple perspectives are time-stamped and synchronized. The relative motion parameters between the target vehicle and the rubber smart buffer barrel are extracted from each viewpoint. Compare the consistency of relative motion parameters measured from different viewpoints and calculate the standard deviation; When the standard deviation is less than the preset threshold, the weighted fusion is performed according to the reciprocal of the measurement accuracy of each viewpoint to obtain the second motion state between the target vehicle and the rubber intelligent buffer. The second motion state includes the fusion distance between the target vehicle and the rubber intelligent buffer, the distance change rate, and the angle between the driving direction and the direction of the rubber intelligent buffer. When the standard deviation is greater than the preset threshold, the weight of the view with large deviation is automatically reduced during the weighted fusion process, which is performed by allocating weights according to the reciprocal of the measurement accuracy of each view. The correlation verification results are obtained by correlating the trend classification results of the second motion state with those of the first motion state.

4. The road vehicle collision avoidance and early warning method based on multimodal data fusion according to claim 3, characterized in that, The specific process of association verification includes: The distance change rate and the included angle are correlated and verified with the trend classification results. When the trend classification result is continuous acceleration, the distance change rate is negative and the included angle is less than the preset included angle, the risk trend is determined to continue and the first correlation verification result is output. When the trend classification result is continuous acceleration, and the distance change rate approaches 0 from a negative value or becomes a positive value, it is determined that the driver has taken braking measures, and the second correlation verification result is output. When the trend classification result is stable driving, but the included angle decreases by more than the preset angle threshold within a unit of time, it is determined that there is an emergency and the third correlation verification result is output.

5. A road vehicle collision avoidance and early warning method based on multimodal data fusion according to claim 2, characterized in that, The third motion state includes the proximity value between the target vehicle and the rubber smart buffer, the target vehicle's heading angle, front wheel steering angle, brake light status, current vehicle speed, and the driver's intention inference result based on the combined inference of the heading angle, front wheel steering angle, brake light status, and speed change trend in the first motion state.

6. The road vehicle collision avoidance and early warning method based on multimodal data fusion according to claim 5, characterized in that, The inference process for determining the driver's intent is as follows: When the speed change trend is stable or decelerating, the brake lights are on, and the vehicle's heading angle deviates from the direction of the rubber intelligent buffer, the driver's intention is to actively avoid danger. When the speed change trend shows continuous acceleration, the brake lights are off, and the vehicle's front angle points towards the rubber smart buffer, the driver's intention is to conclude that no risk is perceived. When the front wheel steering angle changes abruptly, the brake lights are on, and the vehicle's posture is abnormal, the driver's intention is to avoid an emergency collision. When the angle between the target vehicle's driving direction and the direction of the rubber intelligent buffer tank deviates from the vehicle's heading angle and there are no signs of steering operation, it is inferred to be out of control.

7. A road vehicle collision avoidance and early warning method based on multimodal data fusion according to claim 5, characterized in that, The specific process for calculating the probability of a target vehicle colliding with the rubber smart buffer is as follows: The first motion state is used as the trend feature vector, the second motion state as the relative motion feature vector, and the third motion state as the direct state feature vector. Real-time environmental information and historical accident information are used as auxiliary feature vectors. A collision risk prediction model is constructed by concatenating trend feature vectors, relative motion feature vectors, direct state feature vectors, and auxiliary feature vectors into a comprehensive feature vector, which is then input into the collision risk prediction model to calculate the probability of the target vehicle colliding with the rubber smart buffer.

8. A road vehicle collision avoidance and early warning method based on multimodal data fusion according to claim 7, characterized in that, Calculating the probability of a target vehicle colliding with a rubber smart buffer also includes the following steps: The trend classification results of the second motion state are correlated and verified with those of the first motion state to obtain the correlation verification results. Perform a consistency check between the third motion state and the second motion state; When calculating probabilities, the input weights of the trend feature vector and the relative motion feature vector are adjusted based on the consistency verification results, and the contribution of the relative motion feature vector in the collision risk prediction model is adjusted based on the correlation verification results.

9. A road vehicle collision avoidance and early warning method based on multimodal data fusion according to claim 8, characterized in that, The second motion state includes the fusion distance between the target vehicle and the rubber intelligent buffer. The specific process of adjusting the input weights of the trend feature vector and the relative motion feature vector based on the consistency verification result is as follows: The near distance value is compared with the fusion distance. When the difference between the near distance value and the fusion distance is less than or equal to the preset warning distance, the consistency verification result is that the two source measurements are consistent, and the input weights of the relative motion feature vector are maintained. When the difference between the near distance value and the fusion distance is greater than the preset warning distance, the consistency verification result indicates that there is a calibration error or sensor failure, and the input weight of the relative motion feature vector is reduced.

10. A road vehicle collision avoidance and early warning system based on multimodal data fusion, characterized in that, include: The vehicle long-distance motion state analysis module is used to acquire several segments of first video streams from road monitoring equipment within a preset range on the side of the road where the rubber intelligent buffer barrel is located, identify the driving trajectory of the target vehicle heading towards the rubber intelligent buffer barrel in the several segments of first video streams, analyze the driving trajectory, and obtain the first motion state of the target vehicle. The video fusion analysis module is used to acquire a second video stream from at least one fixed monitoring device when the target vehicle enters the field of view of at least one fixed monitoring device that can simultaneously capture the target vehicle and the rubber intelligent buffer barrel, identify the target vehicle in the second video stream, perform cross-view correlation matching with the target vehicle in the first video stream, fuse and analyze the relative motion relationship between the target vehicle and the rubber intelligent buffer barrel, and obtain the second motion state. The vehicle close-range motion state analysis module is used to trigger the camera module on the rubber intelligent buffer barrel to collect a third video stream when the target vehicle enters the warning area of ​​the rubber intelligent buffer barrel, and extract the direct motion state of the target vehicle toward the rubber intelligent buffer barrel to obtain the third motion state of the target vehicle. The early warning analysis module is used to acquire real-time environmental information and historical accident information within the early warning area of ​​the rubber intelligent buffer barrel; it integrates the first motion state, the second motion state, the third motion state, real-time environmental information, and historical accident information to calculate the probability of a target vehicle colliding with the rubber intelligent buffer barrel, and provides graded early warnings for target vehicles heading towards the rubber intelligent buffer barrel based on the probability.

Images