Vehicle emergency state behavior analysis method and system based on multi-modal information fusion

Abstract

The present application relates to a vehicle emergency dynamic behavior analysis method and system based on multi-modal information fusion, and relates to the technical field of traffic data processing. The method comprises the following steps: obtaining multi-modal information of a current emergency braking behavior; determining the cause of the vehicle emergency braking behavior based on the multi-modal information of the current emergency braking behavior; judging whether abnormal driving occurs based on the road information of the current driving trajectory, the weather information, the cause of each historical vehicle emergency braking behavior and the cause of the current vehicle emergency braking behavior, and generating first prompt information if abnormal driving occurs; obtaining vehicle braking state information of the current emergency braking behavior; judging whether the vehicle braking function is abnormal according to the vehicle braking state information of the current emergency braking behavior, and generating second prompt information when it is determined that the vehicle braking function is abnormal. The present application has the advantage of improving the intelligent level of vehicle emergency dynamic behavior analysis.

Description

Technical Field

[0001] This invention relates to the field of traffic data processing technology, and in particular to a method and system for analyzing vehicle dynamic behavior based on multimodal information fusion. Background Technology

[0002] Nowadays, with the improvement of people's living standards, the number of motor vehicles has also increased dramatically. Abnormal driving behavior by drivers poses a serious threat to property and life, and is a major cause of traffic accidents. Therefore, researching how to detect abnormal driving behavior using vehicle trajectory data is of great significance in providing traffic management departments with reliable detection methods. Existing methods for detecting abnormal driving behavior primarily rely on video data and vehicle GPS trajectory data.

[0003] However, video-based abnormal behavior detection is mainly aimed at single-vehicle restricted scenarios, making it difficult to monitor the entire transportation process; while GPS trajectory analysis is mainly based on prior threshold judgment for single vehicles, mostly setting detection thresholds to trigger alarms, lacking in-depth data analysis and information mining steps, and lacking intelligent identification and characteristic analysis functions.

[0004] Therefore, there is a need to provide a method and system for analyzing vehicle dynamic behavior based on multimodal information fusion, in order to improve the intelligence level of vehicle dynamic behavior analysis. Summary of the Invention

[0005] This invention provides a method for analyzing vehicle emergency braking behavior based on multimodal information fusion, comprising: acquiring multimodal information of the current emergency braking behavior; determining the cause of the vehicle's emergency braking behavior based on the multimodal information of the current emergency braking behavior; determining whether abnormal driving has occurred based on road information, weather information, the causes of previous vehicle emergency braking behaviors, and the cause of the current vehicle emergency braking behavior; if so, generating a first prompt message; acquiring vehicle braking status information of the current emergency braking behavior; determining whether the vehicle's braking function is abnormal based on the vehicle braking status information of the current emergency braking behavior, and generating a second prompt message when the vehicle's braking function is determined to be abnormal.

[0006] Furthermore, multimodal information during the emergency braking period is acquired, including: acquiring the working status information of multiple vehicle components; acquiring the road image in front of the vehicle; acquiring obstacle detection information of the vehicle; acquiring the driver's seat image of the vehicle; and acquiring the steering wheel pressure information of the vehicle.

[0007] Furthermore, acquiring vehicle steering wheel pressure information includes: setting multiple pressure detection positions on the vehicle's steering wheel; identifying multiple driving scenarios, where any two driving scenarios correspond to different steering wheel angles; for each driving scenario, acquiring pressure information from multiple sample users at multiple pressure detection positions within the driving scenario, and determining the grip difference value corresponding to the driving scenario based on the pressure information from multiple sample users at multiple pressure detection positions within the driving scenario; identifying key driving scenarios from the multiple driving scenarios based on the grip difference value corresponding to each driving scenario; identifying multiple user types based on the pressure information from multiple pressure detection positions within the key driving scenarios from multiple sample users, and determining multiple key pressure detection positions corresponding to each user type; acquiring the pressure information from multiple pressure detection positions of the current user at the key driving scenarios, determining the current user's user type, and acquiring the vehicle's steering wheel pressure information based on the multiple key pressure detection positions corresponding to the current user's user type, wherein the vehicle's steering wheel pressure information includes the pressure sequence of the multiple key pressure detection positions corresponding to the current user's user type.

[0008] Furthermore, based on multimodal information during the emergency braking period, the cause of the vehicle's abrupt dynamic behavior is determined, including: judging whether the cause of the abrupt dynamic behavior is a vehicle malfunction based on the working status information of multiple vehicle components; if the cause of the abrupt dynamic behavior is not a vehicle malfunction, judging whether the cause of the abrupt dynamic behavior is obstacle avoidance based on the vehicle's obstacle detection information; if the cause of the abrupt dynamic behavior is not obstacle avoidance, judging whether to extract traffic light features based on the electronic map and the vehicle's real-time location; if it is determined to extract traffic light feature sequences, predicting the traffic light image region based on the electronic map and the vehicle's real-time location; and extracting the traffic light feature sequence from the road image in front of the vehicle based on the predicted traffic light image region. The green light feature sequence is used to determine whether the cause of the vehicle's sudden dynamic behavior is a change in traffic signal. If it is determined that the traffic light feature should not be extracted, then based on the vehicle's obstacle detection information, it is determined whether to extract the feature sequence of the preceding vehicle's headlights. If it is determined that the feature sequence of the preceding vehicle's headlights should be extracted, then based on the vehicle's obstacle detection information, the image region of the preceding vehicle's headlights is predicted. Based on the image region of the preceding vehicle's headlights, the feature sequence of the preceding vehicle's headlights is extracted from the road image in front of the vehicle. Based on the feature sequence of the preceding vehicle's headlights, it is determined whether the cause of the vehicle's sudden dynamic behavior is to avoid the vehicle in front. If it is determined that the cause of the vehicle's sudden dynamic behavior is not a change in traffic signal or to avoid the vehicle in front, then based on the vehicle's driver's seat image and the vehicle's steering wheel pressure information, it is determined whether the cause of the vehicle's sudden dynamic behavior is fatigue driving.

[0009] Furthermore, based on the vehicle's driver's seat image and steering wheel pressure information, the system determines whether the cause of the vehicle's sudden dynamic behavior is fatigue driving by analyzing the preceding vehicle's headlight feature sequence. This includes: determining the current user's eye features based on the driver's seat image; determining the pressure change characteristics of each key pressure detection location based on the pressure sequence of multiple key pressure detection locations corresponding to the current user's user type; and determining whether the cause of the vehicle's sudden dynamic behavior is fatigue driving by analyzing the preceding vehicle's headlight feature sequence based on the current user's eye features and the pressure change characteristics of each key pressure detection location.

[0010] Furthermore, based on road information and weather information of the current driving trajectory, the causes of each historical vehicle emergency braking behavior, and the causes of the current vehicle emergency braking behavior, it is determined whether abnormal driving has occurred. This includes: searching for multiple similar historical driving trajectories of other users based on the road information and weather information of the current driving trajectory; and determining whether abnormal driving has occurred based on the vehicle emergency braking behavior data of multiple similar historical driving trajectories of other users, as well as the causes of each historical vehicle emergency braking behavior and the causes of the current vehicle emergency braking behavior. The vehicle emergency braking behavior data includes the number of historical vehicle emergency braking behaviors and the causes of each historical vehicle emergency braking behavior.

[0011] Furthermore, the vehicle braking status information for the current emergency braking behavior is obtained, including: acquiring test data and determining multiple brake vibration detection locations of the vehicle; based on the multiple brake vibration detection locations of the vehicle, obtaining the vehicle braking status information for the current emergency braking behavior, wherein the vehicle braking status information includes the vibration sequence of each brake vibration detection location.

[0012] Furthermore, test data is acquired to determine multiple braking vibration detection locations for the vehicle, including: identifying multiple test scenarios, where the road surface and braking parameters differ between any two test scenarios; determining multiple test locations for the sample vehicle corresponding to the current vehicle; for each test scenario, acquiring the vibration sequence of the sample vehicle corresponding to the current vehicle at multiple test locations under normal braking function and different braking function failures, and determining the key test locations corresponding to the test scenario; and based on the key test locations corresponding to each test scenario, determining multiple braking vibration detection locations for the vehicle.

[0013] Furthermore, based on the vehicle braking status information of the current emergency braking behavior, it is determined whether the vehicle braking function is abnormal, including: establishing and training a function judgment model; for each brake vibration detection location, extracting vibration features of the brake vibration detection location based on the vibration sequence of the brake vibration detection location; and determining whether the vehicle braking function is abnormal based on the vibration features of each brake vibration detection location using the function judgment model.

[0014] This invention provides a vehicle emergency behavior analysis system based on multimodal information fusion, used to execute the aforementioned vehicle emergency behavior analysis method based on multimodal information fusion, comprising: an information acquisition module for acquiring multimodal information of the current emergency braking behavior; a cause determination module for determining the cause of the vehicle emergency braking behavior based on the multimodal information of the current emergency braking behavior; and a behavior analysis module for determining whether abnormal driving has occurred based on road information, weather information, the causes of previous vehicle emergency braking behaviors, and the cause of the current vehicle emergency braking behavior, and if so, generating a first prompt message; the information acquisition module is also used to acquire vehicle braking status information of the current emergency braking behavior; and the behavior analysis module is also used to determine whether the vehicle braking function is abnormal based on the vehicle braking status information of the current emergency braking behavior, and generate a second prompt message when the vehicle braking function is determined to be abnormal.

[0015] Compared to existing technologies, the method and system for analyzing vehicle rapid dynamic behavior based on multimodal information fusion provided in this specification have at least the following advantages:

[0016] 1. Multimodal information fusion methods can comprehensively utilize information from different sensors and systems, such as road images, vehicle positions, meteorological information, and braking vibration data. This information complements each other, providing a more comprehensive reflection of vehicle emergency braking behavior and its underlying causes. By fusing multimodal information, errors and uncertainties that may arise from a single information source can be reduced, thereby improving the accuracy of behavior analysis. Through real-time analysis of vehicle emergency braking behavior and its causes, the system can promptly detect potential safety hazards, such as fatigued driving and braking system malfunctions. When abnormal driving or brake malfunction is detected, the system can immediately generate alerts to remind the driver or relevant maintenance personnel to take appropriate measures, thereby effectively avoiding or reducing traffic accidents.

[0017] 2. Based on steering wheel pressure information from multiple sample users in key driving scenarios, different user types and their corresponding key pressure detection locations can be identified. This facilitates personalized analysis of driver behavior, as different drivers may react and operate differently in emergency situations. By determining the current user's user type and corresponding key pressure detection locations, steering wheel pressure information during emergency braking can be accurately obtained. This information is of significant value for analyzing the current user's driving habits and emergency response capabilities.

[0018] 3. By combining eye features from the driver's seat image with pressure change characteristics from steering wheel pressure information, a comprehensive assessment of the driver's condition can be made from both visual and operational dimensions. Eye features such as blinking frequency and pupil size changes reflect the driver's level of fatigue, while steering wheel pressure changes reflect the driver's operational stability and concentration. The fusion of multi-dimensional information reduces potential misjudgments from a single information source and improves the accuracy of fatigue driving assessment. Real-time warnings help drivers adjust their state in a timely manner, avoiding traffic accidents caused by fatigue driving.

[0019] 4. By analyzing vehicle emergency braking behavior data (including the number of emergency brakes and their causes) from similar historical driving trajectories of multiple other users, normal driving behavior patterns can be established. Simultaneously, combining this with the current user's historical emergency braking behavior data allows for more accurate identification of abnormal behaviors that deviate significantly from the normal pattern. Considering the potential differences in driving habits and emergency response capabilities among drivers, the method achieves personalized analysis by comparing the current user's emergency braking behavior with that of other users on similar driving trajectories. Personalized analysis can more accurately capture each driver's abnormal driving behavior, reducing misjudgments and omissions. Attached Figure Description

[0020] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein:

[0021] Figure 1 This is a flowchart illustrating a method for analyzing vehicle rapid dynamic behavior based on multimodal information fusion in one embodiment of this application;

[0022] Figure 2 This is a flowchart illustrating the acquisition of vehicle steering wheel pressure information in one embodiment of this application;

[0023] Figure 3 This is a block diagram of a vehicle rapid dynamic behavior analysis system based on multimodal information fusion, as shown in one embodiment of this application. Detailed Implementation

[0024] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0025] Figure 1 This is a flowchart illustrating a method for analyzing vehicle abrupt dynamic behavior based on multimodal information fusion, as shown in one embodiment of this application. Figure 1 As shown, the method for analyzing vehicle dynamic behavior based on multimodal information fusion may include the following steps.

[0026] Step 110: Obtain multimodal information of the current emergency braking behavior.

[0027] Specifically, emergency braking refers to the act of a driver rapidly and forcefully pressing the brake pedal while the vehicle is in motion to slow down or stop the vehicle in a very short time. Its core purpose is to avoid collisions and deal with sudden dangers (such as obstacles ahead or pedestrians suddenly running in).

[0028] The reasons why the user suddenly braked include:

[0029] I. Sudden Risks in the Road Environment

[0030] 1. An obstacle suddenly appears ahead.

[0031] Scenario: Pedestrians or animals suddenly cross the road (e.g., children running, pets running into the road). Vehicles ahead suddenly stop (e.g., rear-ending the car in front, swerving to avoid pedestrians). Road debris (e.g., goods falling, construction debris).

[0032] 2. Sudden changes in road conditions

[0033] Scenario: Slippery road surface (rain, snow, standing water, oil stains). Road construction or temporary closure (if warning signs were not set up in advance). Sharp bends, slopes, or blind spots (such as obstacles suddenly appearing at tunnel exits).

[0034] II. Behavior of Traffic Participants

[0035] 1. Other vehicles operating in violation of regulations

[0036] Scenario: The vehicle in front changes lanes illegally (e.g., without using turn signals, cutting in). A vehicle in the adjacent lane suddenly merges into the lane (e.g., distracted driving without checking behind). A vehicle or non-motorized vehicle traveling in the wrong direction enters the lane.

[0037] 2. Pedestrian or non-motorized vehicle behavior is unpredictable.

[0038] Scenario: Pedestrian looking down at their phone without checking road conditions. Non-motorized vehicles suddenly turning or crossing (e.g., food delivery riders rushing to meet deadlines).

[0039] III. Abnormal Vehicle Condition

[0040] 1. Sudden vehicle malfunction

[0041] Scenario: Braking system failure (e.g., brake fluid leak, brake pad jamming). Tire blowout or sudden drop in tire pressure (e.g., running over a sharp object at high speed). Power system failure (e.g., engine stalling, electric vehicle suddenly losing speed).

[0042] 2. Risk of vehicle loss of control

[0043] Scenario: Vehicle skidding (e.g., sudden acceleration / braking on a wet road). Suspension or steering system malfunction causing vehicle drift.

[0044] IV. Driver Factors

[0045] 1. Distracted or fatigued driving

[0046] Scenario: The driver is looking down at their phone, adjusting navigation, or eating. Fatigue while driving leads to slowed reaction time (such as dozing off with eyes closed).

[0047] 2. Misoperation or lack of experience

[0048] Scenario: Novice drivers mistakenly press the accelerator / brake pedal (e.g., mistaking the accelerator for the brake). Lack of familiarity with vehicle performance (e.g., not understanding the braking force required when ABS engages).

[0049] V. Special Scenarios and Extreme Situations

[0050] 1. Natural disasters or emergencies

[0051] Scenario: An earthquake causes road surface cracks or rockfalls. Heavy rain triggers landslides or floods.

[0052] 2. An animal suddenly broke in

[0053] Scene: Animals cross a highway at night.

[0054] Multimodal information refers to different types of data related to emergency braking behavior acquired through multiple sensors or data sources. In emergency braking behavior analysis, multimodal information can provide a more comprehensive and accurate assessment of braking status, compensating for the limitations of single-sensor data.

[0055] The multimodal information of the current emergency braking behavior can include multimodal information from a period of time before emergency braking (e.g., 5s) to a period of time after braking ends (e.g., 5s).

[0056] In some embodiments, step 110 specifically includes:

[0057] The system acquires operational status information for multiple vehicle components, which may be vehicle parts that could cause a user to brake suddenly, such as the braking system, suspension, and steering system. The operational status information for these components may include at least the hydraulic pressure of the master cylinder and slave cylinders, the remaining thickness of the brake pads, the surface temperature of the brake discs, the number of activations, durations, and fault codes of the anti-lock braking system (ABS) and electronic stability program (ESP), suspension height, non-contact monitoring of suspension component temperatures, steering torque, steering angle and angular velocity, operating current of the electric power steering, tire pressure, tire temperature, engine speed and torque, and battery voltage.

[0058] To acquire images of the road in front of the vehicle, specifically, images of the road in front of the vehicle can be acquired using monocular / multi-view cameras (such as forward-looking wide-angle cameras or telephoto cameras).

[0059] Obtain obstacle detection information for the vehicle; specifically, this information can be obtained through onboard radar.

[0060] To obtain an image of the driver's seat in the vehicle, specifically, an image of the driver's seat can be obtained through an in-vehicle camera. The driver's seat image needs to cover the driver's face and upper body.

[0061] Obtain steering wheel pressure information for the vehicle.

[0062] Figure 2 This is a flowchart illustrating the acquisition of vehicle steering wheel pressure information in one embodiment of this application, as shown below. Figure 2 As shown, in some embodiments, obtaining vehicle steering wheel pressure information includes:

[0063] Multiple pressure detection positions are set on the vehicle's steering wheel. Specifically, multiple pressure detection positions can be determined manually based on experience or experimental data. The pressure detection positions can be on the steering wheel, in the positions where the user may hold the steering wheel while driving. At each pressure detection position, a flexible pressure sensor (such as a piezoelectric film or a capacitive sensor) is set to collect pressure information and fit against the surface of the steering wheel to ensure that it does not affect driving operation.

[0064] Multiple driving scenarios are identified, where any two scenarios correspond to different steering wheel angles. Driving scenarios are categorized based on steering wheel angle (e.g., straight driving, turning, U-turn), ensuring each scenario meets different driving operation requirements. For example, Scenario 1: Steering wheel angle within ±5° (straight driving). Scenario 2: Steering wheel angle ±15° (small-angle turn). Scenario 3: Steering wheel angle ±30° (large-angle turn or U-turn). Drivers' grip may differ in different scenarios (e.g., lighter grip for straight driving, firmer grip for turning), requiring separate analysis.

[0065] For each driving scenario, pressure information of multiple sample users at multiple pressure detection locations in the driving scenario is obtained. Based on the pressure information of multiple sample users at multiple pressure detection locations in the driving scenario, the grip difference value corresponding to the driving scenario is determined. The pressure information of the pressure detection location can include pressure values ​​at multiple consecutive time points.

[0066] Based on the grip difference value corresponding to each driving scenario, key driving scenarios are determined from multiple driving scenarios;

[0067] Based on the pressure information of multiple sample users at multiple pressure detection locations in key driving scenarios, multiple user types are identified, and multiple key pressure detection locations corresponding to each user type are determined.

[0068] The system acquires pressure information from multiple pressure detection locations for the current user in key driving scenarios, determines the user type of the current user, and acquires the vehicle's steering wheel pressure information based on the multiple key pressure detection locations corresponding to the current user's user type. The vehicle's steering wheel pressure information includes the pressure sequence of multiple key pressure detection locations corresponding to the current user's user type.

[0069] Specifically, for each driving scenario, each sample user, and each pressure detection location, the maximum, mean, and variance of the pressure values ​​at multiple consecutive time points at the pressure detection location can be calculated.

[0070] For each driving scenario and each sample user, a pressure feature matrix can be constructed based on the maximum, mean, and variance of the pressure values ​​at each pressure detection location within that driving scenario. Each row vector in the pressure feature matrix corresponds to a pressure detection location, and the elements of this row vector include the maximum, mean, and variance of the pressure values ​​at that location. Pressure detection locations where the maximum pressure value is greater than a maximum value threshold and the variance is greater than a variance threshold can be used as the first target pressure detection locations for that driving scenario for that sample user. The first target pressure detection locations for each driving scenario for each sample user are then deduplicated, and the remaining first target pressure detection locations are used as multiple second target pressure detection locations for that sample user. The maximum value threshold and variance threshold can be determined empirically or through experimental analysis.

[0071] For each driving scenario, the grip difference value corresponding to the driving scenario can be calculated based on the pressure feature matrix corresponding to each sample user in that driving scenario. Specifically, based on the pressure feature matrices corresponding to any two sample users in that driving scenario, the grip pressure similarity between the two sample users in that driving scenario is calculated. The average grip pressure similarity between any two sample users in that driving scenario is then calculated to obtain the average grip pressure similarity value. The grip difference value corresponding to the driving scenario is determined based on the average grip pressure similarity value.

[0072] For example, the similarity of grip pressure between two sample users in the same driving scenario can be calculated using the following formula:

[0073] ;

[0074] in, Let m be the similarity of grip pressure between the m-th sample user and the n-th sample user corresponding to the k-th driving scenario. This represents the total number of rows in the pressure feature matrix. This represents the total number of columns in the pressure feature matrix. Let be the value of the element in the i-th row and j-th column of the pressure feature matrix corresponding to the k-th driving scenario for the m-th sample user. This represents the value of the element in the i-th row and j-th column of the pressure feature matrix corresponding to the k-th driving scenario for the n-th sample user.

[0075] Understandable. The formula calculates the squared difference between corresponding element values ​​of two users, reflecting the difference in grip pressure at a specific location. When the grip pressure feature matrices of the two users are identical, the similarity is 1 (because the denominator is 1 + 0 = 1); as the difference increases, the denominator increases, and the similarity decreases. The formula quantifies the difference between the two users by calculating the sum of the squared differences between corresponding elements of their grip pressure feature matrices.

[0076] The grip difference value corresponding to the driving scenario can be calculated using the following formula:

[0077] ;

[0078] in, This represents the grip difference value corresponding to the k-th driving scenario. This represents the average grip pressure similarity for the k-th driving scenario.

[0079] Understandable. It is obtained by calculating the similarity of grip pressure between any two sample users in the k-th driving scenario and then averaging these similarities. It reflects the average similarity of grip pressure between sample users in that driving scenario. (Grip Difference Value) It is by dividing 1 by This calculation method ensures that when the mean similarity of grip pressure is large (i.e., the grip pressure among sample users is relatively similar), the calculation is accurate. A smaller value indicates little difference; when When the pressure is relatively small (i.e., the difference in grip pressure among sample users is relatively large). A larger value indicates a significant difference.

[0080] Driving scenarios where the grip difference value is greater than the grip difference value threshold can be considered as key driving scenarios. The grip difference value threshold can be determined through experience or experimental analysis.

[0081] Based on the similarity of grip pressure between any two sample users for each driving scenario, drivers are classified into different types using clustering algorithms (such as K-means).

[0082] For each user type, multiple second target pressure detection locations corresponding to each sample user included in the user type can be deduplicated, and the remaining second target pressure detection locations can be used as multiple key pressure detection locations corresponding to the user type.

[0083] For each user type, sample users can be drawn from that user type. The pressure feature matrix corresponding to the sampled sample users and the current user in each driving scenario can be calculated. The grip pressure similarity between the sampled sample users and the current user in each driving scenario can be calculated. The average grip pressure similarity between the sampled sample users and the current user in each driving scenario can be calculated to obtain the matching degree between the current user and the user type. The user type with the highest matching degree can be taken as the user type of the current user.

[0084] Step 120: Based on the multimodal information of the current emergency braking behavior, determine the cause of the vehicle's emergency braking behavior.

[0085] Specifically, it includes:

[0086] Based on the working status information of multiple components of the vehicle, determine whether the cause of the vehicle's sudden dynamic behavior is a vehicle malfunction.

[0087] If it is determined that the cause of the vehicle's sudden dynamic behavior is not a vehicle malfunction, then based on the vehicle's obstacle detection information, it is determined whether the cause of the vehicle's sudden dynamic behavior is obstacle avoidance.

[0088] If the cause of the vehicle's sudden dynamic behavior is determined not to be obstacle avoidance, based on the electronic map and the vehicle's real-time location, it is determined whether to extract traffic light features. If it is determined to extract traffic light feature sequences, based on the electronic map and the vehicle's real-time location, the traffic light image area is predicted. Based on the predicted traffic light image area, the traffic light feature sequence is extracted from the road image in front of the vehicle. Based on the traffic light feature sequence, it is determined whether the cause of the vehicle's sudden dynamic behavior is a change in traffic signal. If it is determined not to extract traffic light features, then based on the vehicle's obstacle detection information, it is determined whether to extract the preceding vehicle's headlight feature sequence. If it is determined to extract the preceding vehicle's headlight feature sequence, based on the vehicle's obstacle detection information, the preceding vehicle's headlight image area is predicted. Based on the preceding vehicle's headlight image area, the preceding vehicle's headlight feature sequence is extracted from the road image in front of the vehicle. Based on the preceding vehicle's headlight feature sequence, it is determined whether the cause of the vehicle's sudden dynamic behavior is to avoid the vehicle in front.

[0089] If it is determined that the cause of the vehicle's sudden dynamic behavior is not a change in traffic signal or avoidance of the vehicle in front, then based on the driver's seat image and the vehicle's steering wheel pressure information, the characteristic sequence of the vehicle's headlights in front is used to determine whether the cause of the vehicle's sudden dynamic behavior is fatigue driving.

[0090] Specifically, timestamp synchronization technology is used to align multimodal information to the same time base.

[0091] The vehicle fault determination model can be used to determine whether the cause of the vehicle's sudden dynamic behavior is a vehicle fault based on the working status information of multiple components of the vehicle. The vehicle fault determination model can be a long short-term memory network model.

[0092] Based on the obstacle detection information of the vehicle, it can be determined whether there is an obstacle in front of or between the sides of the vehicle that is less than a first distance threshold. If there is an obstacle, it is determined that the cause of the vehicle’s sudden dynamic behavior is to avoid the obstacle. The first distance threshold can be determined by experience or experimental data.

[0093] Based on the electronic map and the vehicle's real-time location, the system determines whether the current location is near or at a traffic light-controlled intersection. If so, it extracts the traffic light feature sequence. Using the traffic light location information in the electronic map and the vehicle's real-time location, the system can predict the approximate area of ​​the traffic lights in the road image ahead of the vehicle. Based on the predicted traffic light image area, the system extracts the traffic light feature sequence from the road image ahead of the vehicle, such as color and flashing status. Based on the extracted traffic light feature sequence and the vehicle's driving status (such as speed and acceleration), the system determines whether the vehicle's abrupt dynamic behavior is caused by a change in traffic signal (such as a red light).

[0094] If the system determines that traffic light features should not be extracted (i.e., the current location is not close to or at an intersection controlled by traffic lights, or the traffic light features are insufficient to explain the sudden dynamic behavior), it determines whether to extract the preceding vehicle's headlight feature sequence based on the vehicle's obstacle detection information. For example, if the distance between the preceding vehicle and the current vehicle is less than a second distance threshold, it determines to extract the preceding vehicle's headlight feature sequence. The second distance threshold is greater than the first distance threshold, and the second distance threshold can be determined through experience or experimental data. Using obstacle detection information, the approximate area of ​​the preceding vehicle's headlights in the road image ahead of the vehicle can be predicted. Based on the predicted preceding vehicle headlight image area, the system extracts the preceding vehicle's headlight feature sequence from the road image ahead of the vehicle, such as brightness, flashing frequency, and color. Based on the extracted preceding vehicle headlight feature sequence, combined with the vehicle's driving state and obstacle detection information, it determines whether the vehicle's sudden dynamic behavior is caused by avoiding the preceding vehicle. For example, if the preceding vehicle suddenly brakes or changes lanes, and abnormal changes in the preceding vehicle's headlights are detected in the road image ahead of the vehicle (such as brake lights illuminating), the system may determine that the vehicle's sudden dynamic behavior is due to avoiding the preceding vehicle.

[0095] In some embodiments, based on the driver's seat image and the vehicle's steering wheel pressure information, determining whether the cause of the vehicle's sudden dynamic behavior is fatigue driving is based on the characteristic sequence of the vehicle's headlights, including:

[0096] Based on the driver's seat image of the vehicle, the current user's eye features are determined. Specifically, image processing technologies (such as face recognition, eye localization, etc.) are used to extract the driver's eye features from the driver's seat image, including but not limited to the degree of eyelid closure, eye movement trajectory, blinking frequency, etc.

[0097] Based on the pressure sequence of multiple key pressure detection locations corresponding to the current user's user type, determine the pressure change characteristics of each key pressure detection location, such as pressure fluctuation range, pressure change frequency, and pressure peak value.

[0098] Based on the current user's eye characteristics and the pressure change characteristics at each key pressure detection location, the system determines whether the cause of the vehicle's sudden dynamic behavior is fatigue driving by analyzing the preceding vehicle's headlight feature sequence.

[0099] Specifically, a fatigue recognition model can be used to determine whether the cause of the vehicle's sudden dynamic behavior is fatigue driving by judging the characteristic sequence of the headlights of the vehicle in front based on the current user's eye characteristics and the pressure change characteristics of each key pressure detection location. The fatigue recognition model can be a convolutional neural network model.

[0100] In some embodiments, more causes of sudden vehicle behavior can be determined, and the determination of causes of sudden vehicle behavior can be achieved in any feasible manner.

[0101] Step 130: Based on the road information and weather information of the current driving trajectory, the cause of each historical vehicle emergency braking behavior and the cause of the current vehicle emergency braking behavior, determine whether abnormal driving has occurred. If so, generate the first prompt information.

[0102] Specifically, it includes:

[0103] Based on road information (such as road type, speed limit, curve radius, slope, etc.) and meteorological information (such as weather conditions, visibility, temperature, humidity, etc.) of the current driving trajectory, multiple similar historical driving trajectories of other users are searched. For example, based on the matching degree of road features and meteorological conditions, multiple similar historical driving trajectories are filtered out to ensure that the filtered trajectories are highly similar to the current driving trajectory in terms of road conditions and meteorological conditions.

[0104] Based on the vehicle emergency braking behavior data of multiple other users' similar historical driving trajectories, as well as the causes of each historical vehicle emergency braking behavior of the current user and the causes of the current vehicle emergency braking behavior, it is determined whether abnormal driving has occurred. The vehicle emergency braking behavior data includes the number of historical vehicle emergency braking behaviors and the causes of each historical vehicle emergency braking behavior.

[0105] Specifically, if the current user's frequency of sudden braking is significantly higher than that of other users, or if the reasons for sudden braking differ significantly from those of other users (e.g., other users mainly brake suddenly due to changes in traffic signals, while the current user frequently brakes suddenly due to fatigue), then it is determined to be abnormal driving. If abnormal driving is determined, the system will generate an initial warning message to remind the driver to pay attention to driving safety, which may include suggestions such as taking a rest, adjusting driving habits, or paying attention to road conditions ahead.

[0106] An abnormal driving determination model can be used to determine whether abnormal driving has occurred based on the vehicle emergency braking behavior data of multiple similar historical driving trajectories of other users, as well as the causes of each historical vehicle emergency braking behavior of the current user and the causes of the current vehicle emergency braking behavior. The abnormal driving determination model can be a Support Vector Machine (SVM) model.

[0107] Step 140: Obtain vehicle braking status information for the current emergency braking action.

[0108] Specifically, it includes:

[0109] Acquire test data to determine multiple brake vibration detection locations on the vehicle;

[0110] Based on multiple brake vibration detection locations of the vehicle, the vehicle braking status information of the current emergency braking behavior is obtained, wherein the vehicle braking status information includes the vibration sequence of each brake vibration detection location.

[0111] In some embodiments, acquiring test data to determine multiple brake vibration detection locations of the vehicle includes:

[0112] Multiple test scenarios are defined, where any two test scenarios have different road surfaces (e.g., dry road surface, wet road surface, gravel road surface, icy road surface, etc.) and braking parameters (e.g., vehicle speed and turning angle before braking, etc.). These different road surface conditions and braking parameters are combined to form multiple differentiated test scenarios. It is ensured that any two test scenarios differ in at least one of the road surface or braking parameters.

[0113] Multiple test locations are identified for the sample vehicle corresponding to the current vehicle. Specifically, a detailed structural analysis of the sample vehicle is conducted to understand the layout of the braking system and the location of key components. Based on the vehicle structural analysis results, several representative test locations are selected. These locations may include the vicinity of brake discs, brake pads, brake calipers, brake lines, and other components related to the braking system.

[0114] For each test scenario, vibration sequences are acquired from multiple test locations of the sample vehicle corresponding to the current vehicle under normal braking function and different braking function failures (such as brake pad wear, brake fluid leakage, etc.). The key test locations corresponding to the test scenario are determined. The acquired vibration sequences are analyzed and compared to identify the test locations with the greatest difference in vibration characteristics between normal braking function and different failure states. These locations are the key test locations corresponding to that test scenario.

[0115] Based on the key test locations corresponding to each test scenario, multiple braking vibration detection locations for the vehicle are determined. Specifically, the key test locations corresponding to each test scenario are summarized into a list containing all key test locations. After deduplication, the remaining key test locations are used as the multiple braking vibration detection locations for the vehicle.

[0116] Step 150: Based on the vehicle braking status information of the current emergency braking behavior, determine whether the vehicle braking function is abnormal, and generate a second prompt message when it is determined that the vehicle braking function is abnormal.

[0117] In some embodiments, determining whether the vehicle braking function is abnormal based on the vehicle braking status information of the current emergency braking behavior includes:

[0118] Establish and train the functional decision model;

[0119] For each brake vibration detection location, the vibration features of the brake vibration detection location are extracted based on the vibration sequence of the brake vibration detection location.

[0120] The vehicle braking function is determined to be abnormal based on the vibration characteristics of each brake vibration detection location using a functional determination model. The functional determination model can be a support vector machine.

[0121] Specifically, the collected braking vibration sequence is preprocessed, such as by filtering, denoising, and normalization. Relevant vibration features are then extracted from the preprocessed vibration sequence, including time-domain features (such as mean, variance, peak value, etc.), frequency-domain features (such as frequency components, spectral energy, etc.), and time-frequency-domain features (such as wavelet transform coefficients, etc.).

[0122] When a vehicle's braking function is determined to be malfunctioning, a prompt message is generated in a timely manner to remind the driver or relevant maintenance personnel to pay attention to the problem with the braking system and take appropriate measures.

[0123] Figure 3 This is a block diagram of a vehicle rapid dynamic behavior analysis system based on multimodal information fusion, as shown in one embodiment of this application. Figure 3 As shown, a vehicle dynamic behavior analysis system based on multimodal information fusion can include an information acquisition module, a cause determination module, and a behavior analysis module.

[0124] The information acquisition module is used to acquire multimodal information about the current emergency braking behavior;

[0125] The cause determination module is used to determine the cause of the vehicle's emergency braking behavior based on the multimodal information of the current emergency braking behavior.

[0126] The behavior analysis module is used to determine whether abnormal driving has occurred based on road information, weather information, the causes of each historical vehicle emergency braking behavior and the cause of the current vehicle emergency braking behavior. If so, it generates the first prompt message.

[0127] The information acquisition module is also used to acquire vehicle braking status information for the current emergency braking action;

[0128] The behavior analysis module is also used to determine whether the vehicle's braking function is abnormal based on the vehicle's braking status information during the current emergency braking behavior, and to generate a second prompt message when the vehicle's braking function is determined to be abnormal.

[0129] The vehicle dynamic behavior analysis system based on multimodal information fusion can be used to execute the vehicle dynamic behavior analysis method based on multimodal information fusion, which will not be elaborated here.

[0130] Finally, it should be understood that the embodiments described in this specification are merely illustrative of the principles of the embodiments described herein. Other variations may also fall within the scope of this specification. Therefore, alternative configurations of the embodiments described herein are intended to be illustrative rather than limiting, and should be considered consistent with the teachings of this specification. Accordingly, the embodiments described herein are not limited to those explicitly introduced and described herein.

Claims

1. A method for analyzing vehicle rapid dynamic behavior based on multimodal information fusion, characterized in that, include: Obtain multimodal information about the current emergency braking behavior; Based on the multimodal information of the current emergency braking behavior, determine the cause of the vehicle's emergency braking behavior; Based on road information, weather information, the causes of each historical vehicle emergency braking behavior and the cause of the current vehicle emergency braking behavior, determine whether abnormal driving has occurred. If so, generate the first prompt message. Obtain vehicle braking status information for the current emergency braking action; Based on the vehicle braking status information of the current emergency braking behavior, determine whether the vehicle braking function is abnormal, and generate a second prompt message when it is determined that the vehicle braking function is abnormal. This includes determining whether abnormal driving has occurred based on road information, weather information, the causes of each historical vehicle emergency braking incident, and the causes of the current vehicle's emergency braking incident; this includes: Based on road and weather information of the current driving trajectory, find multiple similar historical driving trajectories of other users; Based on the vehicle emergency braking behavior data of multiple other users' similar historical driving trajectories, as well as the causes of each historical vehicle emergency braking behavior of the current user and the causes of the current vehicle emergency braking behavior, it is determined whether abnormal driving has occurred. The vehicle emergency braking behavior data includes the number of historical vehicle emergency braking behaviors and the causes of each historical vehicle emergency braking behavior.

2. The method for analyzing vehicle rapid dynamic behavior based on multimodal information fusion according to claim 1, characterized in that, Obtain multimodal information during the emergency braking period, including: Obtain operational status information for multiple components of the vehicle; Acquire an image of the road in front of the vehicle; Obtain obstacle detection information for the vehicle; Obtain an image of the driver's seat of the vehicle; Obtain steering wheel pressure information for the vehicle.

3. The method for analyzing vehicle rapid dynamic behavior based on multimodal information fusion according to claim 2, characterized in that, Obtain vehicle steering wheel pressure information, including: Multiple pressure detection points are installed on the vehicle's steering wheel; Define multiple driving scenarios, where any two driving scenarios correspond to different steering wheel angles; For each driving scenario, pressure information from multiple pressure detection locations of multiple sample users is obtained. Based on the pressure information from multiple pressure detection locations of multiple sample users in the driving scenario, the grip difference value corresponding to the driving scenario is determined. Based on the grip difference value corresponding to each driving scenario, key driving scenarios are determined from multiple driving scenarios; Based on the pressure information of multiple sample users at multiple pressure detection locations in key driving scenarios, multiple user types are identified, and multiple key pressure detection locations corresponding to each user type are determined. The system acquires pressure information from multiple pressure detection locations for the current user in key driving scenarios, determines the user type of the current user, and acquires the vehicle's steering wheel pressure information based on the multiple key pressure detection locations corresponding to the current user's user type. The vehicle's steering wheel pressure information includes the pressure sequence of multiple key pressure detection locations corresponding to the current user's user type.

4. The method for analyzing vehicle rapid dynamic behavior based on multimodal information fusion according to claim 3, characterized in that, Based on multimodal information during emergency braking, the causes of a vehicle's abrupt dynamic behavior are determined, including: Based on the working status information of multiple components of the vehicle, determine whether the cause of the vehicle's sudden dynamic behavior is a vehicle malfunction. If it is determined that the cause of the vehicle's sudden dynamic behavior is not a vehicle malfunction, then based on the vehicle's obstacle detection information, it is determined whether the cause of the vehicle's sudden dynamic behavior is obstacle avoidance. If the cause of the vehicle's sudden dynamic behavior is determined not to be obstacle avoidance, based on the electronic map and the vehicle's real-time location, it is determined whether to extract traffic light features. If it is determined to extract traffic light feature sequences, based on the electronic map and the vehicle's real-time location, the traffic light image area is predicted. Based on the predicted traffic light image area, the traffic light feature sequence is extracted from the road image in front of the vehicle. Based on the traffic light feature sequence, it is determined whether the cause of the vehicle's sudden dynamic behavior is a change in traffic signal. If it is determined not to extract traffic light features, then based on the vehicle's obstacle detection information, it is determined whether to extract the preceding vehicle's headlight feature sequence. If it is determined to extract the preceding vehicle's headlight feature sequence, based on the vehicle's obstacle detection information, the preceding vehicle's headlight image area is predicted. Based on the preceding vehicle's headlight image area, the preceding vehicle's headlight feature sequence is extracted from the road image in front of the vehicle. Based on the preceding vehicle's headlight feature sequence, it is determined whether the cause of the vehicle's sudden dynamic behavior is to avoid the vehicle in front. If it is determined that the cause of the vehicle's sudden dynamic behavior is not a change in traffic signal or avoidance of the vehicle in front, then based on the driver's seat image and the vehicle's steering wheel pressure information, the characteristic sequence of the vehicle's headlights in front is used to determine whether the cause of the vehicle's sudden dynamic behavior is fatigue driving.

5. The method for analyzing vehicle rapid dynamic behavior based on multimodal information fusion according to claim 4, characterized in that, Based on the driver's seat image and steering wheel pressure information, the system determines whether the cause of the vehicle's sudden dynamic behavior is fatigue driving by analyzing the characteristic sequence of the vehicle's headlights. This includes: Based on the driver's seat image of the vehicle, determine the current user's eye features; Based on the pressure sequence of multiple key pressure detection locations corresponding to the current user's user type, determine the pressure change characteristics of each key pressure detection location. Based on the current user's eye characteristics and the pressure change characteristics at each key pressure detection location, the system determines whether the cause of the vehicle's sudden dynamic behavior is fatigue driving by analyzing the preceding vehicle's headlight feature sequence.

6. The method for analyzing vehicle rapid dynamic behavior based on multimodal information fusion according to any one of claims 1-5, characterized in that, Obtain vehicle braking status information for the current emergency braking action, including: Acquire test data to determine multiple brake vibration detection locations on the vehicle; Based on multiple brake vibration detection locations of the vehicle, the vehicle braking status information of the current emergency braking behavior is obtained, wherein the vehicle braking status information includes the vibration sequence of each brake vibration detection location.

7. The method for analyzing vehicle rapid dynamic behavior based on multimodal information fusion according to claim 6, characterized in that, Acquire test data to determine multiple brake vibration detection locations on the vehicle, including: Multiple test scenarios are defined, where the road surface and braking parameters are different for any two test scenarios; Determine multiple test locations for the sample vehicles corresponding to the current vehicle; For each test scenario, obtain the vibration sequence of the sample vehicle corresponding to the current vehicle under normal braking function and different braking function failures at multiple test locations to determine the key test locations corresponding to the test scenario; Based on the key test locations corresponding to each test scenario, multiple braking vibration detection locations for the vehicle are determined.

8. The method for analyzing vehicle rapid dynamic behavior based on multimodal information fusion according to claim 7, characterized in that, Based on the vehicle's braking status information during the current emergency braking action, determine whether the vehicle's braking function is abnormal, including: Establish and train the functional decision model; For each brake vibration detection location, the vibration features of the brake vibration detection location are extracted based on the vibration sequence of the brake vibration detection location. The functional judgment model determines whether the vehicle's braking function is abnormal based on the vibration characteristics of each brake vibration detection location.

9. A vehicle dynamic behavior analysis system based on multimodal information fusion, characterized in that, The method for performing vehicle rapid dynamic behavior analysis based on multimodal information fusion as described in claim 1 includes: The information acquisition module is used to acquire multimodal information about the current emergency braking behavior; The cause determination module is used to determine the cause of the vehicle's emergency braking behavior based on the multimodal information of the current emergency braking behavior. The behavior analysis module is used to determine whether abnormal driving has occurred based on road information, weather information, the causes of each historical vehicle emergency braking behavior and the cause of the current vehicle emergency braking behavior. If so, it generates the first prompt message. The information acquisition module is also used to acquire vehicle braking status information of the current emergency braking behavior; The behavior analysis module is also used to determine whether the vehicle's braking function is abnormal based on the vehicle's braking status information during the current emergency braking behavior, and to generate a second prompt message when the vehicle's braking function is determined to be abnormal.

Images