Method and system for detecting situational and idiosyncratic road rage in real time

Abstract

The invention discloses a method and a system for detecting situational and idiosyncratic road rage in real time. The basic idea of the method is to mine and utilize the driver emotion, the vehicle state, the traffic condition and other vehicle interaction multi-dimensional data, accurately identify the road rage type through a multi-modal data fusion technology, and take differential prevention and control measures. According to the invention, by integrating the driver emotion, the vehicle state, the traffic condition and the multi-dimensional information of other vehicle interaction, the system can realize more accurate road rage classification and judgment, the error of a single data source is reduced, and the accuracy of road rage judgment is improved. Aiming at different expressions and harmfulness of situational road rage and idiosyncratic road rage, the system adopts differentiated prevention and control measures. The situational road rage is intervened by controlling the vehicle speed and providing emotion regulation suggestions, and the special road rage is used for reducing the risk by strictly limiting the speed and warning other vehicles.

Description

Technical Field

[0001] The present invention relates to a situational and idiosyncratic road rage real-time detection method and system based on multimodal fusion, belonging to the field of traffic safety driving. Background Art

[0002] In recent years, the problem of drivers getting angry while driving, namely "road rage", has become increasingly prominent, and its harmfulness has continued to escalate. Road rage not only leads to increased traffic congestion, but also seriously threatens the safety of the drivers involved, and also poses a great threat to the safety of surrounding pedestrians and vehicles.

[0003] Most current driving behavior monitoring technologies only focus on a single influencing factor, such as the driver's facial expression or the vehicle's motion state, and lack comprehensive analysis of multi-dimensional data. However, the manifestation of road rage is often the result of multiple factors working together, so single-mode monitoring is prone to misjudgment or missed judgment. Summary of the invention

[0004] The purpose of the present invention is to provide a real-time detection method and system for situational and idiosyncratic road rage based on multimodal deep fusion for the field of traffic management and road safety.

[0005] The basic idea of ​​this invention is to mine and utilize multi-dimensional data of driver emotions, vehicle status, traffic conditions and interactions with other vehicles, accurately identify the type of road rage through multimodal data fusion technology, and take differentiated prevention and control measures to provide scientific basis and technical support for traffic safety management.

[0006] The present invention provides a real-time detection method for situational and idiosyncratic road rage, comprising:

[0007] Data collection: Using vehicle sensors and onboard equipment to collect real-time data on the driver’s facial expressions, driving behavior, traffic conditions, and interactions with other vehicles;

[0008] Data preprocessing: Preprocess the collected initial data, including data cleaning, outlier processing and image denoising, and remove samples with substandard data quality to obtain an initial data set that can be used for analysis;

[0009] Feature extraction: Extract multidimensional features from preprocessed data, including:

[0010] Driver emotional characteristics: The probability value of the driver's anger is calculated through the image processing model;

[0011] Driving behavior characteristics: Quantitatively evaluate the driver's driving behavior by calculating the acceleration change index and steering aggressiveness coefficient;

[0012] Traffic condition characteristics: Calculate the current road congestion index by standardizing traffic flow and vehicle speed data;

[0013] Other vehicle interaction characteristics: Real-time monitor and analyze the behavior of other vehicles, calculate the dynamic collision risk index, interaction urgency, and heading conflict coefficient, and calculate the comprehensive interaction characteristic index according to weights;

[0014] Risk calculation: Adopt multi-modal data fusion technology, combine driver emotion characteristics, driving behavior characteristics, traffic condition characteristics, and other vehicle interaction characteristics to calculate the situational road rage risk and trait road rage risk respectively;

[0015] Risk determination: According to the calculated situational road rage risk and trait road rage risk, determine whether the driver has road rage and its type.

[0016] The present invention also provides a real-time detection system for situational and trait road rage, including:

[0017] A data acquisition module for collecting real-time data of the driver's facial expressions, driving behaviors, traffic conditions, and other vehicle interactions by using vehicle sensors and in-vehicle devices;

[0018] A data preprocessing module for preprocessing the collected initial data, including data cleaning, outlier processing, and image denoising, and removing samples with substandard data quality to obtain an initial data set available for analysis;

[0019] A feature extraction module for extracting multi-dimensional features from the preprocessed data, including driver emotion characteristics, driving behavior characteristics, traffic condition characteristics, and other vehicle interaction characteristics;

[0020] A risk calculation module for adopting multi-modal data fusion technology, combining driver emotion characteristics, driving behavior characteristics, traffic condition characteristics, and other vehicle interaction characteristics to calculate the situational road rage risk and trait road rage risk respectively;

[0021] A risk determination module for determining whether the driver has road rage and its type according to the calculated situational road rage risk and trait road rage risk.

[0022] The beneficial effects of the present invention are as follows:

[0023] (1) By integrating multi-dimensional information of driver emotion, vehicle state, traffic condition, and other vehicle interaction, the system can achieve more accurate classification and determination of road rage, reduce the error of a single data source, and improve the accuracy of road rage determination.

[0024] (2)For the different manifestations and hazards of situational road rage and trait road rage, the system has adopted differentiated prevention and control measures. Situational road rage is intervened by controlling the vehicle speed and providing emotional regulation suggestions, while trait road rage reduces risks through stricter speed limits and warning reminders to other vehicles.

[0025] (3)The system can monitor the driver's emotions and driving behaviors in real time, and adopt differentiated prevention and control measures according to the type of road rage to ensure road safety. At the same time, the system evaluates the implementation effect of the prevention and control measures, dynamically adjusts the prevention and control measures, and feedbacks to optimize the prevention and control strategy to achieve the best road rage prevention and control effect.

[0026] (4)Through multi-modal data fusion, the system can overcome individual differences and achieve more personalized emotion determination. The system also has a dynamic adjustment strategy, which can flexibly adjust the detection frequency and prevention and control measures according to the real-time traffic conditions and changes in driver behavior to ensure the real-time and accuracy of the system. Description of the Drawings

[0027] Figure 1 Schematic diagram of the structure of the real-time detection and prevention and control system for situational and trait road rage based on multi-modal deep fusion;

[0028] Figure 2 Schematic diagram of data collection;

[0029] Figure 3 Schematic diagram of the CNN model structure;

[0030] Figure 4 Logic diagram of road rage risk calculation;

[0031] Figure 5 Logic diagram of trait road rage risk calculation based on the LSTM model;

[0032] Figure 6 Technical architecture diagram of situational road rage prevention and control; Figure 7 Technical architecture diagram of trait road rage prevention and control. Detailed Implementation Modes

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the following will further describe the implementation modes of this application in detail with reference to the accompanying drawings.

[0034] As Figure 1 shown, the real-time detection method for situational and trait road rage based on multi-modal deep fusion provided by the embodiment of this application includes the following steps:

[0035] Step 1: Collect the initial data set of the road rage detection and prevention and control system;

[0036] Step 2: Multidimensional feature extraction of road rage;

[0037] Step 3: Calculate the road rage risk;

[0038] Based on the obtained road rage risk, the embodiment of the present application further includes Step 4: Differentiated prevention and control measures for different road rages and dynamic adjustment.

[0039] In Step 1, initial data of the road rage detection and prevention system is collected.

[0040] In some embodiments, the following steps are included:

[0041] Step 1.1: As Figure 2 shown, the system uses vehicle sensors and in-vehicle devices to obtain real-time data of the driver's facial expressions, driving behaviors, traffic conditions, and interactions with other vehicles, specifically as follows:

[0042] (1) The process of obtaining the driver's emotional characteristics is as follows:

[0043] The images of the driver's facial expressions, eye movements, and head postures are collected in real time through an in-vehicle camera.

[0044] (2) The process of obtaining the driver's driving behavior data is as follows:

[0045] The driving behavior is monitored through vehicle sensors, including an acceleration sensor and a steering wheel sensor, to obtain the acceleration a and the steering wheel rotation angle ω.

[0046] (3) The process of obtaining the traffic condition data is as follows:

[0047] The real-time traffic flow parameters, including the average speed v ave and the flow rate Q, are obtained through traffic detectors such as microwave radars and cameras.

[0048] (4) The process of obtaining the data of interactions with other vehicles is as follows:

[0049] The behaviors of surrounding vehicles are monitored through radars and cameras to obtain the lateral distance x and the longitudinal distance y from other vehicles, the driving speed v other and the acceleration a other .

[0050] Step 1.2: Preprocess each item of data in the initial data set, including operations such as data cleaning, outlier processing, and image denoising.

[0051] Step 1.3: Eliminate the samples with unqualified data quality to obtain an initial data set available for analysis.

[0052] In Step 2, for the multidimensional feature extraction of road rage, in some embodiments, the following steps are included:

[0053] Step 2.1: Process the facial expression image using the CNN model to calculate the probability value E of the driver's angry emotion, as Figure 3 shown. The convolutional neural network adopts a two-level convolutional architecture to process the 48×48 pixel three-channel input: the input layer of the first convolutional group receives a 48×48×3 image. Convolution layer 1 uses a 3×3 convolutional kernel (32 channels, ReLU activation), and obtains a 46×46×32 feature map through valid convolution. Pooling layer 1 uses 2×2 max pooling, and the output dimension is reduced to 23×23×32. Convolution layer 2 of the second convolutional group uses a 3×3 convolutional kernel (64 channels, ReLU activation) to output 21×21×64 deep features. Pooling layer 2 precisely adjusts the feature map to 5×5×64. The flattening layer converts the three-dimensional features into a 1600-dimensional vector. The fully connected layer is extended to 1024 neurons, combined with 60% Dropout regularization, and the output layer generates the angry probability value E∈[0,1] through the Sigmoid function.

[0054] Further, this step includes the following two steps:

[0055] Step 2.1.1: Use the convolutional neural network (CNN) to process the facial expression image data of the driver and extract visual features related to emotions. Input the facial image sequence {X t}, calculate to obtain the CNN feature sequence, and the CNN outputs the feature vector: . In the formula, θ CNN —— the weight vectors of the convolutional kernel, pooling layer, and fully connected layer.

[0056] Step 2.1.2: Map the features to the emotion category space, use the Softmax activation function to output the probability distribution value, and obtain the probability value E of the driver's angry emotion.

[0057] Step 2.2: Calculate the acceleration change index J and the steering aggressiveness coefficient a steer through the formula to quantitatively evaluate the driving behavior characteristics of the driver. The feature extraction calculation formula is as follows:

[0058] (1) Acceleration change index J:

[0059]

[0060] (2) Steering aggressiveness coefficient a steer :

[0061]

[0062] In the formula, a x 2 —— longitudinal acceleration;

[0063] ay 2 —— Lateral acceleration;

[0064] μ —— Coefficient of friction;

[0065] g —— Acceleration due to gravity;

[0066] ω —— Steering wheel angle;

[0067] ω max —— Maximum steering wheel angle;

[0068] Step 2.3: Standardize the traffic flow and vehicle speed data, and use the formula to calculate the current road congestion index and analyze and evaluate the traffic conditions. The specific formula is as follows:

[0069] Congestion index:

[0070] Where R —— Congestion index;

[0071] Q —— Current traffic flow;

[0072] q max —— Maximum number of vehicles that can pass through the road per unit time;

[0073] v ave —— Average speed;

[0074] v f —— Free flow speed;

[0075] Step 2.4: Real-time monitor and analyze the behavior of other vehicles, calculate the dynamic collision risk index DCRI, interaction urgency IU, and heading conflict coefficient HCC, and calculate the comprehensive interaction characteristic index M according to the weights. The specific formula is as follows:

[0076]

[0077]

[0078]

[0079] M = 0.5 × DCR + 0.3 × IU + 0.2 × HCC

[0080] Where DCRI —— Dynamic collision risk index;

[0081] IU —— Interaction urgency;

[0082] HCC —— Heading conflict coefficient;

[0083] x —— Lateral distance between the host vehicle and other vehicles;

[0084] y —— Longitudinal distance between the host vehicle and other vehicles;

[0085] v other —— Driving speed of other vehicles;

[0086] a other —— Acceleration of other vehicles;

[0087] λ —— Risk sensitivity;

[0088] δ —— Lateral risk sensitivity coefficient;

[0089] In step 3, as Figure 4 shown in the fusion layer, multimodal fusion of the four aspects of features is performed to calculate the road rage risk. In some embodiments, it includes the following steps:

[0090] Step 3.1: Calculate the situational road rage risk using the Analytic Hierarchy Process (AHP).

[0091] Furthermore, this step includes the following two steps:

[0092] Step 3.1.1: Construct a hierarchical structure model, calculate the weight vector of each factor and perform a consistency test to ensure the rationality of the judgment matrix. Furthermore, this step is divided into the following two steps:

[0093] Step 3.1.1.1: Construct a hierarchical structure model. The situational road rage risk evaluation index is determined by the anger emotion probability value E, the traffic congestion index R, and the comprehensive interaction feature index M, and a judgment matrix A is constructed.

[0094] Step 3.1.1.2: According to the situational road rage judgment matrix A, calculate the corresponding normalized weight vector W and its maximum eigenvalue , and the specific formula is as follows:

[0095]

[0096]

[0097] In the formula, is the geometric mean of each row element, is the i-th weight value of the weight vector W.

[0098] Step 3.1.2: Use the obtained weights to perform a fusion calculation on the anger emotion probability value E, the traffic congestion index R, and the comprehensive interaction feature index M to obtain the situational road rage risk P1, and the specific formula is as follows:

[0099]

[0100] Step 3.2: As Figure 5As shown, the long short-term memory network (LSTM) model is used to calculate the trait road rage risk P2.

[0101] Furthermore, this step includes the following steps:

[0102] Step 3.2.1: Construct an LSTM module. According to the anger emotion probability value E, the acceleration change index J, and the steering aggressiveness coefficient a steer Construct the feature vector X t = [E, J, a steer . The specific formula is as follows:

[0103]

[0104]

[0105]

[0106]

[0107]

[0108]

[0109] In the formula, W f , W i , W C , W o - - Weights of the forget gate, input gate, memory cell, and output gate;

[0110] b f , b i , b c , b o - - Bias terms of the forget gate, input gate, memory cell, and output gate;

[0111] f t - - Forget gate;

[0112] i t - - Input gate;

[0113] C t - - Memory cell update;

[0114] - - Candidate memory cell;

[0115] o t - - Output gate;

[0116] h t - - Hidden state;

[0117] Step 3.2.2: Based on the final hidden state h of the LSTM t, the specific road rage risk P2 is obtained through the fully connected layer weight matrix and the Sigmoid function. The specific formula is as follows:

[0118]

[0119] In the formula, W p —— The weight of the fully connected layer;

[0120] b p —— The bias term of the fully connected layer;

[0121] In step 4, for different differentiated prevention and control measures for road rage and dynamic adjustment, in some embodiments, it includes the following steps:

[0122] Step 4.1: According to the types of situational road rage or trait road rage determined by the system, implement targeted prevention and control strategies respectively.

[0123] Further, this step is divided into two steps:

[0124] Step 4.1.1: When the situational road rage risk P1 > P 1max , the driving behavior belongs to situational road rage, where P 1max is the critical value determined through a statistical test with a significance level of a = 0.05. When it is detected as situational road rage, take prevention and control measures for situational road rage.

[0125] Even further, as Figure 6 , the specific steps are as follows:

[0126] Step 4.1.1.1: Conduct emotional regulation, remind the driver of the safety situation, play soothing music and provide breathing regulation guidance.

[0127] Step 4.1.1.2: Voice prompt to turn on driving assistance, activate Adaptive Cruise Control (ACC), Automatic Emergency Braking (AEB), and turn on Lane Keeping Assist (LKA) to ensure safe driving.

[0128] Step 4.1.1.3: Trigger the speed limit strategy and gradually adjust the vehicle speed within the average speed of the traffic flow.

[0129] Step 4.1.2: When the trait road rage risk P2 > P 2max , the driving behavior belongs to trait road rage, where P 2max is the critical value determined through a statistical test with a significance level of a = 0.05. When it is detected as trait road rage, take prevention and control measures for trait road rage. Further, as Figure 7 , the specific steps are as follows:

[0130] Step 4.1.2.1: Trigger the speed limit strategy to limit the vehicle speed to the maximum speed that can maintain a safe distance, reducing the possibility of dangerous driving.

[0131] Step 4.1.2.2: Upload the vehicle status and driver emotion data to the cloud, formulate a personalized intervention plan based on the driver's historical data and current status, and enable driving assistance.

[0132] Step 4.1.2.3: The vehicle sends a warning signal to other vehicles: the horn is automatically sounded.

[0133] Step 4.2: Dynamically adjust the intensity and combination of prevention and control measures according to the change of the driver's road rage risk. Further, this step includes the following two steps:

[0134] Step 4.2.1: Calculate a sliding average of road rage risk in real time :

[0135]

[0136] In the formula, - a sliding average of situational road rage risk or idiosyncratic road rage risk;

[0137] ——Road rage risk in the current time window t;

[0138] ——Road rage risk in the previous time window t-1;

[0139] ——Road rage risk in the first two time windows t-2.

[0140] Step 4.2.2: Detect whether the driver's road rage risk has decreased and implement corresponding countermeasures. The details are as follows:

[0141] (1) If The risk of road rage among drivers continued to decrease when:

[0142] Gradually restore vehicle control authority, remove speed limits and reduce assistance intensity. Play soothing music and give voice prompts to help drivers relax.

[0143] (2) If , the risk of road rage among drivers was not reduced:

[0144] Maintain current prevention and control measures and trigger upgraded strategies. Suggest the driver to pull over and rest at a safe place ahead through voice prompts. Start the intelligent guided parking function and guide the vehicle into the nearest parking area by adjusting the vehicle's direction and speed.

[0145] Based on the same inventive concept, the present application also provides a real-time detection system for situational and trait road rage, including:

[0146] A data acquisition module, configured to collect real-time data of the driver's facial expressions, driving behaviors, traffic conditions, and interactions with other vehicles by using vehicle sensors and in-vehicle devices;

[0147] A data preprocessing module, configured to preprocess the collected initial data, including data cleaning, outlier processing, and image denoising, and eliminate samples with substandard data quality to obtain an initial data set available for analysis;

[0148] A feature extraction module, configured to extract multi-dimensional features from the preprocessed data, including driver emotion features, driving behavior features, traffic condition features, and interactions with other vehicles features;

[0149] A risk calculation module, configured to adopt a multi-modal data fusion technology, combine driver emotion features, driving behavior features, traffic condition features, and interactions with other vehicles features, and calculate situational road rage risk and trait road rage risk respectively;

[0150] A risk determination module, configured to determine whether the driver has road rage and its type according to the calculated situational road rage risk and trait road rage risk.

[0151] Through multi-modal data fusion, the real-time detection and prevention and control system for road rage can more accurately determine the driver's emotional state and driving behavior, and achieve dynamic prevention and control in combination with external inducing factors. This system can not only effectively reduce the risk of traffic accidents caused by road rage, but also improve road safety and driving experience through intelligent intervention measures.

[0152] The above embodiments only represent several implementation manners of the present invention, and the description thereof is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the inventive concept of the present invention, several modifications and improvements can still be made, and the present invention also intends to include these modifications and improvements.

Claims

1. A real-time detection method for situational and trait road rage, characterized in that, The following steps are involved: Data collection: Using vehicle sensors and onboard equipment to collect real-time data on the driver’s facial expressions, driving behavior, traffic conditions, and interactions with other vehicles; Data preprocessing: Preprocess the collected initial data, including data cleaning, outlier processing and image denoising, and remove samples with substandard data quality to obtain an initial data set that can be used for analysis; Feature extraction: Extract multidimensional features from preprocessed data, including: Driver emotional characteristics: The probability value of the driver's anger is calculated through the image processing model; Driving behavior characteristics: Quantitatively evaluate the driver's driving behavior by calculating the acceleration change index and steering aggressiveness coefficient; Traffic condition characteristics: Calculate the current road congestion index by standardizing traffic flow and vehicle speed data; Other vehicle interaction characteristics: Real-time monitoring and analysis of other vehicle behaviors, calculation of dynamic collision risk index, interaction urgency and heading conflict coefficient, and calculation of comprehensive interaction characteristic index by weight; Risk calculation: Multimodal data fusion technology is used to combine the driver's emotional characteristics, driving behavior characteristics, traffic conditions characteristics and other car interaction characteristics to calculate the situational road rage risk and idiosyncratic road rage risk respectively; Risk determination: Based on the calculated situational road rage risk and idiosyncratic road rage risk, determine whether the driver has road rage and its type.

2. The real-time detection method for situational and trait road rage according to claim 1, characterized in that The vehicle sensors and vehicle-mounted equipment include: In-car camera to capture images of the driver’s facial expressions; Acceleration sensor and steering wheel sensor, used to obtain the driver's driving behavior data; Microwave radar and traffic cameras to obtain data on traffic conditions; Surrounding vehicle monitoring radars and cameras are used to obtain interaction data with other vehicles.

3. A real-time detection method for situational and trait road rage according to claim 1 or 2, characterized in that, Using the Analytic Hierarchy Process to Calculate the Risk of Situational Road Rage P 1. Specifically, it includes: A hierarchical model was constructed to calculate the weight vectors of anger probability value, traffic congestion index and comprehensive interaction characteristic index and conduct consistency test; Fuse and calculate the probability value of anger emotion, traffic congestion index, and comprehensive interaction feature index using the obtained weights to obtain the risk of situational road rage P 1.

4. A real-time detection method for situational and trait road rage according to claim 1 or 2, characterized in that, Calculating the risk of trait road rage using a long short-term memory network model P 2. Specifically, it includes: Construct an LSTM module and construct a feature vector based on the anger probability value, acceleration change index, and steering aggressiveness coefficient; Based on the final hidden state of the LSTM module, obtain the trait road rage risk through the weight matrix of the fully connected layer and the Sigmoid function P 2.

5. The real-time detection method for situational and trait road rage according to claim 1, wherein, In the risk determination step, set the risk threshold for situational road rage P 1max and the risk threshold for trait road rage P 2max , when P 1 > P 1max , it is determined as situational road rage. When P 2 > P 2max , it is determined as trait road rage, where P 1 is the calculated risk of situational road rage, P 2 is the calculated risk of trait road rage.

6. The real-time detection method for situational and trait road rage according to claim 5, characterized in that, When situational road rage is identified, implement the following prevention and control measures: Provide emotional regulation, remind drivers of safety situations, play soothing music and provide breathing regulation guidance; Voice prompts to turn on driving assistance, activate active cruise control, emergency brake assist and lane keeping assist; The speed limit strategy is triggered to gradually adjust the vehicle speed to the average speed of the traffic flow.

7. A real-time detection method for situational and trait road rage according to claim 5, characterized in that, When it is determined to be idiosyncratic road rage, the following prevention and control measures are implemented: Trigger the speed limit strategy to limit the vehicle speed to the maximum speed that can maintain a safe distance; Upload vehicle status and driver emotion data to the cloud, formulate personalized intervention plans based on the driver's historical data and current status, and enable driving assistance; The vehicle sends a warning signal to other vehicles.

8. A real-time detection method for situational and dispositional road rage according to claim 6 or 7, characterized in that It also includes steps to dynamically adjust prevention and control measures: Calculate a sliding average of road rage risk in real time; When the sliding average value continues to decrease, the vehicle control authority will be gradually restored, the speed limit will be lifted, the assistance intensity will be reduced, soothing music will be played, and voice prompts will be given to help the driver relax; When the sliding average does not decrease, maintain the current prevention and control measures and trigger the escalation strategy. Through voice prompts, the driver is advised to pull over and rest at a safe location ahead, and the intelligent parking guidance function is activated to guide the vehicle into the nearest parking area.

9. A real-time detection system for situational and trait road rage, characterized in that, Including: A data acquisition module for collecting real-time data on the driver's facial expressions, driving behaviors, traffic conditions, and interactions with other vehicles using vehicle sensors and in-vehicle devices; A data preprocessing module for preprocessing the collected initial data, including data cleaning, outlier handling, and image denoising, and removing samples with substandard data quality to obtain an initial dataset available for analysis; A feature extraction module for extracting multi-dimensional features from the preprocessed data, including driver emotion features, driving behavior features, traffic condition features, and interactions with other vehicles features; A risk calculation module for calculating the situational road rage risk and the trait road rage risk respectively by using multi-modal data fusion technology and combining driver emotion features, driving behavior features, traffic condition features, and interactions with other vehicles features; A risk determination module for determining whether the driver has road rage and its type based on the calculated situational road rage risk and trait road rage risk.

Images