Abnormal driving behavior judgment method and system based on multiple modes
Through the multimodal data fusion judgment method, combined with the network of adaptive gradient characteristics and self-attention mechanism, the abnormal driving behavior is identified and predicted, which solves the problem of failure to consider the road environment and passenger behavior in the existing technology, and improves the abnormality Accuracy of identification of driving behavior and driving safety.
Patent Information
- Application Number
- CN202410997093.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-12
Smart Images

Figure CN118928425A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of automobile driving behavior safety, and relates to a method and system for judging abnormal driving behavior based on multi-modality. Background Art
[0002] According to statistics, the frequency of traffic accidents is gradually increasing every year, causing great losses to people's lives and property. Inappropriate driving behavior has become the main cause of most traffic accidents. In China, more than 80% of road traffic accidents are related to dangerous driving behavior. Research on identifying abnormal driving behavior has a positive impact on traffic safety.
[0003] The factors that affect road safety are relatively complex. Existing research has classified the factors that affect road safety into three main aspects: people, vehicles, and environment. These include the driver's own data (such as movements, facial expressions, eye movements, etc.), vehicle data (speed, acceleration, GPS), and environmental data (road conditions, weather conditions, external man-made environmental influences), etc. Most of the current research is based on people and vehicles. For example, based on the detection of the driver's own state, the driver's abnormal driving behaviors such as distracted driving, fatigue driving, drunk driving, and fainting are detected. Or based on the detection of the vehicle's motion state, that is, the vehicle's motion data, it is detected whether the vehicle is in abnormal driving behaviors such as sudden acceleration, sudden deceleration, sharp turns, sudden lane changes, speeding, etc.
[0004] However, few studies have considered the impact of road environment, passenger behavior, traffic conditions, etc. on the identification of abnormal driving behavior. More than 93% of traffic accidents in the road traffic system are related to human factors, and the driver's decision-making and behavior in the traffic environment are closely related to traffic safety. This results in a single perception that cannot provide a complete basis for accurate judgment of abnormal driving behavior, which is not conducive to the identification of abnormal driving behavior. Summary of the invention
[0005] In view of this, the purpose of the present invention is to provide a method and system for judging abnormal driving behavior based on multimodality, which can simultaneously monitor the driver's state, vehicle state, and environmental state, and integrate the impact of various factors on driving safety, which is conducive to detecting more abnormal driving behaviors, improving the accuracy of abnormal driving behavior identification in a real road environment, and giving the driver necessary feedback in a timely manner, urging the driver to correct his own driving behavior or avoid situations that affect driving safety in a timely manner. At the same time, it can also provide support for the smart car driving rights transfer strategy, reduce the occurrence of traffic accidents, and improve road safety.
[0006] In order to achieve the above object, the present invention provides the following technical solutions:
[0007] A method for determining abnormal driving behavior based on multimodality comprises the following steps:
[0008] S1. Obtain the status information of the driver and passengers, and build a two-stream spatiotemporal convolutional network that integrates adaptive gradient features to monitor the status of the driver and passengers;
[0009] S2, obtain road environment information, build a multi-detection head cross-stage local network with self-attention mechanism to monitor the road environment;
[0010] S3, obtaining the vehicle motion state and monitoring dangerous driving operation behaviors through a sliding window threshold classification algorithm;
[0011] S4. Based on various monitoring data, predict whether there are any driving behaviors that seriously affect road safety, and classify the behaviors into normal driving, suspected dangerous driving, dangerous driving, and serious dangerous driving behaviors. If there is serious dangerous driving behavior, the driving right will be directly transferred; if there is suspected dangerous driving or dangerous driving behavior, this behavior will be further identified;
[0012] S5. Construct a multi-index threshold division abnormal driving recognition algorithm based on logistic regression to accurately identify abnormal driving behaviors that are suspected of dangerous driving or dangerous driving behaviors by fusing multimodal data.
[0013] Furthermore, in step S1, the yolov8-pose multi-person posture estimation framework is applied, and a two-stream spatiotemporal convolutional network integrating adaptive gradient features is constructed to accurately detect the driver's state and identify whether the passenger has activities that may affect driving safety; it includes the following processes:
[0014] The driver and passenger video sequence is first subjected to the complementary fusion of the adaptive fusion gradient histogram feature module HOG and the color feature module CN for feature extraction, which improves the robustness to illumination changes and enhances the tracking stability when the target object moves rapidly.
[0015] The yolov8-pose single-stage facial skeleton key point recognition technology is used to collect key point information to improve the detection rate;
[0016] The collected key point information is input into the dual-stream spatiotemporal convolution module, and continuous spatial streams and temporal streams are constructed according to the key points. The facial and bone spatial features and temporal features are extracted, and the spatial features and temporal features are fused to perform fatigue recognition based on the driver's face and abnormal behavior recognition based on the skeletons of the driver and passengers.
[0017] Furthermore, in step S2, a multi-detection head cross-stage local network with a self-attention mechanism is constructed to perform multi-task synchronous processing;
[0018] A self-attention mechanism multi-detection head cross-stage local network, which includes a self-attention mechanism module, a cross-stage local backbone network, an auxiliary network, and multiple detection heads;
[0019] The self-attention mechanism module processes the road environment video sequence to screen and filter information and increase attention to scene perception;
[0020] Feature extraction using a cross-stage local backbone network;
[0021] Auxiliary networks that can adapt to different tasks are used to further fuse features;
[0022] The detection head of the network includes multiple modules for performing different detection tasks, wherein the detection head includes: a lane line segmentation detection head for realizing lane line detection, a drivable area segmentation detection head for detecting the drivable area, and a target detection head for vehicle detection, pothole detection and road obstacle detection; the detection head also includes a road wetness and dryness classification detection head for distinguishing different road conditions.
[0023] Further, in step S3, the process is as follows:
[0024] The acquired vehicle motion state data includes at least: speed, acceleration, lateral acceleration, and steering angle;
[0025] The sliding window technology is used to divide the data into a series of time windows. Each window is in units of time length t. In each sliding window, the data index values reflecting abnormal driving behavior are calculated, including: average speed, longitudinal acceleration difference, lateral acceleration difference, and steering angle difference;
[0026] Set the initial value of the indicator threshold, and by comparing the indicator value in each sliding window with the preset threshold, judge in real time whether the vehicle is in an abnormal driving state based only on the vehicle status data. Abnormal driving states include sudden acceleration, sudden deceleration, sharp turn, sudden lane change or speeding;
[0027] M v =mean(v0, v1, ..., v n )
[0028]
[0029] Where: v0, v1..., v n is the speed value at each moment in the sliding window; is the total number of monitoring data in the sliding window; mean is the speed mean in the sliding window; a0, a n are the lateral acceleration and longitudinal acceleration values at the first and last moments in the sliding window, respectively; θ0, θ n They are the steering angle values at the first and last moments in the sliding window respectively.
[0030] Further, in step S4, a preliminary evaluation is performed on each monitoring data to preliminarily determine whether there is any driving behavior that affects road safety in each monitoring data;
[0031] Normal driving is when there is no abnormality in the driver's state, passenger's state, road environment, and vehicle's state;
[0032] When there is serious abnormal driving behavior, the driving right will be directly transferred to the advanced intelligent assisted driving system or the safe operation method will be enforced; serious abnormalities include: the driver fainting, the passenger directly grabbing the steering wheel, and the road environment is extremely dangerous;
[0033] Dangerous driving or suspected dangerous driving includes at least: driver distraction, fatigue driving, suspected sudden acceleration and deceleration, suspected sudden lane changes, potholes on the road ahead, and slippery road conditions.
[0034] Furthermore, in step S5, the dangerous driving or suspected dangerous driving is further accurately identified, which includes the following process:
[0035] Calculate the road environment factor e and the driver state factor d based on the driver state data, the road environment data, and the vehicle motion data;
[0036] According to the road environment factor e and the driver state factor d, the abnormal driving behavior index threshold M is dynamically updated to determine whether there is abnormal driving behavior markABD for the first time; the first abnormal driving behavior mark is introduced as:
[0037]
[0038] Among them, when judging abnormal driving behavior for the first time, the real-time driving index value A<(1-e)(1-d)M0, M0 is the initial setting threshold, and markABD is set to 0; A>(1-e)(1-d)M0, markABD is set to 1;
[0039] For the road environment factor e: the abnormal road state, traffic objects, and traffic event information detected by the road environment monitoring module; for the driver state factor d: the driver fatigue and distraction detected by the driver state monitoring module; use fuzzy logic rules to determine the coefficient α (α < 1) of each type of hazard level; finally, the road environment factor e and the driver state factor d are calculated based on the logistic regression algorithm:
[0040]
[0041] z=∑α i x i , i=1,2,…,n
[0042] e and d are values between 0 and 1. The closer to 1, the more dangerous the current driving environment or the more dangerous the driver's state; z is the linear regression model; x i It is an indicator of road environment factors or driver factors;
[0043] The special situation mark markS is introduced to make a second judgment on whether there is abnormal driving behavior f(markABD, markS), where:
[0044]
[0045] f(markADB, markS)=markADB∧markS
[0046] That is, when there is no abnormal situation, if abnormal driving behavior occurs, it is determined to be abnormal driving; when there is an abnormal situation, the road environment state, vehicle movement state, and driver state are further analyzed to determine whether the driving behavior is abnormal;
[0047] Therefore, the following are given: abnormal driving behavior judgment model including speeding, abnormal driving behavior judgment model for sudden lane change, abnormal driving behavior judgment model for not maintaining a safe distance, abnormal driving behavior judgment model for sudden acceleration, sudden deceleration, and sharp turn, and abnormal line of sight deviation.
[0048] Further, in step S5, the process of judging the speeding abnormal driving behavior is as follows:
[0049] S511: Obtain the lane speed limit V of the vehicle's location under normal road conditions through the vehicle's high-precision map system or other high-precision equipment software max ;
[0050] S512: Simultaneously monitor the road environment, vehicle motion state, and driver state, obtain the vehicle real-time speed V, determine whether the road environment and driver and passenger state are normal, and calculate the road environment factor e and the driver state factor d;
[0051] S513: If the vehicle is on a normal road and the driver and passengers are in normal condition, when the vehicle's real-time speed V is less than the normal road lane speed limit V max , that is, V<V max When the vehicle's real-time speed V is greater than the normal road lane speed limit V max , that is, V>V max When the vehicle is speeding;
[0052] S514: If the vehicle is on an abnormal road or the driver and passenger are in an abnormal state, dynamically update the lane speed limit threshold (1-e) (1-d) V max , when the vehicle's real-time speed is less than the updated lane speed limit threshold, that is, V<(1-e)(1-d)V maxWhen the vehicle's real-time speed is greater than the updated lane speed limit threshold, that is, V>(1-e)(1-d)V max The vehicle is speeding.
[0053] Further, in step S5, the process of determining the abnormal behavior of sudden lane change is as follows:
[0054] S521: Use the acceleration threshold method to realize the recognition of sudden lane change behavior, and pre-set the threshold lateral acceleration a ymax ; Data provided by the vehicle manufacturer or given that the vehicle is basically parallel to the lane and the vehicle width is W c ; Obtain the current lane width W through the vehicle-mounted high-precision map system or other high-precision equipment software r ;
[0055] S522: Simultaneously monitor the road environment, vehicle motion state, and driver state; determine whether the road environment and driver and passenger state are normal, calculate the road environment factor e and the driver state factor d; detect lane lines and obtain the lateral offset d0 of the current vehicle relative to the lane center and the lateral acceleration a of the vehicle y ;
[0056] S523: The distance ΔW between the front wheels of the vehicle and the left and right lane markings in the current state is calculated according to the vehicle lane change identification algorithm ΔW=F(W r , W c , d0), determine whether the vehicle changes lanes;
[0057]
[0058] Where W r , W c Respectively represent the distances between the front wheels on both sides of the vehicle and the left and right lane markings; when ΔW<0, it means the vehicle is in the current lane; ΔW>0, the vehicle deviates from the lane;
[0059] S524: If the vehicle does not change lanes, perform lane crossing detection and lane departure detection to determine whether the vehicle crosses the lane and whether the vehicle deviates from the lane; if the continuous time t1 of the vehicle crossing the lane is greater than the set threshold time t 1max , vehicle crossing lane warning; if the vehicle deviates from the lane for continuous time t2> set threshold time t 2max , vehicle lane departure warning;
[0060] S525: If the vehicle changes lanes, the vehicle is on a normal road and the driver and passengers are in normal condition, when the vehicle lateral acceleration is less than a preset threshold lateral acceleration, i.e., a y <a ymax When the vehicle's lateral acceleration is greater than the preset threshold lateral acceleration, that is, a y>a ymax When , the vehicle's suspected sudden lane change behavior markABD=1;
[0061] S526: If the vehicle changes lanes, the vehicle is on an abnormal road or the driver and passengers are in an abnormal state, dynamically update the threshold lateral acceleration (1-e)(1-d)a ymax , when the vehicle lateral acceleration is less than the dynamically updated threshold lateral acceleration, that is, a<(1-e)(1-d)a ymax When the vehicle's normal lane change behavior markABD=0; when the vehicle's lateral acceleration is greater than the dynamically updated threshold lateral acceleration, that is, a>(1-e)(1-d)a max When , the vehicle's suspected sudden lane change behavior markABD=1;
[0062] S527: If the vehicle is suspected of making an abrupt lane change, determine whether there are special circumstances. If it is a normal road and there is no special situation, markS=1, then the vehicle is determined to be making an abrupt lane change; if it is an abnormal road, markS=0, then the vehicle is determined to be making a normal lane change.
[0063] Further, in step S5, the abnormal driving behavior judgment process of not maintaining a safe vehicle distance is as follows:
[0064] S531: Preset the safe distance L of the driver in normal state max ;
[0065] S532: Simultaneously monitor the road environment and the driver's status, determine whether the road environment and the driver's status are normal, calculate the road environment factor e and the driver's status factor d, and obtain the speed and distance L of the vehicle ahead through road environment monitoring. The vehicle distance calculation formula is:
[0066] L=(F*W) / P
[0067] Where F is the focal length of the camera, W is the width or height of the target, and P refers to the width of the target in pixels in the x direction or the height of the target in the y direction in the image;
[0068] S533: The vehicle is on a normal road and the driver and passengers are in normal condition; when the real-time distance between the vehicle and the vehicle ahead is less than the safe distance, i.e., L<L max When the real-time distance between the vehicle and the vehicle ahead is greater than the safe distance, that is, L>L max When the vehicle is too close to the vehicle;
[0069] S534: The vehicle is on an abnormal road or the driver and passengers are in abnormal condition, and the vehicle safety distance is dynamically updated (1-e)(1-d)L max , when the real-time distance between the vehicle and the vehicle in front is less than the dynamically updated safe distance, that is, L<(1-e)(1-d)Lmax When the real-time distance between the vehicle and the vehicle ahead is greater than the dynamically updated safe distance, that is, L>(1-e)(1-d)L max When the vehicle is too close to the vehicle;
[0070] The judgment process of abnormal driving behaviors such as sudden acceleration, sudden deceleration and sharp turn is as follows:
[0071] S541: Set the thresholds for rapid acceleration, rapid deceleration, rapid turning, and the initial threshold for longitudinal acceleration a xmax , initial threshold value of steering angle θ max ;
[0072] S542: Monitor the road environment, vehicle motion status, and driver and passenger status, determine whether the road environment and driver and passenger status are normal, calculate the road environment factor e and the driver status factor d, and obtain the vehicle's real-time longitudinal acceleration a x , steering angle θ;
[0073] S543: The vehicle is on a normal road and the driver and passengers are in normal condition. When the real-time longitudinal acceleration is less than the initial threshold of longitudinal acceleration and the real-time steering angle is less than the initial threshold of steering angle, that is, a x <a xmax And θ<θ max When the vehicle is running normally, markABD=0; when the real-time longitudinal acceleration is greater than the initial threshold of longitudinal acceleration, that is, a x >a xmax , then the vehicle is suspected of sudden acceleration or deceleration markABD=1; when the real-time steering angle is greater than the initial steering angle threshold, that is, θ>θ max When , the vehicle is suspected to be turning sharply, markABD=1;
[0074] S544: The vehicle is on an abnormal road or the driver and passenger are in an abnormal state. When the real-time longitudinal acceleration is less than the longitudinal acceleration update threshold, and the real-time steering angle is less than the steering angle update threshold, that is, a x <(1-e)(1-d)a xmax And θ<(1-e)(1-d)θ max When the vehicle is running normally, markABD=0; when the real-time longitudinal acceleration is greater than the longitudinal acceleration update threshold, that is, a x >(1-e)(1-d)a xmax , the vehicle is suspected of sudden acceleration or deceleration markABD = 1; when the real-time steering angle is greater than the steering angle update threshold, that is, θ>(1-e)(1-d)θ max When , the vehicle is suspected to be turning sharply, markABD=1;
[0075] S545: For abnormal behaviors such as sudden deceleration and sharp turn, determine whether there are special circumstances. If it is a normal road, there is no special circumstance, markS=1, and the vehicle is determined to be in sudden deceleration and sharp turn behavior; if it is an abnormal road, markS=0, and the vehicle is determined to be in normal lane change behavior;
[0076] S546: For suspected sudden abnormal behavior, road environment monitoring, monitoring the sudden acceleration of the rear vehicle and other abnormal roads, markS = 0, it is normal driving; normal road markS = 1, it is a sudden acceleration abnormal behavior;
[0077] The process of judging abnormal sight deviation:
[0078] S551: Preset the driver's sight deviation time threshold as T max ;
[0079] S552: Monitor the road environment, vehicle motion state, and driver state. Obtain whether the current vehicle is going straight or turning through road environment monitoring-lane line detection. a=1 indicates that the vehicle is on a straight road; a=0 indicates that the vehicle is on a turning road. Obtain data on whether the vehicle steering wheel is turning through vehicle motion state monitoring. b=1 indicates that the vehicle is going straight; b=0 indicates that the vehicle is about to turn or is turning. Obtain driver's line of sight deviation data and line of sight deviation time T through driver state monitoring.
[0080] S553: If the driver's sight deviates from T>T max If the vehicle is on a turning road and the steering wheel is in a turning state, it is normal driving; if the vehicle is on a straight road but the steering wheel is in a turning state, it is abnormal driving; if the vehicle is on a straight road but the steering wheel is in a straight state, it is distracted driving.
[0081] The present invention also proposes a system suitable for the aforementioned multi-modal abnormal driving behavior judgment method, which includes: a driver and passenger state monitoring module, a vehicle motion state monitoring module, a road environment monitoring module, an abnormal driving danger level pre-judgment module and a multi-modal data fusion judgment module, wherein the driver and passenger state monitoring module is used to obtain the state information of the driver and passenger, and construct a dual-stream spatiotemporal convolutional network that integrates adaptive gradient features to monitor the state of the driver and passenger; the vehicle motion state monitoring module is used to obtain the vehicle motion state, and monitor dangerous driving operation behaviors through a sliding window threshold classification algorithm; the road environment monitoring module is used to obtain road environment information, construct an introduction The multi-detection head of the self-attention mechanism monitors the road environment across the local network of the stage; the abnormal driving danger level pre-evaluation module predicts whether there is a driving behavior that seriously affects road safety based on various monitoring data, and divides the behavior into normal driving, suspected dangerous driving, dangerous driving, and serious dangerous driving behavior. If there is a serious dangerous driving behavior, the driving right is directly transferred; if there is suspected dangerous driving or dangerous driving behavior, this behavior is further identified; the multimodal data fusion judgment module accurately identifies abnormal driving behaviors that involve suspected dangerous driving or dangerous driving behaviors by fusing multimodal data through the construction of a multi-indicator threshold division abnormal driving recognition algorithm based on logistic regression.
[0082] The beneficial effects of the present invention are:
[0083] The present invention designs a system to simultaneously monitor the driver status, vehicle status, and environmental status, and designs a multimodal data judgment model, proposes a multi-index dynamic threshold judgment and recognition algorithm based on logistic regression, introduces road environment fusion to consider the impact of various factors on driving safety, and detects abnormal driving behavior from multiple scales.
[0084] The adaptive multi-feature fusion two-stream spatiotemporal convolutional network (2s-HCSTGCN) proposed in the present invention fuses multiple features to increase the robustness to illumination changes and the stability of fast motion tracking of target objects, thereby compensating for the problem of decreased detection capability caused by the inability to utilize temporal features.
[0085] The dangerous driving operation behavior sliding window threshold classification algorithm proposed in the present invention processes a large amount of real-time vehicle driving data with a 1-second sliding window, reducing the amount of calculation.
[0086] The multi-detection head cross-stage local network of the self-attention mechanism proposed in the present invention performs multi-task synchronous processing, effectively reducing computational complexity and improving detection accuracy.
[0087] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0088] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:
[0089] Figure 1 This is a block diagram of the abnormal driving behavior identification system of the present invention;
[0090] Figure 2 The present invention is a driver and passenger adaptive multi-feature fusion dual-stream spatiotemporal convolutional network;
[0091] Figure 3 The invention is a road environment monitoring self-attention mechanism multi-detection head cross-stage local network
[0092] Figure 4 This is the multimodal data fusion algorithm process of the present invention;
[0093] Figure 5 This is a flow chart of identifying overspeeding and abnormal driving behavior of the present invention;
[0094] Figure 6 This is a flow chart for identifying abnormal sudden lane change, vehicle crossing the line, and lane deviation driving behaviors of the present invention;
[0095] Figure 7 This is a flow chart of abnormal driving behavior of not maintaining a safe vehicle distance according to the present invention;
[0096] Figure 8 This is a flow chart of abnormal driving behaviors of rapid acceleration, rapid deceleration and rapid turning according to the present invention;
[0097] Fig. 9 This is a flow chart of abnormal driving behavior of sight deviation according to the present invention. DETAILED DESCRIPTION
[0098] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.
[0099] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.
[0100] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.
[0101] See also Figures 1 to 9 , which is a method and system for judging abnormal driving behavior based on multimodality.
[0102] Example 1
[0103] This embodiment first provides a multi-modal abnormal driving behavior judgment method, such as Figure 1 The abnormal driving behavior recognition system block diagram shown in FIG. 1 includes the following steps:
[0104] S1. Obtain the status information of the driver and passengers, and build a two-stream spatiotemporal convolutional network (2s-HCSTGCN) that integrates adaptive gradient features to monitor the status of the driver and passengers; based on vision, the driver and passenger status monitoring focuses on the impact of changes in vehicle interior lighting on image data and the tracking difficulties caused by the rapid movement of target objects. A two-stream spatiotemporal convolutional network (2s-HCSTGCN) that integrates adaptive gradient features is built to accurately detect the driver's status and identify whether passengers have activities that may affect driving safety.
[0105] S2. Obtain road environment information and construct a multi-head cross-stage local network (Attention Module-Multi Head-CSPNet, AM-MH-CSP) with a self-attention mechanism to monitor the road environment. The multi-head cross-stage local network can perform multi-task synchronous processing, effectively reducing computational complexity and improving detection accuracy.
[0106] S3. Obtain the vehicle's motion status and monitor dangerous driving operations through a sliding window threshold classification algorithm. In view of the huge amount of vehicle driving data and the difficulty in analyzing and processing it, a sliding window threshold classification algorithm is designed to monitor dangerous driving operations.
[0107] S4. Based on various monitoring data, predict whether there are any driving behaviors that seriously affect road safety, and classify the behaviors into normal driving, suspected dangerous driving, dangerous driving, and serious dangerous driving behaviors. If there is serious dangerous driving behavior, the driving right will be directly transferred; if there is suspected dangerous driving or dangerous driving behavior, this behavior will be further identified;
[0108] S5. Construct a multi-index threshold division abnormal driving recognition algorithm based on logistic regression to accurately identify abnormal driving behaviors that are suspected of dangerous driving or dangerous driving behaviors by fusing multimodal data.
[0109] In step S1, the present embodiment monitors the status of the driver and passengers, applies the yolov8-pose multi-person posture estimation framework, and constructs a two-stream spatiotemporal convolutional network (2s-HCSTGCN) integrating adaptive gradient features to accurately detect the driver's status and identify whether the passengers have activities that may affect driving safety; Figure 2 The driver-passenger adaptive multi-feature fusion two-stream spatiotemporal convolutional network shown includes the following processes:
[0110] The driver and passenger video sequence is first subjected to the complementary fusion of the adaptive fusion gradient histogram feature module HOG and the color feature module CN for feature extraction, which improves the robustness to illumination changes and enhances the tracking stability when the target object moves rapidly.
[0111] The fused features are used to extract the corresponding facial key points and bone key points through the facial bounding box and the bone bounding box respectively. The yolov8-pose single-stage facial and bone key point recognition technology is used to collect key point information to improve the detection rate.
[0112] The collected facial key point information and skeleton key point information are input into the dual-stream spatiotemporal convolution module, and continuous spatial stream and temporal stream are constructed according to the key points. The facial and skeleton spatial features and temporal features are extracted, and the spatial features and temporal features are fused to realize fatigue recognition based on the driver's face and abnormal behavior recognition based on the skeleton of the driver and passenger.
[0113] Specifically, the dual-stream spatiotemporal convolution module first constructs a facial spatial graph and a facial temporal graph through the facial key point information, and then obtains the facial spatial stream and facial temporal stream from the facial spatial graph and the facial temporal graph respectively through the graph convolutional network GCN, and fuses the facial spatial stream and the facial temporal stream. Finally, fatigue recognition and gaze deviation recognition are performed based on the fused facial information.
[0114] Similarly, the dual-stream spatiotemporal convolution module first constructs the skeleton spatial graph and the skeleton temporal graph through the skeleton key point information, and then obtains the skeleton spatial stream and the skeleton temporal stream from the skeleton spatial graph and the skeleton temporal graph respectively through the graph convolution network GCN, and fuses the skeleton spatial stream and the skeleton temporal stream. Finally, the driver distraction behavior and passenger dangerous behavior recognition are performed based on the fused skeleton information.
[0115] In this embodiment, the road environment is monitored in step S2. Figure 3 The road environment monitoring self-attention mechanism multi-detection head cross-stage local network shown in the figure constructs a multi-detection head cross-stage local network with the introduction of the self-attention mechanism, performs multi-task synchronous processing, effectively reduces the computational complexity and improves the detection accuracy. The self-attention mechanism multi-detection head cross-stage local network consists of a self-attention mechanism module, a cross-stage local backbone network, an auxiliary network, and multiple specially designed detection heads.
[0116] It includes the following processes:
[0117] Input the collected road environment video sequence into the self-attention mechanism module to process the road environment video sequence, filter out unimportant information, and increase attention to scene perception;
[0118] Then, features are extracted from the data through a shared cross-stage local backbone network. The shared cross-stage local backbone network follows the YOLOv8 backbone network, adopts an improved cross-stage local network (CSP_Darknet53), and replaces the C3 layer module with a C2f layer module with richer gradient flow. Its C2f module combines high-level features with contextual information to further improve network performance.
[0119] Then, the data with preliminary feature extraction is passed to different auxiliary networks Neck respectively to fuse the features extracted from the main network. The Detect Neck auxiliary network is responsible for fusing target detection task features, including vehicle detection, road pothole detection, and road obstacle detection; the Segment Neck auxiliary network is responsible for fusing segmentation detection task features; and the Classification Neck is responsible for fusing classification detection task features.
[0120] The detection head of the network is used to perform different detection tasks. Four detection heads are designed: Detect Head, LaneHead, Drivable Head, and Classification Head. The lane segmentation detection head is used to implement lane detection, the drivable area segmentation detection head is used to detect the drivable area, the target detection head is used for tasks such as vehicle detection, road pothole detection, and road obstacle detection, and the classification detection head is used to detect different road conditions. By integrating these different detection heads, the network can realize the simultaneous detection of multiple tasks, while effectively reducing the computational burden brought by using different network models.
[0121] In this embodiment, vehicle status monitoring is performed in step S3. The acquired vehicle motion status data includes multiple key indicators such as speed, acceleration, lateral acceleration, and steering angle; the sliding window technology is used to divide the data into a series of time windows, each window is in units of time length t, and in each sliding window, a series of data indicator values reflecting abnormal driving behavior are calculated, including: average speed, longitudinal acceleration difference, lateral acceleration difference, and steering angle difference; the initial value of the indicator threshold is set, and by comparing the indicator value in each sliding window with the preset threshold, the vehicle is judged in real time based on the vehicle status data to determine whether the vehicle is in an abnormal driving state such as sudden acceleration, sudden deceleration, sharp turn, sudden lane change or speeding.
[0122] M v ∑mean(v0,v1,...,v n )
[0123]
[0124] Where: v0, v1, ..., v n is the speed value at each moment in the sliding window; n is the total number of monitoring data moments in the sliding window; mean is the mean speed value in the sliding window; a0, a n are the lateral acceleration and longitudinal acceleration values at the first and last moments in the sliding window, respectively; θ0, θ n They are the steering angle values at the first and last moments in the sliding window respectively.
[0125] In step S4, this embodiment makes a preliminary assessment of each monitoring data to determine whether there is any driving behavior that greatly affects road safety in each monitoring data. Normal driving is considered normal only when there is no abnormality in the driver's state, passenger's state, road environment, and vehicle's state. If it is detected that the driver faints, the passenger directly grabs the steering wheel, or the road environment suddenly presents a very dangerous situation that cannot be avoided by humans, it is considered serious abnormal driving and the driving right is directly transferred. Other abnormal situations, including but not limited to: driver distraction, fatigue driving; or suspected abnormal situations, including but not limited to: suspected sudden acceleration and deceleration, suspected sudden lane change, potholes on the road ahead, slippery roads, etc., are considered dangerous driving or suspected dangerous driving.
[0126] In step S5, this embodiment further accurately identifies dangerous driving or suspected dangerous driving. Figure 4 The multimodal data fusion algorithm flow shown includes the following processes:
[0127] Calculate the road environment factor e and the driver state factor d based on the driver state data, the road environment data, and the vehicle motion data;
[0128] According to the road environment factor e and the driver state factor d, the abnormal driving behavior index threshold M is dynamically updated, and it is first determined whether there is an abnormal driving behavior markABD; the first abnormal driving behavior mark is introduced as:
[0129]
[0130] A special situation mark markS is introduced to make a second judgment on whether there is abnormal driving behavior f(markABD, markS), so as to realize the judgment and identification of abnormal driving behavior.
[0131] Introducing special situation flags
[0132]
[0133] Second judgment of abnormal driving
[0134] f(markADB, markS)=markADB∧markS
[0135] When abnormal driving behavior is judged for the first time, the real-time driving index value A<(1-e)(1-d)M0, M0 is the initial setting threshold, and markABD is set to 0; A>(1-e)(1-d)M0, markABD is set to 1. Considering special cases, such as the first judgment of sudden acceleration behavior, if a pothole is detected ahead, markS is set to 0, otherwise markS is set to 1.
[0136] Furthermore, for the road environment factor e: abnormal road conditions (potholes, obstacles, slippery), traffic objects (vehicle density, front and rear vehicle speed / distance, pedestrian density), and traffic event information detected by the road environment monitoring module; for the driver state factor d: driver fatigue and distraction (smoking, using mobile phones, drinking water) detected by the driver state monitoring module. Fuzzy logic rules are used to determine the coefficient α of each type of hazard level (α<1; finally, the road environment factor e and driver state factor d are calculated based on the design of the logistic regression algorithm.
[0137]
[0138] z=Σα i x i , i=1,2,…,n
[0139] e and d are values between 0 and 1. The closer to 1, the more dangerous the current driving environment or the more dangerous the driver's state; z is the linear regression model; x i It is an indicator of road environment factors or driver factors.
[0140] Normal vehicle movement behavior may lead to abnormal driving if it is in an abnormal road environment; abnormal vehicle movement behavior may also be normal driving under different road environments. Therefore, by integrating and analyzing the road environment status, vehicle movement status, and driver status, abnormal driving can be judged more accurately.
[0141] like Figure 5 The speeding abnormal driving behavior identification flow chart shown in the figure, the speeding abnormal driving behavior judgment model is specifically as follows:
[0142] S511: Obtain the lane speed limit V of the vehicle's location under normal road conditions through the vehicle's high-precision map system or other high-precision equipment software max .
[0143] S512: Simultaneously monitor the road environment, vehicle motion state, and driver state. Obtain the vehicle's real-time speed V, determine whether the road environment and driver and passenger state are normal, and calculate the road environment factor e and the driver state factor d.
[0144] S513: If the vehicle is on a normal road and the driver and passengers are in normal condition. When the vehicle's real-time speed V is less than the normal road lane speed limit V max , that is, V<V max When the vehicle's real-time speed V is greater than the normal road lane speed limit V max , that is, V>V max The vehicle is speeding.
[0145] S514: If the vehicle is on an abnormal road or the driver and passengers are in an abnormal state, dynamically update the lane speed limit threshold (1-e) (1-d) V max , when the vehicle's real-time speed is less than the updated lane speed limit threshold, that is, V<(1-e)(1-d)V max When the vehicle's real-time speed is greater than the updated lane speed limit threshold, that is, V>(1-e)(1-d)V max The vehicle is speeding.
[0146] like Figure 6 The sudden lane change abnormality, vehicle crossing the line, and lane deviation driving behavior recognition flow chart shown in the figure, the sudden lane change abnormal behavior judgment model is specifically:
[0147] S521: Use the acceleration threshold method to realize the recognition of sudden lane change behavior, and pre-set the threshold lateral acceleration a ymax The data is given by the vehicle manufacturer or the vehicle is basically parallel to the lane and the vehicle width is W. c The current lane width W is obtained through the vehicle-mounted high-precision map system or other high-precision equipment software. r .
[0148] S522: Simultaneously monitor the road environment, vehicle motion status, and driver status. Determine whether the road environment and driver status are normal, calculate the road environment factor e and the driver status factor d; detect the lane line and obtain the lateral offset a0 of the current vehicle relative to the lane center, the vehicle lateral acceleration a y .
[0149] S523: The distance ΔW between the front wheels of the vehicle and the left and right lane markings in the current state is calculated according to the vehicle lane change identification algorithm ΔW=F(W r , W c , d0), determine whether the vehicle changes lanes. If ΔW>0, the vehicle changes lanes.
[0150] S524: If the vehicle does not change lanes, perform lane crossing detection and lane departure detection to determine whether the vehicle crosses the lane and whether the vehicle deviates from the lane.
[0151] S525: If the vehicle changes lanes, the vehicle is on a normal road and the driver and passengers are in normal condition. When the vehicle lateral acceleration is less than the preset threshold lateral acceleration, i.e., a y >a ymax When the vehicle's lateral acceleration is greater than the preset threshold lateral acceleration, that is, a y >a ymax When , the vehicle's suspected sudden lane change behavior markABD=1.
[0152] S526: If the vehicle changes lanes, the vehicle is on an abnormal road or the driver and passengers are in an abnormal state. Dynamically update the threshold lateral acceleration (1-e)(1-d)a ymax , when the vehicle lateral acceleration is less than the dynamically updated threshold lateral acceleration, that is, a<(1-e)(1-d)a ymax When the vehicle's normal lane change behavior markABD=0; when the vehicle's lateral acceleration is greater than the dynamically updated threshold lateral acceleration, that is, a>(1-e)(1-d)a max When , the vehicle's suspected sudden lane change behavior markABD=1.
[0153] S527: If the vehicle is suspected of sudden lane change, determine whether there are special circumstances. If it is a normal road, there is no special circumstance, markS = 1, and the vehicle is determined to be a sudden lane change; if it is an abnormal road (there are potholes, obstacles, and the distance between the front and the vehicle is too close), markS = 0, and the vehicle is determined to be a normal lane change.
[0154] Further, vehicle lane crossing detection: In the road environment-lane line detection, the vehicle does not change lanes. If the vehicle crosses the line for continuous time t1> the set threshold time t 1max , vehicle crossing line warning.
[0155] Further, vehicle lane departure detection: In the road environment-lane line detection, the vehicle does not change lanes. If the vehicle deviates from the lane for a continuous time t2> the set threshold time t 2max , vehicle lane departure warning.
[0156] Specifically, the vehicle lane change discrimination algorithm ΔW=F(W r , W c , d0),
[0157]
[0158] The two equations represent the distances between the front wheels on both sides of the vehicle and the left and right lane markings. When ΔW<0, it means the vehicle is in the current lane; ΔW>0 means the vehicle has deviated from the lane.
[0159] like Figure 7 The abnormal driving behavior flow chart of not maintaining a safe distance is shown, and the abnormal driving behavior judgment model of not maintaining a safe distance is specifically as follows:
[0160] From the driver's perception to the vehicle stopping, it can be simply divided into three stages: perception-decision-making, braking, and braking effectiveness. The distance the vehicle moves from perception to the braking effectiveness is the safe distance. The driver's perception and reaction abilities are different when they are in a normal state and when they are tired and distracted; and the braking effectiveness results are different when the road environment is normal and abnormal. Therefore, dynamically setting the safe distance can better avoid the risk of vehicle collision.
[0161] S531: Preset the safe distance L of the driver in normal state max .
[0162] S532: Simultaneously monitor the road environment and driver status. Determine whether the road environment and driver status are normal, calculate the road environment factor e and driver status factor d, and obtain the speed and distance L of the vehicle ahead through road environment monitoring.
[0163] S533: The vehicle is on a normal road and the driver and passengers are in normal condition. When the real-time distance between the vehicle and the vehicle ahead is less than the safe distance, that is, L<L max When the real-time distance between the vehicle and the vehicle ahead is greater than the safe distance, that is, L>L max , the vehicle following too closely.
[0164] S534: The vehicle is on an abnormal road or the driver and passengers are in abnormal condition. Dynamically update the vehicle safety distance (1-e)(1-d)L max , when the real-time distance between the vehicle and the vehicle in front is less than the dynamically updated safe distance, that is, L<(1-e)(1-d)L max When the real-time distance between the vehicle and the vehicle ahead is greater than the dynamically updated safe distance, that is, L>(1-e)(1-d)L max , the vehicle following too closely.
[0165] Specifically, the distance detection of the vehicle ahead. The road environment video sequence is preprocessed and input into the improved self-attention mechanism multi-head cross-stage local network (Attention Module-Multi Head-CSPNet, AM-MH-CSP), and the vehicle is identified through the target detection task in the network, and the deepsort algorithm is introduced for vehicle tracking. The detection and tracking results are used to calculate the vehicle speed in real time. The actual width of the vehicle W is set c The ratio of the real distance to the pixel distance is calculated by the pixel width of the detected vehicle, and the pixel distance between the two frames is calculated by the center coordinates of the front and rear frames of each vehicle. The real distance of the vehicle between the two frames is mapped using this ratio and the pixel distance. In the speed measurement algorithm, the real distance of the vehicle and the pixel distance are considered to be in a linear relationship, which is only true when the axis of the surveillance camera is perpendicular to the direction of vehicle movement.
[0166] Vehicle distance calculation formula:
[0167] D=(F*W) / P
[0168] Where D is the distance from the target to the camera (i.e. the vehicle distance), F is the focal length of the camera, W is the width or height of the target, and P refers to the width of the pixels in the x direction or the height of the pixels in the y direction occupied by the target in the image. This formula can be used to calculate the distance to the vehicle in front.
[0169] like Figure 8 The abnormal driving behavior flow chart of sudden acceleration, sudden deceleration and sharp turn is shown, and the abnormal driving behavior judgment model of sudden acceleration, sudden deceleration and sharp turn is specifically as follows:
[0170] S541: Set the thresholds for rapid acceleration, rapid deceleration, and rapid turning. xmax , initial threshold value of steering angle θ max .
[0171] S542: Simultaneously monitor the road environment, vehicle motion status, and driver and passenger status. Determine whether the road environment and driver and passenger status are normal, calculate the road environment factor e and the driver status factor d, and obtain the vehicle's real-time longitudinal acceleration a x , steering angle θ.
[0172] S543: The vehicle is on a normal road and the driver and passengers are in normal condition. When the real-time longitudinal acceleration is less than the initial threshold of longitudinal acceleration, and the real-time steering angle is less than the initial threshold of steering angle, that is, a x <a xmax And θ<θ max When the vehicle is running normally, markABD=0; when the real-time longitudinal acceleration is greater than the initial threshold of longitudinal acceleration, that is, a x >a xmax , then the vehicle is suspected of sudden acceleration or deceleration markABD=1; when the real-time steering angle is greater than the initial steering angle threshold, that is, θ>θ max When , the vehicle is suspected to be making a sharp turn, markABD=1.
[0173] S544: The vehicle is on an abnormal road or the driver and passengers are in an abnormal state. When the real-time longitudinal acceleration is less than the longitudinal acceleration update threshold, and the real-time steering angle is less than the steering angle update threshold, that is, a x <(1-e)(1-d)a xmax And θ<(1-e)(1-d)θ max When the vehicle is running normally, markABD=0; when the real-time longitudinal acceleration is greater than the longitudinal acceleration update threshold, that is, a x >(1-e)(1-d)a xmax , the vehicle is suspected of sudden acceleration or deceleration markABD = 1; when the real-time steering angle is greater than the steering angle update threshold, that is, θ>(1-e)(1-d)θ max When , the vehicle is suspected to be making a sharp turn, markABD=1.
[0174] S545: For suspected abnormal behaviors such as sudden deceleration and sharp turns, determine whether there are special circumstances. If it is a normal road and there is no special circumstance, markS=1, then the vehicle is judged to have sudden deceleration or sharp turns; if it is an abnormal road (there are potholes, obstacles, the distance between the vehicles ahead is too close, and the vehicle ahead suddenly decelerates), markS=0, then the vehicle is judged to have normal lane change behavior.
[0175] S546: For suspected sudden abnormal behavior, road environment monitoring, monitoring abnormal roads such as sudden acceleration of the rear vehicle, if markS=0, it is normal driving; if markS=1 on normal roads, it is sudden acceleration abnormal behavior.
[0176] like Fig. 9 The flow chart of driving behavior with abnormal sight deviation and the model for judging abnormal sight deviation are shown as follows:
[0177] S551: Preset the driver's sight deviation time threshold as T max .
[0178] S552: Simultaneously monitor the road environment, vehicle motion state, and driver state. Through road environment monitoring-lane line detection, obtain whether the current vehicle is going straight or turning (a=1, indicating that the vehicle is on a straight road; a=0, indicating that the vehicle is on a turning road), obtain data on whether the vehicle steering wheel is turning through vehicle motion state monitoring (b=1, indicating that the vehicle is going straight; b=0, indicating that the vehicle is about to turn or is turning), and obtain driver's line of sight deviation data and line of sight deviation time T through driver state monitoring.
[0179] S553: If the driver's sight deviates from T>T max If the vehicle is on a turning road and the steering wheel is in a turning state, it is normal driving; if the vehicle is on a straight road but the steering wheel is in a turning state, it is abnormal driving; if the vehicle is on a straight road but the steering wheel is in a straight state, it is distracted driving.
[0180] Additionally, this embodiment proposes a system applicable to the aforementioned multimodal abnormal driving behavior judgment method, which includes: a driver and passenger status monitoring module, a vehicle motion status monitoring module, a road environment monitoring module, an abnormal driving risk level pre-evaluation module, and a multimodal data fusion judgment module, wherein:
[0181] The driver and passenger status monitoring module is used to obtain the status information of the driver and passenger, and build a two-stream spatiotemporal convolutional network that integrates adaptive gradient features to monitor the status of the driver and passenger;
[0182] The vehicle motion state monitoring module is used to obtain the vehicle motion state and monitor dangerous driving operation behaviors through a sliding window threshold classification algorithm;
[0183] The road environment monitoring module is used to obtain road environment information and build a multi-detection head cross-stage local network with a self-attention mechanism to monitor the road environment.
[0184] The abnormal driving risk level pre-evaluation module predicts whether there are any driving behaviors that seriously affect road safety based on various monitoring data, and classifies the behaviors into normal driving, suspected dangerous driving, dangerous driving, and serious dangerous driving behaviors. If there is serious dangerous driving behavior, the driving right will be directly transferred; if there is suspected dangerous driving or dangerous driving behavior, this behavior will be further identified;
[0185] The multimodal data fusion judgment module constructs a multi-index threshold division abnormal driving recognition algorithm based on logistic regression to accurately identify abnormal driving behaviors that are suspected of dangerous driving or dangerous driving behaviors by fusing multimodal data.
[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.
Claims
1. A multi-modal abnormal driving behavior judgment method, Features: It includes the following steps: S1. Obtain the status information of the driver and passengers, and build a two-stream spatiotemporal convolutional network that integrates adaptive gradient features to monitor the status of the driver and passengers; S2, obtain road environment information, build a multi-detection head cross-stage local network with self-attention mechanism to monitor the road environment; S3, obtaining the vehicle motion state and monitoring dangerous driving operation behaviors through a sliding window threshold classification algorithm; S4. Based on various monitoring data, predict whether there are any driving behaviors that seriously affect road safety, and classify the behaviors into normal driving, suspected dangerous driving, dangerous driving, and serious dangerous driving behaviors. If there is serious dangerous driving behavior, the driving right will be directly transferred; if there is suspected dangerous driving or dangerous driving behavior, this behavior will be further identified; S5. Construct a multi-index threshold division abnormal driving recognition algorithm based on logistic regression to accurately identify abnormal driving behaviors that are suspected of dangerous driving or dangerous driving behaviors by fusing multimodal data.
2. The method for determining abnormal driving behavior based on multimodality according to claim 1, Features: In step S1, the yolov8-pose multi-person pose estimation framework is applied, and a two-stream spatiotemporal convolutional network integrating adaptive gradient features is constructed to accurately detect the driver's state and identify whether the passengers have activities that may affect driving safety; That The process includes: The driver and passenger video sequence is first subjected to the complementary fusion of the adaptive fusion gradient histogram feature module HOG and the color feature module CN for feature extraction, which improves the robustness to illumination changes and enhances the tracking stability when the target object moves rapidly. The yolov8-pose single-stage facial skeleton key point recognition technology is used to collect key point information to improve the detection rate; The collected key point information is input into the dual-stream spatiotemporal convolution module, and continuous spatial streams and temporal streams are constructed according to the key points. The facial and bone spatial features and temporal features are extracted, and the spatial features and temporal features are fused to perform fatigue recognition based on the driver's face and abnormal behavior recognition based on the skeletons of the driver and passengers.
3. The method for determining abnormal driving behavior based on multimodality according to claim 1, Features: In step S2, a multi-detection head cross-stage local network with a self-attention mechanism is constructed to perform multi-task synchronous processing; A self-attention mechanism multi-detection head cross-stage local network, which includes a self-attention mechanism module, a cross-stage local backbone network, an auxiliary network, and multiple detection heads; The self-attention mechanism module processes the road environment video sequence to screen and filter information and increase attention to scene perception; Feature extraction using a cross-stage local backbone network; Auxiliary networks that can adapt to different tasks are used to further fuse features; The detection head of the network includes multiple modules for performing different detection tasks, wherein the detection head includes: a lane line segmentation detection head for realizing lane line detection, a drivable area segmentation detection head for detecting the drivable area, and a target detection head for vehicle detection, pothole detection and road obstacle detection; the detection head also includes a road wetness and dryness classification detection head for distinguishing different road conditions.
4. The abnormal driving behavior judgment method based on multimodality according to claim 1, Features: In step S3, the process is as follows: The acquired vehicle motion state data includes at least: speed, acceleration, lateral acceleration, and steering angle; The sliding window technology is used to divide the data into a series of time windows. Each window is in units of time length t. In each sliding window, the data index values reflecting abnormal driving behavior are calculated, including: average speed, longitudinal acceleration difference, lateral acceleration difference, and steering angle difference; Set the initial value of the indicator threshold, and by comparing the indicator value in each sliding window with the preset threshold, judge in real time whether the vehicle is in an abnormal driving state based only on the vehicle status data. Abnormal driving states include sudden acceleration, sudden deceleration, sharp turn, sudden lane change or speeding; M υ =mean(υ 0 ,u 1 ,...,u n ) Where: 0 ,υ 1 ...,υ n is the speed value at each moment in the sliding window; n is the total number of monitoring data moments in the sliding window; mean is the mean speed value in the sliding window; a 0 , a n are the lateral acceleration and longitudinal acceleration values at the first and last moments in the sliding window respectively; θ 0 ,θ n They are the steering angle values at the first and last moments in the sliding window respectively.
5. The method for determining abnormal driving behavior based on multimodality according to claim 1, Features: In step S4, a preliminary evaluation is performed on each monitoring data to preliminarily determine whether there is any driving behavior that affects road safety in each monitoring data; Normal driving is when there is no abnormality in the driver's state, passenger's state, road environment, and vehicle's state; When there is serious abnormal driving behavior, the driving right will be directly transferred to the advanced intelligent assisted driving system or the safe operation method will be enforced; among them, serious abnormalities include: the driver faints, the passenger grabs the steering wheel directly, the road environment There is an extremely dangerous situation; Dangerous driving or suspected dangerous driving includes at least: driver distraction, fatigue driving, suspected sudden acceleration and deceleration, suspected sudden lane changes, potholes on the road ahead, and slippery road conditions.
6. The method for determining abnormal driving behavior based on multimodality according to claim 1, Features: In step S5, the dangerous driving or suspected dangerous driving is further accurately identified. The process includes: Calculate the road environment factor e and the driver state factor d based on the driver state data, the road environment data, and the vehicle motion data; According to the road environment factor e and the driver state factor d, the abnormal driving behavior index threshold M is dynamically updated to determine whether there is abnormal driving behavior markABD for the first time; the first abnormal driving behavior mark is introduced as: Among them, the first time abnormal driving behavior is judged, the real-time driving index value A<(1-e)(1-d)M 0 , M 0 To set the initial threshold, markABD is set to 0; A>(1-e)(1-d)M 0 , markABD is set to 1; For the road environment factor e: the abnormal road state, traffic objects, and traffic event information detected by the road environment monitoring module; for the driver state factor d: the driver fatigue and distraction detected by the driver state monitoring module; use fuzzy logic rules to determine the coefficient α (α < 1) of each type of hazard level; finally, the road environment factor e and the driver state factor d are calculated based on the logistic regression algorithm: z=∑α i x i ,i=1,2,…,n e and d are values between 0 and 1. The closer to 1, the more dangerous the current driving environment or the more dangerous the driver's state; z is the linear regression model; x i It is an indicator of road environment factors or driver factors; The special situation mark markS is introduced to make a second judgment on whether there is abnormal driving behavior f(markABD, markS), where: f(markADB, markS)=makADB∧markS That is, when there is no abnormal situation, if abnormal driving behavior occurs, it is determined to be abnormal driving; when there is an abnormal situation, the road environment state, vehicle movement state, and driver state are further analyzed to determine whether the driving behavior is abnormal; Therefore, the following are given: abnormal driving behavior judgment model including speeding, abnormal driving behavior judgment model for sudden lane change, abnormal driving behavior judgment model for not maintaining a safe distance, abnormal driving behavior judgment model for sudden acceleration, sudden deceleration, and sharp turn, and abnormal line of sight deviation.
7. The method for determining abnormal driving behavior based on multimodality according to claim 6, Features: In step S5, the process of judging the speeding abnormal driving behavior is as follows: S511: Obtain the lane speed limit V of the vehicle's location under normal road conditions through the vehicle's high-precision map system or other high-precision equipment software max ; S512: Simultaneously monitor the road environment, vehicle motion state, and driver state, obtain the vehicle real-time speed V, determine whether the road environment and driver and passenger state are normal, and calculate the road environment factor e and the driver state factor d; S513: If the vehicle is on a normal road and the driver and passengers are in normal condition, when the vehicle's real-time speed V is less than the normal road lane speed limit V max , that is, V<V max When the vehicle's real-time speed V is greater than the normal road lane speed limit V max , that is, V>V max When the vehicle is speeding; S514: If the vehicle is on an abnormal road or the driver and passenger are in an abnormal state, dynamically update the lane speed limit threshold (1-e) (1-d) V max , when the vehicle's real-time speed is less than the updated lane speed limit threshold, that is, V<(1-e)(1-d)V max When the vehicle's real-time speed is greater than the updated lane speed limit threshold, that is, V>(1-e)(1-d)V max The vehicle is speeding.
8. The method for determining abnormal driving behavior based on multimodality according to claim 6, Features: In step S5, the process of determining abnormal behavior of sudden lane change is as follows: S521: Use the acceleration threshold method to realize the recognition of sudden lane change behavior, and pre-set the threshold lateral acceleration a y max ; Data provided by the vehicle manufacturer or given that the vehicle is basically parallel to the lane and the vehicle width is W c ; Obtain the current lane width W through the vehicle-mounted high-precision map system or other high-precision equipment software r ; S522: Simultaneously monitor the road environment, vehicle motion status, and driver status; determine whether the road environment and driver status are normal, calculate the road environment factor e and the driver status factor d; detect the lane line and obtain the lateral offset d of the current vehicle relative to the lane center 0 , vehicle lateral acceleration a y ; S523: The distance ΔW between the front wheels of the vehicle and the left and right lane markings in the current state is calculated according to the vehicle lane change identification algorithm ΔW=F(W r , W c , d 0 ), determine whether the vehicle changes lanes; Where W r , W c Respectively represent the distances between the front wheels on both sides of the vehicle and the left and right lane markings; when ΔW<0, it means the vehicle is in the current lane; ΔW>0, the vehicle deviates from the lane; S524: If the vehicle does not change lanes, perform lane crossing detection and lane departure detection to determine whether the vehicle crosses the lane and whether the vehicle deviates from the lane; If the vehicle crosses the line for a continuous time t 1 >Set threshold time t 1max , vehicle crossing line warning; If the vehicle deviates from the lane for a continuous time t 2 >Set threshold time t 2max , vehicle lane departure warning; S525: If the vehicle changes lanes, the vehicle is on a normal road and the driver and passengers are in normal condition, when the vehicle lateral acceleration is less than a preset threshold lateral acceleration, i.e., a y <a y max When the vehicle's lateral acceleration is greater than the preset threshold lateral acceleration, that is, a y >a y max When , the vehicle's suspected sudden lane change behavior markABD=1; S526: If the vehicle changes lanes, the vehicle is on an abnormal road or the driver and passengers are in an abnormal state, dynamically update the threshold lateral acceleration (1-e)(1-d)a y max , when the vehicle lateral acceleration is less than the dynamically updated threshold lateral acceleration, that is, a<(1-e)(1-d)a y max When the vehicle's normal lane change behavior markABD=0; when the vehicle's lateral acceleration is greater than the dynamically updated threshold lateral acceleration, that is, a>(1-e)(1-d)a max When , the vehicle's suspected sudden lane change behavior markABD=1; S527: If the vehicle is suspected of making an abrupt lane change, determine whether there is a special situation. If the road is normal and there is no special situation, markS=1, then determine that the vehicle is making an abrupt lane change. If it is an abnormal road, markS=0, and the vehicle is judged to be changing lanes normally.
9. The method for determining abnormal driving behavior based on multimodality according to claim 6, Features: In step S5, the process of determining abnormal driving behavior of not maintaining a safe vehicle distance is as follows: S531: Preset the safe distance L of the driver in normal state max ; S532: Simultaneously monitor the road environment and the driver's status, determine whether the road environment and the driver's status are normal, calculate the road environment factor e and the driver's status factor d, and obtain the speed and distance L of the vehicle ahead through road environment monitoring. The vehicle distance calculation formula is: L=(F*W) / P Where F is the focal length of the camera, W is the width or height of the target, and P refers to the width of the target in pixels in the x direction or the height of the target in the y direction in the image; S533: The vehicle is on a normal road and the driver and passengers are in normal condition; when the real-time distance between the vehicle and the vehicle ahead is less than the safe distance, i.e., L<L max When the real-time distance between the vehicle and the vehicle ahead is greater than the safe distance, that is, L>L max When the vehicle is too close to the vehicle; S534: The vehicle is on an abnormal road or the driver and passengers are in abnormal condition, and the vehicle safety distance is dynamically updated (1-e)(1-d)L max , when the real-time distance between the vehicle and the vehicle in front is less than the dynamically updated safe distance, that is, L<(1-e)(1-d)L max When the real-time distance between the vehicle and the vehicle ahead is greater than the dynamically updated safe distance, that is, L>(1-e)(1-d)L max When the vehicle is too close to the vehicle; The judgment process of abnormal driving behaviors such as sudden acceleration, sudden deceleration and sharp turn is as follows: S541: Set the thresholds for rapid acceleration, rapid deceleration, rapid turning, and the initial threshold for longitudinal acceleration a x max , initial threshold value of steering angle θ max ; S542: Monitor the road environment, vehicle motion status, and driver and passenger status, determine whether the road environment and driver and passenger status are normal, calculate the road environment factor e and the driver status factor d, and obtain the vehicle's real-time longitudinal acceleration a x , steering angle θ; S543: The vehicle is on a normal road and the driver and passengers are in normal condition. When the real-time longitudinal acceleration is less than the initial threshold of longitudinal acceleration and the real-time steering angle is less than the initial threshold of steering angle, that is, a x <a xmax And θ<θ max When the vehicle is running normally, markABD=0; when the real-time longitudinal acceleration is greater than the initial threshold of longitudinal acceleration, that is, a x >a x max , then the vehicle is suspected of sudden acceleration or deceleration markABD=1; when the real-time steering angle is greater than the initial steering angle threshold, that is, θ>θ max When , the vehicle is suspected to be turning sharply, markABD=1; S544: The vehicle is on an abnormal road or the driver and passenger are in an abnormal state. When the real-time longitudinal acceleration is less than the longitudinal acceleration update threshold, and the real-time steering angle is less than the steering angle update threshold, that is, a x <(1-e)(1-d)a x max And θ<(1-e)(1-d)θ max When the vehicle is running normally, markABD=0; when the real-time longitudinal acceleration is greater than the longitudinal acceleration update threshold, that is, a x >(1-e)(1-d)a x max , the vehicle is suspected of rapid acceleration or rapid deceleration markABD = 1; When the real-time steering angle is greater than the steering angle update threshold, that is, θ>(1-e)(1-d)θ max When , the vehicle is suspected to be turning sharply, markABD=1; S545: For abnormal behaviors suspected of sudden deceleration and sharp turning, determine whether there are special circumstances. If it is a normal road and there is no special circumstance, mark S = 1, then determine that the vehicle is in a sudden deceleration behavior or a sharp turning behavior; If it is an abnormal road, markS = 0, and the vehicle is judged to be changing lanes normally; S546: For suspected sudden abnormal behavior, road environment monitoring, monitoring the sudden acceleration of the rear vehicle and other abnormal roads, markS = 0, it is normal driving; normal road markS = 1, it is a sudden acceleration abnormal behavior; The process of judging abnormal sight deviation: S551: Preset the driver's sight deviation time threshold as T max ; S552: Monitor the road environment, vehicle motion state, and driver state. Obtain whether the current vehicle is going straight or turning through road environment monitoring-lane line detection. a=1 indicates that the vehicle is on a straight road; a=0 indicates that the vehicle is on a turning road. Obtain data on whether the vehicle steering wheel is turning through vehicle motion state monitoring. b=1 indicates that the vehicle is going straight; b=0 indicates that the vehicle is about to turn or is turning. Obtain driver's line of sight deviation data and line of sight deviation time T through driver state monitoring. S553: If the driver's sight deviates from T>T max If the vehicle is on a turning road and the steering wheel is in a turning state, it is normal driving; if the vehicle is on a straight road but the steering wheel is in a turning state, it is abnormal driving; if the vehicle is on a straight road but the steering wheel is in a straight state, it is distracted driving.
10. A system applicable to the multi-modal abnormal driving behavior judgment method according to any one of claims 1 to 9, Features: It includes: Driver and passenger status monitoring module, vehicle motion status monitoring module, road environment monitoring module, abnormal driving risk level pre-evaluation module and multi-modal data fusion judgment module, among which, The driver and passenger status monitoring module is used to obtain the status information of the driver and passenger, and build a two-stream spatiotemporal convolutional network that integrates adaptive gradient features to monitor the status of the driver and passenger; The vehicle motion state monitoring module is used to obtain the vehicle motion state and monitor dangerous driving operation behaviors through a sliding window threshold classification algorithm; The road environment monitoring module is used to obtain road environment information and build a multi-detection head cross-stage local network with a self-attention mechanism to monitor the road environment. The abnormal driving risk level pre-evaluation module predicts whether there are any driving behaviors that seriously affect road safety based on various monitoring data, and classifies the behaviors into normal driving, suspected dangerous driving, dangerous driving, and serious dangerous driving behaviors. If there is serious dangerous driving behavior, the driving right will be directly transferred; if there is suspected dangerous driving or dangerous driving behavior, this behavior will be further identified; The multimodal data fusion judgment module constructs a multi-index threshold division abnormal driving recognition algorithm based on logistic regression to accurately identify abnormal driving behaviors that are suspected of dangerous driving or dangerous driving behaviors by fusing multimodal data.
Citation Information
Cited By
Driving behavior specification method and system based on complex scene, medium and equipment
CN119502943A
Bad driving behavior feedback intervention method and device based on driving behavior big data
CN120116946A
Bad driving behavior feedback intervention method and device based on driving behavior big data
CN120116946B
Road health state monitoring method, system and equipment based on dynamic road veins
CN120333506A
Edge fatigue detection method and device based on adaptive threshold adjustment and medium
CN120681147A
