Driver abnormal behavior real-time warning method based on deep learning

Abstract

This application discloses a real-time early warning method for abnormal driver behavior based on deep learning, relating to the field of safe driving technology. It solves the technical problems in existing technologies that do not consider the impact of driver emotional fluctuations on driving and that driver emotion recognition has individual differences. This invention collects behavioral images and road condition information during driving, extracts behavioral features, and obtains a driving behavior record set based on changes in behavioral features and behavior time. Based on the abnormal emotion behavior feature set, a general abnormal emotion recognition model is trained and obtained, which is combined with the behavioral features to obtain a personalized emotion recognition model. Emotional influence factors are set according to different emotion types. A behavioral risk value is obtained based on driving behavior, driving behavior record set, and emotional influence factors. A behavioral risk signal is generated based on the behavioral risk value and road condition information, and an early warning is issued based on the behavioral risk signal.

Description

Technical Field

[0001] This application belongs to the field of safe driving, specifically a real-time early warning method for abnormal driver behavior based on deep learning. Background Technology

[0002] With the rapid development of intelligent and connected vehicles, cars have become a core means of transportation for people's daily travel. However, road traffic safety issues have also become more prominent, among which abnormal driver behavior is one of the main causes of traffic accidents. According to relevant statistics, traffic accidents caused by unsafe driver behavior account for a very high percentage, especially abnormal behaviors such as fatigued driving and distracted driving (such as making or receiving phone calls, looking down at a mobile phone, or deviating from the driver's line of sight). These behaviors can easily lead to slow driver reaction and operational errors, resulting in serious personal injury and property damage.

[0003] The invention patent with application number CN120580672A discloses a driver behavior monitoring and analysis method based on artificial intelligence. This method, during vehicle operation, analyzes the behavior triggers of at least two behavioral atoms based on driver characteristic information and vehicle dynamic information to determine the behavior monitoring time. Within a preset time after the behavior monitoring time, it monitors the behavioral state changes of each behavioral atom, performs behavior chain matching on the driver, and if a match is successful, adjusts the basic weight of any behavioral atom based on real-time lane departure distance, real-time vehicle speed, and the time difference between the behavior triggers of any behavioral atom to obtain a spatiotemporal decay weight. Combining the spatiotemporal decay weight of each behavioral atom with the behavior trigger characteristic value, it obtains a behavioral composite risk value and provides an abnormal warning for driver behavior based on this composite risk value. However, this invention, while providing an abnormal warning based on driver behavior and vehicle dynamic information, does not consider that when drivers are emotionally unstable, they may not trigger excessive behavioral actions but still affect driving. Furthermore, it suffers from inaccurate emotion recognition among different drivers due to differences in appearance. Summary of the Invention

[0004] This application aims to address at least one of the technical problems existing in the prior art; to this end, this application proposes a real-time early warning method for abnormal driver behavior based on deep learning, which is used to solve the technical problem that the prior art does not take into account the impact of driver's emotional fluctuations on driving, and that there are individual differences in driver emotion recognition.

[0005] To achieve the above objectives, the first aspect of this application provides a real-time early warning method for abnormal driver behavior based on deep learning, comprising the following steps: Step 1: Obtain driver behavior images and road condition information, and extract abnormal emotional behavior feature sets from the database; the road condition information includes vehicle driving information and road condition information; Step 2: Extract features from the behavior images to obtain behavior features, record the behavior time corresponding to the behavior features, obtain driving behavior based on behavior features and behavior time, and record it to obtain a driving behavior record set; Step 3: Train the neural network model based on the abnormal emotion behavior feature set to obtain a general abnormal emotion recognition model, and combine it with the behavioral features to obtain a personalized emotion recognition model. Input driving behavior into the personalized emotion recognition model to obtain the emotion type; set emotion influencing factors based on emotion type, and obtain the behavioral risk value based on driving behavior, driving behavior record set and emotion influencing factors. Step 4: Generate behavioral risk signals based on behavioral risk values ​​and road condition information.

[0006] Preferably, the step of extracting behavioral features from the behavioral image includes: The driver's facial images are captured by an in-vehicle camera to obtain behavioral images. Feature extraction is performed on the behavioral images to obtain the driver's facial features. Redundant features are removed, and dimensionality reduction is performed using principal component analysis (PCA) algorithm to generate a fixed-dimensional facial behavioral feature vector. Standardization processing is then performed to obtain the behavioral features.

[0007] Preferably, the step of obtaining and recording driving behavior based on behavioral characteristics and time to obtain a driving behavior record set includes: Extract behavioral features and their corresponding time periods; The behavioral features are linked together according to the chronological order of the behavior, the characteristic change states of the behavioral features are identified, and the driving behavior is combined based on the characteristic change states to obtain a driving behavior record set.

[0008] Preferably, the step of combining driving behaviors based on feature change states to obtain a driving behavior record set includes: Extract features related to changing states and driving behavior; whereby the changing states include the time and magnitude of the change. The time difference between the time nodes of the detected feature changes is recorded and the node change amplitude of each time node is statistically analyzed. A change state threshold is set based on the change time node and the node change amplitude. The system detects the time and magnitude of changes in features and compares them with a threshold for the change state. Based on the comparison results, the time nodes are divided to obtain a driving behavior record set.

[0009] Preferably, the step of setting the change state threshold based on the change time node and the change magnitude of the node includes: Extract the time points of change and the magnitude of change at each point; The nodes are sorted according to the time sequence to obtain the node sorting, and then numbered in the order j, where j = 1, 2, 3...n, and n is a positive integer; Calculate and record the change magnitude between node j and node j-1. When node j+1 is reached, calculate and record the change magnitude between node j+1 and node j. Take the average of the two changes between nodes as the average node change magnitude. Calculate the average node change magnitude between nodes sequentially according to the time nodes. Determine whether the change magnitude between node n+1 and node n is greater than twice the average node change magnitude. If yes, mark node n as a changed node; otherwise, continue calculating the average node change magnitude. When a change node is detected, the change time point corresponding to the change node is recorded, and the change node is used as the initial node to recalculate the average node amplitude to obtain the average change amplitude until the change node is detected again. The change time difference between the two change nodes is calculated. The change time difference is normalized to obtain the change time parameter. The change time parameter is multiplied by the difference between the average change amplitude and the average node amplitude to obtain the change state threshold.

[0010] Preferably, the step of training a neural network model based on an abnormal emotional behavior feature set to obtain a general abnormal emotion recognition model and combining it with behavioral features to obtain a personalized emotion recognition model includes: Abnormal behavioral features and corresponding abnormal emotion types from the abnormal emotional behavior feature set are used as training data and test data. The neural network model is trained using the training data and tested using the test data. The parameters of the neural network model are adjusted based on the test results to obtain a general abnormal emotion model with behavioral features as input data and corresponding emotion types as output data. Standard features are extracted from the general model of abnormal emotions. The difference between the behavioral features and the standard features is calculated to obtain the individual difference. The differences corresponding to different features are normalized to obtain the standard individual difference. The general model of abnormal emotions is weighted and fused according to the standard individual difference to obtain the individual emotion recognition model.

[0011] Preferably, the step of setting emotion influencing factors based on emotion type includes: Extract emotion types; where emotion types include total positive emotion types and total negative emotion types; total positive emotion types include positive emotion types such as relaxation, calmness, and pleasure, and total negative emotion types include negative emotion types such as anger, sadness, and excitement; The basic emotional impact factor during normal driving is set to 1.0. An emotional type-quantification value mapping table is established to quantify the impact of different emotions on driving. The mapping table sets the emotional value and emotional impact degree corresponding to the emotional type. The emotion influence factor QXYi corresponding to emotion type i is calculated using the formula QXYi=QZi(1-QXDi×Ω); where QZj is the emotion value of i, QXDi is the emotion influence degree of i, and Ω is the penalty coefficient set based on the emotion type-quantitative value mapping table.

[0012] Preferably, the step of obtaining the behavioral risk value based on driving behavior, a set of driving behavior records, and emotional influencing factors includes: Extract driving behavior, driving behavior record sets, and emotional influencing factors; Set corresponding weight parameters, and α+β+δ=1; count and record the magnitude and number of emotion changes within the set time period, and calculate the behavioral risk value XWF using the formula XWF=10×[α×(1-QZj)+β×QBC×QBF+δ×(1-QXYmax / QXYi)]; where QBC represents the magnitude of emotion change, QBF represents the number of emotion changes, and QXYmax represents the maximum value of the emotion influence factor.

[0013] Preferably, the step of generating a behavioral risk signal based on behavioral risk values ​​and driving road condition information includes: Extract behavioral risk values ​​and road condition information; whereby road condition information includes vehicle driving information and road condition information. Analyze road condition information to obtain the appropriate vehicle speed, compare the vehicle driving information with the appropriate vehicle speed to obtain the driving speed difference; determine whether the driving speed difference is greater than the set safe speed difference; if yes, directly generate a dangerous speed signal; otherwise, generate a driving signal based on the driving speed difference and the behavioral risk value. The risk speed signal is integrated with the driving signal to obtain the behavioral risk signal.

[0014] Preferably, the generation of driving signals based on driving speed difference and behavioral risk value includes: Extract driving speed difference and behavioral risk value; When the speed difference is greater than 0, the vehicle is determined to be speeding. The normalized speed difference is denoted as Vn, and the speeding risk coefficient CVS is set to e. Vn The speeding risk signal is generated by combining the speeding risk coefficient and the behavioral risk value; when the speed difference is less than or equal to 0, the normal risk signal is generated solely by the behavioral risk value. The overspeed risk signal is combined with the normal risk signal to obtain the driving signal.

[0015] It should be noted that when the speed difference is less than 0, a low speed signal is generated when the current vehicle speed is too low.

[0016] Compared with the prior art, the beneficial effects of this application are: 1. This invention collects images of the driver's behavior while driving and road condition information during vehicle operation. It extracts features from the behavior images to obtain behavioral characteristics and records the corresponding behavior times. Based on changes in behavioral characteristics, the behavior times are divided to obtain a driving behavior record set, accurately capturing the driver's emotional changes and behavioral anomalies, thus more accurately identifying abnormal behavior. A neural network model identifies the emotion types corresponding to the behavioral characteristics and sets emotion influence factors according to different emotion types. Based on driving behavior, the driving behavior record set, and the emotion influence factors, a behavioral risk value is obtained, which can accurately match the driver's current emotional state and driving behavior, further improving the accuracy of risk assessment. Based on the behavioral risk value and road condition information, a behavioral risk signal is generated, and the behavioral risk value is further refined according to road conditions, making the results more accurate and applicable to a wider range of situations.

[0017] 2. This invention detects changes in behavioral features in behavioral images and records the corresponding time points of change. Based on the time points of change and the magnitude of the changes, a threshold value is assigned to the change state. This accurately represents the overall state of the driver's behavior and emotional changes, and can predict risk change trends in advance, providing support for the dynamic adjustment of subsequent risk signals and effectively improving the foresight and effectiveness of early warnings. By obtaining behavioral features from the driver's independent behavioral features and comparing them with standard features in the general abnormal emotion model, a standard individual difference is obtained. The general abnormal emotion model is then weighted and fused based on the standard individual difference to obtain a personalized emotion recognition model. By recognizing emotions while driving through the personalized emotion recognition model, the model and the evaluation system can always keep up with the driver's behavioral changes, improving recognition accuracy and risk assessment accuracy. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the working principle of this application.

[0020] Figure 2 This is a mapping table of emotion type quantification values ​​according to one embodiment of this application. Detailed Implementation

[0021] The technical solutions of this application will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0022] Please see Figure 1 The first aspect of this invention provides a real-time early warning method for abnormal driver behavior based on deep learning, comprising: Step 1: Obtain driver behavior images and road condition information, and extract abnormal emotional behavior feature sets from the database; the road condition information includes vehicle driving information and road condition information.

[0023] For example, the vehicle's built-in camera can capture real-time behavioral images covering the driver's face, neck, and upper shoulders to ensure that facial micro-expressions and basic body movements are captured; and the vehicle's speed and current road conditions can be collected through vehicle dynamic sensors.

[0024] The acquired images are preprocessed, and invalid images, including images with occluded faces, blurred images, and images with heads deviated, are removed in real time. Facial features that affect different emotions during driving are extracted from the facial emotion recognition database.

[0025] It should be added that in low-light scenarios such as at night or in tunnels, the intensity of the supplementary light needs to be adaptively adjusted to ensure the accuracy of image acquisition.

[0026] Step 2: Extract features from the behavior images to obtain behavior features, record the behavior time corresponding to the behavior features, obtain driving behavior based on behavior features and behavior time, and record the driving behavior record set.

[0027] For example, a CNN-LSTM hybrid deep learning model is used to extract features from the behavior image obtained in step one, perform feature fusion, remove redundant features, reduce dimensionality through principal component analysis (PCA) algorithm to generate a fixed-dimensional facial behavior feature vector, complete the standardization process to obtain the behavior features, and record the corresponding behavior time through timestamps.

[0028] Behavioral features are detected according to the time sequence corresponding to the timestamps. It is determined whether the behavioral features have changed. If they have changed, the time and magnitude of the change are recorded. If they have not changed, they are not recorded and invalid images are filtered out.

[0029] It should be added that when identifying changes in behavioral characteristics, a threshold can be set to determine whether a characteristic has changed. For example, normal and rapid characteristic changes such as blinking or shaking can be considered. If the change does not exceed the threshold, the characteristic change will not be recorded. If no change is detected, the saved image and feature can be deleted to ensure that the data is not too redundant.

[0030] The time difference between the time nodes of the feature change is detected, and the node change amplitude of each time node is recorded and counted. The time nodes of change are sorted according to the time order to obtain the node sorting and numbered in the order j, where j=1,2,3……n, and n is a positive integer.

[0031] Behavioral features and corresponding time events within the last 30 minutes are extracted. Time points where significant changes in behavioral features occur (i.e., time points where the change in feature value is ≥0.1, where 0.1 is a preset threshold for feature change, which can be adjusted according to the actual scenario) are selected as the feature change time points to be analyzed. In this embodiment, a total of 6 feature change time points were detected within the last 30 minutes. These were sorted and recorded as time change points. First, the time difference between all original feature change time points was calculated, and then the amplitude change difference between adjacent points was calculated. All feature change time points were sorted according to the order of behavior time and assigned numbers j (j=1,2,3…n, where n is the total number of feature change time points, and n=6 in this embodiment). According to the sorted node number j, the change amplitude between two adjacent nodes (i.e., the change amplitude between node j and node j-1, j≥2) was calculated sequentially, and the average node change amplitude (the average of the change amplitudes of the two most recent adjacent nodes) was calculated in real time. The specific calculation process is as follows: When j=2, the change amplitude between node 2 and node 1 is calculated: |0.52 -0.48| = 0.04; At this point, there is only one change amplitude, so the average node change amplitude is not calculated yet, and the next node is checked; When j=3, the change amplitude between node 3 and node 2 is calculated: |0.61 - 0.52| = 0.09; The average of the two change amplitudes (0.04, 0.09) is taken to obtain the average node change amplitude: (0.04 + 0.09) ÷ 2 = 0.065; When j=4, the change amplitude between node 4 and node 3 is calculated: |0.35 - 0.61| = 0.26; The average of the two most recent change amplitudes (0.09, 0.26) is taken to update the average node change amplitude: (0.09 + 0.26) ÷ 2 = 0.175; When j=5, the change amplitude between node 5 and node 4 is calculated: |0.28 - 0.35| = 0.07; Take the average of the two most recent change amplitudes (0.26, 0.07) and update the average node change amplitude: (0.26 + 0.07) ÷ 2 = 0.165; When j=6, calculate the change amplitude between node 6 and node 5: |0.15 - 0.28| = 0.13; Take the average of the two most recent change amplitudes (0.07, 0.13) and update the average node change amplitude: (0.07 + 0.13) ÷ 2 = 0.10.

[0032] In this embodiment, if node 7 is a changing node, the timestamp corresponding to node 7 is recorded as the changing time point, and node 7 is used as the initial node. The average node change amplitude is recalculated until the changing node 12 is detected again. The change time difference between changing nodes is calculated, and the change time difference is normalized to obtain the change time parameter. The change time parameter is multiplied by the difference between the average change amplitude and the average node amplitude to obtain the change state threshold.

[0033] The time and magnitude of feature changes between each node are compared with the change state threshold. The time of change is compared with the time difference of change, and the magnitude of change is compared with the change state threshold. If at least one difference is large in the comparison results, it is determined to be a feature change. When a feature change is detected, the feature data between the previous changing node and the current changing node are divided to obtain the driving behavior record set.

[0034] Step 3: Train the neural network model based on the abnormal emotion behavior feature set to obtain a general abnormal emotion recognition model, and combine it with the behavioral features to obtain a personalized emotion recognition model. Input driving behavior into the personalized emotion recognition model to obtain the emotion type; set the emotion influencing factor based on the emotion type, and obtain the behavioral risk value based on driving behavior, driving behavior record set and emotion influencing factor.

[0035] For example, abnormal behavioral features and corresponding abnormal emotion types extracted from the database are used as training and testing data. The training data is used to train the neural network model, and the testing data is used to test the trained neural network model. Based on the testing results, the parameters of the neural network model are adjusted to obtain a general abnormal emotion recognition model with behavioral features as input data and corresponding emotion types as output data. From the trained general abnormal emotion recognition model, fixed core standard features are extracted, and historical driving behavior data specific to the target driver is collected. After the same preprocessing process, an individual-specific behavioral feature set is obtained. The individual behavioral features are compared with the general standard features dimension by dimension to obtain the individual difference. Adaptive weighting coefficients are assigned to each feature dimension, following the principle that the larger the difference, the higher the weight. The weighting coefficients are integrated into the general abnormal emotion recognition model to perform weighted fusion correction on the model's feature extraction layer and classification judgment layer, conforming to the target driver's specific behavioral patterns and emotional expression habits, eliminating the group bias of the general model, and finally generating a personalized emotion recognition model adapted to a single driver.

[0036] Specifically, the driver's unique behavioral characteristics data are combined with a general emotion recognition model. This can be achieved by using any one of the following methods: small sample fine-tuning, feature weighted fusion, or transfer learning adaptive adaptation. The underlying feature extraction layer of the model is frozen, and the unique behavioral characteristics data and corresponding personalized emotion annotations are input into the model to obtain a personalized emotion recognition model.

[0037] The driver's current emotion type is obtained by identifying the driver's emotions in the current behavior image. In this embodiment, the emotion type includes a total positive emotion type and a total negative emotion type. The total positive emotion type includes positive emotion types such as relaxation, calmness, and pleasure, while the total negative emotion type includes negative emotion types such as anger, sadness, and excitement.

[0038] The basic emotional impact factor during normal driving is set to 1.0. An emotional type-quantification value mapping table is established to quantify the impact of different emotions on driving. The mapping table sets the emotional value and emotional impact degree corresponding to the emotional type. The emotion influence factor QXYi corresponding to emotion type i is calculated using the formula QXYi=QZi(1-QXDi×Ω); where QZj is the emotion value of emotion i, QXDi is the emotion influence degree of emotion i, and Ω is the penalty coefficient set based on the emotion type-quantitative value mapping table; in this embodiment, the emotion type-quantitative value mapping table has an emotion value range of [0.4, 1.2] for positive emotion types and a value range of [-2, -0.4] for negative emotion types.

[0039] It should be added that different types of emotions have different effects on driving. Positive emotions have a positive effect on driving behavior, but the impact is relatively small. Negative emotions such as anger and sadness have a greater impact on driving behavior, and excessively positive emotions such as excitement can also have a negative impact on driving behavior.

[0040] In this embodiment, α=0.4, β=0.4, and δ=0.2 are set. The behavioral risk value XWF is calculated using the formula XWF=10×[α×(1-QZj)+β×QBC×QBF+δ×(1-QXYmax / QXYi)]. Taking node 7 as an example, the behavioral risk value corresponding to node 7 is calculated to be 54, and the recognized emotion at this time is excitement.

[0041] Step 4: Generate behavioral risk signals based on behavioral risk values ​​and road condition information.

[0042] For example, analyzing the current road condition information, the vehicle speed corresponding to node 7 is 48 km / h, and the road condition information is a city road with good congestion. Therefore, the appropriate speed for the vehicle is 44 km / h. Comparing the vehicle's current driving information with the appropriate speed yields a speed difference of 4. In this embodiment, the set safe speed difference is 8. It is determined whether the speed difference is greater than the set safe speed difference. If it is determined that it is not greater than the set safe speed difference, the speed difference is further compared with 0, and the vehicle is determined to be speeding. The normalized speed difference is 0.4, and the speeding risk coefficient CVS is set to e. 0.4 The speeding risk coefficient is multiplied by the behavior risk value and compared with the risk threshold to generate a behavior risk signal. In this embodiment, if the vehicle is within a reasonable driving speed range and the behavior risk value is less than the risk threshold, no behavior risk signal is generated.

[0043] In other embodiments, if the vehicle's current speed exceeds the vehicle's appropriate speed by a large margin, a dangerous speed signal is immediately generated; similarly, if the vehicle's current speed is too low, an acceleration signal is also required; when the speed difference is less than 0, a normal risk signal is generated directly based on the behavioral risk value.

[0044] It should be noted that the appropriate vehicle speed and the set safe speed difference are both set based on the current road conditions and can be set freely, but must be reasonable and ensure driving safety.

[0045] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0046] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

[0047] Some of the data in the above formula are calculated by removing dimensions and taking their numerical values. The formula is the closest to the real situation obtained by software simulation of a large amount of collected data. The preset parameters and preset thresholds in the formula are set by those skilled in the art according to the actual situation or obtained through simulation of a large amount of data.

[0048] How this application works: This invention collects images of driver behavior while driving and road condition information while the vehicle is in motion. It extracts features from the behavior images to obtain behavioral characteristics and records the corresponding behavior times. Based on the changes in behavioral characteristics, the behavior times are divided to obtain a driving behavior record set. A neural network model is used to identify the emotion types corresponding to the behavioral characteristics, and emotion influence factors are set according to different emotion types. A behavioral risk value is obtained based on driving behavior, driving behavior record set, and emotion influence factors. A behavioral risk signal is generated based on the behavioral risk value and road condition information, and a warning is issued through the behavioral risk signal.

[0049] The above embodiments are only used to illustrate the technical methods of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of this application without departing from the spirit and scope of the technical methods of this application.

Claims

1. A real-time early warning method for abnormal driver behavior based on deep learning, characterized in that, include: Step 1: Obtain images of the driver's behavior and road conditions, and extract a set of abnormal emotional and behavioral features from the database; The driving condition information includes vehicle driving information and road condition information; Step 2: Extract features from the behavior images to obtain behavior features, record the behavior time corresponding to the behavior features, obtain driving behavior based on behavior features and behavior time, and record it to obtain a driving behavior record set; Step 3: Train the neural network model based on the abnormal emotion behavior feature set to obtain a general abnormal emotion recognition model, and combine it with the behavioral features to obtain a personalized emotion recognition model. Input driving behavior into the personalized emotion recognition model to obtain the emotion type; set emotion influencing factors based on emotion type, and obtain the behavioral risk value based on driving behavior, driving behavior record set and emotion influencing factors. Step 4: Generate behavioral risk signals based on behavioral risk values ​​and road condition information.

2. The real-time early warning method for abnormal driver behavior based on deep learning according to claim 1, characterized in that, The step of extracting behavioral features from behavioral images includes: The driver's facial images are captured by an in-vehicle camera to obtain behavioral images. Feature extraction is performed on the behavioral images to obtain the driver's facial features. Redundant features are removed, and dimensionality reduction is performed using principal component analysis (PCA) algorithm to generate a fixed-dimensional facial behavioral feature vector. Standardization processing is then performed to obtain the behavioral features.

3. The real-time early warning method for abnormal driver behavior based on deep learning according to claim 1, characterized in that, The process of obtaining and recording driving behavior based on behavioral characteristics and timing to create a driving behavior record set includes: Extract behavioral features and their corresponding time periods; The behavioral features are linked together according to the chronological order of the behavior, the characteristic change states of the behavioral features are identified, and the driving behavior is combined based on the characteristic change states to obtain a driving behavior record set.

4. The real-time early warning method for abnormal driver behavior based on deep learning according to claim 3, characterized in that, The driving behavior record set obtained by combining driving behaviors based on feature change states includes: Extract features related to changing states and driving behavior; whereby the changing states include the time and magnitude of the change. The time difference between the time nodes of the detected feature changes is recorded and the node change amplitude of each time node is statistically analyzed. A change state threshold is set based on the change time node and the node change amplitude. The system detects the time and magnitude of changes in features and compares them with a threshold for the change state. Based on the comparison results, the time nodes are divided to obtain a driving behavior record set.

5. The real-time early warning method for abnormal driver behavior based on deep learning according to claim 4, characterized in that, The method of setting a change state threshold based on the change time node and the change magnitude of the node includes: Extract the time points of change and the magnitude of change at each point; The nodes are sorted according to the time sequence to obtain the node sorting, and then numbered in the order j, where j = 1, 2, 3...n, and n is a positive integer; Calculate and record the change magnitude between node j and node j-1. When node j+1 is reached, calculate and record the change magnitude between node j+1 and node j. Take the average of the two changes between nodes as the average node change magnitude. Calculate the average node change magnitude between nodes sequentially according to the time nodes. Determine whether the change magnitude between node n+1 and node n is greater than twice the average node change magnitude. If yes, mark node n as a changed node; otherwise, continue calculating the average node change magnitude. When a change node is detected, the change time point corresponding to the change node is recorded, and the change node is used as the initial node to recalculate the average node amplitude to obtain the average change amplitude until the change node is detected again. The change time difference between the two change nodes is calculated. The change time difference is normalized to obtain the change time parameter. The change time parameter is multiplied by the difference between the average change amplitude and the average node amplitude to obtain the change state threshold.

6. The real-time early warning method for abnormal driver behavior based on deep learning according to claim 1, characterized in that, The process of training a neural network model based on abnormal emotional behavior feature sets to obtain a general model for abnormal emotion recognition, and combining it with behavioral features to obtain a personalized emotion recognition model, includes: Abnormal behavioral features and corresponding abnormal emotion types from the abnormal emotional behavior feature set are used as training data and test data. The neural network model is trained using the training data and tested using the test data. The parameters of the neural network model are adjusted based on the test results to obtain a general abnormal emotion model with behavioral features as input data and corresponding emotion types as output data. Standard features are extracted from the general model of abnormal emotions. The difference between the behavioral features and the standard features is calculated to obtain the individual difference. The differences corresponding to different features are normalized to obtain the standard individual difference. The general model of abnormal emotions is weighted and fused according to the standard individual difference to obtain the individual emotion recognition model.

7. The real-time early warning method for abnormal driver behavior based on deep learning according to claim 1, characterized in that, The setting of emotion influencing factors based on emotion type includes: Extract emotion types; where emotion types include total positive emotion types and total negative emotion types; total positive emotion types include positive emotion types such as relaxation, calmness, and pleasure, and total negative emotion types include negative emotion types such as anger, sadness, and excitement; The basic emotional impact factor during normal driving is set to 1.

0. An emotional type-quantification value mapping table is established to quantify the impact of different emotions on driving. The mapping table sets the emotional value and emotional impact degree corresponding to the emotional type. The emotion influence factor QXYi corresponding to emotion type i is calculated using the formula QXYi=QZi(1-QXDi×Ω); where QZj is the emotion value of i, QXDi is the emotion influence degree of i, and Ω is the penalty coefficient set based on the emotion type-quantitative value mapping table.

8. The real-time early warning method for abnormal driver behavior based on deep learning according to claim 1, characterized in that, The method of obtaining behavioral risk values ​​based on driving behavior, driving behavior record sets, and emotional influencing factors includes: Extract driving behavior, driving behavior record sets, and emotional influencing factors; Set corresponding weight parameters, with α+β+δ=1; statistically record the magnitude and frequency of emotion changes within the set time period, and calculate the behavioral risk value XWF using the formula XWF=10×[α×(1-QZj)+β×QBC×QBF+δ×(1-QXYmax / QXYi)]; where QBC represents the magnitude of emotion changes, QBF represents the frequency of emotion changes, QXYmax represents the maximum value of the emotion influence factor, α represents the weight parameter of driving behavior on the behavioral risk value, β represents the weight parameter of the driving behavior record set on the behavioral risk value, and δ represents the weight parameter of the emotion influence factor on the behavioral risk value.

9. The real-time early warning method for abnormal driver behavior based on deep learning according to claim 1, characterized in that, The generation of behavioral risk signals based on behavioral risk values ​​and road condition information includes: Extract behavioral risk values ​​and road condition information; whereby road condition information includes vehicle driving information and road condition information. Analyze road condition information to obtain the appropriate vehicle speed, compare the vehicle driving information with the appropriate vehicle speed to obtain the driving speed difference; determine whether the driving speed difference is greater than the set safe speed difference; if yes, directly generate a dangerous speed signal; otherwise, generate a driving signal based on the driving speed difference and the behavioral risk value. The risk speed signal is integrated with the driving signal to obtain the behavioral risk signal.

10. The real-time early warning method for abnormal driver behavior based on deep learning according to claim 9, characterized in that, The generation of driving signals based on driving speed difference and behavioral risk value includes: Extract driving speed difference and behavioral risk value; When the travel speed difference is greater than 0, it is determined that the vehicle is in an overspeed state, the normalized travel speed difference is marked as Vn, and an overspeed risk coefficient CVS is set as e Vn An overspeed risk signal is generated in combination with the overspeed risk coefficient and the behavior risk value; when the travel speed difference is less than or equal to 0, a normal risk signal is generated only by means of the behavior risk value; The overspeed risk signal is combined with the normal risk signal to obtain the driving signal.

Images