Multi-modal fusion driver anger index real-time detection method and early warning system

Abstract

The invention discloses a multi-modal fusion driver anger index real-time detection method and system, and belongs to the technical field of detection and analysis. Under the combined action of related modules and application processes, electrocardio data of a driver is collected through an intelligent bracelet, the facial expression of the driver is captured through a vehicle-mounted camera, and the voice features of the driver are recorded through an in-vehicle microphone; the advanced data processing and machine learning technology is used for carrying out fusion analysis on the multi-modal data, the emotional state of the driver is evaluated in real time, specifically, the steps of electrocardio data processing, facial expression recognition, voice feature extraction and the like are combined with the multi-modal fusion strategy of the attention mechanism, the anger index of the driver is accurately recognized, and the driver's anger index is accurately recognized. And the corresponding early warning measures are triggered according to the preset threshold value, so that a bus company can find psychological problems of the driver in time and take corresponding intervention measures to maintain the psychological health of the driver, the driving safety of the bus can be effectively improved, the working experience of the driver is improved, and the psychological health of the driver is promoted.

Description

Technical Field

[0001] The present invention relates to the field of detection and analysis technology, in particular to a multi-modal fusion driver anger index real-time detection method and early warning system. Background Art

[0002] With the acceleration of urbanization, traffic congestion is becoming more and more serious. Bus drivers are facing increasing work pressure. Long-term driving fatigue and traffic jams may lead to increased emotional fluctuations in drivers. At present, many bus companies have begun to pay attention to the mental health of drivers, but lack effective monitoring and intervention methods. Existing psychological counseling and training measures are difficult to respond to drivers' emotional changes in real time. According to the "China Urban Public Transport Development Report", in 2019, Chinese urban bus drivers worked an average of more than 8 hours a day, most of which were in a highly tense state. This high-intensity working environment makes drivers more likely to have negative emotions, which in turn affects driving safety. Angry drivers may have a bad attitude towards passengers, and even verbal or physical conflicts, which seriously affect the safety of passengers and the riding experience. Drivers who are in a state of anger for a long time are also prone to psychological problems such as anxiety and depression, which further affects their physical and mental health and work performance.

[0003] Existing driver emotion monitoring technologies mainly rely on single sensor data (such as heart rate), which is difficult to fully reflect the driver's true emotional state. The accuracy of emotion recognition using only electrocardiogram data is only about 70%, which is obviously not enough to meet the needs of practical applications. In addition, existing emotion recognition models are usually static and lack the ability to adapt to dynamic situational changes. For example, drivers may express their emotions differently in different driving environments and conditions, and existing models often cannot capture these changes well. At the same time, existing systems usually adopt a unified standard, ignoring individual differences, and different drivers may have significant differences in their emotional responses in the same situation. Finally, existing systems usually take a long time to process and analyze data, making it difficult to achieve real-time monitoring and instant feedback. Therefore, it is particularly important to develop a multimodal emotion monitoring system that can integrate multiple data sources.

[0004] Purpose of the Invention

[0005] In order to overcome the existing driver emotion monitoring methods, which have the disadvantages as described in the background technology due to technical limitations, the present invention provides a multi-modal fusion driver anger index real-time detection method and early warning system, which can integrate the ECG data collected by a smart bracelet, facial expressions captured by a vehicle-mounted camera, and voice features recorded by a microphone in the vehicle under the joint action of relevant hardware and software technologies, such as ECG, facial expressions, and voice data of the driver driving the vehicle, to achieve high-precision real-time detection of the anger of the bus driver, and comprehensively analyze the driver's emotional state, which can significantly improve the accuracy of emotion recognition and achieve a recognition accuracy of more than 95%, thereby effectively preventing traffic accidents caused by the driver's emotional out-of-control, and improving driving safety and passenger experience.

[0006] The technical solution adopted by the present invention to solve its technical problem is:

[0007] A multimodal fusion driver anger index real-time detection method comprises the following steps: S1: collecting the driver's heart rate (HR) and heart rate variability (HRV) data through a smart bracelet, capturing the driver's facial expression through a high-definition vehicle camera, and recording the driver's voice features through a microphone in the vehicle. The specific voice features include speech speed, pitch, and emotional vocabulary information; S2: extracting features and processing data from the electrocardiogram, video, and voice data obtained in step S1. The feature extraction and data processing include the following sub-steps: (1) electrocardiogram data processing: using the BioSig toolbox to calculate HRV indicators in the time domain and frequency domain. The time domain HRV indicators include SDNN (standard deviation of all normal RR intervals), RMSSD (root mean square of the difference between adjacent RR intervals), and pNN50 (the proportion of adjacent RR intervals with a difference greater than 50 ms). DEAP data are used. The ECG signals and emotion labels of each participant are traversed, and the extracted features are standardized to have zero mean and unit variance. 80% of the acquired data are used for training and 20% for testing. (2) Facial expression data processing: Facial action coding system (FACS) is used for facial expression recognition. FACS decomposes facial expressions into a series of basic action units. Each AU represents a specific facial muscle movement. The emotional state of the bus driver is judged by detecting and analyzing AUs. (3) Speech data processing: Since some emotional words have different meanings in different contexts, segmenting speech into emotion and text may provide more accurate and comprehensive sentiment analysis results. (4) Data fusion. S3: Establish evaluation indicators, divide the degree of anger into multiple levels, and establish a scoring system.

[0008] Furthermore, in the step S2, in the ECG data processing, the SDNN calculation formula is as follows: where RR i is the i-th RR interval, is the average value of all RR intervals, N is the total number of RR intervals; the RMSSD calculation formula is as follows, The pNN50 calculation formula is as follows:

[0009] Frequency domain HRV index analysis is done through power spectral density (PSD). The spectrum includes low frequency (LF) and high frequency (HF), as well as their ratio. Low frequency refers to the power in the range of 0.04-0.15HZ. The formula is High frequency refers to the power in the range of 0.15-0.4HZ, and the formula is The power spectral density is calculated using the periodogram method. The formula is Where X(f) is the Fourier transform of x[n]. The DEAP dataset is used for training and testing. The ECG signals and emotion labels of each participant are traversed, and the extracted features are standardized to have zero mean and unit variance. 80% of the data are used for training and 20% of the data are used for testing. The training set is then trained using a random forest regressor. After training, the test set is used for prediction and the performance of the model is evaluated. Grid Search is used for hyperparameter tuning in the model optimization phase, and the generalization ability of the model is evaluated through K-fold cross validation. After the model is trained, it can detect real-time data streams and score the anger level of bus drivers from 1-100.

[0010] Furthermore, in step S2, in the facial expression data, the FER2013 data set is expanded, and the facial expression recognition annotators give the facial expressions in the image an anger score of 0-100, and the average value is taken after multiple rounds of annotation, and the results of multiple annotators are used for cross-validation; after manually annotating part of the data, the annotated data is used for training to obtain a preliminary emotion recognition model, and then the model is used to predict the unlabeled data, and the prediction results with high confidence are selected for automatic annotation, and finally the automatically annotated data is added to the training set, and the model is retrained. After iteration, a satisfactory number and quality of annotations are achieved, and then the Inception-V3 model is used to extract high-level image features, and the output is a 2048-dimensional feature vector. The AUs extracted by OpenFace are used to construct the feature vector, and the 12 AUs related to anger are mainly extracted: AU04: eyebrows raised, AU05: eyebrows pulled down, AU07: eyelids closed, AU10: upper lip raised , AU12: mouth corners raised, AU15: lip compression, AU17: chin lifted, AU23: mouth corners pulled down, AU24: mouth corners stretched to both sides, AU25: mouth corners raised and stretched, AU26: mouth open, AU27: mouth wide open, the final feature vector will be composed of Inception-V3 features and AUs features, with a total of 2060 dimensions; a multi-layer fully connected neural network is used to map the feature vector to the anger score, and the fully connected layer configuration is: the first layer: 1024 nodes, the activation function is ReLU; the second layer: 512 nodes, the activation function is ReLU; the third layer: 256 nodes, the activation function is ReLU; the output layer: 1 node, the activation function is linear (for regression), the output range is 0-100, the mean square error (MSE) is used as the loss function, the Adam optimizer is used for training, and the mean absolute error (MAE) is used as the evaluation indicator. After evaluating the model performance, the best model weights are saved, and finally the anger score result is obtained.

[0011] Furthermore, in step S2, during the speech data processing, the Librosa library is used to extract Mel-frequency cepstral coefficients (MFCCs). MFCCs can help distinguish different speech emotions. The formula is Where t is the time frame, m is the index of the Mel filter bank, M is the number of Mel filter banks, and f(t) is the frequency. The YIN algorithm is used to extract the fundamental frequency (F0). A high fundamental frequency usually indicates excitement or anger. The formula is Where x(k) is the audio signal, N is the signal length, and τ is the delay; the DeepSpeechASR model is used to convert the audio into text, and the BERT model is used to classify the text into sentiment. The formula is P(y|x)=softmax(W·h+b), where y is the sentiment category (anger, calmness, etc.), x is the input text, h is the output representation of the BERT model, and w and b are the bias terms of the weight matrix; the BERT model can accurately identify the emotional tendency in the text, thereby assisting in judging the driver's anger level. After the extracted features are normalized, the acoustic features are converted into and text features Directly concatenate together to form a multimodal feature vector X = [A; T], where the semicolon represents the concatenation of column vectors, through the formula d X =d A +d T .Define an attention network to calculate the attention weight of the multimodal feature vector X. The attention network consists of a single-layer fully connected network and outputs a weight vector with the same dimension as the feature vector; formula is the weight matrix of the attention network, is the bias term, is the weight vector used to calculate the attention score, is the intermediate vector after the activation function, is the attention weight vector; the calculated attention weight is used to perform weighted fusion on the multimodal feature vector: the formula is X weighted =α⊙X, where ⊙ represents element-by-element multiplication; the LSTM model is a special recurrent neural network (RNN) that can learn long-term dependencies and is suitable for processing time series data; the LSTM model is constructed and trained using the RAVDESS dataset; the state update of the LSTM unit is expressed as: Input gate: i t =σ(W ii x t +b ii +W hi h t-1 +b hi )

[0012] Forget gate: f t =σ(W if x t +b if +W hf h t-1 +b hf )

[0013] Cell state: c t =f t ⊙c t-1 +i t ⊙tanh(Wic x t +b ic +W hc h t-1 +b hc )

[0014] Output gate: o t =σ(W io x t +b io +W ho h t-1 +b ho )

[0015] LSTM cell output: h t =o t ⊙tanh(c t )

[0016] Where: σ is the Sigmoid activation function, is the hyperbolic tangent activation function, ⊙ represents element-by-element multiplication, W and b are the weight matrix and bias term, x t is the input of the current time step, h t-1 is the hidden state of the previous time step, c t-1 is the cell state at the previous time step; the mean square error (MSE) is also used as the loss function to minimize the difference between the predicted anger score and the true label, and finally the anger score result is obtained.

[0017] Furthermore, in the step S2, in the step fusion, data characteristics, model complexity and model performance are considered, and hybrid fusion is used for fusion, which combines the advantages of early fusion and late fusion: 1. It can capture cross-modal correlation information and independently process the characteristics of each modality; 2. It can better utilize multimodal data to improve the generalization ability and robustness of the model; 3. It can reduce the impact of poor quality of single modality data on the overall performance and obtain ECG Y ECG , Facial expression Y face , Voice Y speech After the anger rating results were normalized, they were ensured to be on the same scale for each modality. y is the original score, μ is the mean score, σ is the standard deviation of the score, is the standardized score; customize an attention layer to dynamically adjust the importance of each modality, and the intermediate vector u = tanh (W a X+b a ), calculate the attention score e = w s u, calculate the attention weight α = softmax(e), and perform weighted fusion of the multimodal feature vector X weighted=α⊙X, and the weighted fused feature vector is also input into the LSTM model for training; after the LSTM model, the output of each modality is mapped to a common space h mapped =ReLU(W map X weighted +b map ), and finally fused through a multi-layer fully connected neural network, the formula is h densel =ReLU(W densel h mapped +b densel ),

[0018] h dense2 =ReLU(W dense2 h dense1 +b dense2 ) Where: W map ,W densel ,W dense2 ,W out is the weight matrix, b map ,b densel ,b dense2 ,b out is the bias term, X weighted is the weighted rating vector, is the final predicted anger score.

[0019] Furthermore, in the step 3, in establishing the evaluation index, the anger degree is divided into the following levels and a scoring system is established: 0-20: Very calm, description: the individual is in a completely relaxed and calm state, without any anger, action suggestion: no special attention is needed, continue to maintain a good state; 21-40: Calm, description: the individual has slight mood swings, but remains calm overall, action suggestion: observe and record mood changes, and provide psychological counseling when necessary; 41-60: Mild anger, description: the individual shows mild anger, which may be accompanied by slight physical tension and inappropriate language Full, action suggestions: conduct emotion management training, provide psychological support, and monitor emotional changes; 61-80: Moderate anger, description: the individual shows obvious anger, which may be accompanied by physical actions and verbal aggression, action suggestions: trigger preliminary warning, conduct emergency intervention, including psychological counseling and behavioral guidance, and notify relevant personnel to pay attention to safety; 81-100: Severe anger, description: the individual is in a state of extreme anger, which may be accompanied by violent tendencies and uncontrolled behaviors, action suggestions: regard it as a serious situation, take immediate action, activate the emergency plan, ensure personal safety, and conduct emergency psychological intervention and behavior control.

[0020] A multi-modal fusion driver anger index real-time detection warning system includes a data acquisition module, a data processing module, a warning module, a front-end display module, a voice reminder module, a notification module, a user management module, a log recording module, a data storage module, and a system configuration module in a management end PC.

[0021] Furthermore, the data acquisition module includes a smart bracelet, a vehicle-mounted high-definition camera, and a vehicle-mounted microphone; the smart bracelet is worn on the driver's wrist and is used to collect the driver's electrocardiogram data, and can upload the data to a server via Bluetooth / wifi / cellular data network; the high-definition camera is fixed in front of the driver and is used to collect the driver's facial expression data, and can upload the data to a server via Bluetooth / wifi / cellular data network; the vehicle-mounted microphone is fixed in front of the driver and is used to collect the driver's voice data, and can upload the data to a server via Bluetooth / wifi / cellular data network,

[0022] Furthermore, the data processing module is used to process the acquired data and send it to different emotion recognition models to calculate the anger index respectively, and then use the hybrid fusion method to calculate the anger index and determine whether the anger index exceeds the threshold; the early warning module triggers the corresponding alarm according to the anger index and sends the information to the front end through WebSocket; the front-end display module can display the current anger index and historical trend chart; trigger the corresponding alarm according to the anger index, and send the information to the front end through WebSocket; when the anger index exceeds the threshold, the corresponding alarm information is displayed.

[0023] Furthermore, the voice reminder module generates a voice reminder and plays it through a speaker when the anger index exceeds a threshold; the notification module is used to send text messages and voice messages to notify fleet managers in serious situations; the user management module is used to manage and verify user identities to ensure that only authorized users can access the system; the log recording module is used to record the system's operating status and abnormal conditions for later analysis and maintenance; the data storage module is used to persistently store the anger index and other related data; the system configuration module: is used by the administrator to configure system parameters, such as alarm thresholds, notification methods, etc.

[0024] Compared with the prior art, the beneficial effects of the present invention are as follows: the system of the present invention can effectively reduce traffic accidents caused by emotional out-of-control by timely detecting and intervening in the driver's anger, thereby ensuring the safety of passengers and drivers; helping drivers to better manage their emotions, reduce work pressure, and improve work efficiency and quality of life; providing real-time emotion monitoring and early warning, which helps bus companies to timely detect the driver's psychological problems and take corresponding intervention measures to maintain the driver's mental health. Based on the above, the present invention has a good application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic block diagram of a multi-modal fusion driver anger index real-time detection method of the present invention.

[0026] Figure 2 A flowchart of the operation process of a multimodal fusion driver anger index real-time detection method.

[0027] Figure 3 An architecture diagram of a warning system for real-time detection of driver anger index using multimodal fusion. DETAILED DESCRIPTION

[0028] Figures 1 to 3As shown, a multi-modal fusion driver anger index real-time detection warning system includes a data acquisition module, a data processing module, a warning module, a front-end display module, a voice reminder module, a notification module, a user management module, a log recording module, a data storage module, and a system configuration module. The data acquisition module includes a smart bracelet, a vehicle-mounted high-definition camera, and a vehicle-mounted microphone; the smart bracelet, worn on the driver's wrist, is used to collect the driver's electrocardiogram data, and can upload the data to a server via Bluetooth / wifi / cellular data network; the high-definition camera, fixed in front of the driver, is used to collect the driver's facial expression data, and can upload the data to a server via Bluetooth / wifi / cellular data network; the vehicle-mounted microphone, fixed in front of the driver, is used to collect the driver's voice data, and can upload the data to a server via Bluetooth / wifi / cellular data network. The data processing module is used to process the acquired data and send it to different emotion recognition models to calculate the anger index respectively. Then, the hybrid fusion method is used to calculate the anger index and determine whether the anger index exceeds the threshold. The early warning module triggers the corresponding alarm according to the anger index and sends the information to the front end through WebSocket. The front-end display module can display the current anger index and historical trend chart; trigger the corresponding alarm according to the anger index and send the information to the front end through WebSocket; when the anger index exceeds the threshold, the corresponding alarm information is displayed. Voice reminder module, when the anger index exceeds the threshold, generates a voice reminder and plays it through the speaker; notification module, used to send text messages and voice messages to fleet managers in serious cases; user management module, used to manage and verify user identities to ensure that only authorized users can access the system, specifically using JWT (JSON Web Token) for identity authentication; logging module, used to record the system's operating status and abnormal conditions for later analysis and maintenance, specifically using ELK Stack (Elasticsearch, Logstash, Kibana) for logging and visualization; data storage module, used to persistently store the anger index and other related data, specifically using MySQL for persistent storage of the anger index and other related data; system configuration module: used by administrators to configure system parameters, such as alarm thresholds, notification methods, etc., specifically configured through the Web interface, and configuration information is stored in the database; logging module, records the system's operating status and abnormal conditions for later analysis and maintenance.

[0029] Figures 1 to 3As shown in the figure, a multimodal fusion driver anger index real-time detection method specifically includes the following steps: Step 1: Use a smart bracelet to collect the driver's heart rate (HR) and heart rate variability (HRV) information, and transmit the data to the vehicle-mounted device via Bluetooth or directly upload it to the cloud server; use a high-definition vehicle-mounted camera to capture the driver's facial expression, and transmit the video data to the cloud server via Wi-Fi or cellular network; use the microphone in the car to record the driver's voice features, including speech speed, pitch, and emotional vocabulary information, and transmit the audio data to the cloud server via Wi-Fi or cellular network. Step 2: Filter the ECG data, use a bandpass filter to remove baseline drift and high-frequency noise; perform denoising on the video data, and use a Gaussian filter to improve the image quality; preprocess the video data, use the OpenCV library for face detection, and use the dlib library for face alignment. Step 3: Use the BioSig toolbox to extract ECG data features. ECG data features include HRV indicators in the time domain and frequency domain. The time domain HRV indicators include SDNN (standard deviation of all normal RR intervals), RMSSD (root mean square of the difference between adjacent RR intervals), and pNN50 (the proportion of adjacent RR intervals with a difference greater than 50ms). The formula is as follows: where RR i is the i-th RR interval, is the average of all RR intervals, and N is the total number of RR intervals; Frequency domain HRV index analysis is done through power spectral density (PSD), the spectrum includes low frequency (LF) and high frequency (HF), and their ratio; low frequency refers to the power in the range of 0.04-0.15HZ, the formula is High frequency refers to the power in the range of 0.15-0.4HZ, and the formula is The power spectrum density is calculated using the periodogram method. The formula is Where X(f) is the Fourier transform of x[n]; Facial Action Coding System (FACS) is used for facial expression recognition. FACS decomposes facial expressions into a series of basic action units. Each AU represents a specific facial muscle movement. By detecting and analyzing AUs, the emotional state of the bus driver can be judged. The AUs extracted by OpenFace are used to construct feature vectors. The main 12 AUs related to anger are extracted: AU04: eyebrows raised, AU05: eyebrows pulled down, AU07: eyelids closed, AU10: upper lip raised, AU12: mouth corners raised, AU15: lips compressed, AU17: chin raised, AU23: mouth corners pulled down, AU24: mouth corners stretched to the sides, AU25: mouth corners raised and stretched, AU26: mouth open, AU27: mouth wide open. The Inception-V3 model is used to extract high-level image features and the output is a 2048-dimensional feature vector. The final feature vector will be composed of Inception-V3 features and AUs features, f = [Inception-V3 features, AUs features], with a total of 2060 dimensions. Use the Librosa library to extract Mel frequency cepstral coefficients (MFCCs). MFCCs can help distinguish different speech emotions. The formula is, Where t is the time frame, m is the index of the Mel filter bank, M is the number of Mel filter banks, and f(t) is the frequency. The YIN algorithm is used to extract the fundamental frequency (F0). A high fundamental frequency usually indicates excitement or anger. The formula is,

[0030] Where x(k) is the audio signal, N is the signal length, and τ is the delay. The DeepSpeech ASR model is used to convert the audio into text, and the BERT model is used to classify the text into sentiment. The formula is P(y|x)=softmax(W·h+b), where y is the sentiment category (anger, calm, etc.), x is the input text, h is the output representation of the BERT model, and w and b are the bias terms of the weight matrix. The BERT model can accurately identify the emotional tendency in the text, thereby assisting in judging the driver's anger level. After the extracted features are normalized, the acoustic features are and text features Directly concatenate together to form a multimodal feature vector X = [A; T], where the semicolon represents the concatenation of column vectors. The formula is, d X =d A +d T .

[0031] Figures 1 to 3As shown, step 4, model prediction, this step includes the following steps; ECG feature anger index prediction: use the DEAP dataset for training and testing, traverse the ECG signal and emotion label of each participant, standardize the extracted features to have zero mean and unit variance, use 80% of the data for training, and 20% of the data for testing. Use the random forest regressor to train the training set. After the training is completed, use the test set for prediction and evaluate the performance of the model. In the model optimization stage, grid search is used for hyperparameter tuning, and the generalization ability of the model is evaluated by K-fold cross validation. After the model is trained, the real-time data stream can be detected to score the anger level of the bus driver from 1-100. Facial expression feature anger index prediction: expand the FER2013 dataset, organize experienced facial expression recognition and annotation personnel, give the facial expressions in the image an anger score of 0-100, take the average after multiple rounds of annotation, and use the results of multiple annotation personnel for cross validation. After manually labeling some data, the labeled data is used for training to obtain a preliminary emotion recognition model. The model is then used to predict the unlabeled data, and the prediction results with high confidence are selected for automatic labeling. Finally, the automatically labeled data is added to the training set, and the model is retrained. After iteration, a satisfactory number and quality of labels are achieved. A multi-layer fully connected neural network is used to map the feature vector to the anger score. The fully connected layer configuration is: the first layer: 1024 nodes, the activation function is ReLU; the second layer: 512 nodes, the activation function is ReLU; the third layer: 256 nodes, the activation function is ReLU; the output layer: 1 node, the activation function is linear (for regression), and the output range is 0-100. The mean square error (MSE) is used as the loss function, the Adam optimizer is used for training, and the mean absolute error (MAE) is used as the evaluation indicator. After evaluating the model performance, the best model weights are saved, and the anger score result is finally obtained. Speech feature anger index prediction: Define an attention network to calculate the attention weight of the multimodal feature vector X. The attention network consists of a single-layer fully connected network and outputs a weight vector with the same dimension as the feature vector. is the weight matrix of the attention network, is the bias term, is the weight vector used to calculate the attention score, is the intermediate vector after the activation function, is the attention weight vector. Use the calculated attention weights to perform weighted fusion on the multimodal feature vector: weighted=α⊙X, where ⊙ represents element-by-element multiplication. The LSTM model is a special recurrent neural network (RNN) that can learn long-term dependencies and is suitable for processing time series data. The LSTM model is constructed and trained using the RAVDESS dataset. The state update of the LSTM unit is expressed as:

[0032] Input gate: i t =σ(W ii x t +b ii +W hi h t-1 +b hi )

[0033] Forget gate: f t =σ(W if x t +b if +W hf h t-1 +b hf )

[0034] Cell state: c t =f t ⊙c t-1 +i t ⊙tanh(W ic x t +b ic +W hc h t-1 +b hc )

[0035] Output gate: o t =σ(W io x t +b io +W ho h t-1 +b ho )

[0036] LSTM cell output: h t =o t ⊙tanh(c t )

[0037] Where: σ is the Sigmoid activation function, is the hyperbolic tangent activation function, ⊙ represents element-by-element multiplication, W and b are the weight matrix and bias term, x t is the input of the current time step, h t-1 is the hidden state of the previous time step, c t-1 is the cell state at the previous time step. The mean square error (MSE) is also used as the loss function to minimize the difference between the predicted anger score and the true label, and finally the anger score result is obtained.

[0038] Figures 1 to 3 As shown, step 5, data fusion, get ECG Y ECG , Facial expression Y face , Voice Y speech After the anger rating results were normalized, they were ensured to be on the same scale for each modality. y is the original score, μ is the mean score, σ is the standard deviation of the score, is the normalized score. Customize an attention layer to dynamically adjust the importance of each modality. The intermediate vector u = tanh(W a X+b a ), calculate the attention score e = w s u, calculate the attention weight α = softmax(e), and perform weighted fusion of the multimodal feature vector X weighted =α⊙X, and the weighted fused feature vector is also input into the LSTM model for training. After the LSTM model, the output of each modality is mapped to a common space, and the formula is, mapped =ReLU(W map X weighted +b map ), and finally fused through a multi-layer fully connected neural network, the formula is, h densel =ReLU(W densel h mapped +b densel ),h dense2 =ReLU(W dense2 h dense1 +b dense2 ) Where: W map ,W densel ,W dense2 ,W out is the weight matrix, b map ,b densel ,b dense2 ,b out is the bias term, X weighted is the weighted rating vector, is the final predicted anger score. Step 6, establish evaluation indicators, the anger level can be divided into the following levels, and establish a scoring system: 0-20: Very calm, description: The individual is in a completely relaxed and calm state, without any anger, action suggestion: No special attention, continue to maintain a good state. 21-40: Calm, description: The individual has slight mood swings, but remains calm overall, action suggestion: Observe and record emotional changes, and conduct psychological counseling if necessary. 41-60: Mild anger, description: The individual shows mild anger, which may be accompanied by slight physical tension and verbal dissatisfaction, action suggestion: Conduct emotional management training, provide psychological support, and monitor emotional changes. 61-80: Moderate anger, description: The individual shows obvious anger, which may be accompanied by physical movements and verbal aggression, action suggestion: Trigger initial warnings, conduct emergency intervention, including psychological counseling and behavioral guidance, and notify relevant personnel to pay attention to safety. 81-100: Severe anger, description: The individual is in an extremely angry state, which may be accompanied by violent tendencies and uncontrolled behaviors, action suggestion: Treat it as a serious situation and take immediate action. Initiate emergency plans to ensure personal safety, conduct emergency psychological intervention and behavior control. Step 7: Trigger corresponding alarms according to the anger index; set thresholds. When the anger index exceeds the threshold, trigger alarms and display the current anger index and historical trend charts. Send information to the front end via WebSocket, and the front end uses JavaScript and HTML5 technology to display data; when the anger index exceeds the threshold, generate voice reminders and play through the speaker, and use TTS (Text-to-Speech) technology to generate voice reminders; in serious cases, send SMS and voice messages to notify fleet managers, and use Twilio API to send SMS and voice messages.

[0039] Figure 1 , 2 As shown in Figure 3, through all the above technical solutions, the present invention adopts a multimodal fusion method based on the attention mechanism to comprehensively analyze the driver's emotional state through the ECG data collected by the smart bracelet, the facial expressions captured by the on-board camera, and the voice features recorded by the microphone in the car. This method can significantly improve the accuracy of emotion recognition, reaching a recognition accuracy of more than 95%, and can timely detect and intervene in the driver's anger; help drivers better manage their emotions, reduce work pressure, improve work efficiency and quality of life; provide real-time emotion monitoring and early warning, which helps bus companies to timely detect the driver's psychological problems and take corresponding intervention measures to maintain the driver's mental health, and effectively prevent traffic accidents caused by the driver's emotional out-of-control, thereby improving driving safety and passenger experience.

[0040] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It is obvious to those skilled in the art that the present invention is limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or basic features of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the equivalent elements of the claims are included in the present invention.

[0041] In addition, it should be understood that although the present specification is described according to the implementation mode, the implementation mode does not only include an independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in the embodiments can also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A multi-modal fusion driver anger index real-time detection method, characterized in that: The following steps are included: Step S1, data collection, (1) ECG data: using a smart bracelet to collect the driver’s heart rate and heart rate variability information; (2) Video data: using a high-definition vehicle camera to capture the driver’s facial expressions; (3) Audio data: using the in-car microphone to record the driver’s voice features, including speech speed, pitch, and emotional vocabulary information; Step S2, feature extraction, (1): ECG data processing uses the Biosig toolbox to extract time domain and frequency domain HRV indicators. The time domain HRV indicators are as follows: Standard deviation of all normal intervals: Specifically calculate the standard deviation of all RR intervals, the formula is, The root mean square of the difference between adjacent RR intervals: The root mean square of the difference between adjacent RR intervals is calculated using the formula: The proportion of the period difference between adjacent RRs greater than 50ms: The specific calculation of the proportion of the period difference between adjacent RRs greater than 50ms is as follows: where RR i is the i-th RR interval, is the average of all RR intervals, and N is the total number of RR intervals; Frequency domain HRV indicators are as follows: Power spectral density analysis: The power spectral density is calculated using the periodogram method. The formula is, where X(f) is the Fourier transform of x[n]; Low frequency: Specifically, it is the power in the range of 0.04-0.15Hz, the formula is, High frequency: Specifically, it is the power in the range of 0.15-0.4Hz, and the formula is, LF / HF ratio: specifically the ratio of low frequency to high frequency power; After obtaining the above indicators, the extracted features are standardized to have zero mean and unit variance, and then the DEAP dataset is used for training and testing. The ECG signals and emotion labels of each participant are traversed, and 80% of the data are used for training and 20% of the data are used for testing. The training set is then trained using a random forest regressor, hyperparameters are tuned through grid search, and the generalization ability of the model is evaluated through K-fold cross validation. After training, the real-time data stream is detected to score the anger level of bus drivers from 1-100. (2) Facial expression recognition, specifically including the following processes: Facial expression recognition is performed using the Facial Action Coding System, which decomposes facial expressions into a series of basic action units (AUs), each of which represents a specific facial muscle movement. Expand the FER2013 dataset, organize facial expression recognition annotators, assign anger scores of 0-100 to the facial expressions in the images, take the average value after multiple rounds of annotation, and use the results of multiple annotators for cross-validation; After manually labeling some data, the labeled data is used for training to obtain a preliminary emotion recognition model. The model is then used to predict the unlabeled data, and the prediction results with high confidence are selected for automatic labeling. Finally, the automatically labeled data is added to the training set and the model is retrained. After iteration, a satisfactory number and quality of labels are achieved. Use the Inception-V3 model to extract high-level image features and output a 2048-dimensional feature vector; Use OpenFace to extract 12 AUs related to anger and construct a feature vector: AU04: eyebrows raised, AU05: eyebrows lowered, AU07: eyelids closed, AU10: upper lip raised, AU12: mouth corners raised, AU15: lips compressed, AU17: chin raised, AU23: mouth corners lowered, AU24: mouth corners stretched to both sides, AU25: mouth corners raised and stretched, AU26: mouth open, AU27: mouth wide open. The final feature vector will be composed of Inception-V3 features and AUs features, f = [Inception-V3 features, AUs features], with a total of 2060 dimensions; A multi-layer fully connected neural network is used to map the feature vector to the anger score. The fully connected layer configuration is as follows: the first layer: 1024 nodes, the activation function is ReLU; the second layer: 512 nodes, the activation function is ReLU; The third layer: 256 nodes, the activation function is ReLU; Output layer: 1 node, activation function is linear, output range is 0-100; Use mean square error as the loss function, Adam optimizer for training, and mean absolute error (MAE) as the evaluation indicator; (3) Speech feature extraction and recognition. Since some emotional words may have different meanings in different contexts, speech is segmented into emotions and text to provide more accurate and comprehensive sentiment analysis results. The specific process includes the following: Use Librosa library to extract Mel frequency cepstral coefficients MFCCs. MFCCs can help distinguish different speech emotions. The formula is: Where t is the time frame, m is the index of the Mel filter bank, M is the number of Mel filter banks, and f(t) is the frequency; Use the YIN algorithm to extract the fundamental frequency F0. High fundamental frequency usually indicates excitement or anger. The formula is: Where x(k) is the audio signal, N is the signal length, and τ is the delay; The DeepSpeechASR model is used to convert audio into text, and the BERT model is used to classify the text sentiment. The formula is P(y|x)=softmax(W·h+b), where y is the sentiment category, x is the input text, h is the output representation of the BERT model, and w and b are the bias items of the weight matrix. The BERT model can accurately identify the sentiment tendency in the text, thereby assisting in judging the driver's anger level; an LSTM model is constructed and trained using the RAVDESS dataset. The state update of the LSTM unit is represented as follows: input gate: determines how much information of the current input needs to be updated to the cell state, forget gate: determines which information in the cell state needs to be discarded, cell state: the updated cell state, output gate: determines which information in the cell state needs to be output; Use mean square error (MSE) as the loss function to minimize the difference between the predicted anger score and the true label, and finally get the anger score result; Step S3: Feature fusion, (1) Determine the fusion strategy and adopt hybrid fusion to combine the advantages of early fusion and late fusion. It can not only capture the correlation information across modalities, but also independently process the characteristics of each modality, thereby improving the generalization ability and robustness of the model; (2) The specific fusion method is: Early fusion: concatenate the original features of different modalities to form a high-dimensional feature vector; Late fusion: Process the features of each modality separately to obtain their respective anger scores, and then calculate the attention weight of each modality. The attention weight calculation formula is: Among them, e i is the attention score of the ith modality, which can be calculated by a small neural network and the final anger index is obtained by weighted summation. Among them, Si is the anger score of the i-th mode; Step S4: Establish evaluation indicators, specifically divide the anger level into the following levels, and establish a scoring system: 0-20: Very calm, the individual is in a completely relaxed and calm state, without any anger; 21-40: Calm, the individual has slight mood swings but remains calm overall; 41-60: Mild anger, the individual shows mild anger, which may be accompanied by slight physical tension and verbal dissatisfaction; 61-80: Moderate anger, where the individual displays obvious anger, which may be accompanied by physical and verbal aggression; 81-100: Severe Anger: The individual is in a state of extreme anger, which may be accompanied by violent tendencies and uncontrolled behavior.

2. A multi-modal fusion driver anger index real-time detection method according to claim 1, characterized in that: The hybrid fusion strategy combines the advantages of early fusion and late fusion, which can not only capture the correlation information across modalities, but also process the characteristics of each modality independently, thereby improving the generalization ability and robustness of the model.

3. The multimodal fusion driver anger index real-time detection method according to claim 1 is characterized in that: The anger score calculation method after multimodal data fusion is as follows: calculate the attention weight of each modality and obtain the final anger index through weighted summation.

4. A multi-modal fusion driver anger index real-time detection method according to claim 1, characterized in that: The random forest regressor used in ECG data processing performs hyperparameter tuning through grid search, and the generalization ability of the model is evaluated through K-fold cross validation, thereby improving the accuracy and stability of the model.

5. A multi-modal fusion driver anger index real-time detection method according to claim 1, characterized in that: The specific structure of the multi-layer fully connected neural network used in facial expression recognition is: The first layer has 1024 nodes and the activation function is ReLU; The second layer has 512 nodes and the activation function is ReLU; The third layer has 256 nodes and the activation function is ReLU; The output layer has 1 node, the activation function is linear, and the output range is 0-100.

6. A multi-modal fusion driver anger index real-time detection method according to claim 1, characterized in that: The LSTM model used in speech feature extraction and recognition is able to learn long-term dependencies and is suitable for processing time series data. It is trained using the RAVDESS dataset and uses the mean square error (MSE) as the loss function to minimize the difference between the predicted anger score and the true label.

7. A multi-modal fusion driver anger index real-time detection method according to claim 1, characterized in that: The anger degree in the evaluation index establishment is divided into five levels, and a corresponding scoring system and action suggestions are established to achieve accurate assessment of the driver's emotional state and timely intervention.

8. A multi-modal fusion driver anger index real-time detection warning system, characterized in that: Includes the following modules: Module (1): Data acquisition module, which collects raw data from smart bracelets, high-definition cameras, and microphones, and uploads the data to the server via Bluetooth / WiFi / cellular networks; Module (2): Data processing module, which receives data from sensors, performs filtering, noise reduction, image processing, and extracts main features; uses the trained machine learning model to calculate the anger score, and fuses the anger scores of different modalities to obtain the final anger index; In the data processing module: Filtering: Filtering the ECG data to remove noise; Noise reduction is to reduce the noise of video data to improve image quality; Image processing is the preprocessing of video data, including face detection and alignment; Feature extraction includes the following functions: ECG data: extract features such as SDNN, RMSSD, pNN50, LF, and HF; Facial expression: extract AUs features; Speech features: extract speech speed, pitch, and emotional vocabulary features; feature Standardization: The extracted features are standardized to have zero mean and unit variance; Model prediction: Use the trained random forest regressor, multi-layer fully connected neural network and LSTM model to predict ECG data, facial expressions and voice features respectively to obtain the anger score of each modality; Multimodal fusion: The anger scores of different modalities are fused through the attention mechanism to obtain the final anger index; Module (3): Early warning module, which triggers corresponding alarms according to the anger index and sends the information to the front end through WebSocket; Module (4): front-end display module, which is used to display the current anger index and historical trend chart; trigger corresponding alarms according to the anger index, and send information to the front-end through WebSocket; when the anger index exceeds the threshold, display the corresponding alarm information; Module (5): a voice reminder module, which generates a voice reminder and plays it through a speaker when the anger index exceeds a threshold; Module (6): Notification module, which is used to send SMS or voice messages to fleet managers in serious situations; Module (7): User management module, used to manage and verify user identities to ensure that only authorized users can access the system; Module (8): Log recording module, which records the system's operating status and abnormal conditions for later analysis and maintenance; Module (9): data storage module, which persistently stores anger index and other related data; Module (10): System configuration module: allows administrators to configure system parameters such as alarm thresholds, notification methods, etc.

9. The multi-modal fusion driver anger index real-time detection early warning system according to claim 8 is characterized in that: The multimodal data fusion strategy in the data processing module is specifically hybrid fusion.

Images