A driver emotion collaborative decision method and system in a driving scene

Abstract

The present application relates to the technical field of driver emotion recognition, in particular to a driver emotion collaborative decision-making method and system in a driving scene. The present application first collects four modal data of driver face, voice, body and vehicle CAN bus, pre-processes and aligns them into synchronous multi-modal feature sequences according to time stamps, then extracts deep features of each modal and maps them into valence and / or arousal probability distribution; collects modal quantitative indicators, converts them through reliability scoring, outputs real-time dynamic weights through Softmax normalization, extends each modal probability distribution into a two-dimensional matrix, globally normalizes the weighted sum to obtain a two-dimensional joint probability distribution, finally inputs the distribution sequence of continuous time steps into a time series deep learning model, and outputs an emotion trajectory. With multi-modal avoiding single-modal environmental interference, dynamic weights realizing scene adaptation, two-dimensional fusion improving quantitative precision, and time series modeling capturing emotion time evolution, the present application effectively improves the accuracy of emotion determination.

Description

Technical Field

[0001] This invention relates to the field of driver emotion recognition technology, specifically a driver emotion collaborative decision-making method and system in driving scenarios. Background Technology

[0002] A driver's emotional state directly affects driving behavior and road safety. The valence-arousal two-dimensional emotional space has become the mainstream framework for quantitative analysis of driver emotions because it can quantify the positive and negative tendencies and activation levels of emotions in a refined manner.

[0003] Existing driver valence-arousal emotion analysis methods are insufficient to meet the practical application needs of intelligent driving. Their core shortcomings are: First, they rely heavily on single-modal data, making them susceptible to interference from factors such as insufficient lighting, in-vehicle noise, and limb obstruction in driving scenarios, resulting in poor reliability of emotion quantification results. Second, some multimodal fusion methods use fixed weights, failing to adapt to the varying reliability differences of different modalities under different driving scenarios, leading to poor fusion effects. Third, they focus on the static quantification of emotions at a single moment, lacking temporal modeling capabilities, failing to capture the temporal evolution of emotions, construct continuous valence-arousal emotion trajectories, or detect and predict emotional turning points in the short term, thus hindering the provision of forward-looking decision-making basis for intelligent driving safety interventions. Fourth, the multimodal probabilistic fusion design is crude, with insufficient mapping and connection to the valence-arousal two-dimensional space, resulting in low precision of quantification results, making it difficult to serve as an effective basis for subsequent qualitative emotion determination.

[0004] The aforementioned issues prevent existing methods from achieving real-time, accurate quantification and trajectory prediction of driver emotions in complex driving scenarios such as nighttime, traffic congestion, and highways, thus hindering the practical application of intelligent driving safety intervention systems. Therefore, there is an urgent need to develop a driver valence-arousal emotion trajectory prediction method that adapts to complex driving scenarios and combines multimodal fusion and temporal modeling to improve the accuracy, robustness, and foresight of emotion quantification. Summary of the Invention

[0005] To address the shortcomings of existing methods in complex driving scenarios, such as susceptibility to interference in single-modal approaches, lack of adaptive capabilities in multimodal fusion, and absence of temporal modeling, which hinder accurate quantification and prediction of driver valence-arousal emotional trajectories, this invention provides a driver valence-arousal emotional trajectory prediction method based on multimodal fusion and temporal modeling. Based on this emotional trajectory prediction method, this invention also provides a method for qualitatively determining the driver's current-moment emotion based on heterogeneous graph subgraph matching and a driver emotion collaborative analysis system.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A driver valence-arousal emotion trajectory prediction method based on multimodal fusion and temporal modeling includes the following prediction steps: The system collects driver facial video data, voice signal data, body movement video data, and vehicle CAN bus data, and aligns them according to timestamps after preprocessing to form a synchronous multimodal feature sequence. The single-modal deep features of each modality in the synchronous multimodal feature sequence are extracted and mapped to the probability distribution of valence and / or arousal dimension, respectively. The quantitative indicators of each modality are collected, and after reliability scoring and Softmax normalization, the real-time dynamic weights are output. The probability distribution of each modality is extended into a two-dimensional distribution matrix. After being weighted and summed according to real-time dynamic weights, and then normalized globally, the two-dimensional joint probability distribution of valence and arousal at the current moment is obtained. The sequence, consisting of the two-dimensional joint probability distribution of valence and arousal corresponding to each continuous time step, is input into the temporal deep learning model, and the output is the emotion trajectory of the joint probability distribution of valence and arousal.

[0007] As a further improvement to the above scheme: the facial modality is mapped to a two-dimensional probability distribution of valence and arousal using a lightweight convolutional neural network; the speech modality is mapped to a two-dimensional probability distribution of valence and arousal using an LSTM network; the limb modality is mapped to an arousal distribution with a uniform valence using a GCN; and the vehicle modality is mapped to an arousal distribution with a non-uniform valence using an MLP.

[0008] As a further improvement to the above scheme: the quantitative indicators include the illumination intensity of the facial modality, the signal-to-noise ratio of the speech modality, the range of limb movements of the body modality, and the intensity of driving events of the vehicle modality; the reliability score conversion adopts a piecewise linear mapping function, and the quantitative indicator value x is lower than the preset low threshold T. low The time score is 0, which is higher than the preset high threshold T. high When the score is 1, and it falls between the two, press (xT) low ) / (T high -T low The real-time dynamic weights are calculated and output after being smoothed by a weighted average of the weights of the current frame and the previous 3 frames. The smoothing coefficients are 0.7 for the current frame and 0.3 for the average of the previous 3 frames.

[0009] As a further improvement to the above scheme, the extended rule of the two-dimensional distribution matrix is: two-dimensional unit probability value = valence single-interval probability × arousal single-interval probability.

[0010] As a further improvement to the above scheme: multiply each element of the two-dimensional distribution matrix of each modality at the current moment by the corresponding real-time dynamic weight, and then sum the probability values ​​of the same two-dimensional unit to obtain the initial joint probability distribution matrix; correct the values ​​less than 0 in the initial joint probability distribution matrix to 0 and the values ​​greater than 1 to 1, and then achieve global normalization by dividing all elements in the initial joint probability distribution matrix by the total probability sum, thereby obtaining the valence-arousal two-dimensional joint probability distribution at the current moment.

[0011] A method for qualitatively determining a driver's current emotion based on heterogeneous graph subgraph matching includes the following determination steps: Based on the current moment valence-arousal two-dimensional joint probability distribution and synchronous multimodal feature sequence output by the driver valence-arousal emotion trajectory prediction method based on multimodal fusion and temporal modeling, a heterogeneous behavior flowchart containing modal feature nodes and emotion feature nodes is constructed. Retrieve the pre-trained emotion behavior graph knowledge graph, split it into multiple independent typical emotion subgraphs with emotion labels, and analyze the valence-arousal reference probability distribution and node and edge features of the independent typical emotion subgraphs. Extract the core emotion-related subgraphs from the heterogeneous behavior flowcharts and perform node-edge alignment operations with each typical emotion subgraph. The structural similarity between the core emotion-related subgraph and each typical emotion subgraph is calculated, as well as the emotion similarity between the joint probability distribution of actual valence-arousal corresponding to the core emotion-related subgraph and the reference probability distribution of the typical emotion subgraph. The comprehensive similarity is obtained by weighting according to the preset weights. The emotion label corresponding to the typical emotion subgraph that ranks first and meets the threshold requirement is taken as the emotion judgment result; otherwise, it is judged as an unknown emotion.

[0012] As a further improvement to the above scheme: the construction of the heterogeneous behavior flowchart includes node extraction and attribute assignment, as well as edge construction and attribute assignment; Node extraction and attribute assignment: Four modal feature nodes (facial, speech, body, and vehicle) are extracted from the synchronous multimodal feature sequence, and emotion feature nodes are extracted from the two-dimensional joint probability distribution of valence and arousal. Quantitative features and confidence scores are assigned to each node. Facial nodes correspond to AU action units, speech nodes correspond to MFCC features and signal-to-noise ratio, body nodes correspond to the variance of skeletal keypoint movement and amplitude of movement, vehicle nodes correspond to driving event type and intensity, and emotion feature nodes correspond to the joint probability distribution matrix, peak interval, and peak probability value, and are assigned a unique identifier. Edge construction and attribute assignment: Based on the spatiotemporal and causal relationships of four modalities (face, voice, body, and vehicle), temporal, spatial, and causal relationship edges are constructed, and the relationship type, association weight, and corresponding feature parameters are labeled respectively.

[0013] As a further improvement to the above scheme: taking the emotion feature node in the heterogeneous behavior flowchart as the core, all modal feature nodes that have direct edge connections with the emotion feature node and have an association weight ≥ 0.5 are selected. At the same time, the effective edges between these modal feature nodes are extracted, and redundant nodes and edges that are not related to the emotion feature node are removed to form a core emotion-related subgraph that only contains the core emotion-related features.

[0014] As a further improvement to the above scheme: structural similarity is obtained by weighted summation of cosine similarity of node degree distribution, mean Euclidean distance of node attributes, and edge comprehensive similarity; emotional similarity... The calculation method is as follows =1-JS(P real , ), P real For the actual joint probability distribution, Using the probability distribution as a reference, JS is the divergence function; the overall similarity is... The calculation formula is: α and β are the corresponding weighting coefficients.

[0015] A driver emotion collaborative analysis system is provided for executing the aforementioned driver valence-arousal emotion trajectory prediction method based on multimodal fusion and temporal modeling, and the aforementioned driver current-moment emotion qualitative determination method based on heterogeneous graph subgraph matching. The system includes a data acquisition module, an emotion trajectory prediction module, and an emotion qualitative determination module. The data acquisition module collects driver facial video, voice signals, body movement video, and vehicle CAN bus data. The emotion trajectory prediction module outputs a two-dimensional joint probability distribution of valence-arousal and the emotion trajectory based on the collected data. The emotion qualitative determination module performs heterogeneous graph subgraph matching based on the output of the emotion trajectory prediction module and outputs the emotion determination result.

[0016] Compared with the prior art, the beneficial effects of the present invention are: 1. The emotion trajectory prediction method in this invention is based on four modalities of data: facial, voice, body, and vehicle. It extracts deep features from each modality and maps them to valence and / or arousal probability distributions, thus avoiding the problem of single-modal data being easily affected by the driving environment. At the same time, it combines modality quantification indicators to realize dynamic weight calculation, enabling multimodal fusion to have context-adaptive capabilities. Furthermore, through two-dimensional distribution matrix expansion, weighted summation, and global normalization, it achieves refined probability fusion in the valence-arousal two-dimensional space, improving the accuracy of emotion quantification. Finally, the joint probability distribution sequence of continuous time steps is input into a temporal deep learning model to capture the temporal evolution of emotions and output continuous emotion trajectories. This represents a breakthrough from static quantification at a single moment to dynamic trajectory prediction over all time periods. It not only reflects the continuous changes in the driver's emotions but also provides accurate and reliable quantitative basis for subsequent qualitative judgment of emotions. The overall solution is adaptable to complex driving situations and has both robustness and foresight.

[0017] 2. The qualitative emotion determination method of this invention is closely integrated with the quantitative output of the emotion trajectory prediction method. It constructs a heterogeneous behavior flowchart based on the two-dimensional joint probability distribution of valence-arousal and the synchronous multimodal feature sequence obtained from the emotion trajectory prediction method, achieving precise integration of multimodal emotion features and quantitative results. By extracting core emotion-related subgraphs and eliminating redundant information, it simplifies computational complexity while focusing on core emotion-related features, improving matching efficiency. The method employs a precise one-to-one node-edge alignment to ensure the rigor of subgraph matching. Combining structural similarity and emotional similarity in dual-dimensional calculation and weighting, it obtains a comprehensive similarity, taking into account both the structural association of multimodal features and the quantitative matching of valence-arousal, significantly improving the accuracy of qualitative emotion determination. Finally, it outputs emotion tags based on the comprehensive similarity ranking, with clear determination logic and definite results. Together with the emotion trajectory prediction method, it forms a complete technical closed loop of "quantitative prediction-qualitative determination," transforming abstract emotion quantification results into intuitive emotion tags, adapting to the practical application needs of intelligent driving safety intervention. Attached Figure Description

[0018] Figure 1 This is a flowchart of an emotion trajectory prediction method.

[0019] Figure 2 This is a flowchart of an emotion trajectory prediction method. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see Figure 1 and Figure 2 This embodiment discloses a driver emotion collaborative decision-making method in driving scenarios (comprising a driver valence-arousal emotion trajectory prediction method based on multimodal fusion and temporal modeling, and a driver current-moment emotion qualitative judgment method based on heterogeneous graph subgraph matching). Based on the valence-arousal two-dimensional emotion space, through the collection, processing, fusion, and modeling of multimodal data, it achieves dynamic quantitative prediction of driver emotion trajectories in complex driving scenarios, and completes the qualitative judgment of current-moment emotion based on heterogeneous graph subgraph matching. The method is divided into two main parts: emotion trajectory prediction and current-moment emotion prediction. The steps within each part are executed sequentially, with real-time data synchronization and layer-by-layer integration. All operations are performed online in real-time on the vehicle terminal, adapting to various complex driving scenarios such as nighttime, noise, and obstruction, and possessing both prediction accuracy, stability, and robustness.

[0022] I. Emotional Trajectory Prediction

[0023] Emotion trajectory prediction transforms raw multimodal data into a continuous, smooth two-dimensional joint probability distribution of valence and arousal, while simultaneously detecting and predicting emotional turning points in the short term. It does not directly output specific emotion labels, but rather provides quantitative evidence for subsequent qualitative judgment. This part sequentially executes five steps: multimodal data acquisition and preprocessing, unimodal feature extraction and valence-arousal mapping, context-adaptive weight calculation, construction of the two-dimensional joint probability distribution of valence and arousal, and temporal context modeling. The input to each step is the output of the preceding step, and the overall execution frequency is consistent with the frequency of in-vehicle multimodal data acquisition.

[0024] (I) Multimodal data acquisition and preprocessing

[0025] 1. Multimodal data acquisition

[0026] Four types of core data are collected through onboard hardware: driver facial video data, voice signal data, body movement video data, and vehicle CAN bus data. The vehicle CAN bus data includes vehicle operation-related parameters such as vehicle speed, braking, accelerator, steering angle, and emergency acceleration / braking trigger signals.

[0027] 2. Modal preprocessing

[0028] The collected raw data undergoes targeted preprocessing. Facial and limb motion videos are subjected to frame extraction and image enhancement. Facial videos undergo additional face alignment processing, while limb motion videos undergo additional motion capture of key points of the human skeleton. Speech signal data is subjected to frame segmentation and noise reduction, and acoustic features such as MFCC, fundamental frequency, and energy are extracted. Vehicle CAN bus data is used to extract driving event features such as rapid acceleration, emergency braking, and steering angular velocity.

[0029] 3. Time axis synchronization

[0030] All preprocessed multimodal feature data are precisely aligned according to timestamps to eliminate time differences between modal data, forming a synchronous multimodal feature sequence with a unified time dimension, which serves as the input basis for subsequent steps.

[0031] (II) Unimodal Feature Extraction and Valence-Arousal Mapping

[0032] Based on the data characteristics and emotional expression capabilities of each modality, a dedicated feature extraction network is matched for four modalities: face, voice, body, and vehicle. This completes the extraction of single-modality deep features and maps the single-modality deep features to probability distributions in either the valence or arousal dimension. The face and voice modalities output a two-dimensional probability distribution, while the body and vehicle modalities use the arousal probability distribution as the core, with the valence dimension as an auxiliary or uniform distribution. All probability distributions meet the normalization requirement, i.e., the sum of the probabilities of all intervals within the distribution is 1.

[0033] 1. Facial expression feature extraction and mapping

[0034] A lightweight convolutional neural network (such as EfficientFace) is used to extract facial action units (such as AU4 frowning, AU12 smiling, AU43 eye closing, etc., typical emotion-related action units in driving scenarios) and overall expression features. After L2 normalization, the extracted features are input into two independent fully connected layer branches, which are mapped to valence probability distributions respectively. and arousal probability distribution .

[0035] For example, when driver facial AU4 (frowning) is detected to be activated but AU12 (smiling) is not activated, It will exhibit a distribution characteristic of "0.8 probability in extremely negative intervals, 0.15 probability in negative intervals, and 0.05 probability in other intervals". It exhibits a distribution characteristic of "0.7 probability in the high arousal interval, 0.25 probability in the medium arousal interval, and 0.05 probability in the other intervals".

[0036] (2) Speech feature extraction and mapping

[0037] The preprocessed speech acoustic features (such as MFCC, fundamental frequency, and energy) are processed using a sliding window method to form a continuous speech temporal feature sequence. This sequence is then input into an LSTM network to mine temporal-dependent features (such as the trend of speech intonation changing from flat to abrupt). The hidden layer output of the last time step of the network is extracted as the speech temporal depth feature. After L2 normalization, these features are input into two independent fully connected layer branches, which are mapped to valence probability distributions, respectively. and arousal probability distribution .

[0038] For example, when a driver complains in a rapid and loud voice due to poor road conditions, The probability of the extremely negative interval can reach 0.9. The probability of reaching an extremely high arousal level can reach 0.85; while when the driver is communicating smoothly, The probability of the neutral interval is 0.8. The probability of the mid-wake interval is 0.8.

[0039] 3. Limb movement feature extraction and mapping

[0040] Using key points of the human skeleton (such as multiple core key points like the shoulder, elbow, wrist, and waist) as nodes and physiological connections between key points (such as shoulder-elbow and elbow-wrist) as edges, a dynamic graph structure of the human skeleton is constructed. The input graph convolutional network (GCN) extracts the spatial association and dynamic change features of limb movements. After global pooling and L2 normalization, the input is a fully connected layer branch, which is mapped to an arousal probability distribution. The valence dimension is divided into intervals and generated into uniform distributions with equal probability (e.g., 5 intervals generate an equal probability of 0.2), which are marked as having no information contribution.

[0041] For example, when a driver tenses their limbs and grips the steering wheel tightly due to an emergency maneuver, The probability of extremely high arousal intervals was 0.8, while the combined probability of the remaining intervals was 0.2; however, when the driver relaxed while driving, The probability of low arousal intervals is 0.75, and the valence dimension maintains a uniform distribution of 0.2 across all intervals.

[0042] 4. Vehicle Information Feature Extraction and Mapping

[0043] Driving event features (such as sudden braking, sudden acceleration, steering angular velocity, vehicle speed fluctuations, etc.) are concatenated into an original feature vector, which is then processed by max-min normalization and input into a lightweight multilayer perceptron (MLP) to extract deep features of driving events. These features are then L2 normalized and input into a fully connected layer branch, where they are mapped to an arousal probability distribution. Based on prior knowledge of driving scenarios, we formulate driving event-valence tendency mapping rules (such as emergency braking events corresponding to negative valence tendency, and steady driving at a constant speed corresponding to neutral valence tendency), and transform the tendency weights into a non-uniform valence distribution, which are then marked as auxiliary information contributions.

[0044] For example, when the vehicle detects an emergency braking event (brake pedal pressure > 80%), The probability of high arousal interval is 0.9, and the valence distribution shows the characteristics of "0.7 in the extremely negative interval, 0.2 in the negative interval, and 0.1 in the remaining intervals"; when the vehicle is traveling at a constant and stable speed (speed fluctuation < 5km / h), The probability of the low arousal interval is 0.8, the probability of the neutral interval of the valence distribution is 0.8, and the total probability of the other intervals is 0.2.

[0045] 5. Results Integration

[0046] The valence and arousal probability distribution results of the four modalities are integrated by timestamp to form a time-synchronized single-modal probability distribution sequence. After outlier filtering, the sequence is stored according to modality.

[0047] For example, the valence and arousal probability distribution results of the four modalities are integrated using a unified 100ms timestamp to form a time-synchronized single-modal probability distribution sequence, such as "at t=100ms, facial..." ,voice limbs ,vehicle After filtering outliers (such as probability values ​​<0 or >1), they are stored according to modality.

[0048] (III) Scenario-Adaptive Weight Calculation

[0049] A real-time environment perception module is constructed. By using quantitative indicators that are strongly bound to the reliability of each modality, the feature extraction quality of each modality under the current driving scenario is evaluated. The normalized real-time dynamic weights are calculated and output. The sum of the weights is 1, so that the high reliability modality dominates the subsequent multimodal fusion and the low reliability modality is weakened. At the same time, a threshold triggering mechanism is set to realize the dynamic refresh of the weights.

[0050] 1. Initialization of the real-time environment perception module

[0051] The module establishes a one-to-one correspondence between loading modal-reliability metrics, including low, high, and refresh thresholds for each metric. For example, facial illumination intensity has a low threshold of 50 lux, a high threshold of 200 lux, and a refresh threshold of 20 lux; speech signal-to-noise ratio has a low threshold of 0 dB, a high threshold of 20 dB, and a refresh threshold of 10 dB. It also presets basic priority weights for each modality, such as face 0.4, speech 0.3, limbs 0.2, and vehicle 0.1. Finally, it initializes a weight buffer queue (e.g., with a capacity of 10 frames) and connects it to the weight output channel for subsequent steps, completing the basic module configuration.

[0052] 2. Real-time indicator collection

[0053] Reliability quantification metrics for each modality are collected at a preset execution frequency (e.g., 10Hz). For the facial modality, this is measured by illumination intensity; for the speech modality, by signal-to-noise ratio (SNR); for the limb modality, by the amplitude of limb movements; and for the vehicle modality, by the intensity of driving events. For example, at a given timestamp, the facial illumination intensity is 30 lux, the speech SNR is 25 dB, the limb movement amplitude has a variance of 600 pixels, and the vehicle driving event intensity is 0.9. All metrics are bound to this unified timestamp, achieving precise time alignment with the single-modality probability distribution sequence.

[0054] 3. Verification of indicator validity

[0055] The collected indicator values ​​undergo range and mutation checks to filter outliers. Range checks determine whether an indicator is within a preset value range. For example, if a light intensity of -5 lux or a signal-to-noise ratio of 50 dB exceeds a reasonable range, it is considered abnormal. Single-indicator anomalies are replaced with the average of the last three frames. For instance, if the variance of a body movement amplitude is 1200 pixels (exceeding a high threshold), it is replaced with the average of the variances of the last three frames (550, 600, and 580 pixels), which is 577. Multiple indicator anomalies (≥2 modal indicators) trigger a fallback mechanism, directly activating the basic priority weights of 0.4 for face, 0.3 for voice, 0.2 for body, and 0.1 for vehicle.

[0056] 4. Reliability score conversion

[0057] A piecewise linear mapping function is used to transform the validated valid index values ​​into raw reliability scores in the [0,1] interval. Index values ​​below a low threshold T are considered valid. low The original reliability score is 0, which is higher than the high threshold T. high The original reliability score is 1; scores between these two are calculated using the formula Score=(xT) low ) / (T high -T lowThe calculation is as follows: For example, after verification, the facial illumination intensity is 100 lux, and the original reliability score is (100-50) / (200-50)≈0.33; the voice SNR is 25dB, which is higher than the high threshold of 20dB, and the original reliability score is 1; the body movement amplitude is 577 pixels with variance, and the original reliability score is (577-100) / (1000-100)≈0.53; the vehicle driving event intensity is 0.9, which is higher than the high threshold of 0.8, and the original reliability score is 1.

[0058] (5) Calculate the original weight scores

[0059] The base score is obtained by multiplying the original reliability score by the modality's basic priority weight. Then, a scenario fine-tuning coefficient is introduced to meet the specific needs of driving scenarios. For example, the facial fine-tuning coefficient is 0.8 and the vehicle fine-tuning coefficient is 1.5 in nighttime scenarios, the vehicle fine-tuning coefficient is 1.5 in highway scenarios, and the voice fine-tuning coefficient is 1.2 in congested scenarios. The base score is adjusted by the coefficient to obtain the unnormalized original weight score.

[0060] For example, in a nighttime scene, the original reliability score for a face is 0 × the basic priority weight 0.4 = 0. After multiplying by the fine-tuning coefficient 0.8, the original weight score for the face is 0. The original reliability score for a voice is 1 × the basic priority weight 0.3 = 0.3. After multiplying by the fine-tuning coefficient 1.0, the original weight score for the voice is 0.3. The original reliability score for a limb is 0.53 × the basic priority weight 0.2 = 0.106. After multiplying by the fine-tuning coefficient 1.1, the original weight score for the limb is 0.117. The original reliability score for a vehicle is 1 × the basic priority weight 0.1 = 0.1. After multiplying by the fine-tuning coefficient 1.5, the original weight score for the vehicle is 0.15.

[0061] 6. Softmax normalization

[0062] Input the original weight scores of all modalities into the Softmax function, and then... Complete normalization processing (RScore) i RScore is the original weight score for the i-th mode. j (where the original weight score is the j-th mode), the real-time dynamic weights w for each mode are obtained. face w voice w body w vehicle Ensure that the sum of the weights is 1. Perform weighted average smoothing on the real-time dynamic weights (e.g., the weight of the current frame accounts for 0.7, and the average weight of the previous 3 frames accounts for 0.3) to avoid sudden changes in weights.

[0063] For example, using the original weight scores of 0, 0.3, 0.117, and 0.15 for the nighttime scene mentioned above, e 0 =1、e 0.3 ≈1.3499, e0.117 ≈1.124, e 0.15 ≈1.1618, summing gives 4.6357; therefore, the real-time dynamic weight is: w face =1 / 4.6357≈0.216、w voice =1.3499 / 4.6357≈0.291、w body =1.124 / 4.6357≈0.242、w vehicle =1.1618 / 4.6357≈0.251, the weighted sum is 1.

[0064] 7. Dynamic weight refresh

[0065] The system monitors changes in metric values ​​in real time. If the absolute value of the difference between any modal metric value and the previous frame exceeds the refresh threshold, the entire process of metric acquisition and normalization is immediately re-executed, and all weights are refreshed. If no refresh is triggered, the current weights remain unchanged. For example, the refresh threshold for facial illumination intensity is 20 lux. If the previous frame's illumination intensity was 50 lux and the current frame's was 25 lux, the difference 25 > 20, triggering a full-process weight refresh. The refresh threshold for voice SNR is 10 dB. If the previous frame's was 22 dB and the current frame's was 18 dB, the difference 4 < 10, so no refresh is triggered. Similarly, if the intensity of a vehicle driving event was 0.2 in the previous frame and 0.6 in the current frame, the difference 0.4 > the refresh threshold of 0.3, triggering a weight refresh.

[0066] 8. Weight Output and Storage

[0067] The real-time dynamic weights of each modality are output to the joint probability distribution construction module below in a preset format (such as JSON format). At the same time, the weights are bound to the corresponding timestamps and stored in the weight cache queue. The data of the last 10 or last 30 frames are retained for anomaly backtracking. If the emotion fusion result is found to be abnormal at a certain moment, the changes in indicators and weights at that moment and the previous few frames can be backtracked through the cache queue to locate the cause of the problem.

[0068] (iv) Constructing a two-dimensional joint probability distribution of valence and arousal

[0069] The unimodal valence-arousal probability distribution is fused with real-time dynamic weights. The valence-arousal two-dimensional joint probability distribution at the current moment is obtained by expanding the single-dimensional distribution into two dimensions and weighting and summing. This joint probability distribution comprehensively describes the driver's current emotional state while retaining the confidence and contribution of each modality. All probability calculations meet the normalization requirements.

[0070] 1. Initialization before fusion

[0071] The discrete interval partitioning rules for valence and arousal are loaded. Valence is typically divided into 5 intervals (V1 extremely negative, V2 negative, V3 neutral, V4 positive, V5 extremely positive), and arousal is divided into 5 intervals (A1 extremely low, A2 low, A3 medium, A4 high, A5 extremely high). A 5×5 two-dimensional probability grid is constructed, and the zero matrix of the two-dimensional probability grid is initialized and used as the joint probability matrix. The single-modal probability distribution of the current timestamp and the real-time dynamic weights normalized by Softmax are extracted. The sum of the weights is checked to see if it is 1 (error ≤ 0.005). If not, the weight results are retrieved again.

[0072] For example, load the interval rules of valence V1[-1,-0.6), V2[-0.6,-0.2), V3[-0.2,0.2), V4[0.2,0.6), V5[0.6,1], and arousal A1[0,0.2), A2[0.2,0.4), A3[0.4,0.6), A4[0.6,0.8), A5[0.8,1], initialize the zero matrix of a 5×5 two-dimensional probability grid; extract the real-time dynamic weights wface=0.220, wvoice=0.291, wbody=0.242, wvehicle=0.251 at the current moment, the weight sum is 1.004, the error is ≤0.005, and it is determined to be a valid weight.

[0073] 2. Single-modal two-dimensional distribution extension

[0074] All single-modal probability distributions are uniformly extended into a two-dimensional probability distribution matrix that matches a 5×5 two-dimensional grid. The extension rule is: two-dimensional unit probability value = valence single-interval probability × arousal single-interval probability. Facial and voice modalities are directly extended according to the two-dimensional distribution, while limb and vehicle modalities are extended according to their own arousal and valence distributions. All two-dimensional distribution matrices are standardized and verified to ensure that the total probability sum is 1.

[0075] For example, facial modal valence distribution Arousal distribution Therefore, the probability value of the two-dimensional element (V1, A4) is 0.8 × 0.7 = 0.56, and the probability value of (V2, A3) is 0.15 × 0.25 = 0.0375. The sum of the probabilities of all elements is 1, and the standardization verification passes.

[0076] 3. Weighted fusion

[0077] Multiply each element of the two-dimensional probability distribution matrix of each mode by the real-time dynamic weight of the corresponding mode to obtain the weighted two-dimensional distribution matrix of each mode; perform element-wise weighted summation on all weighted two-dimensional distribution matrices to obtain the unnormalized initial joint probability distribution matrix.

[0078] For example, taking the two-dimensional unit (V1, A5) for calculation, the weighted probability of the face unit is equal to the original probability of face (V1, A5) 0.04 × 0.220 = 0.0088, the weighted probability of the speech unit is equal to the original probability of speech (V1, A5) 0.65 × 0.291 = 0.18915, the weighted probability of the limb unit is equal to the original probability of limb (V1, A5) 0.16 × 0.242 = 0.03872, and the weighted probability of the vehicle unit is equal to the original probability of vehicle (V1, A5) 0.8 × 0.251 = 0.2008. Then the initial joint probability of this unit is 0.0088 + 0.18915 + 0.03872 + 0.2008 = 0.43747. The initial joint probability of all 5×5 units is calculated according to this rule, forming the initial joint probability distribution matrix.

[0079] 4. Post-processing and validity verification

[0080] The initial joint probability distribution matrix is ​​globally normalized. The total probability sum S of the initial joint probability distribution matrix is ​​calculated. If S≠1, all elements are divided by S to ensure that the total probability sum is 1. Outlier probability values ​​are filtered out, with values ​​less than 0 corrected to 0 and values ​​greater than 1 corrected to 1. If the total probability sum changes after correction, lightweight normalization is performed. The peak probability value and peak two-dimensional interval of the joint distribution are extracted, and the contribution ratio of each mode in the peak interval is calculated. The joint distribution is validated for validity; if the validation is not satisfied, a backtracking mechanism is triggered, and the fusion calculation is re-executed.

[0081] For example, the total probability sum of all units in the initial joint probability distribution matrix is ​​S=1.002. Dividing each unit probability value by 1.002 for global normalization, the normalized unit probability (V1, A5) is 0.43747 / 1.002≈0.4366, which is the maximum value in the matrix, i.e., the peak probability value ≈0.4366. The peak two-dimensional interval is V1 (extremely negative) + A5 (extremely high). The contribution percentage of each modality within this peak interval is calculated as follows: vehicle ≈0.2008 / 0.43747≈45.9%, speech ≈0.18915 / 0.43747≈43.2%, limb ≈8.8%, and face ≈2.1%. Verification shows that all unit probability values ​​∈ [0,1] and the total probability sum = 1, thus the validity verification passes.

[0082] 5. Result Output and Storage

[0083] The normalized initial joint probability distribution matrix is ​​output to the temporal context modeling module in a preset format, while storing the core sentiment features (peak probability, peak interval, contribution ratio of each modality), the real-time dynamic weight of the current modality, and other traceability information. After binding the timestamp, it is stored in the cache, and the cached data is associated with the previous single-modal distribution and weight data.

[0084] For example, the initial joint probability distribution matrix of 5×5 is flattened into a one-dimensional vector by rows and output to step 5; the core information is stored as "timestamp t=100ms, peak probability 0.4366, peak interval V1A5, vehicle contribution 45.9%, voice contribution 43.2%, real-time dynamic weight [0.220,0.291,0.242,0.251]", and after binding the timestamp, a joint probability distribution sequence is formed and stored in the joint distribution cache queue, and associated with the single-modal distribution and weight data at that time.

[0085] (v) Temporal Context Modeling

[0086] By taking the joint probability distribution sequence of continuous time steps as input, a time-series deep learning model is used to capture the evolution and dependencies of emotions in the time dimension, complete the smoothing optimization of the original emotional trajectory, detect the emotional turning point at the current moment, and predict the potential emotional turning points in the short term. The output smooth joint probability distribution at the current moment serves as the core quantitative basis for subsequent qualitative judgment.

[0087] 1. Initialization of the timing modeling module

[0088] Load pre-trained GRU or Transformer temporal models on demand, perform lightweight model optimization based on in-vehicle hardware (e.g., convert FP16 to INT8, prune redundant network layers) to ensure that inference speed matches execution frequency (e.g., 10Hz); define core temporal parameters, including sliding window N (generally 10 / 20 steps), prediction step size K (generally 3 / 5 steps), and sliding window movement step size S=1 step; unify the model input and output formats, flatten the 5×5 joint probability matrix into a 25-dimensional vector, with the model input layer dimension being N×25 and the output layer dimension being 1×25; load the emotion trajectory smoothing judgment threshold (e.g., the difference in peak probability between adjacent temporal steps ≤ 0.1 is considered normal fluctuation) and emotion turning point judgment rules (interval cross-domain, temporal change rate > 0.2, peak probability mutation amplitude > 0.15, satisfying any one of these is a turning point); initialize the temporal cache queue and output channel.

[0089] For example, a lightweight GRU model is loaded and INT8 quantization is completed, achieving an inference speed of 15Hz to meet the 10Hz execution requirement; N=10 steps (corresponding to 1 second), K=3 steps (corresponding to 300ms), and S=1 step are defined; the model input is a 10×25 feature sequence, and the output is a 1×25 current smooth vector and a 3×25 prediction vector; the inflection point judgment rules are loaded: valence / arousal interval cross-domain, rate of change > 0.2, peak mutation > 0.15; a temporal buffer queue with a capacity of 13 is initialized (10 steps of input + 3 steps of prediction).

[0090] 2. Time series data preprocessing

[0091] The normalized joint probability distribution matrix is ​​flattened into a 25-dimensional one-dimensional vector and stored in a time-series cache queue in time stamp order. The queue adopts a first-in-first-out rule and always retains the latest N time-series step data. The cached data is validated and outlier filtered. If the time stamp jumps, linear interpolation is performed to complete it. The latest N time-series step data in the cache queue are extracted, concatenated into a sliding window input feature sequence, and after global normalization, converted into a tensor format that can be recognized by the vehicle model (such as a [1,10,25] Tensor tensor).

[0092] For example, the joint probability distribution matrix at each time step is flattened into a 25-dimensional vector and stored in a time-series cache queue according to timestamps of t=10ms, 20ms...100ms. When the queue length exceeds 10, the earliest t=10ms vector is removed. If a missing t=70ms timestamp is detected, it is filled in by linear interpolation using vectors of t=60ms and t=80ms. Ten 25-dimensional vectors from t=10ms to t=100ms are extracted, concatenated into a 10×25 sliding window sequence, and converted into a [1,10,25] Tensor tensor after global normalization, which is used as input to the GRU model.

[0093] 3. Forward inference of time-series models

[0094] The sliding window feature tensor is input into the time series model. The GRU model mines time-dependent features through forget gate, input gate, and output gate, filters out meaningless historical features, and captures key time-series change features. The model outputs the original smooth joint probability vector at the current moment and the original predicted joint probability vector for the next K steps. The vectors are restored to a 5×5 original smooth matrix and spliced ​​together to form the original smooth emotion trajectory and the original future emotion prediction sequence.

[0095] For example, the tensor [1,10,25] is input into the GRU model. The model captures the temporal features of the arousal interval from A3 to A5 from t=60ms to t=100ms through a gating mechanism. It outputs the original smoothed 25-dimensional vector at t=100ms and three predicted 25-dimensional vectors at t=110ms, 120ms, and 130ms. All vectors are restored to 5×5 original smoothed matrices. The original smoothed matrices from t=10ms to t=100ms are concatenated to form the original emotion trajectory, and the matrix from t=110ms to t=130ms forms the original prediction sequence.

[0096] 4. Secondary smoothing of emotional trajectory

[0097] The original smoothing matrix and the prediction matrix output by the time series model are physically restored to ensure that the total probability sum is 1 and the probability value ∈ [0,1]. The original smoothed emotion trajectory is optimized twice using the moving weighted average method. The smoothing window is set to M=3 steps, and weighting coefficients are assigned to each time series step within the window (0.6 for the current step, 0.3 for the previous step, and 0.1 for the previous two steps, with the sum of coefficients = 1). The second smoothing value is calculated by element-wise weighted average. The same smoothing rule is applied to the predicted sequence of the next K steps to eliminate meaningless fluctuations, preserve the true trend of emotion evolution, and generate the final smoothed valence-arousal joint probability distribution emotion trajectory.

[0098] For example, the original smoothing matrix at t=100ms is restored to ensure that the total probability sum is 1; a second smoothing is performed with M=3 steps, and the second smoothing matrix at t=100ms = 0.6 × t=100ms original matrix + 0.3 × t=90ms original matrix + 0.1 × t=80ms original matrix. After element-wise calculation, the probability of the peak interval V1A5 is optimized from 0.4366 to 0.442, and the fluctuation is smoother; the same weighting rule is applied to the prediction matrix from t=110ms to t=130ms to complete the smoothing of the prediction sequence.

[0099] 5. Detection and prediction of emotional turning points

[0100] Based on preset judgment rules, the core emotional features of the smoothed valence-arousal joint probability distribution between the current moment and the previous moment are extracted. The temporal change rate and the amplitude of peak probability mutation are calculated to determine whether an interval crossover has occurred, thus completing the real-time detection of the emotional turning point at the current moment. The core emotional features of the future K-step prediction sequence are extracted, the continuous evolution trend is analyzed, and potential short-term emotional turning points are predicted. The potential turning points are assigned confidence scores (high ≥0.8, medium 0.5~0.8, low <0.5) according to the model output probability, and the intervention priority is marked.

[0101] For example, examining the characteristics of the current t=100ms and the previous t=90ms: the arousal interval at t=90ms is A3 with a peak probability of 0.28; the arousal interval at t=100ms crosses to A5 with a peak probability of 0.442. The rate of change is (0.442-0.28) / 0.28≈0.579>0.2, satisfying the turning point determination rule. Therefore, t=100ms is determined to be a "turning point of increased arousal + decreased valence". Analyzing the predicted sequence for the next 3 steps, from t=110ms to t=130ms, the arousal remains at A5 and the valence remains at V1, with no new interval crossings or abrupt changes. Therefore, there is no potential emotional turning point in the next 3 steps. If the predicted valence at t=120ms crosses from V1 to V2 with a rate of change of 0.25>0.2, and the model outputs a confidence level of 0.85, then it is determined to be a high-confidence potential turning point, marked as a low-intervention priority of "negative valence mitigation".

[0102] 6. Result Validation

[0103] The rationality of the smoothed valence-arousal joint probability distribution emotion trajectory and turning point results is verified to ensure that the trajectory evolution trend conforms to prior knowledge of the driving scenario, the turning point is strongly correlated with the driving event, and the prediction trend is continuous. Abnormal results are corrected, and continuous abnormalities trigger the emergency mechanism and output an alarm. The core verification rules include: the trajectory has no anti-common sense evolution (such as a sudden drop in arousal after emergency braking), the turning point must correspond to the driving event (such as an increase in arousal corresponding to emergency braking / acceleration), and the prediction sequence has no abrupt change trend (such as predicting A5 in one step and A2 in the next step).

[0104] For example, verifying the turning point at t=100ms: corresponding to the three emergency braking events of the vehicle from t=80ms to t=100ms, the emotional trajectory gradually rises from A3 to A5, which is consistent with the prior knowledge of the driving scenario, and the predicted sequence trend is continuous, so the verification passes; if the arousal level is detected to suddenly rise from A1 to A5 at t=50ms, but no driving event is triggered, it is judged as a counterintuitive evolution, and the trajectory at this moment is corrected using the average features of the previous 3 frames; if trajectory anomalies occur for 5 consecutive frames, the emergency mechanism is triggered, the turning point prediction is paused, only the smooth emotional trajectory is output, and a temporal modeling anomaly alarm is output to the vehicle system.

[0105] II. Current Moment Emotion Prediction

[0106] The current moment's emotion prediction serves as the qualitative judgment and result implementation layer, representing the final execution stage of the entire method. Utilizing heterogeneous graph subgraph matching technology, it combines the quantitative results of the emotion trajectory prediction with the initially acquired multimodal feature data, and performs deep matching with a predefined emotion behavior graph knowledge graph. This transforms the abstract valence-arousal joint probability distribution into concrete and interpretable driver emotion labels, the results of which can be directly integrated into the intelligent driving safety intervention system. This section, based on subgraph matching of the behavior flowchart and the emotion behavior graph, includes initialization configuration, heterogeneous graph construction, subgraph extraction and alignment, similarity calculation, and emotion determination.

[0107] 1. Graph matching initialization

[0108] The nodes and edges of the loaded heterogeneous graph are defined using unified rules, explicitly including four categories of modal feature nodes: face, voice, limbs, and vehicle. Specifically, face node attributes include AU action units and expression feature extraction confidence; voice node attributes include MFCC feature vectors and signal-to-noise ratio (SNR) value; limb node attributes include skeletal keypoint movement variance and movement amplitude; and vehicle node attributes include driving event type and event intensity. Emotion feature node attributes include the original smoothing matrix, peak valence / arousal interval, and peak probability value. Temporal relationship edge attributes include relationship type (synchronous / sequential), time difference, and association weight; spatial relationship edge attributes include relationship type (associated / independent), spatial association degree, and association weight; and causal relationship edge attributes include relationship type (causal / correlated), causal confidence, and association weight. The loading of structural similarity and emotion similarity fusion weights (α=0.5, β=0.5 for general scenarios, and adjusted to α=0.4, β=0.6 for single-modal failure scenarios such as night / occlusion), probability distribution normalization rules for JS divergence calculation, and structural similarity sub-index weights (node ​​degree distribution similarity 0.3, node attribute similarity 0.4, edge type / weight similarity 0.3). The mapping relationship between typical driving scenarios (calm, tension, anger, pleasure, fatigue) and emotional behavior graph subgraphs is solidified, and the valence-arousal reference range of each typical emotion is clarified (e.g., tension is negative valence + high / very high arousal, calm is neutral valence + low arousal) and typical behavioral pattern characteristics (e.g., anger corresponds to glaring face, high voice volume, and rough driving). The minimum threshold for comprehensive similarity (0.6) and the effective threshold for emotional similarity (0.5) are loaded. The graph matching algorithm is lightweighted and optimized based on the computing resource limitations of the vehicle end. The greedy matching algorithm is used to replace the complex graph isomorphism algorithm, and the maximum number of nodes and the maximum number of edges in the graph structure are limited to ≤10 and ≤20, respectively. A graph structure cache queue with a capacity of 10 frames is initialized, as well as a standardized output channel for connecting to the vehicle intelligent driving safety intervention system.

[0109] 2. Construct a driver behavior flowchart

[0110] Based on the smoothed valence-arousal joint probability distribution at the current moment, a heterogeneous behavior flowchart G is constructed. real =(V real E real ); Node set V real It includes five core node categories: facial, voice, body, vehicle, and emotion feature nodes. According to rules, all nodes are assigned quantified features, confidence scores, and other attributes, and are assigned unique identifiers. For example, the facial node is assigned an AU4 (frowning) feature value of 0.85 and an extracted confidence score of 0.7; the vehicle node is assigned two instances of sudden braking and a driving event intensity of 0.9; and the emotion feature node is assigned a 5×5 original smooth matrix with a peak interval of extremely negative valence (V1) + extremely high arousal (A5). The edge set E... realBased on the spatiotemporal and causal relationships constructed using multimodal and emotional features, all edges are assigned attributes such as relationship type, association weight, and causal confidence. For example, a causal relationship edge is constructed between a vehicle braking node and a facial frowning node, labeled as causal, with an association weight of 0.9 and a causal confidence of 0.85. A spatial relationship edge is constructed between facial and speech nodes, labeled as association, with a spatial association degree of 0.9 and an association weight of 0.8. The behavior flowchart is then lightweighted and pruned, removing meaningless edges with an association weight <0.1 and invalid nodes with a feature extraction confidence <0.05 to complete the initial validity check. The core checks include whether it contains the five types of core nodes and whether there are valid edge connections between nodes. If the check fails, the previous frame's valid behavior flowchart is used as a temporary replacement.

[0111] 2. Retrieve and analyze the emotion-behavior knowledge graph.

[0112] Retrieve the pre-trained emotion and behavior knowledge graph G from the vehicle's local storage. ref This graph is a typical emotion knowledge graph trained offline and accessed online, consisting of several independent subgraphs representing typical emotions such as calmness, tension, and anger. (The k-th typical emotion subgraph) is composed of; the knowledge graph is split into independent typical emotion subgraphs according to emotion labels, and the node set, edge set, and emotion reference features of each subgraph are analyzed, such as the analysis For each subgraph, typical attributes of vehicle nodes are extracted: emergency braking event intensity ≥ 0.8; and typical attributes of facial nodes are AU4 feature value ≥ 0.8. Simultaneously, the valence-arousal reference probability distribution of each subgraph is extracted (e.g., ...). The total probability of the negative valence interval is 0.7, and the total probability of the high / extremely high arousal interval is 0.8); all typical emotion subgraphs are standardized and aligned. If a subgraph has missing nodes / edges (such as some subgraphs without voice nodes), virtual nodes / edges are constructed to supplement them. The virtual node attribute value is marked as 0 and the virtual edge association weight is marked as 0 to ensure that it is completely consistent with the node type and edge type of the behavior flowchart.

[0113] 3. Core subgraph extraction and cross-graph alignment

[0114] Using the emotion feature nodes of the behavior flowchart as the core, modal feature nodes and their corresponding edges that are directly connected to them and have an association weight ≥ 0.5 are extracted to form the core emotion-related subgraph of the behavior flowchart. Remove redundant nodes / edges that are not strongly related to the emotion feature nodes; categorize by node type. With each To achieve precise one-to-one alignment of nodes, such as... facial node correspondence Facial nodes, The virtual vehicle node corresponds to For vehicle nodes, the matching weight is set to 0 during virtual node matching. Based on the node alignment results, type and position alignment across graph edges is completed, such as... The causal relationship edge from the vehicle node to the face node corresponds to For causal edges at the same position, virtual edges are matched with the corresponding edge and marked with a matching weight of 0; finally, the alignment result is verified to ensure... With each The number of nodes and edges are completely consistent, the node-edge correspondence is unique, and there are no mismatches or omissions.

[0115] 4. Two-dimensional similarity calculation: Based on the cross-graph alignment results, calculate respectively With each Structural similarity Similarity to emotions Both dimensions have scores in the range of [0,1], and the higher the score, the higher the matching degree.

[0116] 5. Structural similarity calculation

[0117] Calculated from three sub-dimensions: node degree distribution, node attributes, and edge type / weight, first extract... and The node degree distribution vector is used to obtain the node degree distribution similarity through cosine similarity; then, for nodes that are aligned one-to-one, the Euclidean distance similarity of their quantized feature vectors is calculated, and the average of all node similarities is taken as the node attribute similarity; finally, for aligned edges, the Jaccard coefficient of the edge type and the Pearson correlation coefficient of the edge weight are calculated, and the weighted average is taken as the comprehensive similarity of a single edge, and the average of all edge similarities is taken as the edge type / weight similarity. The comprehensive structural similarity is obtained by weighted summation according to the preset sub-index weights, and after normalization, it is mapped to the [0,1] interval, such as... and The similarity of node degree distribution is 0.88, the similarity of node attributes is 0.85, and the similarity of edge type / weight is 0.86. The overall structural similarity is 0.3×0.88+0.4×0.85+0.3×0.86=0.862.

[0118] 6. Emotional Similarity Calculation: Extraction Actual valence-arousal joint probability distribution P real and Reference probability distribution After performing uniform normalization on both to ensure the total probability sum is 1, the JS divergence between them is calculated; through =1-JS(P real , The JS divergence is converted into sentiment similarity. If the calculation result is negative, it is corrected to 0 to ensure that the score is ∈ [0,1]. and If the JS divergence is 0.12, then the sentiment similarity is 1-0.12=0.88.

[0119] 7. Combining similarity scoring with sentiment assessment

[0120] Structural similarity and emotional similarity are weighted and fused according to preset fusion weights to obtain a comprehensive similarity score. The overall similarity of all typical emotion subgraphs is sorted from high to low. The emotion label corresponding to the first-ranked subgraph is taken as the emotion determination result, and its overall similarity value is the preliminary confidence level. If the overall similarity of the first-ranked subgraph is lower than the preset minimum threshold (0.6), it is determined to be an unknown emotion.

[0121] For example, in a nighttime scene, α=0.4 and β=0.6. and The overall similarity is 0.4×0.862+0.6×0.88=0.8728, which is much higher than 0.52 for the calm subgraph and 0.75 for the angry subgraph. The initial emotion label is tension, with an initial confidence level of 0.8728. If the highest overall similarity in a certain scene is 0.55, which is lower than 0.6, it is directly judged as an unknown emotion.

[0122] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for predicting driver valence-arousal emotion trajectory based on multimodal fusion and temporal modeling, characterized in that, The prediction steps include the following: The system collects driver facial video data, voice signal data, body movement video data, and vehicle CAN bus data, and aligns them according to timestamps after preprocessing to form a synchronous multimodal feature sequence. The single-modal deep features of each modality in the synchronous multimodal feature sequence are extracted and mapped to the probability distribution of valence and / or arousal dimension, respectively. The quantitative indicators of each modality are collected, and after reliability scoring and Softmax normalization, the real-time dynamic weights are output. The probability distribution of each modality is extended into a two-dimensional distribution matrix. After being weighted and summed according to real-time dynamic weights, and then normalized globally, the two-dimensional joint probability distribution of valence and arousal at the current moment is obtained. The sequence, consisting of the two-dimensional joint probability distribution of valence and arousal corresponding to each continuous time step, is input into the temporal deep learning model, and the output is the emotion trajectory of the joint probability distribution of valence and arousal.

2. The driver valence-arousal emotion trajectory prediction method based on multimodal fusion and temporal modeling according to claim 1, characterized in that, The facial modality is mapped to a two-dimensional probability distribution of valence and arousal using a lightweight convolutional neural network. The speech modality is mapped to a two-dimensional probability distribution of valence and arousal using an LSTM network. The limb modality is mapped to an arousal distribution with a uniform valence using a GCN. The vehicle modality is mapped to an arousal distribution with a non-uniform valence using an MLP.

3. The driver valence-arousal emotion trajectory prediction method based on multimodal fusion and temporal modeling according to claim 1, characterized in that, Quantitative indicators include illumination intensity for the facial modality, signal-to-noise ratio for the speech modality, limb movement amplitude for the body modality, and driving event intensity for the vehicle modality; the reliability score conversion uses a piecewise linear mapping function, and the quantitative indicator value x is lower than a preset low threshold T. low The time score is 0, which is higher than the preset high threshold T. high When the score is 1, and it falls between the two, press (xT) low ) / (T high -T low The real-time dynamic weights are calculated and output after being smoothed by a weighted average of the weights of the current frame and the previous 3 frames. The smoothing coefficients are 0.7 for the current frame and 0.3 for the average of the previous 3 frames.

4. The driver valence-arousal emotion trajectory prediction method based on multimodal fusion and temporal modeling according to claim 1, characterized in that, The extended rule of the two-dimensional distribution matrix is: two-dimensional unit probability value = valence single-interval probability × arousal single-interval probability.

5. The driver valence-arousal emotion trajectory prediction method based on multimodal fusion and temporal modeling according to claim 1, characterized in that, The initial joint probability distribution matrix is ​​obtained by multiplying each element of the two-dimensional distribution matrix of each modality at the current moment by the corresponding real-time dynamic weight, and then summing the probability values ​​of the same two-dimensional unit. The values ​​less than 0 in the initial joint probability distribution matrix are corrected to 0, and the values ​​greater than 1 are corrected to 1. Then, global normalization is achieved by dividing all elements in the initial joint probability distribution matrix by the total probability sum, thereby obtaining the valence-arousal two-dimensional joint probability distribution at the current moment.

6. A method for qualitatively determining a driver's current emotion based on heterogeneous graph subgraph matching, characterized in that, The following determination steps are included: Based on the current moment valence-arousal two-dimensional joint probability distribution and synchronous multimodal feature sequence output by the driver valence-arousal emotion trajectory prediction method based on multimodal fusion and temporal modeling according to any one of claims 1-5, a heterogeneous behavior flowchart containing modal feature nodes and emotion feature nodes is constructed. Retrieve the pre-trained emotion behavior graph knowledge graph, split it into multiple independent typical emotion subgraphs with emotion labels, and analyze the valence-arousal reference probability distribution and node and edge features of the independent typical emotion subgraphs. Extract the core emotion-related subgraphs from the heterogeneous behavior flowcharts and perform node-edge alignment operations with each typical emotion subgraph. The structural similarity between the core emotion-related subgraph and each typical emotion subgraph is calculated, as well as the emotion similarity between the joint probability distribution of actual valence-arousal corresponding to the core emotion-related subgraph and the reference probability distribution of the typical emotion subgraph. The comprehensive similarity is obtained by weighting according to the preset weights. The emotion label corresponding to the typical emotion subgraph that ranks first and meets the threshold requirement is taken as the emotion judgment result; otherwise, it is judged as an unknown emotion.

7. The method for qualitatively determining a driver's current emotion based on heterogeneous graph subgraph matching according to claim 6, characterized in that, The construction of heterogeneous behavior flowcharts includes node extraction and attribute assignment, as well as edge construction and attribute assignment. Node extraction and attribute assignment: Four modal feature nodes (facial, speech, body, and vehicle) are extracted from the synchronous multimodal feature sequence, and emotion feature nodes are extracted from the two-dimensional joint probability distribution of valence and arousal. Quantitative features and confidence scores are assigned to each node. Facial nodes correspond to AU action units, speech nodes correspond to MFCC features and signal-to-noise ratio, body nodes correspond to the variance of skeletal keypoint movement and amplitude of movement, vehicle nodes correspond to driving event type and intensity, and emotion feature nodes correspond to the joint probability distribution matrix, peak interval, and peak probability value, and are assigned a unique identifier. Edge construction and attribute assignment: Based on the spatiotemporal and causal relationships of four modalities (face, voice, body, and vehicle), temporal, spatial, and causal relationship edges are constructed, and the relationship type, association weight, and corresponding feature parameters are labeled respectively.

8. The method for qualitatively determining a driver's current emotion based on heterogeneous graph subgraph matching according to claim 6, characterized in that, Taking the emotion feature node in the heterogeneous behavior flowchart as the core, all modal feature nodes that have direct edge connections with the emotion feature node and have an association weight ≥ 0.5 are selected. At the same time, the effective edges between these modal feature nodes are extracted, and redundant nodes and edges that are not related to the emotion feature node are removed to form a core emotion-related subgraph that only contains the core emotion-related features.

9. The method for qualitatively determining a driver's current emotion based on heterogeneous graph subgraph matching according to claim 6, characterized in that, Structural similarity is obtained by weighted summation of cosine similarity of node degree distribution, mean Euclidean distance of node attributes, and edge comprehensive similarity; emotional similarity... The calculation method is as follows =1-JS(P real , ), P real For the actual joint probability distribution, Using the probability distribution as a reference, JS is the divergence function; the overall similarity is... The calculation formula is: α and β are the corresponding weighting coefficients.

10. A driver emotion collaborative analysis system, characterized in that, The method for predicting driver valence-arousal emotion trajectory based on multimodal fusion and temporal modeling as described in any one of claims 1-5, and the method for qualitatively determining driver's current emotion based on heterogeneous graph subgraph matching as described in any one of claims 6-9, comprises a data acquisition module, an emotion trajectory prediction module, and an emotion qualitative determination module. The data acquisition module is used to acquire driver facial video, voice signals, body movement video, and vehicle CAN bus data. The emotion trajectory prediction module outputs a two-dimensional joint probability distribution of valence-arousal and the emotion trajectory based on the acquired data. The emotion qualitative determination module performs heterogeneous graph subgraph matching based on the output of the emotion trajectory prediction module and outputs the emotion determination result.

Images