Driver multi-modal emotion prediction method and system for intelligent cockpit complex environment

Abstract

The application discloses a driver multi-modal emotion prediction method and system for a smart cockpit complex environment, and the method comprises the following steps: 1, obtaining a multi-modal emotion sample sequence containing text, vision and voice modes, and through preprocessing and mode missing mask processing, obtaining an incomplete multi-modal observation sequence; 2, extracting bottom layer features of each mode, and obtaining Gaussian distribution parameters through variational encoding; 3, using an external prototype memory matrix to perform memory anchoring, retrieving semantic prototypes, obtaining semantic confidence, and calibrating mode mean; 4, combining physical variance and semantic confidence to calculate fusion weight, and generating initial global agent features; 5, performing cross-modal information injection and feedback gating through a closed-loop semantic modulation network, and recursively purifying bottom layer noise; using the final global agent features to output emotion prediction values, and combining second-order Wasserstein distance and variance stability constraints to perform model training. The application can improve the accuracy and robustness of emotion prediction in a complex cockpit environment.

Description

Technical Field

[0001] This invention belongs to the fields of intelligent driving assistance, emotion computing and multimodal signal processing technology. Specifically, it relates to a method and system for predicting driver multimodal emotions in the complex environment of intelligent cockpits. It integrates variational coding, nonparametric memory anchoring, multilayer dynamic network inference and geometric manifold learning technology, and is applicable to the detection and evaluation of driver emotions, fatigue, distraction and other states in intelligent cockpits. Background Technology

[0002] With the rapid development of intelligent connected vehicle technology, the intelligent cockpit, as the core carrier of human-vehicle interaction, relies heavily on accurate perception of the driver's state to achieve human-machine collaborative driving and improve driving safety. The driver's emotional changes and physiological states, such as fatigue, distraction, anxiety, or anger, directly affect driving behavior and decisions. Real-time detection and identification of these states can provide a basis for intelligent cockpit safety warnings, personalized interactions, and assisted driving decisions, forming an important foundation for the practical application of intelligent driving assistance systems.

[0003] Multimodal fusion detection technology integrates multi-dimensional information such as facial images, speech audio, text semantics, physiological signals, or driving operations. Compared with single-modal detection methods, it has the advantages of richer information sources and more comprehensive state representation, and has become an important research direction in the field of driver state detection. Existing methods typically collect multimodal observation data of the driver, extract features, fuse modalities, and then input the data into a prediction model to predict the driver's emotion or state. Under ideal conditions such as stable lighting, low environmental noise, and unobstructed driver face, this type of method can achieve good prediction results.

[0004] However, in the complex driving environment of real-world intelligent cockpits, existing multimodal driver state detection methods still have many problems, and their output results are difficult to reliably provide a basis for intelligent cockpit control systems and driving safety warning modules. These problems are mainly reflected in the following aspects: 1) Insufficient information in feature representation: Existing methods usually represent the multimodal features of drivers as a single feature vector or direct classification results, lacking characterization of feature probability distribution, confidence range and modal reliability, making it difficult to reflect the uncertainty of observation data of different modalities in complex environments; 2) Poor model interpretability: Traditional methods focus on outputting the final prediction result, making it difficult to explain the contribution of each modal feature to the prediction result, and also difficult to reflect the impact of feature drift, noise interference or modal missingness on the model output, which is not conducive to the debugging and optimization of intelligent cockpit perception system. 3) Insufficient noise and interference resistance: Existing methods usually lack feature calibration and noise suppression mechanisms for the complex cockpit environment. They have limited ability to handle feature defects and distribution drift caused by factors such as backlight, occlusion, strong vehicle noise, wind noise, sensor abnormalities or interaction interruptions. Noise interference in the observation data will affect model training and prediction, leading to a decrease in detection accuracy.

[0005] The aforementioned problems render traditional multimodal driver emotion prediction methods insufficiently robust in real-world intelligent cockpit environments, and their detection accuracy falls short of practical application requirements, limiting their large-scale deployment in the field of intelligent driving. Therefore, there is an urgent need for a multimodal driver emotion prediction method and system designed for the complex environment of intelligent cockpits. This method should be able to perform uncertainty modeling, feature calibration, and robust fusion of features across various modalities under incomplete multimodal observation conditions, thereby improving the stability and accuracy of driver emotion state prediction in complex environments. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a method and system for predicting driver multimodal emotions in complex environments of intelligent cockpits. The aim is not only to achieve accurate prediction of driver emotions but also to probabilistically represent and calibrate multimodal features in complex environments, providing information such as feature confidence and distribution range. This significantly improves the information content and interpretability of the model, benefiting researchers in optimizing intelligent cockpit perception systems and designing precise driving safety warning strategies.

[0007] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: The present invention provides a method for predicting multimodal driver emotions in complex environments of intelligent cockpits, characterized by the following steps: Step 1: Obtain a multimodal sentiment sample sequence of any target object containing text, visual, and speech modalities within the training batch. After performing time alignment, feature normalization, and sentiment label mapping, mask the preprocessed multimodal sentiment sample sequence using a modality missing mask matrix to obtain an incomplete multimodal observation sequence. ; Step 2: Use the multimodal feature extraction module to extract features from the incomplete multimodal observation sequence. The low-level features of each mode are extracted, and the variational Gaussian distribution parameters are obtained to obtain the latent variables of each mode. ; Step 3: Utilize the memory anchoring calibration module based on a pre-built external prototype memory matrix. ,from Select the semantic prototypes that match each modality and their corresponding semantic confidence scores. Used to find the original mean in the parameters of a Gaussian distribution. Perform calibration to obtain the mean value of the m-th modality after calibration. ; Step 4: Use the dynamic proxy generation module to perform a two-factor dynamic evaluation of the physical variance and semantic confidence in the Gaussian distribution parameters to generate initial global proxy features. ; Step 5: The closed-loop semantic modulation module uses the initial global proxy features. Using an L-layer bidirectional closed-loop semantic modulation network as the modulation center, the low-level features of each modality are recursively refined to obtain the final global proxy features. ; Step 6: Utilize the sentiment prediction output module to perform analysis using a multilayer perceptron. Regression prediction is performed to obtain the sentiment prediction value. ; Step 7: Construct the joint loss function This includes: constructing a geometric alignment loss function. Emotional regression loss and KL divergence loss ; Step 8: Based on the joint loss function The multimodal sentiment prediction network, consisting of a multimodal feature extraction module, a memory anchoring calibration module, a dynamic agent generation module, a closed-loop semantic modulation module, and a sentiment prediction output module, is iteratively trained until the joint loss function is reached. The training continues until convergence, resulting in a trained multimodal sentiment prediction model. This model is used to detect incomplete multimodal observation sequences and output driver sentiment prediction values.

[0008] The multimodal emotion prediction method for drivers in complex environments of intelligent cockpits, as described in this invention, is characterized in that step 2 includes the following steps: Step 2.1, After feature extraction for each modality is performed in the input multimodal encoder, the feature is then uniformly projected to dimension 1 through a fully connected layer. The shared subspace is used to obtain the underlying features of each modality. ; Step 2.2: Extract the low-level features of the m-th mode. The input is processed in a variational encoder, and the Gaussian distribution parameters of the m-th modal feature are output, including: the original mean of the m-th modality. Sum of logarithmic variance ,according to Obtain the standard deviation of the m-th mode. ,and Thus, the mean of any multimodal sentiment sample sequence can be obtained. and standard deviation M represents the total number of modes; Step 2.3: Obtain the latent variables of the m-th mode according to equation (1). : (1) It is standard Gaussian noise.

[0009] Furthermore, step 3 includes the following steps: Step 3.1, with For the query vector, the external prototype memory matrix is ​​used. Perform cosine similarity calculation to retrieve similarities with... Best matching semantic prototype ; Step 3.2: Extract the highest cosine similarity score obtained from the retrieval and use it as the semantic confidence score of the m-th modality. ; Step 3.3, using equation (2) to... Perform calibration to obtain the mean value of the m-th modality after calibration. : (2) In equation (2), This indicates the preset anchoring coefficient.

[0010] Furthermore, step 4 includes the following steps: Step 4.1: Calculate the fusion weight of the m-th mode using equation (3). : (3) Step 4.2, for Normalization is performed to obtain the normalized weights of the m-th mode. ; Step 4.3: Generate initial global proxy features using equation (4). : (4) In equation (4), This represents the total number of modes.

[0011] Furthermore, step 5 includes the following steps: Step 5.1, in the... In layer modulation, the first Global proxy features output by the layer As the query vector, it absorbs the low-level features of each modality in layer l through a cross-modal attention mechanism. The effective information in the equation is used to obtain the injection features corresponding to the m-th mode using equation (5). : (5) In equation (5), This represents the cross-modal attention function corresponding to the m-th modality; Indicates the first The global proxy features output by the layer; Indicates the first The bottom-level features of the m-th mode in the layer; when season = ; Step 5.2: Use equation (6) to obtain the first... Global proxy features output by the layer : (6) In equation (6), This represents the normalized fusion weight of the m-th modality; Representation layer normalization; Indicates a fully connected mapped network; Indicates residual connection; Step 5.3: The gating signal generation unit will generate the first... Global proxy features output by the layer Input feedback gating network, and through After activation function processing, the suppression signal corresponding to the m-th mode is generated using equation (7). : (7) In equation (7), and They represent the first Feedback gating network parameters corresponding to each mode; Step 5.4: The recursive purification calculation unit uses equation (8) to obtain the first... The m-th mode of the bottom layer features : (8) In equation (8), This indicates bitwise multiplication. Step 5.5, Assign to Then, return to step 5.1 and continue execution until completion. Layer modulation, ultimately obtaining the first The final global proxy feature output by the layer .

[0012] Furthermore, step 7 includes: Step 7.1: Following steps 1-2.2, obtain the [number]th [item] within the training batch. The mean of a multimodal sentiment sample sequence and standard deviation and the The mean of a multimodal sentiment sample sequence and standard deviation Therefore, the first equation (9) is used to calculate the second equation. The multimodal emotion sample sequence and the first The squared second-order Wasserstein distance term between the multimodal sentiment sample sequences : (9) In equation (9), and Let represent the Gaussian distributions corresponding to the i-th and j-th multimodal sentiment sample sequences, respectively; Step 7.2: Construct the first equation using equation (10). The multimodal emotion sample sequence and the first Variance-stable regularization term among multimodal sentiment sample sequences : (10) Step 7.3: Construct the first equation using equation (11). The multimodal emotion sample sequence and the first The total distance term between the multimodal sentiment sample sequences : (11) In equation (11), These are preset weighting coefficients.

[0013] Step 7.4: Assign the true sentiment label to the i-th multimodal sentiment sample sequence within the training batch. and the Real sentiment labels for a multimodal sentiment sample sequence Discretize them separately to obtain the corresponding results. Emotion category identifier and Emotion category identifier Therefore, the geometric alignment loss function is constructed using equation (12). : (12) In equation (12), N represents the total number of multimodal sentiment sample sequences in the training batch; (·) indicates an indicator function; This represents the preset category separation interval threshold; Step 7.5: Construct the emotional regression loss using equation (13). : (13) Step 7.6: Construct the KL divergence loss using equation (14). : (14) In equation (14), This represents the posterior distribution of the i-th multimodal sentiment sample sequence within the training batch, obtained through a variational encoder. Indicates the standard Gaussian prior distribution; This represents the KL divergence.

[0014] Step 7.7: Construct the task loss function using equation (15). : (15) In equation (15), express The corresponding weighting coefficients, express The corresponding weighting coefficients; Step 7.8: Obtain the joint loss function using equation (16). : (16) In equation (16), express The corresponding weighting coefficients.

[0015] The present invention provides a multimodal driver emotion prediction system for complex environments in intelligent cockpits, characterized by the application of the aforementioned method and including: Multimodal feature extraction module: used to extract features from incomplete multimodal observation sequences. The underlying features of each modality are extracted, and the variational Gaussian distribution parameters are obtained to obtain the latent variables of each modality; wherein, the incomplete multimodal observation sequence X is obtained from the public multimodal sentiment analysis dataset in the training stage through time alignment, feature standardization, sentiment label mapping and modality missing masking, and in the application stage it is obtained from multi-source sensor data in the smart cockpit through time alignment, feature standardization and modality missing representation processing; Memory anchoring calibration module: based on a pre-built external prototype memory matrix ,from The semantic prototypes that match each modality and their corresponding semantic confidence scores are selected and used to calibrate the original mean in the Gaussian distribution parameters to obtain the calibrated mean of each modality. The dynamic proxy generation module performs a two-factor dynamic evaluation of the physical variance and semantic confidence in the Gaussian distribution parameters to generate initial global proxy features. Closed-loop semantic modulation module: It is used to recursively refine the low-level features of each modality through an L-layer bidirectional closed-loop semantic modulation network with the initial global agent features as the modulation center, so as to obtain the final global agent features. Sentiment prediction output module: used to perform regression prediction on the final global proxy features and output sentiment prediction values; Geometric manifold alignment module: used during the model training phase to construct the basic distance term, variance-stabilized regularization term, and total distance term based on the Gaussian representation of samples within the training batch, and to construct the geometric alignment loss function based on the consistency of the true sentiment labels of the training samples; The model training module is used to iteratively train the multimodal sentiment prediction system based on the joint loss function until the joint loss function converges, thereby obtaining the trained multimodal sentiment prediction model. This trained multimodal sentiment prediction model is used in the application phase to receive incomplete multimodal observation sequences obtained in real-time within the intelligent cockpit. It also outputs the driver's emotion prediction value.

[0016] Compared with existing technologies, the beneficial effects of this invention are reflected in: 1. This invention uses variational coding to probabilistically represent multimodal features such as text, vision, and speech, and uses a pre-constructed external prototype memory matrix to perform memory anchoring calibration on the original mean of each modality. It can explicitly represent the uncertainty of each modality feature in complex cockpit environments such as light occlusion, environmental noise, sensor anomalies, or interaction interruptions, and suppress feature drift caused by modality loss or noise interference, thereby improving the representation stability of incomplete multimodal observation data.

[0017] 2. This invention combines physical variance and semantic confidence for a two-factor dynamic evaluation to generate initial global proxy features. Using these initial global proxy features as the modulation center, a closed-loop semantic modulation network is used to inject cross-modal information and recursively refine the underlying features of each modality. As a result, the model can dynamically adjust the fusion contribution based on the reliability of different modalities in the current sample, and weaken noisy and invalid responses in the underlying features, thereby improving the robustness and accuracy of multimodal sentiment prediction in complex environments.

[0018] 3. This invention introduces a geometric alignment loss function based on second-order Wasserstein distance and variance stabilization regularization during the model training phase. It also combines sentiment regression loss and KL divergence loss to construct a joint loss function, making similar sentiment samples closer together in the latent distribution space and maintaining a certain distance between dissimilar sentiment samples. Therefore, the model improves the inter-class discrimination ability of fine-grained sentiment states and enhances the stability and interpretability of the trained multimodal sentiment prediction model in the complex environment of intelligent cockpits. Attached Figure Description

[0019] Figure 1 This is a flowchart of a driver multimodal emotion prediction method for complex environments in intelligent cockpits, according to the present invention. Figure 2 This is a schematic diagram of the module structure of the multimodal emotion prediction system of the present invention; Figure 3 This is a schematic diagram of the hardware deployment topology of the method of the present invention in the complex environment of an intelligent cockpit. Detailed Implementation

[0020] In this embodiment, a driver multimodal emotion prediction method for the complex environment of a smart cockpit is proposed. During the model training phase, training samples can be constructed using publicly available multimodal emotion analysis datasets, and complex situations such as occlusion, noise, sensor anomalies, or interaction interruptions in the smart cockpit can be simulated by artificially constructing modality missing masks. Alternatively, during the application phase, real-time multimodal observation data of the driver within the smart cockpit can be received to output driver emotion prediction values. This method does not simply concatenate text, visual, and speech features; instead, it robustly models incomplete multimodal observation data under complex environments through probabilistic representation, memory anchoring calibration, dynamic proxy generation, and closed-loop semantic modulation, thereby improving the accuracy, stability, and interpretability of driver emotion prediction. Specifically, the method includes the following steps: Step 1: Obtain a multimodal emotion sample sequence of the target object containing text, visual, and speech modalities, and generate an incomplete multimodal observation sequence. .like Figure 1 As shown in the flowcharts for "Constructing a multimodal sentiment sample sequence" and "Generating an incomplete multimodal observation sequence X", this step is used to generate the input data required for model training and real-time applications.

[0021] During the model training phase, the multimodal sentiment sample sequence of the target object includes text, visual, speech, and sentiment label data from a publicly available multimodal sentiment analysis dataset. After time alignment, feature standardization, and sentiment label mapping of the data, a modality missing mask matrix is ​​used for masking to obtain an incomplete multimodal observation sequence X, which simulates modality missing caused by occlusion, noise, sensor anomalies, and interaction interruptions in a smart cockpit.

[0022] In the application phase, the real-time multimodal observation data of the driver acquired within the intelligent cockpit undergoes time alignment, feature standardization, and modality missing characterization processing to obtain an incomplete multimodal observation sequence X. Combined with... Figure 3 The hardware deployment topology shown involves a cockpit infrared / RGB camera capturing the driver's facial images, head posture, and gaze; a cockpit microphone array capturing the driver's voice and cockpit acoustic signals; and an in-vehicle voice recognition module generating text modal information. After the aforementioned video streams, audio streams, and text information are input into the core computing terminal, they are uniformly organized into an incomplete multimodal observation sequence X, thus ensuring that the missing modal samples constructed during the training phase remain consistent with the complex inputs of the real cockpit during the application phase.

[0023] Step 2: Use the multimodal feature extraction module to extract the low-level features of each modality from the incomplete multimodal observation sequence X, and obtain the variational Gaussian distribution parameters to obtain the latent variables of each modality.

[0024] like Figure 1 As shown in the flowchart for "Extracting Modal Features and Outputting Gaussian Distribution Parameters," this step is used to convert the incomplete multimodal observation sequence X into computable, fusionable modal features that can characterize uncertainty. In terms of system structure, this step corresponds to... Figure 2 The "multimodal feature extraction module" in this embodiment addresses the differences in data sources, sampling frequencies, and feature dimensions among text, visual, and speech modalities. First, it extracts the low-level features of each modality separately, then maps them uniformly into a shared subspace of dimension d to obtain the first feature. Low-level features of each modality .

[0025] Step 2.1: After inputting X into the multimodal encoder for feature extraction of each modality, the feature is then uniformly projected to a shared subspace of dimension d through a fully connected layer to obtain the low-level features of each modality. .

[0026] The multimodal encoder includes a visual encoder, a speech encoder, and a text encoder, which are used to extract features from the visual modality, speech modality, and text modality, respectively. In a preferred embodiment, the visual encoder uses ResNet-50, the speech encoder uses Wav2Vec 2.0, and the text encoder uses BERT. In cases where on-board computing power is limited or the deployment environment is different, a lightweight encoder with the same feature extraction function can be used instead. Preferably, the dimension d is 256 to facilitate unified fusion and calibration between different modalities.

[0027] Step 2.2: Extract the low-level features of the m-th mode. The input is processed in a variational encoder, and the Gaussian distribution parameters of the m-th modal feature are output, including: the original mean of the m-th modality. Sum of logarithmic variance ,according to Obtain the standard deviation of the m-th mode. ,and Thus, the mean of any multimodal sentiment sample sequence can be obtained. and standard deviation ,and M represents the total number of modes; Step 2.3: According to equation (1), we obtain the first... Latent variables of each modality : (1) For standard Gaussian noise. Equation (1) employs a reparameterization technique to make the latent variables... It can participate in end-to-end backpropagation training while maintaining random sampling characteristics. Through this step, Figure 2 The multimodal feature extraction module not only outputs the low-level features of each modality, but also outputs Gaussian distribution parameters that reflect modal uncertainty, providing a probabilistic feature basis for subsequent memory anchoring calibration, dynamic agent generation and closed-loop semantic modulation.

[0028] Step 3: Utilize the memory anchoring calibration module based on a pre-built external prototype memory matrix. ,from Select the semantic prototypes that match each modality and their corresponding semantic confidence scores. Used to find the original mean in the parameters of a Gaussian distribution. Calibration was performed to obtain the mean value of each mode after calibration. .

[0029] like Figure 1 As shown in the flowcharts for "Retrieving Semantic Prototypes and Obtaining Semantic Confidence" and "Anchoring and Calibrating Modal Means and Suppressing Feature Drift," this step is used to introduce an external semantic prototype as a stable reference based on probabilistic feature representation, and to calibrate feature drift caused by modality loss, noise interference, or sensor anomalies. In terms of system structure, this step corresponds to... Figure 2 The "memory anchoring calibration module" in the text. Due to the original mean obtained in step 2... It may still be affected by the cabin environment. For example, the visual modality may deviate from the true facial expression features due to backlighting, and the speech modality may deviate from the true emotional features of speech due to wind noise. Therefore, relying solely on the current observation data may lead to misjudgments in subsequent fusion. This embodiment uses an external prototype memory matrix. It provides non-parametric semantic anchors, enabling each modality feature to converge toward a known stable semantic distribution.

[0030] Step 3.1, with For the query vector, the external prototype memory matrix is ​​used. Perform cosine similarity calculation to retrieve similarities with... Best matching semantic prototype .

[0031] In practical implementation, the external prototype memory matrix This can be achieved through offline clustering before model training. For example, first, extract multimodal latent features from samples in the training set that have clear emotional or driving state labels; then, cluster them according to emotional or state categories such as "calm," "fatigue," "anger," "anxiety," and "distraction"; finally, store each cluster center as a semantic prototype in an external prototype memory matrix. In this context, each semantic prototype can be understood as a standard anchor point in the latent space for a typical driver's emotional state.

[0032] When performing a search, with the first The original mean of each modality As the query vector, calculate its relationship with the external prototype memory matrix respectively. The cosine similarity between the semantic prototypes is calculated, and the semantic prototype with the highest cosine similarity is selected as the semantic prototype. Cosine similarity is used to measure the degree of consistency between the current modal feature direction and the semantic prototype direction. Even if the amplitude of different modal features changes due to noise, it can still determine their semantic attribution relatively stably.

[0033] Combination Figure 3 In the smart cockpit application scenario shown, when the images captured by the cockpit's infrared / RGB cameras experience localized distortion due to changes in lighting, the original mean value of the visual modality is... It may deviate from the true semantic meaning of facial expressions; when wind noise or tire noise is mixed into the audio stream captured by the cockpit microphone array, the original mean of the speech modality may be affected. Shifts may also occur. This can be achieved by retrieving the external prototype memory matrix. This embodiment can find the closest stable semantic prototype for the above-mentioned disturbed features, thereby providing a basis for subsequent mean calibration.

[0034] Step 3.2: Extract the highest cosine similarity score obtained from the retrieval and use it as the first... Semantic confidence of each modality .

[0035] Semantic confidence Used to indicate the current number The original mean of each modality With external prototype memory matrix The highest cosine similarity score indicates the degree of similarity between the best-matching semantic prototypes. A high highest cosine similarity score indicates that although the current modal feature may have some physical noise, it is still highly consistent with the emotional state of a stable driver in the semantic space. A low highest cosine similarity score indicates that the current modal feature may have deviated significantly from the existing semantic prototype, and its reliability should be reduced.

[0036] The advantage of this setup is that this embodiment does not solely rely on the physical quality of sensors to determine modality reliability, but further introduces semantic-level matching relationships. For example, even if a certain speech signal has strong energy and low physical variance, if its semantic features do not match any emotional prototype, then this modality should not be assigned excessive weight. Conversely, even if a visual modality has slight occlusion, if its facial expression features are still highly consistent with the semantic prototype of "fatigue," then this visual modality still has high semantic reference value. Therefore, semantic confidence... Able to compare with the standard deviation in subsequent steps This combination enables a two-dimensional assessment of modal reliability.

[0037] like Figure 1 As shown, semantic confidence The acquisition of this comes after "retrieval of semantic prototypes" and will further participate in the process of "calculating fusion weights and generating initial global proxy features". Figure 2 In the system architecture shown, the semantic confidence level output by the memory anchoring calibration module This will be passed to the dynamic proxy generation module for subsequent weight fusion. The calculation.

[0038] Step 3.3, using equation (2) to... Perform calibration to obtain the first Mean after modal calibration : (2) In equation (2), This indicates the preset anchoring coefficient.

[0039] Equation (2) represents the original mean of the m-th mode. semantic prototype obtained from retrieval Anchoring fusion is performed to obtain the calibrated mean. .in, Used to control the calibration strength of the semantic prototype to the original mean. In specific implementation, It can be pre-set based on training and validation results, or it can be used as a learnable parameter in model training; when the cockpit environment is relatively stable and modal loss is minimal. A smaller value can be chosen to allow the model to retain more of the current observation features; when the cabin environment noise is strong and modal loss is significant, A larger value can be selected, allowing the model to rely more on semantic prototypes for calibration.

[0040] After processing by equation (2), the calibrated mean It no longer relies entirely on the currently perturbed observation features, but is instead influenced by the external prototype memory matrix. The provided stable semantic anchor constraints. This method is a non-parametric memory anchoring method, which does not require designing rules separately for each noise case, nor does it require retraining the model during the inference phase. Therefore, it is suitable for deployment on... Figure 3 The in-vehicle intelligent computing platform or domain controller / ECU shown.

[0041] The technical effect of this step is that, on the one hand, through semantic prototypes... Calibrling modal means affected by occlusion, noise, or sensor anomalies can suppress unconstrained feature drift in the latent space; on the other hand, semantic confidence can be used to... Describing the degree of matching between the current modality and the stable semantic prototype provides a semantic basis for the subsequent dynamic agent generation module to calculate the fusion weights. Therefore, Figure 1 The subsequent process of "calculating fusion weights and generating initial global proxy features" no longer relies solely on the physical variance of the modality itself, but combines semantic confidence and physical uncertainty simultaneously, thereby improving the robustness and interpretability of multimodal sentiment prediction in complex intelligent cockpit environments.

[0042] Step 4: Use the dynamic proxy generation module to perform a two-factor dynamic evaluation of the physical variance and semantic confidence in the Gaussian distribution parameters to generate initial global proxy features. .

[0043] like Figure 1 As shown in the flowchart "Calculate fusion weights and generate initial global proxy features", this step is used to determine the fusion contribution of each modality based on its reliability in the current sample; in terms of system structure, this step corresponds to Figure 2 The "dynamic proxy generation module" in the text. Among them, the standard deviation... Used to characterize the Uncertainty of a modality affected by occlusion, noise, or sensor malfunction, semantic confidence Used to characterize the The degree of matching between a modality and its external semantic prototype.

[0044] Step 4.1: Calculate the first step using equation (3). Fusion weights of each modality : (3) In equation (3), Indicates the first Semantic confidence of each modality Let represent the standard deviation of the m-th mode. From equation (3), we can see that the fusion weights... With semantic confidence Positively correlated with standard deviation Negative correlation. Therefore, when a modality has a high semantic matching degree and low physical uncertainty, it will receive a higher weight in the fusion process; when a modality is greatly affected by noise or occlusion, its fusion weight will be reduced accordingly.

[0045] Step 4.2, for After normalization, we obtain the first... Modal normalized weights So that the modal weights satisfy: In the formula, This represents the total number of modes. By normalizing, the fusion contribution of different modes can be constrained within a uniform scale, preventing a single mode from unreasonably dominating due to differences in numerical range.

[0046] Step 4.3: Generate initial global proxy features using equation (4). : (4) In equation (4), Indicates the first The mean value after modal calibration. Due to The memory anchoring calibration has already been performed in step 3. Furthermore, it combines physical variance and semantic confidence, thus generating initial global proxy features. It can reliably represent the current multimodal emotional state of the driver. This initial global proxy feature is then input. Figure 2 The closed-loop semantic modulation module shown is used for subsequent cross-modal information injection and recursive purification.

[0047] Step 5: The closed-loop semantic modulation module uses the initial global proxy features. Using an L-layer bidirectional closed-loop semantic modulation network as the modulation center, the low-level features of each modality are recursively refined to obtain the final global proxy features. .

[0048] like Figure 1As shown in the flowcharts for "Cross-modal information injection and feedback gating" and "Recursive purification of modal features and suppression of noise response," this step utilizes the initial global proxy features. Multi-layer modulation is performed on the low-level features of each mode; in terms of system structure, this step corresponds to Figure 2 The "closed-loop semantic modulation module" in the text. Through this module, on the one hand, the global proxy features can absorb effective information from each modality, and on the other hand, it can generate suppression signals in reverse to weaken the noise response in the low-level features of each modality.

[0049] Step 5.1: In the modulation of the l-th layer, the first layer... Global proxy features output by the layer As the query vector, it absorbs the first... Layer-by-layer features of each modality The effective information in the data is used to obtain the injection features corresponding to the m-th modality. As shown in equation (5): (5) In equation (5), This represents the cross-modal attention function corresponding to the m-th modality; Indicates the first The global proxy features output by the layer; Indicates the first The first in the layer The underlying features of each modality; when season = .

[0050] Through equation (5), global proxy features It can extract relevant information related to the current driver's emotional state from text, visual, and speech modalities. For example, the visual modal can provide information such as eye closure and facial expression changes, the speech modal can provide information such as speech rate, pitch, and energy changes, and the text modal can provide semantic expression information. The cross-modal attention mechanism ensures that the above information is not simply concatenated, but is selectively injected based on its relevance to global agent features.

[0051] Step 5.2: Utilize normalized fusion weights Injecting features for each modality Perform scaling and fusion, and combine with the global proxy features from the previous layer. After performing residual updates, layer normalization, and fully connected mapping, the global proxy features output by layer l are obtained. As shown in equation (6): (6) In equation (6), Indicates the number of modes; This represents the normalized fusion weight of the m-th modality; Representation layer normalization; This represents a fully connected mapping network; in equation (6), the normalized fusion weights... Derived from step 4, the injection intensity of different modalities is related to their physical variance and semantic confidence. This allows reliable modalities to have a greater impact on global proxy features, while the impact of modalities more affected by noise or occlusion on global proxy features is correspondingly reduced. Residual updates and layer normalization are used to maintain the stability of the multilayer modulation process and avoid weakening of effective semantic information in recursive computation.

[0052] Step 5.3: The gating signal generation unit will generate the first... Global proxy features output by the layer The input feedback gating network, after being processed by the Sigmoid activation function, generates the suppression signal corresponding to the m-th mode. As shown in equation (7): (7) In equation (7), and They represent the first Feedback gating network parameters corresponding to each mode; suppression signal This is used to indicate which dimensions in the low-level features of the m-th modality should be preserved and which dimensions should be weakened. Because It is a global proxy feature It is generated in reverse, so it can adaptively modulate the underlying features of each modality according to the overall sentiment semantics of the current sample, instead of using fixed rules to filter noise.

[0053] Step 5.4: The recursive purification calculation unit uses equation (8) to obtain the first... The m-th mode of the bottom layer features : (8) In equation (8), Indicates bitwise multiplication; by suppressing the signal For the first Noise bits and invalid responses in the low-level features of each modality are suppressed to obtain purified modal features. For example, when there is strong environmental noise in the speech modality, the noise-related feature dimensions can be suppressed; when there are abnormal responses caused by occlusion areas in the visual modality, the corresponding feature dimensions can also be weakened.

[0054] Step 5.5, let Then return to step 5.1 and continue execution until completion. Layer modulation, ultimately obtaining the first The final global proxy feature output by the layer .

[0055] Through multi-layer recursive modulation, the closed-loop semantic modulation module can continuously update the global proxy features and use the updated global proxy features to reverse-engineer the low-level features of each modality. For example... Figure 2 As shown, this process forms a closed-loop structure from global proxy features to modal features, and then from modal features back to global proxy features. Compared to unidirectional feature fusion, this step can gradually reduce noise response and enhance effective semantic response during multi-layer computation, thereby obtaining a final global proxy feature that is more suitable for use by the sentiment prediction output module. .

[0056] Step 6: Utilize the sentiment prediction output module to perform analysis using a multilayer perceptron. Regression prediction is performed to obtain the sentiment prediction value. ; like Figure 1 As shown in the flowchart "Output sentiment prediction results", this step is used to output the final global proxy features after closed-loop semantic modulation. Converted into driver sentiment prediction values. Combined Figure 2 In the system architecture shown, the sentiment prediction output module is located after the closed-loop semantic modulation module, and its input is the result obtained in step 5. The output is a sentiment prediction value. .because Having integrated effective information from text, visual, and speech modalities, and undergone multi-layered recursive refinement, this predicted value can more stably reflect the driver's current emotional state. In practical implementation, the emotional prediction value... It can represent continuous emotional intensity, or it can be further mapped to discrete emotional states such as fatigue, calmness, and anger, depending on the application requirements.

[0057] Step 7: Construct the joint loss function This includes: constructing a geometric alignment loss function. Emotional regression loss and KL divergence loss ; like Figure 1 As shown in the flowchart "Constructing the Joint Loss Function and Training the Model," this step is used to jointly optimize sentiment prediction error, latent distribution constraints, and geometric relationships between samples during the model training phase. Figure 2In the geometric manifold alignment module and model training module, this step constrains the error between the predicted value and the real sentiment label by sentiment regression loss, and constrains the difference between the variational posterior distribution and the standard Gaussian prior distribution by KL divergence loss. Furthermore, it uses geometric alignment loss to constrain the relative positions of different training samples in the Gaussian distribution space.

[0058] Step 7.1: Following steps 1-2.2, obtain the [number]th [item] within the training batch. The mean of a multimodal sentiment sample sequence and standard deviation and the The mean of a multimodal sentiment sample sequence and standard deviation Therefore, the first equation (9) is used to calculate the second equation. The multimodal emotion sample sequence and the first The second-order Wasserstein squared distance term between multimodal sentiment sample sequences : (9) In equation (9), and Let represent the Gaussian distributions corresponding to the i-th and j-th multimodal sentiment sample sequences, respectively.

[0059] Step 7.2: Construct the first equation using equation (10). The multimodal emotion sample sequence and the first Variance-stable regularization term among multimodal sentiment sample sequences : (10) Step 7.3: Construct the first equation using equation (11). The multimodal emotion sample sequence and the first The total distance term between the multimodal sentiment sample sequences : (11) In equation (11), These are preset weighting coefficients.

[0060] Step 7.4: Assign the true sentiment label to the i-th multimodal sentiment sample sequence within the training batch. and the Real sentiment labels for a multimodal sentiment sample sequence Discretize them separately to obtain the corresponding results. Emotion category identifier and Emotion category identifier Therefore, the geometric alignment loss function is constructed using equation (12). : (12) In equation (12), N represents the total number of multimodal sentiment sample sequences in the training batch; (·) indicates an indicator function; This indicates the preset category separation interval threshold.

[0061] Step 7.5: Construct the emotional regression loss using equation (13). : (13) Step 7.6: Construct the KL divergence loss using equation (14). : (14) In equation (14), Let represent the posterior distribution of the i-th multimodal sentiment sample sequence within the training batch, obtained through a variational encoder. Indicates the standard Gaussian prior distribution; This represents the KL divergence.

[0062] Step 7.7: Construct the task loss function using equation (15). : (15) In equation (15), express The corresponding weighting coefficients, express The corresponding weighting coefficients.

[0063] Step 7.8: Obtain the joint loss function using equation (16). : (16) In equation (16), express The corresponding weighting coefficients; Through the aforementioned joint loss function, the model training module can simultaneously optimize prediction accuracy, latent distribution stability, and geometric separability between samples. For example... Figure 2 As shown, the geometric manifold alignment module introduces the Gaussian distribution representation of samples within the training batch into the loss calculation, making similar emotion samples closer in the distribution space and maintaining a preset interval between different emotion categories, thereby improving the stability and discriminative ability of the emotion prediction model in complex cockpit environments.

[0064] Step 8: During the model training phase, based on the joint loss function... The multimodal sentiment prediction network, consisting of a multimodal feature extraction module, a memory anchoring calibration module, a dynamic agent generation module, a closed-loop semantic modulation module, and a sentiment prediction output module, is iteratively trained until the joint loss function is reached. The process continues until convergence is achieved, resulting in a trained multimodal emotion prediction model. In the application phase, the incomplete multimodal observation sequence X, obtained in real-time within the smart cockpit, is input into the trained multimodal emotion prediction model, which outputs the driver's emotion prediction value. .

[0065] In practice, during the training phase, the multimodal sentiment data is divided into a training set, a validation set, and a test set. Steps 1 to 7 are then followed to complete the forward computation, loss calculation, and parameter updates. When the validation set loss stabilizes, the trained multimodal sentiment prediction model is obtained.

[0066] In the application phase, such as Figure 3 As shown, the cockpit camera, microphone array, and in-vehicle voice recognition module acquire visual, speech, and text information, respectively, and input them into the in-vehicle intelligent computing platform. The in-vehicle intelligent computing platform calls the trained model to output the driver's emotion prediction value ŷ, and triggers corresponding safety warnings, vibration alerts, and voice interaction feedback based on the prediction value.

[0067] To further verify the effectiveness and technical performance of the proposed method under incomplete multimodal data conditions, the publicly available CMU-MOSI multimodal sentiment analysis dataset was selected for experimental verification, and compared with existing representative multimodal sentiment analysis methods. During the experiment, under the same data partitioning, input modality settings, and evaluation metrics, the performance of different methods under random modality missing conditions was tested to evaluate the adaptability of each method to incomplete multimodal input. The average robustness performance comparison results of each method under the random modality missing condition of the CMU-MOSI dataset are shown in Table 1. Table 1: Comparison of average robustness performance of the proposed system (Bi-GeM) and existing representative methods under the random modality missing condition of the CMU-MOSI dataset.

[0068] As shown in Table 1, the Bi-GeM system of this invention achieves good overall robustness under the condition of missing random modalities in the CMU-MOSI dataset. Specifically, the system achieves superior results in five-class classification accuracy, seven-class classification accuracy, and correlation coefficient, reaching 40.90%, 35.71%, and 0.528, respectively. In terms of binary classification accuracy and mean absolute error, the system's performance is close to that of P-RMF. These results indicate that the method of this invention, while maintaining relatively stable basic sentiment prediction performance, can further improve the ability to distinguish fine-grained sentiment states and the correlation between the predicted results and the actual sentiment labels.

Claims

1. A method for predicting multimodal driver emotions in the complex environment of an intelligent cockpit, characterized in that, Includes the following steps: Step 1: Obtain a multimodal sentiment sample sequence of any target object containing text, visual, and speech modalities within the training batch. After performing time alignment, feature normalization, and sentiment label mapping, mask the preprocessed multimodal sentiment sample sequence using a modality missing mask matrix to obtain an incomplete multimodal observation sequence. ; Step 2: Use the multimodal feature extraction module to extract features from the incomplete multimodal observation sequence. The low-level features of each mode are extracted, and the variational Gaussian distribution parameters are obtained to obtain the latent variables of each mode. ; Step 3: Utilize the memory anchoring calibration module based on a pre-built external prototype memory matrix. ,from Select the semantic prototypes that match each modality and their corresponding semantic confidence scores. Used to find the original mean in the parameters of a Gaussian distribution. Perform calibration to obtain the mean value of the m-th modality after calibration. ; Step 4: Use the dynamic proxy generation module to perform a two-factor dynamic evaluation of the physical variance and semantic confidence in the Gaussian distribution parameters to generate initial global proxy features. ; Step 5: The closed-loop semantic modulation module uses the initial global proxy features. Using an L-layer bidirectional closed-loop semantic modulation network as the modulation center, the low-level features of each modality are recursively refined to obtain the final global proxy features. ; Step 6: Utilize the sentiment prediction output module to perform analysis using a multilayer perceptron. Regression prediction is performed to obtain the sentiment prediction value. ; Step 7: Construct the joint loss function This includes: constructing a geometric alignment loss function. Emotional regression loss KL divergence loss ; Step 8: Based on the joint loss function The multimodal sentiment prediction network, consisting of a multimodal feature extraction module, a memory anchoring calibration module, a dynamic agent generation module, a closed-loop semantic modulation module, and a sentiment prediction output module, is iteratively trained until the joint loss function is reached. The training continues until convergence, resulting in a trained multimodal sentiment prediction model. This model is used to detect incomplete multimodal observation sequences and output driver sentiment prediction values.

2. The method for predicting multimodal driver emotions in complex environments of intelligent cockpits according to claim 1, characterized in that, Step 2 includes the following steps: Step 2.1, After feature extraction for each modality is performed in the input multimodal encoder, the feature is then uniformly projected to dimension 1 through a fully connected layer. The shared subspace is used to obtain the underlying features of each modality. ; Step 2.2: Extract the low-level features of the m-th mode. The input is processed in a variational encoder, and the Gaussian distribution parameters of the m-th modal feature are output, including: the original mean of the m-th modality. and logarithmic variance ,according to Obtain the standard deviation of the m-th mode. ,and Thus, the mean of any multimodal sentiment sample sequence can be obtained. and standard deviation M represents the total number of modes; Step 2.3: Obtain the latent variables of the m-th mode according to equation (1). : (1) It is standard Gaussian noise.

3. The method for predicting driver multimodal emotions in complex environments of intelligent cockpits according to claim 2, characterized in that, Step 3 includes the following steps: Step 3.1, with For the query vector, the external prototype memory matrix is ​​used. Perform cosine similarity calculation to retrieve similarities with... Best matching semantic prototype ; Step 3.2: Extract the highest cosine similarity score obtained from the retrieval and use it as the semantic confidence score of the m-th modality. ; Step 3.3, using equation (2) to... Perform calibration to obtain the mean value of the m-th modality after calibration. : (2) In equation (2), This indicates the preset anchoring coefficient.

4. The method for predicting multimodal driver emotions in complex environments of intelligent cockpits according to claim 3, characterized in that, Step 4 includes the following steps: Step 4.1: Calculate the fusion weight of the m-th mode using equation (3). : (3) Step 4.2, for Normalization is performed to obtain the normalized weights of the m-th mode. ; Step 4.3: Generate initial global proxy features using equation (4). : (4) In equation (4), This represents the total number of modes.

5. A method for predicting multimodal driver emotions in complex environments of intelligent cockpits according to claim 4, characterized in that, Step 5 includes the following steps: Step 5.1, in the... In layer modulation, the first Global proxy features output by the layer As the query vector, it absorbs the low-level features of each modality in layer l through a cross-modal attention mechanism. The effective information in the equation is used to obtain the injection features corresponding to the m-th mode using equation (5). : (5) In equation (5), This represents the cross-modal attention function corresponding to the m-th modality; Indicates the first The global proxy features output by the layer; Indicates the first The bottom-level features of the m-th mode in the layer; when season = ; Step 5.2: Use equation (6) to obtain the first... Global proxy features output by the layer : (6) In equation (6), This represents the normalized fusion weight of the m-th modality; Representation layer normalization; Indicates a fully connected mapped network; Indicates residual connection; Step 5.3: The gating signal generation unit will generate the first... Global proxy features output by the layer Input feedback gating network, and through After activation function processing, the suppression signal corresponding to the m-th mode is generated using equation (7). : (7) In equation (7), and They represent the first Feedback gating network parameters corresponding to each mode; Step 5.4: The recursive purification calculation unit uses equation (8) to obtain the first... The m-th mode of the bottom layer features : (8) In equation (8), This indicates bitwise multiplication. Step 5.5, Assign to Then, return to step 5.1 and continue execution until completion. Layer modulation, ultimately obtaining the first The final global proxy feature output by the layer .

6. The method for predicting multimodal driver emotions in complex environments of intelligent cockpits according to claim 5, characterized in that, Step 7 includes: Step 7.1: Following steps 1-2.2, obtain the [number]th [item] within the training batch. The mean of a multimodal sentiment sample sequence and standard deviation and the The mean of a multimodal sentiment sample sequence and standard deviation Therefore, the first equation (9) is used to calculate the second equation. The multimodal emotion sample sequence and the first The squared second-order Wasserstein distance term between the multimodal sentiment sample sequences : (9) In equation (9), and Let represent the Gaussian distributions corresponding to the i-th and j-th multimodal sentiment sample sequences, respectively; Step 7.2: Construct the first equation using equation (10). The multimodal emotion sample sequence and the first Variance-stable regularization term among multimodal sentiment sample sequences : (10) Step 7.3: Construct the first equation using equation (11). The multimodal emotion sample sequence and the first The total distance term between the multimodal sentiment sample sequences : (11) In equation (11), Preset weighting coefficients; Step 7.4: Assign the true sentiment label to the i-th multimodal sentiment sample sequence within the training batch. and the Real sentiment labels for a multimodal sentiment sample sequence Discretize them separately to obtain the corresponding results. Emotion category identifier and Emotion category identifier Therefore, the geometric alignment loss function is constructed using equation (12). : (12) In equation (12), N represents the total number of multimodal sentiment sample sequences in the training batch; (·) indicates an indicator function; This represents the preset category separation interval threshold; Step 7.5: Construct the emotional regression loss using equation (13). : (13) Step 7.6: Construct the KL divergence loss using equation (14). : (14) In equation (14), This represents the posterior distribution of the i-th multimodal sentiment sample sequence within the training batch, obtained through a variational encoder. Indicates the standard Gaussian prior distribution; Indicates the KL divergence; Step 7.7: Construct the task loss function using equation (15). : (15) In equation (15), express The corresponding weighting coefficients, express The corresponding weighting coefficients; Step 7.8: Obtain the joint loss function using equation (16). : (16) In equation (16), express The corresponding weighting coefficients.

7. A multimodal emotion prediction system for drivers in complex intelligent cockpit environments, characterized in that, It is used to perform any one of the methods of claims 1-6, and includes: Multimodal feature extraction module: used to extract features from incomplete multimodal observation sequences. The underlying features of each modality are extracted, and the variational Gaussian distribution parameters are obtained to obtain the latent variables of each modality; wherein, the incomplete multimodal observation sequence X is obtained from the public multimodal sentiment analysis dataset in the training stage through time alignment, feature standardization, sentiment label mapping and modality missing masking, and in the application stage it is obtained from multi-source sensor data in the smart cockpit through time alignment, feature standardization and modality missing representation processing; Memory anchoring calibration module: based on a pre-built external prototype memory matrix ,from The semantic prototypes that match each modality and their corresponding semantic confidence scores are selected and used to calibrate the original mean in the Gaussian distribution parameters to obtain the calibrated mean of each modality. The dynamic proxy generation module performs a two-factor dynamic evaluation of the physical variance and semantic confidence in the Gaussian distribution parameters to generate initial global proxy features. Closed-loop semantic modulation module: It is used to recursively refine the low-level features of each modality through an L-layer bidirectional closed-loop semantic modulation network with the initial global agent features as the modulation center, so as to obtain the final global agent features. Sentiment prediction output module: used to perform regression prediction on the final global proxy features and output sentiment prediction values; Geometric manifold alignment module: used during the model training phase to construct the basic distance term, variance-stabilized regularization term, and total distance term based on the Gaussian representation of samples within the training batch, and to construct the geometric alignment loss function based on the consistency of the true sentiment labels of the training samples; The model training module is used to iteratively train the multimodal sentiment prediction system based on the joint loss function until the joint loss function converges, thereby obtaining the trained multimodal sentiment prediction model. This trained multimodal sentiment prediction model is used in the application phase to receive incomplete multimodal observation sequences obtained in real-time within the intelligent cockpit. It also outputs the driver's emotion prediction value.

Images