Multi-modal data driven driver emotion recognition method and system based on brain inspiration

Abstract

The invention discloses a brain inspiration-based multi-modal data-driven driver emotion recognition method and system. Feature extraction is performed on the face video data and the multi-source heterogeneous driving behavior data after pulse processing through a constructed pulse neural network brain-like model of a driver face video feature extraction network, a multi-source heterogeneous driving behavior feature extraction network and a multi-modal feature fusion network; and outputting a cross-modal pulse attention feature fusion result, and identifying the emotion of the driver. Experiments show that the method is remarkably superior to an existing method in the aspects of accuracy, real-time performance and robustness, and the problems that single-modal recognition accuracy is insufficient, facial video and multi-source heterogeneous driving behavior modal feature extraction is insufficient, and a modal fusion strategy is low in efficiency are effectively solved. In addition, the pulse neural network brain-like method significantly accelerates the calculation speed, enhances the anti-interference capability of the model, and provides an efficient emotion recognition solution for an intelligent driving system.

Description

Technical Field

[0001] The present invention belongs to the field of image recognition technology and relates to driver emotion recognition based on multimodal data, and specifically to a method and system for driver emotion recognition driven by multimodal data based on brain inspiration. Background Art

[0002] Driver emotion recognition is a core technology in intelligent driving systems, playing a crucial role in improving driving safety, optimizing human-machine interaction, and enabling personalized services. As a key component of autonomous driving and advanced driver assistance systems, this technology monitors the driver's mental state in real time and promptly identifies high-risk emotions such as anger, fear, and sadness. Research has shown that negative emotions can slow driver reaction times and increase the risk of accidents. Real-time warning and intervention through emotion recognition systems can effectively reduce the incidence of traffic accidents caused by negative emotions. Furthermore, emotion recognition technology can empower intelligent driving systems to deliver a more user-friendly interactive experience. For example, the system can dynamically adjust the in-vehicle environment (such as temperature, lighting, and music) based on the driver's emotional state, or alleviate driving stress through voice interaction, thereby enhancing driving comfort and safety. Furthermore, long-term emotion monitoring data can provide a valuable basis for driver behavior analysis, helping to build accurate driver profiles. This has broad application prospects in insurance pricing, professional driver assessment, and fleet safety management.

[0003] Traditional driver emotion recognition methods mostly rely on a single modality, such as facial video or driving behavior. This single-dimensional analysis method struggles to fully capture a driver's complex emotional state, limiting the accuracy of emotion recognition. In driver emotion recognition methods based on facial video, existing technologies do not fully exploit the spatiotemporal features of video sequences, making it difficult to effectively capture the dynamic characteristics of subtle changes in facial expressions. Driver emotion recognition methods based on driving behavior primarily rely on manually designed feature extractors. This artificially engineered feature engineering fails to fully exploit the deep emotional information contained in behavioral data.

[0004] The current multimodal fusion methods in the field of driver emotion recognition mainly face two technical bottlenecks: on the one hand, they fail to fully exploit the complementary information characteristics between heterogeneous multi-source modal data such as vision, physiological signals and speech in the feature extraction stage; on the other hand, they lack dynamic perception mechanisms for cross-modal correlation features and effective screening mechanisms for redundant information in the feature fusion stage. This not only leads to insufficient collaborative optimization of cross-modal semantic representations, but also causes the magnitude of model parameters and computational complexity to rise simultaneously, ultimately resulting in a significant attenuation of recognition accuracy and generalization performance, and at the same time it is difficult to meet the real-time response requirements of the vehicle system for real-time emotion monitoring.

[0005] In summary, finding a new method with excellent real-time performance, high accuracy and strong robustness to replace the existing driver emotion recognition method has become an urgent problem that needs to be solved. Summary of the Invention

[0006] In response to the shortcomings of the existing technology, the present invention proposes a brain-inspired multimodal data-driven driver emotion recognition method and system. In the process of feature extraction and fusion, a brain-like spiking neural network (SNN) is introduced. The membrane potential characteristics of biological neurons are reproduced based on the LIF neuron model. An event-driven pulse mechanism is adopted to respond with low latency and efficiently process real-time data streams, significantly improving the recognition speed, accuracy and robustness of driver emotions.

[0007] The brain-inspired multimodal data-driven driver emotion recognition method includes the following steps: Step 1: Collect driver facial video data and multi-source heterogeneous driving behavior data while the vehicle is in motion. Perform frame-by-frame face alignment on the driver's facial video data to ensure the face is positioned correctly. Interpolate the multi-source heterogeneous driving behavior data to map the original data to a new time series, ensuring a consistent temporal distribution. The processed facial video data, multi-source heterogeneous driving behavior data, and their corresponding labels serve as training samples for the model's supervised learning.

[0008] The multi-source heterogeneous driving behavior data includes a time series of vehicle steering wheel position, accelerator pedal position, brake pedal force, forward acceleration, lateral acceleration, forward speed, lateral speed, and vertical speed.

[0009] Step 2: Construct a brain-inspired multimodal driver emotion recognition spiking neural network model, including a driver facial video feature extraction network, a multi-source heterogeneous driving behavior feature extraction network, and a multimodal feature fusion network. The facial video features are extracted from the pulsed facial video data and the multi-source heterogeneous driving behavior data. F FV and multi-source heterogeneous driving behavior characteristics F DB After fusion, the output is the seven-category driver emotion recognition results of surprise, fear, disgust, happiness, sadness, anger, and neutrality.

[0010] An image feature extraction network and a time series feature extraction network were constructed. A spike neuron layer (SN) was added after the convolutional layer to serve as the driver facial video feature extraction network and the multi-source heterogeneous driving behavior feature extraction network. The spike neuron layer used multistep leaky integrate-and-fire neurons (Multistep LIF) for data processing.

[0011] The multimodal feature fusion network is a cross-modal pulse attention network, which introduces multi-step leaky integral release neurons in the multi-head attention mechanism. F FV As a query matrix, multi-source heterogeneous driving behavior features F DB As key matrix and value matrix, generate enhanced multi-source heterogeneous driving behavior information features with driver facial video Attn FV->DB ; Based on multi-source heterogeneous driving behavior characteristics F DB As the query matrix, facial video features F FV As key matrix and value matrix, generate enhanced driver facial video features with driving behavior information Attn DB->FV ;Will Attn FV->DB and Attn DB->FV Output fusion features after splicing F , and finally output the driver’s emotion classification through the fully connected layer.

[0012] Step 3: Input the training samples obtained in step 1 into the brain-inspired multimodal driver emotion recognition spiking neural network model constructed in step 2 to predict the driver's emotion classification results. .

[0013] During the training process, the total task is divided into three subtasks. Subtask 1 uses facial video features independently. F FV For sentiment classification, subtask 2 independently uses multi-source heterogeneous driving behavior features F DB Perform sentiment classification, while subtask 3 uses fusion features F For sentiment classification, the total loss function Loss is:

[0014]

[0015]

[0016]

[0017]

[0018]

[0019] in, yRepresents the label corresponding to the training sample, CrossEntropyLoss(‧) represents the cross entropy loss. Loss1, Loss2 and Loss3 represent the loss functions of subtask 1, subtask 2 and subtask 3 respectively. α 、 β 、 γ is the weight coefficient of the corresponding loss. 、 Represent the classification results of subtask 1 and subtask 2 respectively.

[0020] Step 4: Deploy the trained multimodal driver emotion recognition pulse neural network model on the vehicle-mounted embedded edge computing platform, collect driver facial videos and multi-source heterogeneous driving behavior data in real time, input the trained model, perform feature extraction and fusion, and output the recognition results.

[0021] A brain-inspired multimodal data-driven driver emotion recognition system includes a data acquisition module, a feature extraction and fusion module, and a recognition module.

[0022] The data acquisition module is used to collect the driver's facial video data and multi-source heterogeneous driving behavior data, align the facial video data by frame, and form a consistent time distribution of the multi-source heterogeneous driving behavior data through interpolation, and then transmit it to the feature extraction and fusion module.

[0023] The feature extraction and fusion module extracts features from the facial video data and multi-source heterogeneous driving behavior data using the trained multi-modal driver emotion recognition pulse neural network model to generate facial video features. F FV and multi-source heterogeneous driving behavior characteristics F DB , output the fusion feature F to the recognition module.

[0024] The recognition module outputs a driver emotion recognition result based on the fusion feature F. The driver emotion recognition result is specifically one of surprise, fear, disgust, happiness, sadness, anger, and neutrality.

[0025] The present invention has the following beneficial effects: 1. This method introduces a spiking neuron layer that only emits pulse signals when necessary, significantly accelerating computational speed, improving computational efficiency, and reducing resource consumption. Furthermore, its sparse emission mechanism enhances the model's robustness to noise, making it more resilient to interference. Furthermore, the spiking neuron layer's temporal modeling capabilities better reflect the temporal characteristics of driver emotion changes.

[0026] 2. We propose using facial video and multi-source, heterogeneous driving behavior data as non-invasive inputs to a driver emotion recognition model. Unlike traditional invasive driver emotion recognition methods based on electroencephalogram (EEG), electrocardiogram (ECG), and electrical skin signals, this method does not require direct contact with the driver, resulting in greater comfort and safety. Furthermore, the collected data is highly correlated with the driver's emotions, enabling more precise emotion recognition with a high recognition accuracy rate.

[0027] 3. In the feature fusion process, the complementarity and redundancy of inter-modal features are fully considered, and the model is optimized in combination with multi-task learning strategies to achieve deep fusion and collaborative modeling of multimodal features, effectively improving the overall performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Schematic diagram of face alignment for driver's facial video frame; Figure 2 Schematic diagram of the test sample; Figure 3 Schematic diagram of the multimodal driver emotion recognition spiking neural network model; Figure 4 Schematic diagram of the network structure for extracting driver facial video features; Figure 5 Schematic diagram of the network structure for extracting multi-source heterogeneous driving behavior features; Figure 6 Schematic diagram of the multimodal feature fusion network structure. DETAILED DESCRIPTION

[0029] The present invention will be further explained below with reference to the accompanying drawings; Example 1 This embodiment provides a brain-inspired multimodal data-driven driver emotion recognition method, the specific steps of which are as follows: Step 1: Collect the driver's facial video data and multi-source heterogeneous driving behavior data during vehicle driving.

[0030] The driver's facial video data is collected using a near-infrared camera. This is because the near-infrared camera can obtain clear and stable driver's facial images in low light, normal light, and strong light environments, while the visible light camera performs poorly. Using near-infrared video can enhance the robustness of the model.

[0031] like Figure 1 As shown, face alignment is performed frame by frame for the driver's facial video data to ensure that the face is located in the standard position. The specific steps are as follows: s1.1、Use the 68-point facial key point detector of Dlib library to accurately locate the facial feature points in each frame image and obtain the key point coordinate set P={( xi , y i )},in( x i , y i ) is the i The coordinates of the feature points i =1,2,…68. Calculate the left eye center based on the detected feature point coordinates of the left and right eyes C L ( x L , y L ) and the center of the right eye C R ( x R , y R )'s exact location:

[0032]

[0033] in, n L 、 n R Represents the number of feature points for the left and right eyes respectively, Left 、 Righ t represents the feature point sets of the left and right eyes respectively.

[0034] s1.2. Calculate the center of the left eye C L ( x L , y L ) and the center of the right eye C R ( x R , y R ) d x and vertical distance d y , thus obtaining the angle between the left and right eyes θ , as the rotation angle of the current frame:

[0035]

[0036]

[0037] Compare the actual distance between the left and right eyes with the set target distanceD x The ratio between s , as the image scaling factor:

[0038] Then calculate the target translation ( t x , t y ).

[0039] s1.3, according to the rotation angle θ , Zoom s and the translation amount ( t x , t y ), construct the affine transformation matrix M , process the original image frame to obtain the aligned face image frame: .

[0040] s1.4. To improve computational efficiency while maintaining recognition accuracy and ensure that the model can effectively capture key emotional changes, each driver's facial video data is evenly divided into 8 segments. Two frames are randomly selected from each segment, thus converting each video into a facial image sequence containing 16 frames and a size of 112×112 as the facial video sample. X FV ∈R 16×3×112×112 .

[0041] The multi-source heterogeneous driving behavior data is collected for 3 seconds at a frequency of 60 times per second by on-board sensors, including the vehicle steering wheel position, accelerator pedal position, brake pedal force, forward acceleration, lateral acceleration, forward speed, lateral speed and vertical speed. Therefore, a section of multi-source heterogeneous driving behavior data can be represented as a matrix of size 8×180. Such high-frequency sampling and multi-dimensional data structure can provide rich spatiotemporal information and provide reliable data support for the emotion recognition model. The vehicle steering wheel position, accelerator pedal position, brake pedal force, and forward and lateral acceleration are collected by the steering wheel angle sensor, position sensor, force sensor and accelerometer respectively, and the forward speed, lateral speed and vertical speed are calculated by the accelerometer. Since there are uneven timestamps between the original collected multi-source data, the time distribution is aligned by the following method: s1.5. Generate a new time series with a time range of 0 to 3 seconds and 180 data points evenly distributed.

[0042] s1.6, for adjacent original data d t+ d t-, calculate the interpolation data d corresponding to the new time point tc tc :

[0043] Among them, d t+ >d tc >d t- Through the interpolation method, the original data is mapped to the new time series so that the data has a consistent time distribution.

[0044] s1.7. Perform normalization to prevent large values ​​of certain features from affecting model training, thereby improving training efficiency and obtaining multi-source heterogeneous driving behavior samples. X DB ∈R 8×180 .

[0045] After the above preprocessing, a total of 3500 sample pairs consisting of facial video samples and multi-source heterogeneous driving behavior samples were obtained. Typical sample data are as follows: Figure 2 As shown in the figure, the dataset includes driving scenarios such as urban roads, highways, rural roads, and tunnels under various weather and lighting conditions. It covers factors such as age, gender, driving experience, and educational background, with a balanced distribution of samples across categories. During dataset construction, each sample pair was labeled with the driver's emotion category based on the triggering emotion. 2,800 sample pairs were randomly selected as the training set, and the remaining 700 sample pairs were used as the test set.

[0046] Step 2: Build Figure 3 The brain-inspired multimodal driver emotion recognition spiking neural network model shown in the figure includes a driver facial video feature extraction network, a multi-source heterogeneous driving behavior feature extraction network, and a multimodal feature fusion network. It extracts facial video features and multi-source heterogeneous driving behavior features respectively, and outputs seven-category driver emotion recognition results after fusion.

[0047] The driver's facial video feature extraction network is shown as Figure 4 As shown, it includes a cascaded multi-layer convolutional network, a spatial feature extraction network, and a temporal feature extraction network: s2.1, for facial video samples X FV ∈R 16×3×112×112 ,First, the image is pulsed frame by frame, and then preliminary feature extraction is performed through a multi-layer convolutional network.

[0048] The multi-layer convolutional network performs a 7×7 convolution kernel on the input image with a stride of 2 and a padding of 3, mapping the image to a 64-channel feature space. The input then passes through the Spike Neuron Layer (SN), which uses multi-step leaky integral firing neurons (Multistep LIF) for processing. Next, a 3×3 maximum pooling layer (stride of 2) is used to complete the initial downsampling operation. The data after the initial downsampling is then f 0 enters 4 consecutive pulse residual convolution blocks, each of which consists of two consecutive convolutional layers, convolution batch normalization (Convolution BatchNormalization, ConvBN) and a pulse neuron layer:

[0049] Among them, ConvBN(‧) represents the convolution batch normalization operation; SN(‧) represents the spiking neuron layer; f a-1 Indicates the a The input feature map of the pulse residual convolution block is f a It is a The pulse residual convolution block outputs the feature map, a=1,2,3,4. Conv 3×3 (‧) represents a convolutional layer of size 3×3. Finally, the output of the fourth consecutive residual convolution block is subjected to adaptive average pooling to output a preliminary feature map. M ∈R 512×1×1 .

[0050] s2.2, the preliminary feature map M Flattened to a one-dimensional sequence M f ∈R 512 , and then input into the spatial feature extraction network. The spatial feature extraction network includes spatial position embedding and spatial encoder. In spatial position embedding, the feature sequence M f with learnable position embeddings e p Add and encode the spatial information of each position in the feature map to help the network understand the spatial relationship between different positions as the input of the spatial encoder z p 0 :

[0051] The spatial encoder includes L consecutive encoding layers, each layer including a spiking self-attention mechanism and a feedforward neural network. l In the layer spatial encoder, the query vector is first generated through layer normalization LN(‧), linear transformation and LIF neurons , key vector Sum vector :

[0052]

[0053]

[0054] in, 、 、 Respectively represent l The first layer of the spatial encoder k The weight matrix of the multi-head attention head, where k =1,2,…, K , K Represents the total number of attention heads.

[0055] No. p The self-attention weight vector of feature points :

[0056] Rule No. k The output of a multi-head attention for:

[0057] in, , indicating the k In the attention head p The feature point pair p The attention weight of each feature point. represents the latent dimension of the attention head.

[0058] After the outputs of K attention heads are spliced ​​on the feature dimension, they are passed through the linear projection matrix W l Transformed, and mapped through layer normalization and feedforward neural network MLP(‧), and then through the pulse neuron layer and residual connection, the first l The output of the layer spatial encoder:

[0059]

[0060] After L layers of spatial encoders, the encoded result is obtained ,Will Q Encoding results 、 … Splicing in the spatial dimension to obtain the feature map Mr ∈R H×W×C , and then calculate the feature map Mr Spatial feature embedding :

[0061] Where Q represents a one-dimensional sequence M f GAP(‧) represents the global average pooling operation.

[0062] s2.3. Embedding spatial features Input temporal feature extraction network. The temporal feature extraction network includes temporal positional embedding and T Layer Temporal Encoder. In the temporal position embedding, the spatial features are embedded and time coding Add to encode the time position:

[0063] in, Represents a facial video sample X FV A sequence of 16 frames of images consisting of spatial feature embedding results, Represents the introduced learnable variables. Input the time encoder, the structure of the time encoder is the same as that of the space encoder. As facial video features F FV .

[0064] The multi-source heterogeneous driving behavior feature extraction network is as follows: Figure 5 As shown: s2.4, first input multi-source heterogeneous driving behavior data X DB ∈R 8×180 Pulse a time series:

[0065] Where Pulse(‧) represents the time series pulse operation, hRepresents a pulsed time series.

[0066] s2.5. Then, through two consecutive Inception residual modules and a global average pooling layer (GlobalAverage Pooling), the Inception residual module consists of three Inception units. The input signal d of the Inception residual module is residually connected with the feature InceptioUnit(d) processed by the second Inception module, and then input into the third Inception unit for further feature extraction, resulting in the output ResidualBlock(d) of the Inception residual module:

[0067] The input of each Inception unit first passes through a one-dimensional convolution layer Conv1() with a convolution kernel size of 1 and a stride of 1, and then enters three one-dimensional convolution layers Conv1d() with convolution kernel sizes of 39, 19, and 9 and strides of 19, 9, and 4 respectively. At the same time, the Inception unit also introduces a one-dimensional maximum pooling operation MaxPool1d() and a one-dimensional convolution layer in parallel, and after each convolution layer, a convolution batch normalization layer ConvBN() and a spike neuron layer SN() are designed to obtain the output of four channels. h 1. h 2. h 3. h 4:

[0068]

[0069]

[0070]

[0071] Among them, the subscript represents the size of the convolution kernel, h In Represents the input of the Inception unit.

[0072] The output features of each channel are then concatenated and normalized through the one-dimensional batch normalization layer BatchNorm1d() to obtain the output InceptioUnit(d) of the Inception unit:

[0073] Concat(‧) represents the concatenation operation.

[0074] s2.6. Finally, the model introduces a global average pooling layer GAP() to generate an output feature vector representing the multi-source heterogeneous driving behavior characteristics F DB :

[0075] The multimodal feature fusion network is as follows Figure 6 As shown, the core of the network adopts the pulse attention mechanism: s2.7, facial video features are transformed through the fully connected layer F FV and multi-source heterogeneous driving behavior characteristics F DB Mapped to a unified dimension, then, the Unsqueeze operation is used to insert a new dimension, expressed as and .

[0076] s2.8, using facial video features as queries and multi-source heterogeneous driving behavior features as keys and values, input facial video modality features into the cross-modal pulse attention layer, so that it is mapped to Q FV 、 K DB and V DB :

[0077]

[0078]

[0079] in, W Q 、 W K 、 W V is a learnable linear mapping matrix, which is divided into multiple heads. Subsequently, the pulse attention mechanism performs attention weighted calculations on each head in parallel, and finally concatenates the outputs of each head together:

[0080]

[0081] in, Heads Indicates the number of attention heads. Output features Attn FV->DB It is an enhanced multi-source heterogeneous driving behavior information feature with driver’s facial video.

[0082] s2.8. Similarly, using multi-source heterogeneous driving behavior features as queries and driver facial video features as keys and values, we can obtain enhanced driver facial video features with multi-source heterogeneous driving behavior information. Attn DB->FV .

[0083] Reduce the dimension of the output features and perform concat operation (‧) to obtain the final fusion features F :

[0084] Finally, the driver's emotion classification results are output :

[0085] Where FC(‧) represents a fully connected neural network.

[0086] Step 3: Input the training set samples in step 1 into the brain-inspired multimodal driver emotion recognition spiking neural network model constructed in step 2, set the initial learning rate and learning rate decay strategy, perform multi-task iterative training on the model, and use the SGD optimizer to automatically adjust the weight parameters of the neural network to continuously reduce the loss function value, and finally obtain a trained model.

[0087] The overall task is divided into three subtasks to supervise the different modules to fully learn features. Subtask 1 independently uses features from the facial video modality for classification, subtask 2 independently uses features from multi-source heterogeneous driving behavior modalities for classification, and subtask 3 uses the fused features for classification:

[0088]

[0089]

[0090]

[0091]

[0092]

[0093] in, 、 Represent the classification results of subtask 1 and subtask 2 respectively. y Represents the label of the input sample pair, CrossEntropyLoss(‧) represents the cross entropy loss. Loss1, Loss2, Loss3, and Loss represent the loss functions of subtask 1, subtask 2, subtask 3, and the total task respectively. α 、β 、 γ is the weight coefficient of the corresponding loss.

[0094] Step 4: Compare the performance of this method with different models on the test set. The experimental environment is: Intel® Core™ i9-13900K CPU, dual RTX4090 GPUs with 48GB of video memory, Ubuntu 20.04 operating system, and Pytorch deep learning framework. Accuracy (Acc) and F1 score were selected as evaluation metrics for the various emotion recognition results. The results are shown in Tables 1 and 2: Table 1

[0095] Table 2

[0096] The data in Tables 1 and 2 show that our proposed method achieves the best overall performance, with an average accuracy of 68.74% and a Macro F1 score of 68.73% across seven emotion categories. Both accuracy and robustness surpass those of other algorithms. It demonstrates a particularly significant advantage in the recognition of disgust (Acc: 69.17%, F1: 73.17%) and happiness (Acc: 73.24%, F1: 73.74%). Other methods, such as CogEmoNet and Emotion-FAN, perform reasonably well on some emotions, but lack overall balance. Former-DFER performs poorly on most emotions. The multimodal method M3DFEL demonstrates strong competitiveness but significantly underperforms our proposed method in anger recognition.

[0097] In addition, the average inference time of different models on the test set is compared, and the results are shown in Table 3: Table 3

[0098] The smaller the inference time, the stronger the model's real-time performance. The data in Table 3 show that this method performs excellently in terms of inference time, at only 9.74 milliseconds, significantly outperforming other recognition algorithms. CogEmoNet's inference time is 28.31 milliseconds, and Emotion-FAN's is 30.64 milliseconds. Other algorithms, such as CEFLNET, IAL, and MARLIN, all have inference times greater than 30 milliseconds. Compared to MARLIN's inference time of 92.11 milliseconds and CEFLNET's 89.04 milliseconds, this method significantly improves real-time performance and possesses strong real-time processing capabilities. This demonstrates the superior real-time performance, high accuracy, and strong robustness of this method in the driver emotion recognition task.

[0099] Example 2 Based on Example 1, this embodiment uses the Transformer improved by the pulse neuron layer as the driver's facial video feature extraction network and the multi-source heterogeneous driving behavior feature extraction network to extract facial video features. F FV and multi-source heterogeneous driving behavior characteristics F DB , and then input into the multimodal feature fusion network for feature fusion, using the fusion feature F Carry out driver emotion classification. The specific improvement process is as follows: Step 1: Replace the continuous activation function in Transformer with a pulse neuron model to implement event-driven sparse computing:

[0100]

[0101]

[0102] where τ represents the membrane time constant of the spiking neuron, X [ t ] represents the input current corresponding to time step t. Θ() represents the Heaviside step function, which outputs 0 when the input value is negative and 1 when it is negative. H [ t ] exceeds the threshold voltage V th When an action potential is triggered S ( t ), and then the membrane potential is reset to the resting potential V reset ; If no action potential is triggered, the membrane potential will remain unchanged at the current value.

[0103] Step 2: Restructure the residual module order, eliminate integer-floating point multiplication operations, and ensure that all calculations comply with the pulse drive principle:

[0104] in, O l-1 、 O l Represent the input and output of the residual block respectively. Before the convolution calculation begins, the input data O l-1First, a spike operation is performed to ensure that the data input to the convolutional layer consists of features consisting only of 0s and 1s, ensuring that the calculation process conforms to the pulse-driven calculation rules. After the convolution calculation is completed, the subsequent normalization operation strictly limits the output to a range of 0 to 1. As a result, in the reconstructed residual module, the data is always forwarded as a sequence of 0s and 1s, which conforms to the pulse-driven calculation characteristics of the spiking neural network.

[0105] Step 3: Replace the self-attention mechanism in the Transformer with a pulse form to generate a pulsed query matrix Q, key matrix K, and value matrix V, eliminating floating-point operations:

[0106]

[0107]

[0108] in X' Represents the features of the input attention calculation layer. The attention calculation process at this time can be expressed as:

[0109] in s Denotes the scaling factor. Due to the pulsed operation, the query matrix Q, key matrix K, and value matrix V are all composed of 0s and 1s. The attention calculation process is implemented by masking operations and event-driven accumulation, avoiding floating-point multiplication operations. This feature significantly reduces computational energy consumption.

[0110] Step 4: According to the pulse neuron model in step 1, the action potential S ( t ) is controlled by the Heaviside step function Θ() at the membrane potential H [ t ] exceeds the threshold voltage V th However, the step function is not differentiable at the threshold, which makes the traditional gradient descent method ineffective. The Sigmoid derivative has a significant gradient only near the threshold, which is more consistent with the sensitivity of neurons to membrane potential changes. Therefore, Sigmoid ( αH [ t ]) as a differentiable approximation of the Heaviside step function Θ(), which enables spiking neural networks to be trained using gradient descent:

[0111] in α represents the approximation coefficient.

[0112] The aforementioned transformation process exhibits clear decoupling characteristics. Activation function replacement and gradient approximation serve as fundamental operator modifications, applicable to any ANN architecture containing nonlinear activations. Residual reconstruction and attention pulsing, on the other hand, are specific module adaptations of the Transformer. After this transformation, all feature transfer is accomplished via binary pulse trains, and operations like convolution and attention are transformed into counting and logical operations on pulse events. This establishes a closed-loop computational flow of "pulse generation → pulse propagation → pulse accumulation," building a discrete event processing mechanism that closely resembles biological neural networks. This fundamental shift from traditional numerical computation to an event-driven paradigm provides a theoretically comprehensive deployment solution for low-power scenarios such as automotive equipment.

[0113] Example 3 Based on Example 1, this embodiment provides a brain-inspired multimodal data-driven driver emotion recognition system, including a data acquisition module, a feature extraction and fusion module, and a recognition module.

[0114] The data acquisition module is used to collect the driver's facial video data and multi-source heterogeneous driving behavior data.

[0115] The feature extraction and fusion module extracts features from the facial video data and multi-source heterogeneous driving behavior data using the trained multi-modal driver emotion recognition pulse neural network model to generate facial video features. F FV and multi-source heterogeneous driving behavior characteristics F DB , output the fusion feature F to the recognition module.

[0116] The recognition module outputs the driver emotion recognition result based on the fusion feature F.

[0117] The above specific embodiments are used to illustrate the present invention rather than to limit the present invention. Any modifications and changes made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A brain-inspired multimodal data-driven driver emotion recognition method collects multimodal driver behavior data during vehicle driving, extracts features by mode, and then uses a multimodal feature fusion network to fuse features. Finally, based on the fused features F Output the recognition results of the driver's emotions , characterized in that: Collect driver's facial video data and multi-source heterogeneous driving behavior data as multimodal driver behavior data; Construct a feature extraction network based on deep learning, replace the continuous activation function of the convolutional layer with a pulse neuron model, and use multi-step leaky integration to release neurons for data processing; pulse the multimodal driver behavior data and input it into the feature extraction network of the corresponding modality to extract facial video features F FV and multi-source heterogeneous driving behavior characteristics F DB ; The multimodal feature fusion network is a cross-modal pulse attention network, which introduces multi-step leaky integral release neurons into the multi-head attention mechanism to generate enhanced multi-source heterogeneous driving behavior information features with driver facial videos across modalities. Attn FV->DB and enhanced driver facial video features with driving behavior information Attn DB->FV , output fusion feature F=Concat( Attn FV->DB , Attn DB->FV ).

2. The brain-inspired multimodal data-driven driver emotion recognition method according to claim 1, characterized in that: The driver's facial video data is collected by a near-infrared camera; The multi-source heterogeneous driving behavior data includes vehicle steering wheel position, accelerator pedal position, brake pedal force, forward acceleration, lateral acceleration, forward speed, lateral speed and vertical speed; among them, the vehicle steering wheel position, accelerator pedal position, brake pedal force, and forward and lateral acceleration are collected by a steering wheel angle sensor, position sensor, force sensor and accelerometer respectively; the forward speed, lateral speed and vertical speed are calculated by an accelerometer.

3. The brain-inspired multimodal data-driven driver emotion recognition method according to claim 1, characterized in that: For the driver's facial video data, face alignment is performed frame by frame to ensure that the face is in the standard position and generate facial video samples X FV ; For multi-source heterogeneous driving behavior data, the original collected data is mapped to a new time series through interpolation, so that the multi-source driving behavior data has a consistent time distribution and generates multi-source heterogeneous driving behavior samples. X DB .

4. The brain-inspired multimodal data-driven driver emotion recognition method according to claim 3, characterized in that: For facial video samples X FV Perform feature extraction. The specific steps are as follows: s2.1, for facial video samples X FV First, the image is pulsed by frame, and after a convolution kernel mapping, a pulse neuron layer and a maximum pooling layer are used to complete the initial downsampling operation. Then, the data after the initial downsampling is f 0 enters multiple consecutive pulse residual convolution blocks and outputs preliminary feature maps M ; s2.2, the preliminary feature map M Flattened to a one-dimensional sequence M f , then M f with learnable position embeddings e p Add and encode the spatial information of each position in the feature map to obtain the spatial position feature z p 0 , the spatial position feature z p 0 Input L consecutive spatial encoding layers, use the pulse self-attention mechanism and feedforward neural network to encode the spatial position, and obtain the feature map Mr ; s2.

3. Calculating feature maps Mr Spatial feature embedding x ' and with time coding Add together and get Input T consecutive temporal encoding layers, output facial video features through the pulse self-attention mechanism and feedforward neural network F FV .

5. The brain-inspired multimodal data-driven driver emotion recognition method according to claim 3, characterized in that: For multi-source heterogeneous driving behavior samples X DB Perform feature extraction. The specific steps are as follows: Input multi-source heterogeneous driving behavior data X DB The time series is pulsed, and then two consecutive Inception residual modules and a global average pooling layer GAP () are used for feature extraction to output multi-source heterogeneous driving behavior features. F DB : ； The Inception residual module includes three Inception units InceptioUnit(): ； Among them, d and ResidualBlock(d) represent the input and output of the Inception residual module respectively; Pulse() represents the pulse operation.

6. The brain-inspired multimodal data-driven driver emotion recognition method according to claim 3, characterized in that: Transformer improved by spiking neuron layer for facial video samples X FV and multi-source heterogeneous driving behavior samples X DB Feature extraction is performed separately, and the specific improvement process is as follows: Replace the continuous activation function in Transformer with a spike neuron model. When the membrane potential of the spike neuron model H [ t ] exceeds the threshold voltage V th When the action potential is triggered S ( t ), and then the membrane potential is reset to the resting potential V reset If no action potential is triggered, the membrane potential will remain unchanged at the current moment; Reconstruct the residual module calculation order: ； in, O l-1 、 O l Respectively represent l The input and output of the residual block, SN l (‧) indicates the spike neuron layer, ConvBN l (‧) represents the convolution batch normalization operation; The self-attention mechanism in Transformer is replaced with a pulse form, generating a pulsed query matrix Q, key matrix K, and value matrix V, and outputting the attention feature SSA(Q,K,V).

7. The brain-inspired multimodal data-driven driver emotion recognition method according to claim 1, characterized in that: Labeling facial video samples X FV and multi-source heterogeneous driving behavior samples X DB Corresponding emotion labels y , and the recognition results of the driver's emotions output by the network Compare and calculate the cross entropy loss Loss3; Calculate only facial video features separately F FV and using only multi-source heterogeneous driving behavior features F DB Results of emotion recognition 、 With emotion labels y The cross entropy losses Loss1 and Loss2 are used, and the total loss function Loss is set to the weighted sum of Loss1~Loss3. The parameters of the feature extraction network and the multimodal feature fusion network are trained.

8. The brain-inspired multimodal data-driven driver emotion recognition method according to claim 7, characterized in that: Set the initial learning rate and learning rate decay strategy, and use the SGD optimizer to automatically adjust the network weight parameters.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 8.

10. A brain-inspired, multimodal data-driven driver emotion recognition system, characterized by: Used to implement the method according to any one of claims 1 to 8, comprising a data acquisition module, a feature extraction and fusion module, and a recognition module; The data acquisition module is used to collect the driver's facial video data and multi-source heterogeneous driving behavior data to generate facial video samples X FV and multi-source heterogeneous driving behavior samples X DB Then it is transmitted to the feature extraction and fusion module; The feature extraction and fusion module uses the trained feature extraction network to extract facial video samples. X FV and multi-source heterogeneous driving behavior samples X DB Perform feature extraction and output the fusion feature F through the trained multimodal feature fusion network; The recognition module outputs the driver emotion recognition result based on the fusion feature F.

Images