Multi-mode driver abnormal state detection method and system based on three-expert network

Abstract

The invention discloses a multi-modal driver abnormal state detection method and system based on a three-expert network. The method comprises the following steps: acquiring an acquired face or upper body video image, an electroencephalogram time sequence signal and a myoelectricity time sequence signal of a driver; the collected three modal signals are preprocessed; inputting the preprocessed three modal signals into a three-expert network for feature extraction to respectively obtain a visual space feature sequence, an electroencephalogram time domain feature sequence and a myoelectricity activity feature sequence; and carrying out feature fusion on the visual space feature sequence, the electroencephalogram time domain feature sequence and the myoelectricity activity feature sequence by adopting a cross-modal attention fusion mechanism to obtain a fusion feature vector, and outputting a driver state classification result after global average pooling, full connection layer and focus loss function optimization are carried out on the fusion feature vector. According to the method, deep cross-modal fusion of visual, electroencephalogram and myoelectricity three-modal signals is realized, and the recognition accuracy and robustness of the abnormal state of the driver are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of driver state detection technology, specifically to a multimodal driver abnormal state detection method and system based on a three-expert network. Background Technology

[0002] Driving has become an indispensable mode of transportation in modern society, providing people with an efficient and convenient means of mobility and promoting economic development and social exchange. However, with the rapid increase in the number of motor vehicles and the continuous increase in road traffic density, the frequency of traffic accidents remains high, posing a serious threat to public safety and the lives and property of the people. These accidents cause huge casualties and economic losses, and road traffic injuries have become a major burden on global public health.

[0003] In these traffic accidents, human factors are the primary cause, accounting for over 90%. Common human factors include speeding, drunk driving, fatigued driving, distracted driving, and emotional distress. Fatigue leads to slowed reaction time, decreased attention, and impaired judgment; according to research, the fatality rate in major traffic accidents caused by fatigued driving is extremely high. Distracted driving (such as using a mobile phone, adjusting devices, or talking to passengers) is equally deadly; data from the National Highway Traffic Safety Administration (NHTSA) shows that distracted driving is a major cause of fatal collisions. In an emotionally agitated state, drivers are prone to aggressive actions, further amplifying the risk of accidents. These abnormal states often intertwine and are more pronounced during prolonged driving or in complex road conditions, significantly reducing a driver's perception and ability to respond to the road environment, thus becoming a major hidden danger in traffic accidents.

[0004] In practical applications for detecting abnormal driver conditions, the following prominent shortcomings still exist:

[0005] 1. Most solutions rely heavily on a single modality, especially the visual modality, which leads to a significant drop in detection performance in low light, occlusion (such as sunglasses or hats), or privacy-protected scenarios; pure physiological modality solutions are susceptible to motion artifacts and lack stability.

[0006] 2. Physiological signal acquisition devices are often highly invasive, requiring special headbands or multiple electrode attachments, which can be uncomfortable to wear for extended periods. Furthermore, the processing of EEG and EMG signals is highly complex, making it difficult to maintain reliable operation during actual driving.

[0007] 3. The vehicle operation parameter scheme is based on indirect reasoning, which makes it difficult to capture early physiological abnormalities and has limited ability to distinguish between different abnormal states at a fine level.

[0008] 4. Most existing multimodal solutions lack sufficient fusion depth, often employing simple feature layer concatenation or decision layer weighting. This fails to fully exploit the deep complementary relationships between visual spatial information, EEG temporal dynamics, and EMG activity patterns, resulting in limited recognition accuracy and generalization ability in complex abnormal states (such as fatigue accompanied by emotional fluctuations) or cross-individual scenarios. Furthermore, their computational complexity is high, making real-time performance unsuitable for in-vehicle embedded deployments. Summary of the Invention

[0009] To address the shortcomings of existing technologies, this invention provides a multimodal driver abnormality detection method based on a three-expert network, which achieves deep cross-modal fusion of visual, EEG, and EMG signals, significantly improving the accuracy and robustness of driver abnormality detection.

[0010] In a first aspect, embodiments of the present invention provide a multimodal driver abnormal state detection method based on a three-expert network, comprising:

[0011] Acquire video images of the driver's face or upper body, electroencephalogram (EEG) time-series signals, and electromyogram (EMG) time-series signals;

[0012] The three modal signals were preprocessed to obtain preprocessed visual video frames, preprocessed EEG timing signals, and preprocessed EMG timing signals.

[0013] The three modal signals after preprocessing are input into a three-expert network for feature extraction. The three-expert network includes a visual spatial expert branch, an EEG temporal expert branch, and an electromyographic activity expert branch. The preprocessed visual video frames are input into the visual spatial expert branch to extract visual spatial feature sequences, the preprocessed EEG temporal signals are input into the EEG temporal expert branch to extract EEG temporal feature sequences, and the preprocessed electromyographic temporal signals are input into the electromyographic activity expert branch to extract electromyographic activity feature sequences.

[0014] A cross-modal attention fusion mechanism is used to fuse visual spatial feature sequences, EEG temporal feature sequences, and electromyographic activity feature sequences to obtain a fused feature vector. After optimization of the fused feature vector by global average pooling, fully connected layers, and a focus loss function, the driver state classification result is output.

[0015] Furthermore, the specific method for inputting the preprocessed visual video frames into the visual spatial expert branch to extract the visual spatial feature sequence includes:

[0016] The preprocessed visual video frames are divided into patches of a set size, and after flattening, they are linearly projected into a patch marker sequence.

[0017] The patch-labeled sequence is input into a multi-layer Transformer block, each layer including a multi-head self-attention module and a feedforward network. Spatial topological features are extracted step by step through residual connections and layer normalization, and at least one spatial feature from facial expression, eye movement trajectory and head pose is extracted.

[0018] Furthermore, the specific method for inputting the preprocessed EEG temporal signal into the EEG temporal domain expert branch to extract the EEG temporal domain feature sequence includes:

[0019] The preprocessed EEG time-series signal is subjected to channel expansion and local pattern extraction through an initial one-dimensional convolutional block to obtain the extracted feature sequence.

[0020] The extracted feature sequence is input into a dilated temporal convolutional network block for processing to obtain the feature sequence.

[0021] Input the feature sequence into a long sequence spatial model block or an alternative Transformer encoder, and output the EEG time-domain feature sequence.

[0022] Furthermore, the specific method for inputting the preprocessed electromyographic timing signal into the electromyographic activity expert branch to extract the electromyographic activity feature sequence includes:

[0023] The preprocessed electromyographic timing signal is input into a stacked one-dimensional ConvNeXt block to extract upper limb muscle tension and activation pattern features, and outputs an electromyographic activity feature sequence.

[0024] Furthermore, the specific method for feature fusion using a cross-modal attention fusion mechanism includes:

[0025] Independent linear projections were performed on the visual spatial feature sequence, the EEG temporal feature sequence, and the electromyographic activity feature sequence, respectively.

[0026] The visual spatial feature projection results are used as query vectors to represent the context of current visual attention; the EEG temporal feature projection results are used as key vectors to represent the state of neural activity; and the electromyographic activity feature projection results are used as value vectors to represent the specific actions performed by the muscles.

[0027] The scaling dot product attention mechanism is used for computation, so that the context of visual attention dominates the weighted attention to EEG and EMG physiological signals;

[0028] Multi-head parallel expansion is employed to capture cross-modal complementary relationships in different subspaces.

[0029] Furthermore, the methods for preprocessing and synchronizing the three acquired modal signals specifically include:

[0030] The three modal signals are filtered, denoised, and timestamp aligned, and then a sliding window is used for segmentation.

[0031] Secondly, embodiments of the present invention provide a multimodal driver abnormal state detection system based on a three-expert network, used to implement the multimodal driver abnormal state detection method based on a three-expert network provided in the first embodiment above, the system comprising:

[0032] The data acquisition module is used to acquire video images of the driver's face or upper body, EEG timing signals, and EMG timing signals.

[0033] The preprocessing module is used to preprocess the three modal signals acquired to obtain preprocessed visual video frames, preprocessed EEG timing signals, and preprocessed EMG timing signals.

[0034] The feature extraction module employs a three-expert network to extract features from the three modal signals after preprocessing. The three-expert network includes a visual spatial expert branch, an EEG temporal expert branch, and an electromyography (EMG) activity expert branch. It is used to input the preprocessed visual video frames into the visual spatial expert branch to extract visual spatial feature sequences, input the preprocessed EEG temporal signals into the EEG temporal expert branch to extract EEG temporal feature sequences, and input the preprocessed EMG temporal signals into the EMG activity expert branch to extract EMG activity feature sequences.

[0035] The cross-modal attention fusion module is used to fuse visual spatial feature sequences, EEG temporal feature sequences, and electromyographic activity feature sequences using a cross-modal attention fusion mechanism to obtain a fused feature vector. After the fused feature vector is optimized by global average pooling, fully connected layers, and a focus loss function, the driver state classification result is output.

[0036] Furthermore, the feature extraction module includes a visual spatial feature extraction unit, an EEG temporal feature extraction unit, and an electromyography activity feature unit. The visual spatial feature extraction unit is used to divide the preprocessed visual video frame into patches of a set size, and after flattening, linearly project it into a patch marker sequence.

[0037] The patch-marked sequence is input into a multi-layer Transformer block, each layer including a multi-head self-attention module and a feedforward network. Spatial topological features are extracted step by step through residual connections and layer normalization, and at least one spatial feature among facial expression, eye movement trajectory and head pose is extracted.

[0038] The EEG temporal feature extraction unit is used to perform channel expansion and local pattern extraction on the preprocessed EEG temporal signal through an initial one-dimensional convolutional block to obtain the extracted feature sequence.

[0039] The extracted feature sequence is input into a dilated temporal convolutional network block for processing to obtain the feature sequence.

[0040] Input the feature sequence into a long sequence spatial model block or an alternative Transformer encoder, and output the EEG time-domain feature sequence;

[0041] The electromyographic activity feature unit is used to input the preprocessed electromyographic timing signal into a stacked one-dimensional ConvNeXt block to extract upper limb muscle tension and activation pattern features, and output an electromyographic activity feature sequence.

[0042] Thirdly, another embodiment of the present invention provides an electronic device including a processor, an input device, an output device, and a memory, wherein the processor, the input device, the output device, and the memory are interconnected, the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to invoke the program instructions to execute the method described in the first embodiment of the present invention.

[0043] Fourthly, another embodiment of the present invention provides a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in the first embodiment of the present invention.

[0044] The beneficial effects of this invention are:

[0045] This invention provides a multimodal driver abnormality detection method based on a three-expert network. Through deep cross-modal attention fusion of visual, EEG, and EMG modalities, it significantly improves the accuracy and robustness of driver abnormality detection. The visual modality provides spatial behavioral context, the EEG modality captures neural temporal dynamics, and the EMG modality reflects changes in muscle activity. The complementary nature of these three modalities enables high-precision recognition under varying lighting conditions, slight occlusion, or complex conditions. Experimental results demonstrate that the average accuracy is higher than existing shallow multimodal solutions.

[0046] This method transfers and optimizes the three-expert network structure, constructing visual spatial expert branches, EEG temporal expert branches, and EMG activity expert branches respectively. It also designs a cross-modal attention fusion mechanism with visual features as queries, EEG features as keys, and EMG features as values, to achieve efficient deep feature interaction and complementary enhancement. This solves the shortcomings of existing technologies, such as insufficient fusion depth and difficulty in mining deep complementary relationships between multiple modalities.

[0047] The proposed method features a highly efficient three-expert network structure that supports real-time in-vehicle embedded reasoning. It also addresses class imbalance through focus loss, making it highly practical and easy to integrate into driver assistance systems. This enables early detection and intervention of abnormal states, significantly reducing the risk of traffic accidents and demonstrating significant application value and promising prospects for wider adoption.

[0048] The multimodal driver abnormal state detection system based on a three-expert network provided in this embodiment of the invention has the same inventive concept and the same beneficial effects as the multimodal driver abnormal state detection method based on a three-expert network provided in this embodiment of the invention, and will not be described again here. Attached Figure Description

[0049] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0050] Figure 1 The diagram shows a flowchart of a multimodal driver abnormal state detection method based on a three-expert network provided by an embodiment of the present invention.

[0051] Figure 2 The flowchart illustrates the fusion of extracted feature sequences using a cross-modal attention mechanism.

[0052] Figure 3 A schematic diagram of the structure of the EEG time-domain expert branch is shown;

[0053] Figure 4 A schematic diagram of the electromyographic activity expert branch is shown;

[0054] Figure 5 A schematic diagram of the structure of the visual space expert branch is shown;

[0055] Figure 6 The diagram shows a structural schematic of a multimodal driver abnormal state detection system based on a three-expert network according to another embodiment of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0058] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0059] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items, as well as all possible combinations, and includes such combinations.

[0060] like Figure 1 The diagram illustrates a multimodal driver abnormality detection method based on a three-expert network according to a first embodiment of the present invention, comprising the following steps:

[0061] Acquire video images of the driver's face or upper body, electroencephalogram (EEG) time-series signals, and electromyogram (EMG) time-series signals;

[0062] The three modal signals were preprocessed to obtain preprocessed visual video frames, preprocessed EEG timing signals, and preprocessed EMG timing signals.

[0063] The three modal signals after preprocessing are input into a three-expert network for feature extraction. The three-expert network includes a visual spatial expert branch, an EEG temporal expert branch, and an electromyographic activity expert branch. The preprocessed visual video frames are input into the visual spatial expert branch to extract visual spatial feature sequences, the preprocessed EEG temporal signals are input into the EEG temporal expert branch to extract EEG temporal feature sequences, and the preprocessed electromyographic temporal signals are input into the electromyographic activity expert branch to extract electromyographic activity feature sequences.

[0064] A cross-modal attention fusion mechanism is used to fuse visual spatial feature sequences, EEG temporal feature sequences, and electromyographic activity feature sequences to obtain a fused feature vector. After optimization of the fused feature vector by global average pooling, fully connected layers, and a focus loss function, the driver state classification result is output.

[0065] The specific methods for preprocessing the three types of signals collected include: bandpass filtering (20Hz-50Hz for EEG, 20Hz-500Hz for EMG), denoising, and timestamp alignment. Then, samples are segmented using a fixed-length sliding window (e.g., 2000ms with a step size of 200ms) to ensure that the state is consistent within each window, resulting in preprocessed visual video frames, preprocessed EEG time-series signals, and preprocessed EMG time-series signals.

[0066] The preprocessed signals from the three modalities are input into a three-expert network architecture for feature extraction. This three-expert network includes a visual-spatial expert branch, an EEG temporal expert branch, and an EMG activity expert branch. The visual-spatial expert branch employs residual blocks and a two-dimensional convolutional neural network backbone (such as a ResNet structure) to effectively extract visual-spatial topological features such as facial expressions, eye movements, and head posture, capturing the spatial contextual information of driver behavior. The EEG temporal expert branch uses dilated temporal convolutional network blocks (TCN blocks) and long-sequence state-space model blocks (Mamba blocks) to efficiently process long-sequence neural activity signals, extracting dynamic EEG patterns such as alpha and theta waves, capturing early temporal signs of fatigue or emotional fluctuations, and outputting a temporal EEG feature sequence. The EMG activity expert branch uses one-dimensional ConvNeXt blocks. ConvNeXt is a modern convolutional neural network (ConvNet) designed to re-explore the potential of convolutional networks by drawing inspiration from the design principles of Transformers. The electromyography (EMG) activity expert branch is dedicated to multi-channel EMG signal processing, extracting physiological activity features such as upper limb muscle tension and activation patterns, and outputting an EMG activity feature sequence that reflects muscle changes under distraction or emotional excitement. A Mamba architecture with linear complexity is introduced to resolve the contradiction between processing long-sequence physiological data and real-time warning. Determining driving status requires analyzing long-term physiological evolution; the computational complexity of the traditional Transformer architecture increases quadratically with sequence length (O(N)). 2 This leads to high latency in in-vehicle embedded computing platforms. This invention innovatively employs a selective state-space model (Mamba) in the time expert branch, reducing the time complexity of long sequence processing to a linear level (O(N)). This not only significantly reduces the computational overhead of in-vehicle hardware but also ensures that the system can achieve millisecond-level real-time early warning response when it detects signs of distraction or fatigue.

[0067] like Figure 2 The diagram illustrates a flowchart of the cross-modal attention mechanism for fusing extracted feature sequences. First, each output feature sequence is independently linearly projected. Then, the visual spatial feature projection result is used as the query matrix Q, representing the context of the current visual attention; the EEG temporal feature projection result is used as the key matrix K, representing the state of neural activity; and the electromyographic activity feature projection result is used as the value matrix V, representing the specific action performed by the muscle.

[0068] Attention calculation employs a scaled dot product attention mechanism, as shown in formula (1):

[0069]

[0070] in, The key vector dimension is used. This mechanism allows the visual spatial context to dominate the weighted attention to EEG and EMG physiological signals. By using the macroscopic environmental context captured visually as the query vector, the system is guided to perform weight allocation and feature alignment in the physiological feature pool. This fusion strategy can automatically focus on the physiological signal features most relevant to the current driving behavior, effectively filtering motion artifact noise caused by vehicle bumps, and achieving deep, high-quality multimodal information fusion.

[0071] Further multi-head expansion is adopted, as shown in formulas (2) and (3):

[0072]

[0073] in, The number of heads is the number of heads in this embodiment. =8. The multi-head mechanism allows for the parallel capture of cross-modal complementary relationships across different subspaces. The fusion result is processed through residual connections, layer normalization, and linear projection to output a unified fused feature vector. This effectively enables vision-driven deep interaction, ensuring complementary enhancement of information from the three modalities.

[0074] The detailed structure of the EEG temporal domain expert branch is as follows: Figure 3 As shown. This branch receives the preprocessed EEG temporal signal, first performs channel expansion and local pattern extraction through an initial one-dimensional convolutional block to obtain the extracted feature sequence; then it enters a four-layer dilated temporal convolutional network (TCN) block, with dilation rates of 1, 2, 4, and 8 respectively, to exponentially expand the receptive field and obtain the feature sequence. The receptive field calculation for this process is shown in formula (4):

[0075]

[0076] in, The kernel size is [size]. The number of layers is specified. This design efficiently captures long-term temporal dependencies in EEG signals, such as power variations in alpha and theta waves. Subsequently, the feature sequence is input into a long-sequence state-space model block (Mamba block) or an alternative Transformer encoder to further model global temporal dynamics, ultimately outputting a temporal feature sequence of EEG. This branch is specifically designed to extract early signs of fatigue or emotional disturbances in neural activity.

[0077] The detailed structure of the electromyographic activity expert branch is as follows: Figure 4 As shown. This branch receives the electromyographic timing signal and directly inputs it into the stacked one-dimensional ConvNeXt block. This block adopts a depthwise separable convolution structure, and the calculation process is shown in formula (5):

[0078]

[0079] In this architecture, DWConv is a depthwise convolution (kernel size 7, grouped convolution), and PWConv is a 1×1 point convolution used for channel mixing. This structure, combined with residual connections, ensures smooth gradient propagation. Optionally, a channel attention module is then added to further enhance the weights of key muscle channels, ultimately outputting an electromyographic activity (EMG) feature sequence. This branch is specifically designed to capture changes in upper limb muscle tension and identify activation patterns under emotional excitement or fatigue. Traditional EMG detection schemes typically extract only one-dimensional time-frequency features, ignoring the spatial distribution information of the electrode array on the human epidermis. The EMG activity branch in this embodiment maps high-density sensor data into a two-dimensional spatial activation map, capturing the spatial fingerprint of coordinated muscle group contractions through a residual convolutional network. This enables precise differentiation of similar action sequences (such as normal gear shifting and limb tremors caused by fatigue), greatly enhancing the representational ability of refined driving behavior.

[0080] The detailed structure of the visual space expert branch is as follows: Figure 5 As shown. This branch receives a sequence of video frames and first divides each frame into fixed-size patches using a block embedding module. After flattening, the patches are linearly projected into a patch-marked sequence. Then, it enters a multi-layer Transformer block, each layer including a multi-head self-attention network (calculated as in formulas (1) and (3)) and a feedforward network (MLP). Spatial topological features, such as facial expressions, eye movements, and head poses, are extracted step by step through residual connections and layer normalization. This branch is dedicated to providing the spatial context of driver behavior, supporting the accurate identification of distracted actions. By mining high-density electrode topological features through the spatial expert branch, the analytical accuracy of refined abnormal actions is improved.

[0081] The feature sequences output by the three expert branches are fused through a cross-modal attention mechanism, and the resulting fused feature vector is input into the classification head. This classification head includes a global average pooling layer, a fully connected layer, a dropout layer, and a softmax activation function. It is optimized using a focus loss function, as shown in Equation (6).

[0082]

[0083] in, The predicted probability of the correct category. As a balance factor, This is a focused parameter. The loss function effectively mitigates the class imbalance problem where normal driving samples far outnumber abnormal states, ultimately outputting the driver state classification result.

[0084] This invention provides a multimodal driver abnormality detection method based on a three-expert network. Through deep cross-modal attention fusion of visual, EEG, and EMG modalities, it significantly improves the accuracy and robustness of driver abnormality detection. The visual modality provides spatial behavioral context, the EEG modality captures neural temporal dynamics, and the EMG modality reflects changes in muscle activity. The complementary nature of these three modalities enables high-precision recognition under varying lighting conditions, slight occlusion, or complex conditions. Experimental results demonstrate that the average accuracy is higher than existing shallow multimodal solutions.

[0085] This method transfers and optimizes the three-expert network structure, constructing visual spatial expert branches, EEG temporal expert branches, and EMG activity expert branches respectively. It also designs a cross-modal attention fusion mechanism with visual features as queries, EEG features as keys, and EMG features as values, to achieve efficient deep feature interaction and complementary enhancement. This solves the shortcomings of existing technologies, such as insufficient fusion depth and difficulty in mining deep complementary relationships between multiple modalities.

[0086] The proposed method features a highly efficient three-expert network structure that supports real-time in-vehicle embedded reasoning. It also addresses class imbalance through focus loss, making it highly practical and easy to integrate into driver assistance systems. This enables early detection and intervention of abnormal states, significantly reducing the risk of traffic accidents and demonstrating significant application value and promising prospects for wider adoption.

[0087] like Figure 6 The diagram illustrates the overall structure of a multimodal driver abnormality detection system based on a three-expert network, according to another embodiment of the present invention. The system includes: a data acquisition module for acquiring video images of the driver's face or upper body, EEG timing signals, and EMG timing signals; a preprocessing module for preprocessing the acquired three modal signals to obtain preprocessed visual video frames, preprocessed EEG timing signals, and preprocessed EMG timing signals; and a feature extraction module that uses a three-expert network to extract features from the preprocessed three modal signals. The three-expert network includes a visual spatial expert branch, an EEG temporal expert branch, and an EMG activity expert branch, and is used to input the preprocessed visual video frames into the visual spatial... The system extracts visual spatial feature sequences from the inter-branch expert network, inputs the preprocessed EEG temporal signal into the EEG temporal domain expert network to extract EEG temporal feature sequences, and inputs the preprocessed EMG temporal signal into the EMG activity expert network to extract EMG activity feature sequences. The cross-modal attention fusion module is used to fuse the visual spatial feature sequences, EEG temporal feature sequences, and EMG activity feature sequences using a cross-modal attention fusion mechanism to obtain a fused feature vector. After the fused feature vector is optimized by global average pooling, fully connected layers, and a focus loss function, the driver state classification result is output.

[0088] The data acquisition module includes a driver-facing camera, a wearable EEG headband, and a wearable EMG armband. The camera is used to acquire video images of the face or upper body. The wearable EEG headband acquires EEG timing signals through multi-channel dry electrodes, and the wearable EMG armband acquires upper limb EMG timing signals through multi-channel surface electrodes. The three modalities are acquired synchronously in real time and then transmitted to the preprocessing module. The preprocessing module performs bandpass filtering (20Hz-50Hz for EEG, 20Hz-500Hz for EMG), denoising, and timestamp alignment. Then, a fixed-length sliding window (e.g., 2000ms, 200ms step) is used to segment the samples, ensuring consistency within each window, resulting in preprocessed visual video frames, preprocessed EEG timing signals, and preprocessed EMG timing signals. The three modal signals after preprocessing are input into the feature extraction module in parallel. The feature extraction module uses a three-expert network architecture to extract features from the three modal signals after preprocessing. Finally, the driver state classification results are output after being integrated by the cross-modal attention fusion module, including categories such as normal driving, fatigue (mild / severe), and distraction (phone operation / other actions).

[0089] The three-expert network includes a visual spatial expert branch, an EEG temporal expert branch, and an electromyography (EMG) activity expert branch. The preprocessed visual video frames are input into the visual spatial expert branch, which uses residual blocks and a two-dimensional convolutional neural network backbone (such as a ResNet structure) to effectively extract visual spatial feature sequences such as facial expressions, eye movements, and head posture, thereby capturing the spatial context information of the driver's behavior.

[0090] The preprocessed EEG time-series signal is input into the EEG time-domain expert branch, which uses dilated temporal convolutional network blocks (TCN Blocks) and long sequence state space model blocks (Mamba Blocks) to efficiently process long-term neural activity signals, extract EEG dynamic patterns such as alpha waves and theta waves, capture early temporal signs of fatigue or emotional fluctuations, and obtain EEG time-domain feature sequences.

[0091] The preprocessed electromyographic timing signal is input into the electromyographic activity expert branch, which uses a one-dimensional ConvNeXt block dedicated to multi-channel electromyographic signal processing. It extracts physiological activity features such as upper limb muscle tension and activation patterns to obtain an electromyographic activity feature sequence that reflects muscle changes under distraction or emotional excitement.

[0092] Three expert branches output visual spatial feature sequences, EEG temporal feature sequences, and EMG activity feature sequences, respectively, which are then input into a cross-modal attention fusion module. This module uses the visual spatial feature sequence as the query, the EEG temporal feature sequence as the key, and the EMG activity feature sequence as the value. Through a multi-head attention mechanism, it achieves deep interactive fusion dominated by vision, allowing spatial contextual information to guide the weighted enhancement of physiological signals. Simultaneously, the temporal / activity features of EEG and EMG complementarily feed back into the visual representation, forming a unified fusion feature vector. This fusion method fully utilizes the complementarity of the three modalities, avoids information loss caused by shallow splicing, and improves the robustness of recognition under complex abnormal states.

[0093] The execution process of each module can be carried out according to the steps of the multimodal driver abnormal state detection method based on a three-expert network provided in the first embodiment, and will not be described in detail in this embodiment.

[0094] This invention provides a multimodal driver abnormal state detection system based on a three-expert network. Through deep cross-modal attention fusion of visual, EEG, and EMG modalities, it significantly improves the accuracy and robustness of abnormal state detection. The visual modality provides spatial behavioral context, the EEG modality captures neural temporal dynamics, and the EMG modality reflects changes in muscle activity. The complementary nature of these three modalities enables high-precision recognition under varying lighting conditions, slight occlusion, or complex conditions. Experimental results demonstrate that the average accuracy is higher than existing shallow multimodal solutions.

[0095] While retaining the advantages of visual modality, EEG and EMG are introduced as physiological supplements to effectively alleviate the environmental sensitivity problem of pure visual solutions. At the same time, portable headband and armband devices are used, which are comfortable to wear, have little interference, are suitable for long-term real driving monitoring, and achieve reliable detection in all weather conditions.

[0096] By fully exploring the characteristics of each modality through expert branch design and integrating the attention mechanism of modules, we can achieve efficient deep interaction, avoid information loss and computational redundancy in existing solutions, and improve cross-individual generalization ability and the accuracy of distinguishing fine-grained states (such as mild fatigue and severe fatigue).

[0097] The three-expert network structure is highly efficient, supports real-time inference in vehicle-mounted embedded systems, and handles class imbalance through a focus loss function. It is highly practical and easy to integrate into driver assistance systems, enabling early detection and intervention of abnormal states, significantly reducing the risk of traffic accidents. It has significant application value and promising prospects for promotion.

[0098] Another embodiment of the present invention provides an electronic device, which includes a processor, an input device, an output device, and a memory. The processor, the input device, the output device, and the memory are interconnected. The memory is used to store a computer program, which includes program instructions. The processor is configured to call the program instructions to execute the method described in the first embodiment above.

[0099] It should be understood that, in the embodiments of the present invention, the processor may be a Central Processing Unit (CPU), but it may also be other general-purpose processors, graphics processing units (GPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0100] Input devices may include touchpads, microphones, etc., and output devices may include displays (LCDs, etc.), speakers, etc.

[0101] The memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store information about the device type.

[0102] In specific implementations, the processor, input device, and output device described in the embodiments of the present invention can execute the implementation methods described in the method embodiments of the present invention, or they can execute the implementation methods described in the system embodiments of the present invention, which will not be repeated here.

[0103] In another embodiment of the present invention, a computer-readable storage medium is provided, which stores a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the method described in the first embodiment above.

[0104] The computer-readable storage medium can be an internal storage unit of the terminal described in the foregoing embodiments, such as the terminal's hard drive or memory. The computer-readable storage medium can also be an external storage device of the terminal, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the terminal. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of the terminal. The computer-readable storage medium is used to store the computer program and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0105] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0106] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the terminals and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0107] In the several embodiments provided in this application, it should be understood that the disclosed terminals and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices or units, or may be electrical, mechanical or other forms of connection.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A multimodal driver abnormal state detection method based on a three-expert network, characterized in that, include: Acquire video images of the driver's face or upper body, electroencephalogram (EEG) time-series signals, and electromyogram (EMG) time-series signals; The three modal signals were preprocessed to obtain preprocessed visual video frames, preprocessed EEG timing signals, and preprocessed EMG timing signals. The three modal signals after preprocessing are input into a three-expert network for feature extraction. The three-expert network includes a visual spatial expert branch, an EEG temporal expert branch, and an electromyographic activity expert branch. The preprocessed visual video frames are input into the visual spatial expert branch to extract visual spatial feature sequences, the preprocessed EEG temporal signals are input into the EEG temporal expert branch to extract EEG temporal feature sequences, and the preprocessed electromyographic temporal signals are input into the electromyographic activity expert branch to extract electromyographic activity feature sequences. A cross-modal attention fusion mechanism is used to fuse visual spatial feature sequences, EEG temporal feature sequences, and electromyographic activity feature sequences to obtain a fused feature vector. After optimization of the fused feature vector by global average pooling, fully connected layers, and a focus loss function, the driver state classification result is output.

2. The method as described in claim 1, characterized in that, The specific method for inputting the preprocessed visual video frames into a visual spatial expert branch to extract visual spatial feature sequences includes: The preprocessed visual video frames are divided into patches of a set size, and after flattening, they are linearly projected into a patch marker sequence. The patch-labeled sequence is input into a multi-layer Transformer block, each layer including a multi-head self-attention module and a feedforward network. Spatial topological features are extracted step by step through residual connections and layer normalization, and at least one spatial feature from facial expression, eye movement trajectory and head pose is extracted.

3. The method as described in claim 2, characterized in that, The specific method for inputting the preprocessed EEG temporal signal into the EEG temporal domain expert branch to extract the EEG temporal domain feature sequence includes: The preprocessed EEG time-series signal is subjected to channel expansion and local pattern extraction through an initial one-dimensional convolutional block to obtain the extracted feature sequence. The extracted feature sequence is input into a dilated temporal convolutional network block for processing to obtain the feature sequence. Input the feature sequence into a long sequence spatial model block or an alternative Transformer encoder, and output the EEG time-domain feature sequence.

4. The method as described in claim 3, characterized in that, The specific method for inputting the preprocessed electromyographic timing signal into the electromyographic activity expert branch to extract the electromyographic activity feature sequence includes: The preprocessed electromyographic timing signal is input into a stacked one-dimensional ConvNeXt block to extract upper limb muscle tension and activation pattern features, and outputs an electromyographic activity feature sequence.

5. The method as described in claim 4, characterized in that, The specific methods for feature fusion using a cross-modal attention fusion mechanism include: Independent linear projections were performed on the visual spatial feature sequence, the EEG temporal feature sequence, and the electromyographic activity feature sequence, respectively. The visual spatial feature projection results are used as query vectors to represent the context of current visual attention; the EEG temporal feature projection results are used as key vectors to represent the state of neural activity; and the electromyographic activity feature projection results are used as value vectors to represent the specific actions performed by the muscles. The scaling dot product attention mechanism is used for computation, so that the context of visual attention dominates the weighted attention to EEG and EMG physiological signals; Multi-head parallel expansion is employed to capture cross-modal complementary relationships in different subspaces.

6. The method as described in claim 1, characterized in that, The specific methods for preprocessing and synchronizing the three acquired modal signals include: The three modal signals are filtered, denoised, and timestamp aligned, and then a sliding window is used for segmentation.

7. A multimodal driver abnormal state detection system based on a three-expert network, characterized in that, For implementing the multimodal driver abnormal state detection method based on a three-expert network as described in any one of claims 1-6, the system comprises: The data acquisition module is used to acquire video images of the driver's face or upper body, EEG timing signals, and EMG timing signals; The preprocessing module is used to preprocess the three modal signals acquired to obtain preprocessed visual video frames, preprocessed EEG timing signals, and preprocessed EMG timing signals. The feature extraction module employs a three-expert network to extract features from the three modal signals after preprocessing. The three-expert network includes a visual spatial expert branch, an EEG temporal expert branch, and an electromyography (EMG) activity expert branch. It is used to input the preprocessed visual video frames into the visual spatial expert branch to extract visual spatial feature sequences, input the preprocessed EEG temporal signals into the EEG temporal expert branch to extract EEG temporal feature sequences, and input the preprocessed EMG temporal signals into the EMG activity expert branch to extract EMG activity feature sequences. The cross-modal attention fusion module is used to fuse visual spatial feature sequences, EEG temporal feature sequences, and electromyographic activity feature sequences using a cross-modal attention fusion mechanism to obtain a fused feature vector. After the fused feature vector is optimized by global average pooling, fully connected layers, and a focus loss function, the driver state classification result is output.

8. The system as described in claim 7, characterized in that, The feature extraction module includes a visual spatial feature extraction unit, an EEG temporal feature extraction unit, and an electromyography activity feature unit. The visual spatial feature extraction unit is used to divide the preprocessed visual video frame into patches of a set size, and after flattening, linearly project it into a patch marker sequence. The patch-marked sequence is input into a multi-layer Transformer block, each layer including a multi-head self-attention module and a feedforward network. Spatial topological features are extracted step by step through residual connections and layer normalization, and at least one spatial feature among facial expression, eye movement trajectory and head pose is extracted. The EEG temporal feature extraction unit is used to perform channel expansion and local pattern extraction on the preprocessed EEG temporal signal through an initial one-dimensional convolutional block to obtain the extracted feature sequence. The extracted feature sequence is input into a dilated temporal convolutional network block for processing to obtain the feature sequence. Input the feature sequence into a long sequence spatial model block or an alternative Transformer encoder, and output the EEG time-domain feature sequence; The electromyographic activity feature unit is used to input the preprocessed electromyographic timing signal into a stacked one-dimensional ConvNeXt block to extract upper limb muscle tension and activation pattern features, and output an electromyographic activity feature sequence.

9. An electronic device comprising a processor, an input device, an output device, and a memory, wherein the processor, input device, output device, and memory are interconnected, and the memory is used to store a computer program, the computer program comprising program instructions, characterized in that, The processor is configured to invoke the program instructions to perform the method as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1-6.

Images