Multi-mode driver abnormal behavior detection method giving consideration to privacy protection

Abstract

The invention discloses a multi-mode driver abnormal behavior detection method giving consideration to privacy protection. Performing privacy protection processing on the face video of the driver by adopting a dynamic visual mask technology based on block compressed sensing, and hiding a non-key region while keeping key physiological region information; synchronously extracting facial expression spatio-temporal features and rPPG physiological signal features from the video data after privacy protection; bimodal feature interaction is realized through a cross-modal complementary module based on a Mama framework, and the influence of redundant information is reduced by using a selective scanning mechanism; and a multi-level adaptive fusion module is adopted to carry out progressive feature fusion, so that end-to-end detection of abnormal behaviors of the driver is realized. According to the invention, only the vehicle-mounted camera is used as a data source, no additional sensor equipment is needed, and good practicability and deployment convenience are achieved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent traffic safety, and relates to computer vision, deep learning and signal processing technology, and in particular to a multi-modal driver abnormal behavior detection method considering privacy protection. BACKGROUND

[0002] In recent years, with the rapid growth of motor vehicle ownership and the increasing complexity of road environment, the situation of traffic safety is becoming more and more severe. According to the Minnesota 2023 traffic safety report, about 94% of traffic accidents are closely related to the abnormal driving behavior of drivers, which has brought heavy life safety hazards and economic losses to the society. When drivers are in a negative emotional state such as anger, dissatisfaction or anxiety, the incidence of dangerous behaviors such as speeding, lane changing, tailing and emergency braking significantly increases, and such behaviors often lack premonitions, which easily leads to vehicle out of control and the increase of collision risk.

[0003] Currently, researchers have been committed to detecting abnormal driving behavior from various data sources. The common data sources are as follows: vehicle behavior-based data, driver physiological signal-based data and driver behavior performance-based data. The method based on vehicle dynamics data detects abnormal driving behavior by analyzing data such as steering angle, vehicle speed, acceleration and lane deviation. The method based on physiological signals evaluates the cognitive and emotional state of the driver by analyzing electroencephalogram (EEG), electrocardiogram (ECG), electromyogram (EMG) and other physiological signals. The method based on driver behavior uses visual cues such as facial expressions, eye movements and head posture, combined with computer vision and deep learning technology to monitor the driver's state. Compared with the above methods, the method based on driver behavior performance has the characteristics of strong real-time, non-contact adaptability and the like, and has become the most popular detection method at present. This method mainly analyzes behavior characteristics such as facial expressions, eye movement trajectories, head postures and yawning, combined with computer vision and deep learning technology, to realize real-time detection of the mental state of the driver. However, this method mainly relies on vehicle-mounted cameras to collect face images, which has a certain risk of driver privacy leakage. SUMMARY

[0004] To solve the above technical problems, a technical solution is needed that can protect the privacy of drivers and accurately detect abnormal driving behavior.

[0005] To achieve the above purpose, the application provides the following technical solutions:

[0006] A multi-modal driver abnormal behavior detection method considering privacy protection, comprising the following steps:

[0007] Step 1: The driver's face video is processed for privacy protection by using a dynamic visual mask method based on block compressed sensing, the coordinates of the key physiological regions are determined through face key point detection, a mask matrix is constructed to retain the key physiological region information, and the non-key physiological region (NEPR) is processed by compressed encoding to obtain the face video data after visual privacy protection;

[0008] Step 2: For the face video data obtained in step 1, a pre-trained ConvNeXt-T network is used to extract face expression spatial features, and the spatial feature sequence is input into a 1D convolution block to build time correlation to obtain visual modality spatiotemporal features;

[0009] Step 3: For the face video data obtained in step 1, an unsupervised method OMIT is used to extract a remote photoplethysmography signal sequence, and a 1D convolution block is used to process the remote photoplethysmography signal sequence to obtain physiological modality spatiotemporal features;

[0010] Step 4: The visual modality spatiotemporal features obtained in step 2 and the physiological modality spatiotemporal features obtained in step 3 are input into a cross-modal complementary module MCCM, and a selective scanning mechanism of a state space model SSM is used to realize interaction of the dual modality features to obtain a complementary feature representation;

[0011] Step 5: The complementary feature representation obtained in step 4 is input into a multi-level adaptive fusion module MLAF for progressive feature fusion between different levels of the feature pyramid to obtain a fusion feature;

[0012] Step 6: The fusion feature obtained in step 5 is input into a classifier to output a driver abnormal behavior detection result.

[0013] Further, the specific process of step 1 of the present application is: adjusting the size of the original video to be divisible by a preset block size B; determining the position coordinates of the key physiological regions through face key point detection, constructing a corresponding mask matrix, and multiplying the mask matrix with a Gaussian random observation matrix element by element to generate a final observation matrix; and performing block compressed sensing processing on the original video based on the final observation matrix to realize visual privacy protection.

[0014] Further, the specific process of step 2 of the present application is: using a pre-trained ConvNeXt-T network as a spatial feature extraction backbone network and adding a squeeze-and-excitation SE block to enhance the channel attention mechanism; extracting time sequence features through two one-dimensional convolution blocks, the one-dimensional convolution block including a 1D CNN, a BN and a ReLU, and adding a dropout operation to prevent overfitting.

[0015] Further, the specific process of step 3 of the present application is: performing face alignment according to the face key points obtained in step 1, extracting the area with high signal quality in the face through dynamic area selection, converting the RGB signal into a remote photoelectric plethysmogram (rPPG) signal by using a QR decomposition algorithm, and extracting sequence features by using two consecutive one-dimensional convolution blocks.

[0016] Further, the specific process of step 4 of the present application is: performing linear mapping on the two kinds of modal features obtained in steps 2 and 3, respectively, for convolution processing and gate adjustment; sequentially performing one-dimensional deep separable convolution, batch normalization (BN) and SiLU activation function processing; splicing the processed features, and further fusing them through a shared state selection module (SSM) and a gating mechanism; and obtaining a multi-modal complementary feature representation through residual connection.

[0017] Further, the specific process of step 5 of the present application is: sequentially passing the complementary features obtained in step 4 through three one-dimensional convolution blocks to extract deep semantic information, and using a multi-level adaptive fusion strategy to gradually fuse the shallow local features into the deep global features, so as to reduce the information difference between high-dimensional and low-dimensional features.

[0018] The present application has the beneficial effect that a multi-modal driver abnormal behavior detection method considering privacy protection is proposed, which is based on driver face video data, synchronously extracts facial expression features and rPPG signals, and fuses the two modalities to realize end-to-end detection. Different from traditional multi-modal methods, the present application only relies on homologous input and does not require additional sensor data, and has higher practicability and deployment flexibility. In order to reduce the interference of redundant information between modalities, a cross-modality complementary module MCCM based on Mamba structure is designed, a selective scanning mechanism is introduced to effectively model the complex correlation between modalities, and deep complementary features are promoted. In order to realize smooth fusion, a multi-level adaptive fusion module MLAF is further proposed, which gradually fuses the information of different levels of feature pyramid from top to bottom, reduces the information gap between high-dimensional and low-dimensional features, and at the same time, in order to protect privacy, the present application uses a dynamic visual masking method based on block compressed sensing, which realizes accurate masking for key physiological regions, ensures the integrity of the rPPG signal, and effectively prevents visual privacy leakage. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 It is a flowchart of the method of the present application;

[0020] Figure 2 It is a visual privacy protection diagram of NEPR under different block sizes;

[0021] Figure 3 It is a flowchart of the MCCM module;

[0022] Figure 4 This is a flowchart of the MLAF module;

[0023] Figure 5 This is a confusion matrix diagram on the MAHNOB-HCI dataset in this embodiment. Detailed Implementation

[0024] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] This invention aims to address three key issues in existing driver abnormal behavior detection technologies: first, single-modal detection methods lack accuracy and struggle to cope with complex real-world driving environments; second, in-vehicle monitoring systems lack effective privacy protection mechanisms, posing a risk of driver facial information leakage; and third, traditional multimodal methods require additional sensor equipment, increasing system complexity and deployment costs.

[0026] See Figure 1 The present invention provides a multimodal driver abnormal behavior detection method that takes into account privacy protection, comprising the following steps:

[0027] Step 1: The driver's facial video is processed for privacy protection using a dynamic visual masking method based on block compression perception. The coordinates of key physiological regions are determined by facial key point detection, and a mask matrix is ​​constructed to retain the information of key physiological regions. Non-key physiological regions (NEPR) are compressed and encoded to obtain the facial video data with visual privacy protection.

[0028] In this embodiment, driver facial video data is obtained from the publicly available dataset MAHNOB-HCI, and OpenFace is used to perform facial detection and keypoint extraction on the video. OpenFace is an open-source facial analysis tool based on deep learning that can accurately detect 68 facial keypoints. These keypoints are labeled according to the Multi-PIE standard, specifically including: 17 facial contour points, 10 eyebrow points, 12 eye points, 9 nose points, and 20 lip region points. Based on the extracted keypoints, key physiological regions can be further subdivided, mainly the cheek region. This region is defined by a closed polygon formed by sequentially connecting keypoints P1, P29, P17, P14, P31, P4, and P1, and serves as the key region for subsequent analysis.

[0029] Compressive sensing theory shows that when a signal is sparse in a certain transform domain, it can be accurately recovered by measuring much less than the Nyquist sampling rate. In this theory, the sampling process of the signal is to project the original high-dimensional signal into a low-dimensional space through the observation matrix to obtain the sampled observation value.

[0030] In this example, the input image is divided into several image blocks of size B x B, and at the same time, in order to ensure that the image size can be divided by an integer, the input image size is adjusted so that:

[0031]

[0032] where H and W represent the height and width of the image respectively, and B represents the size of the block. In order to accurately block the image, it is adjusted to H' and W'. The adjusted image size is Secondly, block compressive sampling coding is carried out, and the observation matrix Φ used is composed of several different image block observation matrices , and since it does not involve the reconstruction problem, the size of the observation matrix in subsequent research is the same as the size of the input image , where each image block observation matrix Φ i is a Gaussian random observation matrix, and in order to solve the problem of feature loss in BCS sampling coding caused by negative values in the Gaussian random observation matrix, the following linear transformation is performed on Φ:

[0033]

[0034]

[0035] where Ф0is a Gaussian random observation matrix, and a global translation transformation and peak normalization are performed on it to obtain Ф i . The final observation matrix Ф is composed of Ф1-Ф i (i = H'W' / B 2 ). The process of final block compressive sampling coding can be represented as:

[0036] Y = ΨΦX (4)

[0037] where Ψ is a mask matrix, Φ is an observation matrix. X is the input image data, Y is the final obtained privacy protected face video data, and the visual concealment effect under different block sizes is as shown in Figure 2 , Figure Two The four columns are the visual concealment data under the conditions of B = 0, 4, 8, and 12, respectively. As the block size increases, the visual concealment effect is better. The subsequent experimental analysis is carried out under the premise of B = 12. The head-mounted device is an electroencephalogram acquisition device, and the physiological data such as electroencephalogram and electrocardiogram during the data set production have no effect on our experiment.

[0038] Step 2: For the visual-invisible face video data obtained in step 1, a pre-trained ConvNeXt-T network is used to extract face expression spatial features, and then the spatial feature sequence is input into a 1D convolution block to construct temporal correlation, and finally the spatio-temporal features of the visual modality are obtained:

[0039] In this embodiment, the privacy-protected face video data obtained in step 1 is input into a pre-trained lightweight convolutional neural network ConvNeXt-T to extract spatial features of facial expressions, and then sent to a one-dimensional convolution block to further extract temporal features, and finally output the spatio-temporal joint features of facial expressions. Specifically, ConvNeXt-T is used as the backbone network for spatial feature extraction, followed by a Squeeze-and-Excitation (SE) module to introduce a channel attention mechanism to enhance the network's focus on relevant feature channels. The SE module, as a channel self-attention mechanism, can adaptively adjust the weights of each channel, thereby strengthening key channels and suppressing irrelevant information. To improve the model's generalization ability and reduce the risk of overfitting, a transfer learning strategy is adopted: first, pre-train ConvNeXt-T on the ImageNet-22K dataset, and then fine-tune it on the facial expression recognition dataset FER2013 to improve its ability to recognize expression features.

[0040] In the temporal feature extraction stage, two consecutive one-dimensional convolution blocks (1D CNN) are used to model the temporal sequence of spatial features. 1D convolution is suitable for processing sequence data by sliding the convolution kernel along the time axis to extract local temporal features. The 1D convolution block used consists of a one-dimensional convolution layer, a batch normalization (BN) layer, and a ReLU activation function, and introduces a Dropout operation to enhance the network's non-linear representation ability and suppress overfitting. The convolution kernel size is uniformly set to 3, and the number of channels decreases layer by layer, finally outputting the spatio-temporal feature representation of the driver's facial expressions.

[0041] Step 3: For the visual-invisible face video data obtained in step 1, an unsupervised method OMIT is used to extract the rPPG signal sequence, and then a 1D convolution block is used to process the rPPG signal sequence, and finally the spatio-temporal features of the physiological modality are obtained:

[0042] rPPG is a non-contact physiological signal extraction technology based on video images, which uses the absorption and reflection changes of light on the skin to detect the periodic changes in blood volume, thereby reflecting the dynamic characteristics of blood flow during heart contraction and relaxation. By analyzing the subtle color changes in the facial skin area (such as the cheeks or forehead), the rPPG signal can be obtained.

[0043] In this embodiment, the privacy-protected face video data obtained in step 1 is input into the unsupervised rPPG extraction method OMIT to extract the rPPG signal, and then the signal is input into the 1D convolution block to extract the sequence feature. Specifically, the OMIT method first realizes face detection and alignment based on the face key points obtained in step 1, then performs dynamic region selection to adaptively select the face region with high signal quality, and in this process, the key physiological region without compression encoding can be accurately selected, so as to not affect the rPPG signal extraction quality. Finally, the RGB signal is converted into the rPPG signal through QR decomposition, and the conversion steps are as follows:

[0044] A = QR (5)

[0045] QR decomposition is performed on the RGB signal, wherein is a column orthogonal matrix, is an upper triangular matrix. Then the first column S of Q is taken to construct a projection matrix P:

[0046] P = I n -SS T (6)

[0047] where I n is an n*n identity matrix; the second column of the projected signal Y = PA contains the rPPG component. Finally, a FIR band-pass filter is used to reduce spectral leakage.

[0048] The final rPPG signal is time series data corresponding to the original video frame number. Considering the periodicity of the rPPG signal, two consecutive one-dimensional convolution blocks (1D CNN) are used to extract its features. Among them, the first layer uses a larger convolution kernel (kernel size = 20) to capture the periodic pattern, and the second layer uses a smaller convolution kernel to further extract local detail features. The extracted rPPG features can be used for emotion classification tasks, and are complementary to the facial expression features.

[0049] Step 4: input the facial expression spatio-temporal features and rPPG signal features into the cross-modal complementary module MCCM, realize the interaction of the two modal features through the selective scanning mechanism of the state space model SSM, and obtain the feature representation after the interaction:

[0050] The flowchart of the MCCM module is shown in Figure 3 In this embodiment, the facial expression features and rPPG signal features obtained in steps 2 and 3 are input into the MCCM for cross-modal feature complementation, wherein B represents batchsize, L represents time series length, and D represents feature dimension. First, the input features of the two modalities are linearly mapped respectively, and are divided into two parts for convolution processing and gate adjustment respectively:

[0051] X,G=Proj(F A ),Proj(F B ) (7)

[0052] X1,X2=Split(X);G1,G2=Split(G) (8)

[0053] where Proj denotes a linear mapping that shares the input features F A ,F B . X, G denotes the mapped feature vectors; Split denotes a linear split that divides X, G into two parts. The split X1, G1 is used for convolution processing, and X2, G2 is used for gate adjustment. Subsequently, in order to capture the local temporal features within the modal, 1D depth separable convolution, BN, SiLU are sequentially passed through:

[0054]

[0055] Subsequently, the local features of the two modalities are spliced, input into the shared SSM and subjected to gate operation to obtain:

[0056]

[0057] Finally, the features after modeling are mapped back to the two modal space, and are connected in residual connection with the original input to enhance stability:

[0058] [F' A ,F' B ]=Split(Proj(F SSM ))+[F A ,F B ] (11)

[0059] where DWConv denotes 1D depth separable convolution; BN denotes batch norm; SiLu is an activation function; SSM denotes state space modeling. Finally, through the cross-modal feature interaction of the MSSM module, the information complementarity between the two modalities is realized, irrelevant information is reduced, and the features after interaction are subsequently sent to the multi-level adaptive fusion module for feature fusion.

[0060] Step 5: input the feature representation after the interaction into the multi-level adaptive fusion module MLAF, and perform progressive feature fusion between different levels of the feature pyramid to obtain the fused features:

[0061] The flowchart of the MLAF module is shown in Figure 4 , and Figure 4 is a schematic diagram of the MLAF module, and the input is the output of the MCCM (i.e. the module introduced in step 4, the flowchart is as shown inFigure 3 The two input features pass through three 1D convolution blocks; the output of each 1D convolution is input into the FM for fusion; the output of the 3-layer FM is linearly projected and spliced to obtain the final output feature. Further, in step 4, after the features of the two modalities are interacted via the multi-modal shared attention module (MCCM), a three-layer one-dimensional convolution block (1D CNN) is used to process the interaction features layer by layer to extract deeper semantic information. To achieve the smoothness of the feature fusion process, a multi-level adaptive fusion strategy is introduced, which gradually fuses the shallow local features into the deep global features through a top-down layer-by-layer fusion method, thereby effectively alleviating the information difference between high-dimensional and low-dimensional features after fusion, and improving the completeness and discriminability of the fusion representation.

[0062] First, the features after the interaction of the two modalities are processed by a 1D convolution block to obtain output features where j e {1, 2, 3} represents the output of the three 1D convolution blocks. Then the input FM is fused for feature fusion, which is a cross-modal attention interaction mechanism based on transformer, which calculates the attention of the rPPG modality to the facial expression modality, which will combine the facial expression and the rPPG modality, and finally output the fused features to be able to fully understand the facial information. First, the feature matrix W KA ,W VA ,W QB is used to project the facial expression modality to obtain K A and V A , and the rPPG modality is projected to obtain Q B , as shown in the following formula:

[0063] K A =W KA ·F A , V A =W VA ·F A (12)

[0064] Q B =W QB ·F B (13)

[0065] Then the attention weights of the two modalities are calculated, and multiplied by V A to obtain the final attention matrix:

[0066]

[0067] After passing through an MLP and a residual link, the final fused features are obtained:

[0068] F f =MLP(FATT )+F ATT (15)

[0069] Output feature of the final three-layer fusion module Where i∈{1,2,3} is obtained by step-by-step fusion to obtain the final output feature:

[0070]

[0071]

[0072] Where W KA , W VA , W QB are linear transformation matrices, the parameters of which are updated by back propagation in neural network training, and the initial values of which are randomized. K A , V A , Q B represent key, value, and query, respectively; is a three different representation of feature linear mapping. Softmax is a normalization function; d K represents the vector dimension of Q B , K A ; MLP is a feedforward neural network composed of several fully connected layers, activation functions, and dropout; Concat represents vector concatenation, Linear represents linear mapping, and C is the feature dimension. After the semantic information of the visual and rPPG modalities is interacted and fused by the FM, the loss of shallow information is reduced.

[0073] Step 6: input the fusion feature into the classifier to output the driver abnormal behavior detection result:

[0074] Finally, the fusion feature obtained in step 5 is input into the fully connected layer after pooling processing, and the driver emotion classification task is completed. This embodiment is trained and verified on the MAHNOB-HCI dataset. In order to verify the effectiveness of this embodiment, the confusion matrix of the classification result is as shown in Figure 5 , wherein Figure 5 (a), (b), and (f) respectively represent the confusion matrices of emotion classification using rPPG signals, facial expressions, and fusion of rPPG and facial expressions. The effectiveness of multi-modal fusion can be verified by comparing the three subgraphs; wherein Figure 5(c) means no MCCM, MLAF module, (d) means only MCCM module, (e) means only MLAF module, the effectiveness of the MCCM, MLAF module proposed in steps 4, 5 can be verified by comparing subgraphs (c)-(f). This dataset provides two forms of labels: continuous emotion scores (range from 1 to 9) in two dimensions of Valence and Arousal, and nine categories of discrete emotion labels (such as anger, joy, etc.). It is generally believed that high arousal, low valence emotions (such as anger, anxiety) have a greater impact on driving behavior. Compared with continuous models, discrete emotion models are more intuitive and avoid the problem of ambiguous emotion classification caused by subjective interpretation differences. Therefore, in order to improve the accuracy of abnormal driving behavior recognition, combined with the nine categories of discrete emotion labels in the dataset, they are divided into three categories according to the degree of influence on driving behavior: (1) no influence class: neutral, joy2; (2) slight influence class: disgust, surprise, amusement; (3) serious influence class: anger, fear, anxiety, sadness.

[0075] The above examples are only used to illustrate the design idea and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the present application and to implement it, and the protection scope of the present application is not limited to the above examples. Therefore, any equivalent changes or modifications made in accordance with the principles and design ideas disclosed by the present application are within the protection scope of the present application.

Claims

1. A multimodal driver abnormal behavior detection method that balances privacy protection, characterized in that, Includes the following steps: Step 1: The driver's facial video is processed for privacy protection using a dynamic visual masking method based on block compression perception. The coordinates of key physiological regions are determined by facial key point detection, and a mask matrix is ​​constructed to retain the information of key physiological regions. Non-key physiological regions (NEPR) are compressed and encoded to obtain the facial video data with visual privacy protection. Step 2: For the facial video data obtained in Step 1, input the pre-trained ConvNeXt-T network to extract facial expression spatial features, input the spatial feature sequence into a 1D convolutional block to construct temporal correlation, and obtain the visual modality spatiotemporal features. Step 3: For the facial video data obtained in Step 1, the unsupervised method OMIT is used to extract the remote photoplethysmography (PPG) signal sequence. The PPG signal sequence is then processed using 1D convolutional blocks to obtain the spatiotemporal features of the physiological modality. Step 4: Input the visual modal spatiotemporal features obtained in Step 2 and the physiological modal spatiotemporal features obtained in Step 3 into the cross-modal complementarity module MCCM. Through the selective scanning mechanism of the state space model SSM, the interaction of the two modal features is realized to obtain the complementary feature representation. Step 5: Input the complementary feature representation obtained in Step 4 into the multi-level adaptive fusion module MLAF, and perform progressive feature fusion between different levels of the feature pyramid to obtain fused features; Step 6: Input the fused features obtained in Step 5 into the classifier and output the abnormal driver behavior detection results.

2. The multimodal driver abnormal behavior detection method according to claim 1, which takes into account privacy protection, is characterized in that, Step 1 is specifically as follows: The original video is resized so that it is divisible by a preset block size B; the location coordinates of key physiological areas are determined by facial key point detection, a corresponding mask matrix is ​​constructed, and the mask matrix is ​​multiplied element-wise with the Gaussian random observation matrix to generate the final observation matrix. Based on the final observation matrix, the original video is subjected to block-based compressed sensing processing to achieve visual privacy protection.

3. The multimodal driver abnormal behavior detection method according to claim 1, which takes into account privacy protection, is characterized in that... Step 2 is as follows: a pre-trained ConvNeXt-T network is used as the backbone network for spatial feature extraction, and a squeezed excitation SE block is added to enhance the channel attention mechanism; temporal features are extracted through two one-dimensional convolutional blocks, including 1DCNN, BN and ReLU, and dropout operation is added to prevent overfitting.

4. A multimodal driver abnormal behavior detection method that balances privacy protection as described in claim 1, characterized in that, Step 3 is as follows: facial alignment is performed based on the facial key points obtained in Step 1. Regions with high signal quality in the face are extracted through dynamic region selection. The RGB signal is converted into a remote photoplethysmography (PPG) signal using the QR decomposition algorithm. Sequence features are extracted using two consecutive one-dimensional convolutional blocks.

5. A multimodal driver abnormal behavior detection method that balances privacy protection as described in claim 1, characterized in that, Step 4 is as follows: The two modal features obtained in Step 2 and Step 3 are linearly mapped for convolution processing and gating adjustment; they are then processed sequentially by one-dimensional depthwise separable convolution, batch normalization, and SiLU activation function; the processed features are concatenated and further integrated with the gating mechanism through a shared state selection module; and multimodal complementary feature representations are obtained through residual connections.

6. A multimodal driver abnormal behavior detection method that balances privacy protection as described in claim 1, characterized in that, Step 5 is as follows: The complementary features obtained in step 4 are sequentially processed through three layers of one-dimensional convolutional blocks to extract deep semantic information. A multi-level adaptive fusion strategy is then adopted to gradually fuse shallow local features into deep global features in order to reduce the information difference between high- and low-dimensional features.

Images