A driver behavior detection method based on deep learning

Abstract

This invention relates to the field of deep learning technology, and particularly to a deep learning-based driver behavior detection method. The method includes calculating differential features from internal state features and external environment features, processing the differential features using one-dimensional temporal convolution, and then performing residual connections. Next, a feature vector sequence containing temporal context is calculated based on a cross-attention mechanism. An adaptive spatiotemporal channel attention weight matrix is ​​generated based on the feature vector sequence containing temporal context and the internal state features. This weight matrix is ​​then used to fuse the feature vector sequence containing temporal context and the internal state features. The fused features are then enhanced using a feedforward neural network and nonlinear transformation. The enhanced features are then input into two classification heads composed of fully connected layers for driver emotion recognition and driver behavior recognition. This invention mitigates look-ahead bias through causal masking, enabling the modeling of complex driving behaviors.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a deep learning-based method for detecting driver behavior. Background Technology

[0002] Among the many factors contributing to traffic accidents, human factors, especially dangerous or abnormal driving behaviors of drivers (such as fatigued driving, making or receiving phone calls, smoking, and distracted driving such as deviating from the driver's line of sight), are the core causes of accidents. Traditional driver condition monitoring relies heavily on onboard mechanical sensors or contact-based physiological signal acquisition devices, which have limitations such as high cost, complex operation, and easy interference with normal driving. In recent years, with the rapid development of artificial intelligence technology, non-contact behavior detection technology based on computer vision and deep learning has gradually become a research hotspot in the fields of Intelligent Transportation Systems (ITS) and Advanced Driver Assistance Systems (ADAS) due to its advantages of low cost, strong robustness, and ease of deployment, providing a new technical path for solving the problem of driver behavior monitoring.

[0003] With the rapid development of intelligent transportation systems and advanced driver assistance systems (ADAS), driver state detection has risen from a peripheral auxiliary function to a core technical link in ensuring road safety. In recent years, multimodal multi-task learning models (such as MMTL-UniAD) have shown significant advantages in the field of driver perception. By integrating multimodal data such as driver facial and body posture, as well as multi-view images from outside the vehicle, they have improved the system's comprehensive perception capability of complex driving scenarios. However, existing mainstream driver state monitoring models still have the following limitations in practical applications and multimodal data fusion: existing models lack temporal logic modeling and suffer from forward-looking bias and misalignment of the field of view. From the perspective of cognitive psychology, individual psychological stress stems from the cognitive assessment of external environmental stimuli and one's own coping resources. In real driving scenarios, the generation mechanism of external dynamic traffic environment stimuli and driver reactions (emotions, behavioral performance) has clear "temporal asymmetry" and "lag". However, existing models have not fully explored the temporal relationships in the dataset during the feature fusion stage, ignoring the time dimension. For example, the MMTL-UniAD model only uses simple spatial dimension feature splicing and global attention mechanisms for feature fusion. This approach fails to fully leverage the unidirectional dependence of driver behavior on the traffic environment. This leads to the model's tendency to use future traffic environment information to infer the current driver state during offline training, resulting in look-forward bias and information leakage. Consequently, the offline training objective cannot be strictly aligned with the field of view of the online real-time inference scenario, limiting the model's generalization ability in unknown dynamic scenarios to some extent.

[0004] In summary, current mainstream driver state monitoring models (such as MMTL-UniAD and traditional Transformer architecture) have the following main drawbacks:

[0005] 1. Existing look-ahead bias and vision misalignment: When handling multimodal fusion, existing models typically employ bidirectional symmetrical attention or simple global feature concatenation, neglecting the temporal dimension. This mechanism easily utilizes traffic environment information from future moments to aid in inferring the current state during offline training, leading to information leakage and look-ahead bias, causing the offline training objective to misalign with the vision of online real-time inference.

[0006] 2. Lack of temporal logic and fine-grained motion perception: The standard Transformer cross-attention mechanism is essentially permutation-invariant and does not adequately model temporal logic. Furthermore, traditional cross-attention only focuses on the static feature mapping of the current moment, ignoring the temporal motion gradients of adjacent time steps, making it difficult to capture fine-grained features of small driver movements and short-lived events, as well as instantaneous dynamic changes in the external environment. Summary of the Invention

[0007] To address the problems existing in the prior art, this invention proposes a driver behavior detection method based on deep learning, which specifically includes the following steps:

[0008] The driver's facial image, posture image, and environmental image are acquired, and the driver's internal state features and the external environment features in which the driver is located are extracted from them. The driver's internal state features include the driver's appearance features and the driver's skeletal movement features.

[0009] The internal state features and external environment features of consecutive frames are stacked into time series tensors, and each is input into a learnable absolute temporal position encoding matrix.

[0010] The difference features between adjacent frames of the time series tensors of internal state features and external environment features are calculated respectively. After processing the difference features by one-dimensional temporal convolution, the residual connection is used to superimpose them with the corresponding time series tensors to obtain enhanced internal state features and enhanced external environment features.

[0011] The enhanced internal state features are mapped to query vectors, and the enhanced external environment features are mapped to value vectors and key vectors. Based on the cross-attention mechanism, a feature vector sequence containing temporal context is calculated.

[0012] An adaptive spatiotemporal channel attention weight matrix is ​​generated based on the feature vector sequence containing temporal context and the internal state features. The feature vector sequence containing temporal context and the internal state features are then fused based on this weight matrix. Finally, the fused features are enhanced through a feedforward neural network and nonlinear transformation.

[0013] The enhanced features are then input into two classification heads consisting of fully connected layers to perform driver emotion recognition and driver behavior recognition.

[0014] Compared with the prior art, the present invention has the following beneficial effects:

[0015] 1. By introducing a causal masking mechanism to eliminate look-ahead bias, this technique embeds a causal masking matrix into the cross-modal attention mechanism. Leveraging the asymptotic properties of the Softmax function, the attention weights of the external environment in future time steps are forced to zero, cutting off the reverse-temporal gradient flow. This not only eliminates look-ahead bias but also achieves view alignment between offline training and online inference.

[0016] 2. Fine-grained motion perception and temporal relationship construction: By injecting learnable absolute temporal position codes into the feature sequence, the physical order within the time window is anchored, solving the permutation invariance problem. Simultaneously, a spatiotemporally constrained attention module (FM-TCA) for fine-grained motion perception is introduced, and one-dimensional temporal convolution (Conv1D) is used to extract motion difference features between adjacent time steps, capturing the dynamic causal relationship between driver actions and environmental abrupt changes. Attached Figure Description

[0017] Figure 1 This is a network architecture diagram of a deep learning-based driver behavior detection method according to the present invention.

[0018] Figure 2 This is a schematic diagram of the spatiotemporal constrained attention module (FM-TCA Module) for fine-grained motion perception of the present invention;

[0019] Figure 3 The mask matrix of this invention Attention weight matrix that conforms to time constraints A schematic diagram. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] This invention proposes a driver behavior detection method based on deep learning, which specifically includes the following steps:

[0022] The driver's facial image, posture image, and environmental image are acquired, and the driver's internal state features and the external environment features in which the driver is located are extracted from them. The driver's internal state features include the driver's appearance features and the driver's skeletal movement features.

[0023] The internal state features and external environment features of consecutive frames are stacked into time series tensors, and each is input into a learnable absolute temporal position encoding matrix.

[0024] The difference features between adjacent frames of the time series tensors of internal state features and external environment features are calculated respectively. After processing the difference features by one-dimensional temporal convolution, the residual connection is used to superimpose them with the corresponding time series tensors to obtain enhanced internal state features and enhanced external environment features.

[0025] The enhanced internal state features are mapped to query vectors, and the enhanced external environment features are mapped to value vectors and key vectors. Based on the cross-attention mechanism, a feature vector sequence containing temporal context is calculated.

[0026] An adaptive spatiotemporal channel attention weight matrix is ​​generated based on the feature vector sequence containing temporal context and the internal state features. The feature vector sequence containing temporal context and the internal state features are then fused based on this weight matrix. Finally, the fused features are enhanced through a feedforward neural network and nonlinear transformation.

[0027] The enhanced features are then input into two classification heads consisting of fully connected layers to perform driver emotion recognition and driver behavior recognition.

[0028] This embodiment describes the present invention in three parts, specifically including:

[0029] 1. The first part involves the explicit logical decoupling of the multimodal input stream. In this stage, the invention logically divides the input multimodal data into two independent streams: an "internal state stream" and an "external environment stream." The internal state stream contains facial, body posture, and skeletal key point features reflecting the driver's ontological attributes; the external environment stream contains multi-view exterior image features reflecting the semantics of the traffic scene. This decoupling establishes the basic topological structure for feature interaction, namely, using the environmental context as the index (Key / Value) and the driver's current state as the query object (Query), thus providing a structural foundation for subsequently modeling the unidirectional dependency relationship of "traffic environment affecting driver behavior."

[0030] 2. The second part is the alignment and construction of temporal features. In order to perform temporal modeling in subsequent modules, this invention needs to stack the feature vectors of consecutive frames into a time series tensor in this part.

[0031] 3. The third part is the fine-grained motion-aware spatiotemporally constrained attention module, which is the core algorithm component proposed in this technology. In real driving scenarios, dangerous behaviors of drivers (such as operating the central control, making or receiving phone calls, and nodding while fatigued) often have fine-grained characteristics, characterized by small movement amplitude and short occurrence time; at the same time, external traffic environment stimuli (such as sudden braking by the vehicle in front) are also instantaneous dynamic changes. Traditional cross-attention mechanisms usually only focus on the static feature mapping of the current moment, ignoring the temporal motion gradient between adjacent time points. In order to more accurately capture this fine-grained dynamic causal relationship, a motion perception and adaptive fusion strategy is introduced on the basis of the spatiotemporal constraint mechanism, proposing the fine-grained motion-aware spatiotemporally constrained attention module (Fine-grained Motion-aware TCA, FM-TCA). This module not only mitigates forward-looking bias through causal masking, but also achieves modeling of complex driving behaviors through motion excitation and Self-Adaptive Spatio-Temporal Fusion (SA-STF) gating.

[0032] The network structure of the entire driver behavior detection is as follows: Figure 1 As shown, the specific steps include:

[0033] 1. Explicit logic decoupling of multimodal input streams.

[0034] This embodiment logically divides the input multimodal data into two independent streams: an "internal state stream" and an "external environment stream." The internal state stream contains facial, body posture, and skeletal keypoint features reflecting the driver's ontological attributes; the external environment stream contains multi-view exterior image features reflecting the semantics of the traffic scene. This decoupling establishes the basic topology for feature interaction, namely, using the environmental context as the index (Key / Value) and the driver's current state as the query object (Query), thus providing a structural foundation for subsequently modeling the unidirectional dependency relationship of "traffic environment influencing driver behavior."

[0035] 1. Driver status feature extraction.

[0036] Driver state characteristics are a direct reflection of a driver's physiological state, psychological emotions, and behavioral intentions. The data sources processed in this invention cover the driver's facial, body, and skeletal data. Let the time length of the input video sequence be T. For any time t∈{1,...,T} in the sequence, the following feature extraction process can be defined:

[0037] 1) Driver's appearance feature extraction:

[0038] The driver's appearance features are mainly captured by observing subtle changes in facial expressions and overall body posture (such as leaning forward or sideways). Since the facial and body regions have clear spatial locality in images, this embodiment uses a convolutional neural network as a feature extractor.

[0039] definition for The RGB image of the face after constant face detection, cropping, and alignment, in which... This defines the height and width of the RGB image of the face. Similarly, define... For isomorphic body region images, where The height and width of the body region image are given. To extract deep semantic features, this embodiment inputs the image into a pre-trained CNN backbone network. The feature extraction process can be represented as a nonlinear mapping function. :

[0040]

[0041]

[0042] in, This represents the number of visual feature channels. (Function) It is composed of a series of convolution operators, nonlinear activation functions, and pooling operators. Specifically, for the ... Feature map of layer , its first The convolution operation for each output channel is defined as:

[0043]

[0044] in, This represents a two-dimensional discrete convolution operation; For the first Layer connection The input channel and the first The convolutional kernel weights for each output channel; For the next level (the first) The output of the first layer) Each channel feature map; For the first Layer The bias term corresponding to each output channel; It is the ReLU nonlinear activation function.

[0045] Final feature vector It is obtained by performing global average pooling on the output of the last convolutional layer. This step eliminates the sensitivity of spatial dimensions and only retains the semantic strength of the channel dimension.

[0046] 2) Skeletal movement characteristics:

[0047] Skeleton data, composed of a three-dimensional coordinate sequence of key points on the human body, possesses high abstraction and robustness, and is insensitive to changes in lighting and clothing texture. It is a key clue for recognizing specific driver actions (such as making a phone call, drinking water, or operating the center console). To capture the evolution of skeleton sequences in time and space, this invention uses a temporal constrained 3D convolutional neural network (TCON) to extract features of gestures and postures.

[0048] set up The input is a sequence of gesture skeleton nodes, where This refers to the number of key points in the hand (e.g., 21 hand nodes), where 3 represents... Three-dimensional coordinates; similarly, For the body posture skeleton sequence, the hand skeleton spatial coordinate sequence With body posture skeletal space coordinate sequence Concatenation is used as input.

[0049] Unlike 2D CNNs, 3D CNNs also have receptive fields in the temporal dimension. A 3D convolution operation is defined as a weighted sum of the input tensor and the 3D convolution kernel. This invention divides the three-dimensional index of the feature map into a temporal dimension. With spatial dimension For the first Layer Each channel feature map in spatiotemporal coordinates value at The formula for calculating temporally constrained 3D convolution is:

[0050]

[0051] In this formula, Represents the generation of the first The dimensions of the 3D convolutional kernel for each output channel feature map in the time, height, and width dimensions; It is the first In the layer, connect the first The feature map of the input channel and the first input channel feature map The convolution kernels of the output channel feature maps are in relative coordinates. Weight parameters at the location; For the first Layer (i.e., the previous layer) Each output channel feature map, in absolute spatiotemporal coordinates The sampled value at that location; It is a non-linear activation function; Indicates the first The first in the layer The bias terms correspond to the feature maps of each output channel. To ensure information constraints in the temporal dimension, the 3D CNN employs asymmetric left-side zero-padding in the temporal dimension, that is, inserting several virtual frames with all zeros at the very beginning of the entire time series. This ensures that the features extracted from frame t depend only on the frames input at time t and before time t; a temporally constrained 3D convolution is constructed to ensure... The output at time depends only on and the previous input frame.

[0052] By stacking multiple 3D convolutional layers, the network can extract high-level spatiotemporal semantics such as "a hand moving rapidly to the right" or "a head slowly drooping." The final skeleton feature representation is as follows:

[0053]

[0054] in, The feature mapping function represents a temporally constrained 3D convolutional neural network. This is a sequence of gesture skeleton nodes; For body posture skeletal sequence; The final fused output skeleton feature channel dimension; This represents the global spatial average pooling operation, which eliminates the sensitivity to spatial resolution and compresses the multidimensional spatiotemporal feature map output by 3D convolution into a unified one-dimensional feature vector.

[0055] 2. Internal integration characteristics.

[0056] Before feature concatenation, the temporal resolution mismatch between heterogeneous networks must be addressed. 2D CNNs extract facial and body features as instantaneous, frame-by-frame static representations, while temporally constrained 3D CNNs process a video clip containing multiple frames (e.g., a length of...). If directly spliced, the two will experience a severe misalignment in their temporal receptive fields. To ensure that heterogeneous features are within the same range... To ensure synchronization, this invention sets the time dimension stride to 1 in the 3D CNN network design and disables global temporal pooling. This means the output frame rate of the 3D CNN remains consistent with the input frame rate. Subsequently, this invention only extracts the 3D CNN output tensor corresponding to the current time step in the time dimension. The slices are used as the final skeleton motion features. Through this alignment strategy, the present invention ensures that the three modal features involved in the splicing maintain consistency in the cutoff time of their receptive fields.

[0057] After alignment is completed, in order to construct a unified, high-dimensional driver state representation, this invention defines a fusion function. Its structure typically consists of feature concatenation followed by a multilayer perceptron (MLP), used to map the concatenated high-dimensional features to a unified hidden layer dimension of the model. . Driver's internal integrated feature vector at time step The calculation is as follows:

[0058]

[0059] in, This indicates a vector concatenation operation. Indicates that the driver is All observable, time-aligned internal state information at any given moment.

[0060] 3. Extraction of traffic environment features.

[0061] External traffic environment features are derived from the vehicle's left-view, front-view, and right-view cameras. Feature extraction in this part is crucial because, in the temporal constraint model of this invention, it is the "cause" leading to changes in the driver's state. To efficiently process high-resolution multi-view environmental images and extract traffic features with global semantics, this invention employs a multi-axis region attention network (MARNet).

[0062] The traditional Vision Transformer (ViT) employs a global self-attention mechanism, whose computational complexity is the square of the image size. This computational cost is unacceptable when processing high-resolution driving videos. MARNet reduces the complexity to linear by introducing a multi-axis attention mechanism, decomposing global attention into two sparse forms: block attention and grid attention. It retains the ability to capture local details and global dependencies.

[0063] 1) Integration of environmental characteristics:

[0064] for A collection of multi-view environmental images at different times , The left-hand diagram represents the driver at time t. This represents the driver's forward graph at time t. Representing the right-hand image of the driver at time t, this invention first utilizes a weighted MARNet to extract spatial feature maps from each viewpoint:

[0065]

[0066] Since MARNet is a visual backbone network, its output With two-dimensional spatial resolution ( ) and number of channels Directly substituting this into subsequent sequence summation calculations will lead to a mismatch in tensor dimensions. To smoothly reduce the 3D spatial feature map to a one-dimensional semantic vector, this invention introduces a spatial global average pooling layer (GAP) after the MARNet output. This compresses the feature map in the spatial dimension, thereby eliminating spatial dimension sensitivity and preserving global traffic semantics.

[0067]

[0068] Subsequently, the multi-view features are integrated into a unified environmental feature vector through the spatial fusion module. Considering the varying impacts of different perspectives on driver state, this invention employs a learnable weighted summation mechanism:

[0069]

[0070] in, These are learnable viewpoint weight parameters. It is a feature mapping function containing a linear projection layer, responsible for mapping the channel dimensions after pooling. Projected onto the model's unified hidden layer dimension .

[0071] II. Alignment and Construction of Temporal Features.

[0072] To perform temporal modeling in subsequent modules, this invention requires stacking the feature vectors of consecutive frames into a time-series tensor. A driver internal state sequence matrix is ​​defined. and external environment sequence matrix as follows:

[0073]

[0074]

[0075] Here The time window length is defined. This step not only tensors the data but, more importantly, aligns the driver's state with the external traffic environment context in the time dimension. However, the standard Transformer cross-attention mechanism is inherently permutation-invariant. Although this invention will introduce an upper triangular causal mask to cut off future information in the subsequent TCA Module, for known historical information with a mask of 0 (i.e.,... arrive Without special processing, the model will be completely unable to distinguish the chronological order of historical events. Within the model's receptive field, the attention output generated by the environmental stimulus sequence [sudden braking, horn honking, normal driving] and [normal driving, horn honking, sudden braking] will be indistinguishable. This loss of temporal order will prevent the model from meeting the temporal constraints required by this invention.

[0076] To address this issue and ensure the model's time-series awareness, this invention requires injecting temporal position encoding before the feature sequence is input into the attention module. Here, this invention introduces a learnable absolute temporal position encoding matrix. And then add it to the original feature sequence element by element:

[0077]

[0078]

[0079] Through this operation, each feature vector in the sequence matrix not only contains rich multimodal spatial semantics but is also anchored to its physical order within the time window. Injection of position-encoded features... and This will serve as the direct input to the temporal constraint attention module in the next section, providing a data foundation for mining unidirectional temporal dependencies.

[0080] III. Spatiotemporal Constraint Attention Module for Fine-Grained Motion Perception (FM-TCA Module).

[0081] The spatiotemporal constrained attention module is the core algorithm component proposed in this patent. In real-world driving scenarios, dangerous driver behaviors (such as operating the central control, making or receiving phone calls, and nodding due to fatigue) often exhibit fine-grained characteristics with small amplitude and short duration; simultaneously, external traffic environment stimuli (such as sudden braking by the vehicle in front) are also instantaneous dynamic abrupt changes. Traditional cross-attention mechanisms typically only focus on the static feature mapping of the current moment, ignoring the temporal motion gradient between adjacent time points. To more accurately capture such fine-grained dynamic causal relationships, this invention introduces a motion perception and adaptive fusion strategy based on the spatiotemporal constrained mechanism, proposing a fine-grained motion-aware spatiotemporal constrained attention module (Fine-grained Motion-aware TCA, FM-TCA). This module not only mitigates forward-looking bias through causal masking but also achieves modeling of complex driving behaviors through motion excitation and self-adaptive spatio-temporal fusion (SA-STF) gating. The FM-TCA network structure is as follows: Figure 2As shown, it specifically includes:

[0082] 1. Motion excitation branch: This branch enhances the driver's internal state sequence matrix and the external environment sequence matrix respectively. Specifically, it constructs a difference sequence through the two matrices, processes the difference sequence, and then connects it with the residual of the original matrix to obtain the enhanced data.

[0083] 2. Temporally Constrained Attention Branch: This branch fuses the enhanced driver internal state sequence matrix and external environment sequence matrix through a cross-attention mechanism and temporal constraints.

[0084] 3. Cross-modal causal fusion strategy branch: This branch adaptively evaluates the fused features and assigns dynamic fusion weights to different channels to obtain the final features used for classification.

[0085] In the standard Transformer cross-attention architecture, the query, key, and value are typically obtained by linearly projecting the input feature sequence. However, this approach only measures the similarity of static semantics and struggles to capture dynamic changes. Inspired by the Temporal Motion Attention (TMA) mechanism, this invention introduces a temporal motion activation branch in the feature mapping stage. This branch extracts the dynamic gradient of the driver's actions and the external environment by calculating the feature differences between adjacent time steps, allowing the attention mechanism to find the correlation between changes in the traffic environment and changes in the driver's actions. First, the driver's internal state sequence is defined. and external environment sequence Fine-grained motion difference features in the time dimension and For the first in the sequence The motion difference of a frame is calculated as follows:

[0086]

[0087]

[0088] in, They represent the first Motion difference characteristics of the driver's internal state and external environment at any given time. They represent the first The original driver's internal state feature vector and external environment feature vector at each moment. This represents the current time step (video frame index), and its value range is... .when To ensure consistent sequence lengths, the following settings are used: and . This represents the total length of the time window (total number of frames) of the input multimodal video sequence.

[0089] Subsequently, to smooth the difference features and promote the exchange of motion information between adjacent time intervals, this invention employs one-dimensional temporal convolution (Conv1D) to process the motion difference sequence and superimposes it back into the original static features as residuals, thereby obtaining enhanced features with motion perception capabilities. and :

[0090]

[0091]

[0092] In the formula, This represents the enhanced driver's internal state sequence and external environment sequence, incorporating temporal motion stimulus information; the matrix dimension is [missing information]. . This is the original static feature sequence matrix with dimension . . They represent respectively to , Perform one-dimensional temporal convolution operations. The kernel size is usually set to 3, which aims to smooth the motion difference features of adjacent frames in the temporal dimension and realize information exchange between channels.

[0093] Finally, through a learnable projection weight matrix and The enhanced features are mapped to the attention subspace. Here, the Query and Key incorporate motion stimulus information, while the Value retains the specific semantic content of the original environment.

[0094]

[0095]

[0096]

[0097] in These represent the query matrix, key matrix, and value matrix in the cross-attention mechanism, respectively. The learnable linear projection weight matrix is ​​responsible for mapping the enhanced features to a dimension of 1. In the attention query and key subspace; This represents a learnable linear projection weight matrix, responsible for mapping the original environment features to a dimension of 1. In the numerical subspace; This represents the unified hidden layer feature dimension of the model; This represents the feature dimension of keys and values ​​in the attention mechanism. Through this mapping mechanism, the model can not only match multimodal spatial semantics in subsequent dot product calculations, but also assign higher attention weights to abrupt features in the temporal domain.

[0098] After acquiring data including motion sensing and Next, the attention score needs to be calculated. Because... The operation has the property of being fully connected, the first The driver's state in the frame may be related to the first frame. Frame (where) The traffic environment context generates correlations. In order to follow the temporal relationship, this invention constructs an upper triangular mask matrix. Mask matrix The element is defined as follows:

[0099]

[0100] The attention calculation process after introducing a mask is as follows:

[0101] 1) Calculate the original score and apply the mask:

[0102]

[0103] This step sets the entire upper triangular portion of the matrix to negative infinity.

[0104] 2) Normalized weights:

[0105]

[0106] Here we use The significance lies in the asymptotic properties of the Softmax function ( After Softmax normalization, the attention weights assigned to future frames are reduced to zero. The generated weight matrix... It is a lower triangular matrix, such as Figure 3 As shown. In gradient backpropagation, this means that feature gradients from the future traffic environment cannot be propagated back to the current driver state, thus blocking the reverse-temporal flow of gradients and eliminating look-forward bias.

[0107] To obtain an attention weight matrix that meets the temporal constraints Subsequently, this invention performs weighted aggregation on the environmental feature Values ​​to obtain a feature vector sequence containing temporal context. :

[0108]

[0109] Traditional Transformer architectures typically employ residual connections (such as...) This invention aims to fuse cross-modal features. However, in driver behavior recognition tasks, features from different channels often represent different fine-grained semantics. Furthermore, the influence of the external environment on the driver is dynamic; at certain times (e.g., normal cruising), the model should rely more on driver ontological features, while at other times (e.g., emergency avoidance), the supplementation of environmental features is crucial. Therefore, this invention draws on the core ideas of Adaptive Spatiotemporal Fusion (SA-STF) and Spatial Channel Attention (SCA) to introduce a channel-gated adaptive cross-modal fusion strategy. This strategy can adaptively evaluate the joint importance of internal and external information at the current moment and assign dynamic fusion weights to different channels.

[0110] Step 1: Feature assembly and cross-channel interaction. The calculated environmental context... By linear projection Mapping back to the model dimension and comparing it with the original driver's internal features The concatenation is performed along the channel dimension. Subsequently, one-dimensional convolution is used for cross-channel feature compression and interaction, and the GELU activation function is used to extract nonlinear joint features. :

[0111]

[0112]

[0113] in, This represents a concatenation operation along the feature dimension. Responsible for splicing Dimensions recompressed to .

[0114] Step 2: Generate adaptive channel-gated weights. For the interacting features... Dimension expansion is performed again using one-dimensional convolution, followed by the Sigmoid activation function. Map the output value to Intervals are used to generate adaptive spatiotemporal channel attention weight matrices. :

[0115]

[0116] The weight matrix Dynamically reflects the environmental context at each time step. The contribution of each feature channel to the inference of the current driver's state.

[0117] Step 3: Adaptive residual weighted fusion. Utilizing the generated adaptive weights... The environmental context is weighted element-wise at the channel level, and finally the residuals are superimposed onto the original internal features. Above, and perform layer normalization:

[0118]

[0119] This design allows the network to adaptively learn the weight allocation of spatiotemporal attention during training. When there is severe occlusion or irrelevant noise in the external environment, the network automatically reduces... The value of allows the model to smoothly regress to a state that relies solely on the original driver features, effectively preventing feature degradation.

[0120] Step 4: Nonlinear Feature Enhancement. Finally, a feedforward neural network (FFN) is used to further transform the fused features nonlinearly to enhance their expressive power.

[0121]

[0122]

[0123] The final output of the enhanced features It incorporates fine-grained motion information, spatiotemporal constraint logic, and adaptive cross-modal channel weights, and will then be fed into a fully connected classification head for the final driver emotion recognition (DER) and behavior recognition (DBR) tasks.

[0124] Because the masking mechanism eliminates interference from future information during the feature fusion stage, the predictions made by the classification head are entirely based on temporal dependencies. This not only makes the model more rigorous in engineering logic, but also enables it to align with the real-world perspective of online inference when facing unknown dynamic traffic environments, inferring the current state based on the sequence of events that have occurred. This significantly improves the robustness and generalization ability of the system in actual deployments.

[0125] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A driver behavior detection method based on deep learning, characterized in that, Specifically, the following steps are included: The driver's facial image, posture image, and environmental image are acquired, and the driver's internal state features and the external environment features in which the driver is located are extracted from them. The driver's internal state features include the driver's appearance features and the driver's skeletal movement features. The internal state features and external environment features of consecutive frames are stacked into time series tensors, and each is input into a learnable absolute temporal position encoding matrix. The difference features between adjacent frames of the time series tensors of internal state features and external environment features are calculated respectively. After processing the difference features by one-dimensional temporal convolution, they are superimposed with the corresponding time series tensors by residual connection to obtain enhanced internal state features and enhanced external environment features. The enhanced internal state features are mapped to query vectors, and the enhanced external environment features are mapped to value vectors and key vectors. Based on the cross-attention mechanism, a feature vector sequence containing temporal context is calculated. An adaptive spatiotemporal channel attention weight matrix is ​​generated based on the feature vector sequence containing temporal context and the internal state features. The feature vector sequence containing temporal context and the internal state features are then fused based on this weight matrix. Finally, the fused features are enhanced through a feedforward neural network and nonlinear transformation. The enhanced features are then input into two classification heads consisting of fully connected layers to perform driver emotion recognition and driver behavior recognition.

2. The driver behavior detection method based on deep learning according to claim 1, characterized in that, The process of extracting the driver's appearance features includes: At each time step, obtain the RGB image of the face after face detection, cropping and alignment, and its isomorphic body region image; The image is input into a pre-trained convolutional neural network to extract the corresponding feature vector. The convolutional neural network includes multiple layers of convolutional modules consisting of convolution operators, non-linear activation functions, and pooling operators. The feature map output by the last convolutional module is used to obtain the corresponding feature vector through global average pooling.

3. The driver behavior detection method based on deep learning according to claim 1, characterized in that, The process of extracting the driver's skeletal motion features includes: Extract the spatial coordinate sequence of the driver's hand skeleton from the image and body posture skeletal spatial coordinate sequence Concatenation serves as the underlying data; To ensure information constraints in the time dimension, asymmetric left-side zero-padding is used to construct a temporally constrained 3D convolution, ensuring... The output at time depends only on and the previous input frame; The time step is set to 1, and the global time pooling operation is canceled, only the output tensor corresponding to the current time step is extracted in the time dimension. The slice is used as the skeleton action feature at the current moment; The three-dimensional index of the input tensor is divided into a time dimension. With spatial dimension Temporally constrained 3D convolutional neural networks consist of multiple cascaded 3D convolutional modules. The lower-level data is used as the input to the first 3D convolutional module, and so on. The 3D convolutional module receives multiple input feature maps from the previous layer and generates new output feature maps. For the ... Layer The feature map generated by each output channel, in spatiotemporal coordinates value at Represented as: in, Represents the generation of the first The dimensions of the 3D convolutional kernel for each output channel feature map in the time, height, and width dimensions; It is the first In the layer, connect the first The feature map of the input channel and the first input channel feature map The convolution kernels of the output channel feature maps are in relative coordinates. Weight parameters at the location; For the first Layer Each output channel feature map, in absolute spatiotemporal coordinates The sampled value at that location; It is a non-linear activation function; Indicates the first The first in the layer The bias term corresponding to the feature map of each output channel.

4. The driver behavior detection method based on deep learning according to claim 1, characterized in that, The process of extracting the characteristics of the driver's external environment includes: for A collection of multi-view environmental images at different times The spatial feature maps of each viewpoint are extracted using MARNet with shared weights. A spatial global average pooling layer is introduced after the MARNet output to compress the feature map in the spatial dimension; The spatial fusion module integrates multi-view features into a unified environmental feature vector, that is, it takes into account the differences in the impact of different views on the driver's state and adopts a learnable weighted summation mechanism. The environmental feature vector obtained by weighted summation is projected onto the model's unified hidden layer dimension through a feature mapping function that includes a linear projection layer.

5. The driver behavior detection method based on deep learning according to claim 1, characterized in that, The processes of obtaining enhanced internal state characteristics and enhanced external environment characteristics include: Stacking the feature vectors of consecutive frames into a time series tensor yields the driver's internal state sequence matrix. and external environment sequence matrix ; Introduce a learnable absolute temporal position encoding matrix And it is superimposed on the driver's internal state sequence matrix in an element-wise addition manner. and external environment sequence matrix middle; Calculate the driver's internal state sequence matrix in frame t. and external environment sequence matrix The difference between two adjacent frames is obtained and and order , The internal state sequence matrix of the driver is obtained. and external environment sequence matrix Difference sequences of the same length; One-dimensional temporal convolution is used to process the two motion difference sequences separately, and then they are compared with the corresponding driver internal state sequence matrices. and external environment sequence matrix When combined, they enhance both the internal state characteristics and the external environment characteristics.

6. The driver behavior detection method based on deep learning according to claim 1, characterized in that, The process of calculating a feature vector sequence containing temporal context based on the cross-attention mechanism includes: Construct an upper triangular mask matrix mask matrix The element in the i-th row and j-th column represents the relationship between the i-th frame image and the j-th frame image, defined as: Calculate attention score using mask matrix Represented as: in, This represents the query vector obtained by enhancing the internal state feature mapping. To obtain key vectors by enhancing the feature mapping of the external environment transpose, The dimension of the key vector; Using the Softmax function Normalization is performed to obtain the attention weight matrix that meets the temporal constraints. By using an attention weight matrix that conforms to temporal constraints to weight the value vectors, a sequence of feature vectors containing temporal context is obtained.

7. The driver behavior detection method based on deep learning according to claim 1, characterized in that, The process of generating adaptive weights based on feature vector sequences containing temporal context and internal state features includes: The feature vector sequence containing temporal context is mapped back to the model dimension via linear projection and compared with the driver's internal state features. The components are stitched together along the channel dimension. One-dimensional convolution is used for cross-channel feature compression and interaction, and the GELU activation function is used to extract nonlinear joint features. ; Nonlinear joint characteristics Dimensionality expansion is performed using one-dimensional convolution, and the output value is mapped to a sigmoid activation function. Intervals are used to generate adaptive spatiotemporal channel attention weight matrices. .

8. The driver behavior detection method based on deep learning according to claim 7, characterized in that, The feature vector sequence containing temporal context and internal state features are fused based on an adaptive spatiotemporal channel attention weight matrix, including: in, Features of fusion; This represents element-wise multiplication; Representation layer normalization.

9. The driver behavior detection method based on deep learning according to claim 8, characterized in that, Enhancement of fused features through feedforward neural networks and nonlinear transformations includes: in, This represents the output of the feedforward neural network; , , , These are the trainable parameters in the feedforward neural network; This represents the GELU activation function; Features enhanced through feedforward neural networks and nonlinear transformations; This indicates the regularization layer.

Images