Multi-mode driver behavior monitoring method and system based on environment self-adaption

Abstract

The invention discloses a multi-mode driver behavior monitoring method and system based on environment self-adaption. The method comprises the steps that S1, a front RGB driving image, a front infrared driving image and an inclined side face infrared driving image are collected; s2, carrying out geometric transformation on the lateral infrared view angle image; s3, carrying out feature extraction; s4, calculating an attention map of the front view angle to the side view angle and an attention map of the side view angle to the front view angle through a bidirectional cross attention mechanism; s5, inputting the fused infrared fusion feature map into a pre-trained first attitude estimation network; s6, inputting the front RGB driving image into a pre-trained second attitude estimation network; s7, dynamically generating an infrared dominant gating weight coefficient and a visible light dominant gating weight coefficient according to the illumination intensity scalar; and S8, inputting the key point coordinates into a pre-trained time sequence behavior classification model, and classifying behaviors of the driver according to a predefined rule. According to the method, the problem of geometric misalignment between multi-view infrared images is effectively solved.

Description

Technical Field

[0001] This invention relates to the field of dangerous driving behavior detection technology, specifically to a multimodal driver behavior monitoring method and system based on environment adaptation. Background Technology

[0002] With the development of the automotive industry and the rise in people's living standards, the usage and reliance on vehicles are increasing year by year, making road traffic safety a hot topic. As the direct controllers of vehicles, drivers' driving status plays a decisive role in the vehicle's operation. Therefore, when distracted driving or fatigued driving occurs, the probability of traffic accidents increases significantly. Thus, the development of driver behavior monitoring technology is of great significance for preventing traffic accidents and protecting people's personal safety.

[0003] Current driver behavior detection methods suffer from the following main drawbacks: When using a single-view infrared camera, there is a serious occlusion problem, and the geometric differences between images from different viewpoints are large. Traditional affine transformations or homography transformations have poor alignment effects, leading to inaccurate key point localization. When fusing infrared and visible light images, most methods use simple weighted averaging or feature stitching, failing to fully consider the dynamic changes in the reliability of each modality under different lighting conditions, resulting in unstable fusion effects in scenarios with sudden changes in lighting. There is a lack of an efficient fusion mechanism that can simultaneously model the geometric relationships between viewpoints and the semantic correlations between features. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multimodal driver behavior monitoring method and system based on environment adaptation. It effectively solves the geometric misalignment problem between multi-view infrared images, achieves intelligent and adaptive fusion of infrared and visible light information, and significantly improves the robustness of driver attitude estimation under complex lighting conditions.

[0005] The technical solution of the present invention is as follows: A multimodal driver behavior monitoring method based on environment adaptation includes the following steps: S1 acquires frontal RGB driving images, frontal infrared driving images, and oblique side infrared driving images respectively; S2 employs a view alignment module based on deformable convolution to perform geometric transformation on the oblique side infrared driving image, aligning it spatially with the front infrared driving image. S3 uses a shared-weight encoder network to extract features from the aligned oblique side infrared driving image and the front infrared driving image respectively, generating dual-view feature maps with consistent semantic representation, namely the front view feature map and the side view feature map. S4 inputs the extracted dual-view features into a cross-view attention fusion module, calculates the attention map of the front view to the side view and the attention map of the side view to the front view through a bidirectional cross-attention mechanism, and then fuses them. S5 inputs the fused infrared feature map into a pre-trained first attitude estimation network to realize driver attitude estimation in infrared fusion mode. S6 inputs a frontal RGB driving image into a pre-trained second pose estimation network to achieve driver pose estimation in RGB mode. S7 calculates the current ambient light intensity scalar based on the frontal RGB driving image, dynamically generates infrared-dominated gating weight coefficients and visible light-dominated gating weight coefficients based on the ambient light intensity scalar, and outputs a set of optimal driver key point coordinates that are robust to illumination (key points are a technical term in human pose estimation technology, used to characterize various key parts of the human body, such as wrists, elbows, etc.). S8 inputs the key point coordinates into a pre-trained temporal behavior classification model, and classifies the driver's behavior according to predefined rules. The behavior categories include: focused driving, using a mobile phone, and eating.

[0006] Furthermore, step S2 includes the following steps: The frontal infrared driving image and the oblique side infrared driving image are concatenated along the channel dimension into a feature tensor; The offset field required for deformable convolution is predicted by an offset prediction network consisting of three convolutional layers. By applying deformable convolution to the oblique side infrared driving image, a non-rigid spatial alignment is achieved, making it consistent with the frontal infrared driving image in terms of spatial structure.

[0007] Furthermore, step S3 includes the following steps: A backbone network with shared weights is used as an encoder to perform feature processing on the aligned side view feature map and the features of the frontal infrared driving image, respectively, to generate a dual-view feature map with consistent semantic representation. The encoder network structure is adapted to single-channel infrared image input and includes a 7×7 convolutional layer followed by batch normalization and ReLU activation function; a 3×3 max pooling layer; two residual blocks; and a 3×3 convolutional layer followed by batch normalization and ReLU activation function.

[0008] Furthermore, step S4 includes the following steps: It employs a bidirectional cross-attention mechanism, comprising two symmetrical CrossAttention sub-modules. Each CrossAttention sub-module includes a query convolutional layer, a key convolutional layer, and a value convolutional layer, and each consists of three 1×1 convolutional layers. The query convolutional layer projects the input features onto the corresponding key channels (in the attention mechanism, there are "query", "key", and "value", and here the key channel refers to the number of channels of the key), while the key convolutional layer and the value convolutional layer project features from another perspective onto the same dimension. The Query and Key are multiplied by matrix to obtain the Energy matrix. The Energy matrix is ​​then Softmax normalized in the last dimension to obtain the attention weights. The attention weights are then multiplied by the Value matrix to obtain the context features. The dimensions of the features are adjusted, and finally added to the original query features (referring to the input features of the query convolutional layer) and multiplied by a learnable scaling factor gamma to obtain the attention output of the branch. The attention outputs in the front-to-side and side-to-front directions are concatenated with the original dual-view features in the channel dimension to form fused features, which are then output through a 1×1 convolutional layer for dimensionality reduction.

[0009] Furthermore, step S5 includes the following steps: The fused features are fed into the backbone feature extraction network built on the HRNet structure. The backbone feature extraction network contains an initial layer and four residual layers. The initial layer includes 7x7 convolution, batch normalization, ReLU activation function and max pooling. Each residual layer includes multiple Bottleneck blocks. The fused features are processed through the initial layer of the backbone network. The initial layer uses a 7×7 convolutional kernel to perform preliminary feature transformation and downsampling. A batch normalization layer and a ReLU activation function are connected after the 7×7 convolutional kernel. The high-dimensional features extracted by the backbone network are fed into the feature adaptation layer, which consists of 1×1 convolutional kernels, and a batch normalization layer and a ReLU activation function are connected after the 1×1 convolutional kernels. A lightweight Transformer module is used to enhance the features processed by the feature adaptation layer. The Transformer module is based on the Swin Transformer architecture, with a depth of 1, containing 4 attention heads, and using a 4×4 local window for self-attention calculation to recalibrate and enhance the spatial feature relationships. The prediction head decodes the enhanced features of the Transformer module into a key point heatmap. The prediction head consists of two convolutional layers in sequence. The first layer uses a 3×3 convolutional kernel, and the second layer uses a 1×1 convolutional kernel. The number of output channels corresponds to the predefined key points. The heatmap is converted into key point coordinates and visibility confidence using a post-processing operation based on differentiable softargmax.

[0010] Furthermore, step S6 includes the following steps: The input is a frontal RGB driving image; The initial convolutional layer of the network encoder uses a 7×7 convolutional kernel; The backbone network is used to extract multi-level features from the image processed by the initial convolutional layer to obtain a high-dimensional feature representation. The backbone network contains an initial layer and four residual layers. The initial layer includes 7x7 convolution, batch normalization, ReLU activation function and max pooling. Each residual layer includes multiple Bottleneck blocks. The feature adaptation layer is a 1×1 convolution, used to adjust and fuse the channel dimensions of the high-dimensional feature representation; The lightweight Transformer module is connected to the feature adaptation layer to enhance the global context awareness of the features. Its hyperparameters of depth, number of heads, and window size are the same as those of the corresponding module in the first pose estimation network. The prediction head processes the features after they have been processed by the feature adaptation layer or the lightweight Transformer module through a series of convolutional operations, and outputs a key point heatmap. A softargmax-based decoding method is used to convert the key point heatmap into key point coordinates and visibility confidence.

[0011] Furthermore, step S7 includes the following steps: A quantitative assessment of the lighting conditions in the current driving environment is performed: a light intensity scalar is calculated from the frontal RGB driving image, with a value range of [0, 1]. After obtaining the light intensity scalar, the visible light mode weight rgb_weight and the infrared fusion mode weight ir_weight are dynamically calculated using a preset piecewise linear function; the piecewise linear function is defined by a low light threshold and a high light threshold. The obtained weight values ​​are then used to fuse the coordinate data using a weighted average algorithm. ; Where fused_coords are the coordinates of the key points after fusion, kp_rgb are the coordinates of the key points obtained from the RGB image, and kp_ir are the coordinates of the key points obtained from the infrared image.

[0012] The visibility of the fused model is set to the maximum value of the visibility predicted by the RGB and infrared models, thus ensuring that key points can still be detected even in occlusion situations.

[0013] Furthermore, step S7 includes the following steps: For the keypoint data of each frame dimension in the input sequence, feature normalization is first performed. Taking the neck keypoint coordinates as the origin, the neck coordinates are subtracted from all keypoint coordinates to obtain the normalized position relative to the neck. Calculate a set of predefined multimodal temporal feature vectors, including: head position and velocity features, relative hand position features, distance between hand and mouth features, elbow angle features, head posture features, and visibility features; The feature vector is fed into a first-in, first-out (FIFO) buffer; based on the buffer's fullness, one of the following two methods is adaptively selected: The heuristic rule-based method: When there are not enough frames in the buffer, predefined expert rules are used to make real-time judgments on the latest calculated single-frame features. The rules include judgment rules for playing mobile phone, making and receiving calls, eating in the car, dozing off, and covering mouth and yawning. The deep learning-based approach works as follows: When the required number of frames collected in the buffer is met, a pre-trained deep learning temporal model is activated. This model is input into a bidirectional LSTM network for temporal modeling, outputting the hidden states in both directions at each time step. These hidden states are then concatenated to obtain a feature tensor. An attention network is then used to calculate the weights for this sequence. The attention network consists of a fully connected layer and a softmax function. It receives the output of the LSTM and calculates the importance weight at each time step. The LSTM output features and attention weights are weighted and summed to obtain a fixed context vector. The context vector is fed into the fully connected layer classifier, which outputs the prediction scores for six categories: [focused driving, playing on the phone, making or receiving calls, eating in the car, dozing off, and covering the mouth while yawning]. The highest score is taken as the final behavior classification result.

[0014] A multimodal driver behavior monitoring system based on environment adaptation includes: an image acquisition module, a viewpoint alignment module, a feature encoding module, a cross-viewpoint fusion module, an illumination adaptive gating module, a pose estimation module, and a behavior classification module; The image acquisition module is used to acquire frontal RGB, frontal infrared, and oblique side infrared images; The feature extraction module is used to extract features from RGB images; The view alignment module, based on deformable convolution, is used for geometric alignment of oblique side infrared images; The encoder module is used to extract features from the aligned dual-view infrared image; The cross-view attention fusion module is used to fuse dual-view infrared features through a bidirectional cross-attention mechanism; The illumination adaptive gating module is used to dynamically fuse infrared and visible light features based on the ambient light intensity; The pose estimation module is used to predict keypoint coordinates based on the fused features; The behavior classification module is used to identify driving behavior based on key point coordinates.

[0015] The beneficial effects of this invention are as follows: This invention achieves the function of dual-view complementaryness through multi-view infrared fusion technology. When one hand is blocked, the other view can still capture key information. Its cross-attention mechanism can dynamically adjust the weight of each view to enhance the representation of important areas.

[0016] The light-adaptive weighted technology ensures environmental robustness. The decision-making mode of infrared features + RGB features ensures accuracy in well-lit environments while also taking into account recognition quality in low-light environments. The linear interpolation weight allocation logic avoids abrupt weight changes, ensuring stable performance in scenarios with sudden changes in lighting and ensuring the continuity of behavior recognition.

[0017] The bidirectional cross-attention technology enables accurate modeling of geometric relationships. Through geometric transformation learning, it can automatically learn deformations caused by viewpoint differences, which is superior to traditional affine transformations. It also has the function of semantic information complementarity, ensuring that different viewpoints have a consistent understanding of the same behavior.

[0018] It systematically solves the key defects of existing technologies, and achieves a qualitative leap in terms of occlusion handling, environmental adaptability, and fusion effect, providing a more reliable and accurate technical solution for driver behavior monitoring.

[0019] The present invention effectively solves the technical problems of inaccurate geometric alignment from multiple perspectives, insufficient feature fusion, and poor adaptability to ambient lighting through the above-mentioned solution. Attached Figure Description

[0020] Figure 1 This is an overall flowchart of the method of the present invention; Figure 2 This is the RGB input image for this invention; Figure 3 This is the infrared input diagram of the present invention. Detailed Implementation

[0021] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0022] A multimodal driver behavior monitoring method based on environment adaptation includes the following steps: S1: Acquire frontal RGB driving image, frontal infrared driving image, and oblique side infrared driving image respectively.

[0023] The implementation process of step S1 is as follows: Preferably, multiple sensors arranged within the vehicle's cockpit are used to synchronously acquire images. Specifically, this includes: a visible light camera positioned 45° forward to acquire a frontal RGB driving image, an infrared camera positioned 45° forward to acquire a frontal infrared driving image, and an infrared camera positioned 45° to the left front to acquire a side-view infrared driving image. The image resolution is preferably 640x480 pixels, and the frame rate is preferably 25-30 fps. All cameras require joint calibration after installation to obtain their respective intrinsic and extrinsic parameters. During program initialization, connections must be established with three video sources. When cyclically reading frames, frame rate control (e.g., a target of 15 FPS) and performance monitoring parameters must be set. This invention requires the two infrared perspectives to be corrected and aligned; if not calibrated, calibration can be attempted every 100 frames.

[0024] S2: A view alignment module based on deformable convolution is used to perform geometric transformation on the oblique side infrared driving image so that it is spatially aligned with the front infrared driving image.

[0025] The implementation process of step S2 is as follows: The frontal infrared driving image and the oblique side infrared driving image have dimensions of [B, 256, H / / 4, W / / 4], where B is the batch size, and H and W are the height and width of the original image. These two images are concatenated along the channel dimension to form a 512-dimensional feature tensor. This tensor is then passed through an offset prediction network consisting of three convolutional layers. The specific structure is as follows: the first layer is a 3×3 convolution with 512 input channels, 256 output channels, padding=1, followed by a ReLU activation function; the second layer is a 3×3 convolution with 256 input and output channels, padding=1, followed by a ReLU activation function; the third layer is a 1×1 convolution with 256 input channels and 18 output channels, corresponding to the offsets of the 3×3 deformable convolution kernel in the x and y directions at each spatial location, predicting the offset field required for the deformable convolution. This deformable convolutional kernel is 3×3 in size with padding of 1 and uses 256 output channels. By applying this deformable transformation to the side view features, it achieves non-rigid spatial alignment, ensuring that they are consistent with the front view features in spatial structure. This eliminates the coordinate deviation of key points (such as elbows) caused by the 45° view difference. This module fully utilizes the adaptability of deformable convolution to geometric deformation to achieve fine, non-linear view geometric alignment.

[0026] S3: Using a shared-weight encoder network, feature extraction is performed on the aligned oblique side infrared driving image and the front infrared driving image respectively to obtain the side view feature map and the front view feature map.

[0027] The implementation process of step S3 is as follows: This step is performed by the feature encoding module. Preferably, a shared-weight backbone network is used as the encoder to process the features of the aligned oblique side infrared driving image and the frontal infrared driving image respectively, generating a dual-view feature map with consistent semantic representation. The encoder network structure is specially designed to adapt to single-channel infrared image input. First, there is a 7×7 convolutional layer with 1 input channel, 32 output channels, a stride of 2, and padding of 3, followed by batch normalization and ReLU activation function; next is a 3×3 max pooling layer with a stride of 2 and padding of 1; then two residual blocks, the first residual block increases the number of channels from 32 to 64 and downsamples (stride of 2), the second residual block keeps the number of channels unchanged at 64; finally, there is a 3×3 convolutional layer with 128 input channels, 256 output channels, and padding of 1, followed by batch normalization and ReLU activation function. The entire encoding process achieves 4x downsampling, ultimately outputting a 256-dimensional feature map, resulting in a frontal infrared feature map and a diagonal infrared feature map with dimensions [B, 256, Hf, Wf]. Here, B is the batch size, Hf = H / / 4, and Wf = W / / 4 represent the height and width of the feature map after 4x downsampling by the encoder.

[0028] S4: Input the extracted dual-view features into a cross-view attention fusion module, calculate the attention map of the front view to the side view and the attention map of the side view to the front view through a bidirectional cross-attention mechanism, and then fuse them.

[0029] The implementation process of step S4 is as follows: This step employs a bidirectional cross-attention mechanism, comprising two symmetrical CrossAttention sub-modules. Each CrossAttention sub-module consists of three 1×1 convolutional layers: the query convolutional layer projects the input features onto a 256-dimensional key channel, while the key and value convolutional layers project features from the other perspective onto the same dimension. The attention calculation process uses the standard scaled dot product attention mechanism, expressed by the following formula: ; Where Q comes from the current viewpoint features, K and V come from another viewpoint features, and d is the key channel dimension of 256. The specific implementation process is as follows: First, the spatial dimensions of the feature map with shape [B, 256, Hf, Wf] are flattened and adjusted to [B, 256, L] (L = Hf * Wf), and the dimensional order is transformed to obtain Query, Key, and Value with shape [B, L, 256]. Then, the Query and Key are multiplied by matrix to obtain the Energy matrix with shape [B, L, L]. The Energy is Softmax normalized on the last dimension to obtain the attention weights. Then, the attention weights are multiplied by the Value to obtain the context features with shape [B, L, 256]. The dimensions of this feature are adjusted and restored to the shape [B, 256, Hf, Wf]. Finally, it is added to the original query features (residual connection) and multiplied by a learnable scaling factor gamma to obtain the attention output of this branch. The attention outputs from both directions (front to side and side to front) are concatenated with the original dual-view features in the channel dimension to form a 1024-dimensional fused feature ([B, 1024, Hf, Wf]), which is then reduced to a 512-dimensional output ([B, 512, Hf, Wf]) through a 1×1 convolutional layer.

[0030] S5: Input the fused infrared feature map into a pre-trained first attitude estimation network to realize driver attitude estimation in infrared fusion mode.

[0031] The implementation process of step S5 is as follows: Features of the frontal RGB driving image described in S1 are extracted using another feature encoder. This encoder has the same structure as the encoder in S3 but with independent weights, and transfer learning is performed using a pre-trained model. The output is a visible light feature map. To ensure dimensionality matching, it has the same spatial dimensions Hf and Wf as the infrared multi-view fused feature map output in S4.

[0032] Furthermore, the implementation process of step S5 is as follows: The fused features are fed into a backbone feature extraction network. In a preferred embodiment, this backbone network is constructed based on the HRNet architecture. Its initial layer preferably uses a 7×7 convolutional kernel with a stride of 2 and padding of 3 for preliminary feature transformation and downsampling, followed by a batch normalization layer and a ReLU activation function. The high-dimensional features extracted by the backbone network then pass through a feature adaptation layer. This layer preferably consists of a 1×1 convolutional kernel, used to increase the number of feature channels to 1024 dimensions to achieve deeper feature fusion and enhanced expressive power, followed by a batch normalization layer and a ReLU activation function. To enhance the model's ability to model global spatial dependencies, this invention preferably introduces a lightweight Transformer module to enhance the features. This module is based on the Swin Transformer architecture, with a depth configuration of 1, containing 4 attention heads, and using a 4×4 local window for self-attention computation. This module is responsible for recalibrating and enhancing spatial feature relationships. Finally, the network decodes the processed features into a keypoint heatmap through a prediction head. The prediction head consists of two convolutional layers in sequence: the first layer uses a 3×3 convolutional kernel and outputs 128-dimensional features; the second layer uses a 1×1 convolutional kernel, and the number of output channels corresponds to the number of predefined keypoints, which is 8 in this invention. This generates the final heatmap output, with dimensions [B, 8, H', W'], where H' = Hf / 8 and W' = Wf / 8.

[0033] To convert the heatmap into accurate keypoint coordinates and visibility confidence scores, the network employs a post-processing operation based on differentiable softargmax. The computation process can be represented as follows: ; in, τ The temperature hyperparameter is preferred to be 0.05. Subsequently, the normalized coordinates of the key points are obtained by calculating the expected value of this probability distribution with respect to the grid coordinates. ; Visibility confidence of key points v The result is obtained by linearly scaling and processing the maximum value of the heatmap using the sigmoid function: ; in, γ and β These are learnable parameters or preset scalars.

[0034] S6: Input the frontal RGB driving image into a pre-trained second pose estimation network to achieve driver pose estimation in RGB mode.

[0035] The implementation process of step S6 is as follows: The second pose estimation network receives a color image captured by a front-facing RGB camera as input, with dimensions [B, 3, H, W]. This network has a similar core architecture to the first pose estimation network in S5, but its input layer is configured to directly receive three-channel RGB input data.

[0036] Specifically, the initial convolutional layer of the encoder in the network preferably uses a 7×7 convolutional kernel with 3 input channels to adapt to the number of color channels in an RGB image. The image undergoes multi-level feature extraction via the backbone network to obtain a high-dimensional feature representation. Subsequently, the features are passed through a feature adaptation layer, preferably a 1×1 convolution, to adjust and fuse the channel dimensions. Optionally, a lightweight Transformer module is fed in to enhance global context awareness; its hyperparameters, such as depth, number of heads, and window size, are the same as the preferred values ​​for the module described in S5.

[0037] Finally, the features are fed into a prediction head, which outputs a keypoint heatmap with dimensions [B, 8, H', W'] through a series of convolutional operations. Subsequently, the same softargmax-based differentiable decoder described in S5 is applied to convert the heatmap into accurate keypoint coordinates and visibility confidence.

[0038] S7: Calculate the current ambient light intensity scalar based on the frontal RGB driving image, dynamically generate infrared-dominant gating weight coefficients and visible light-dominant gating weight coefficients according to the ambient light intensity scalar, and output a set of optimal driver key point coordinates that are robust to illumination.

[0039] The implementation process of step S7 is as follows: First, the lighting conditions of the current driving environment need to be quantitatively assessed. Specifically, a scalar of light intensity is calculated from the frontal RGB image, with a value ranging from [0, 1]. In a preferred embodiment, the scalar is calculated as follows: the RGB image is converted to the HSV color space, and its lightness channel, i.e., the V channel, is extracted; then, the statistical median of all pixel values ​​in the lightness channel is calculated, and the median is divided by 255 to normalize, thereby obtaining the light intensity scalar light_level. Its calculation formula can be briefly expressed as: ; After obtaining the scalar value of illumination intensity, this invention dynamically calculates the visible light mode weight (rgb_weight) and the infrared fusion mode weight (ir_weight) using a preset piecewise linear function. The weight calculation function is defined by a low illumination threshold and a high illumination threshold, with preferred values ​​of 0.3 and 0.7, respectively. ; ; ; The obtained weight values ​​are then used to fuse the coordinate data using a weighted average algorithm. ; Where fused_coords represents the fused keypoint coordinates, kp_rgb represents the keypoint coordinates obtained from the RGB image, and kp_ir represents the keypoint coordinates obtained from the infrared image. The visibility of the fused image is set to the maximum value of the visibility predicted by the RGB and infrared models, thus ensuring that keypoints can still be detected even in occlusion situations.

[0040] S8: Input the coordinates of the key points into a pre-trained temporal behavior classification model, and classify the driver's behavior according to predefined rules. Behavior categories include, but are not limited to: focused driving, using mobile phone, eating, etc.

[0041] The implementation process of step S8 is as follows: The temporal behavior classification model receives the illumination-robust keypoint coordinate sequence output from step S7 as input. This invention integrates a dual-strategy architecture combining a heuristic rule-based method and a deep learning model-based method to ensure reliable behavior recognition across different time windows.

[0042] Specifically, the behavior classification process is performed according to the following steps: First, temporal feature extraction and calculation are performed. For each frame of keypoint data in the input sequence, with dimensions [8, 3], the model first performs feature normalization: taking the neck keypoint coordinates as the origin, the neck coordinates are subtracted from all keypoint coordinates to obtain their normalized positions relative to the neck.

[0043] ; Based on this, a set of predefined multimodal temporal feature vectors is calculated, which preferably consists of the following 48 features: 1. Head position and velocity characteristics: normalized head position (2D) and its movement velocity between consecutive frames (1D).

[0044] ; 2. Hand relative position feature: The position vector (4D) of the left and right wrist key points relative to the head key points.

[0045] 3. Hand-mouth distance feature: Assuming the mouth is located at the position 0.1 below the head key point, calculate the Euclidean distance (2D) between the left and right wrists and the preset mouth.

[0046] 4. Elbow angle feature: According to the coordinates of the shoulder, elbow, and wrist points, use the cosine theorem to calculate the joint angles of the left and right elbows respectively (2D).

[0047] ; where A is the shoulder, B is the elbow, and C is the wrist 5. Head pose feature: The vector (2D) of the head key point relative to the neck key point.

[0048] 6. Visibility feature: The visibility confidence of all 8 key points (8D).

[0049] The feature vector is fed into a first-in-first-out (FIFO) buffer, and its capacity (window size) is preferably set to 30 frames. The classification decision adaptively selects one of the following two methods according to the buffer filling state: Heuristic rule-based method: When the number of frames in the buffer is insufficient, the system uses predefined expert rules to make real-time judgments on the latest calculated single-frame features. The rules include but are not limited to: Playing with mobile phone: The hand is close to the lower part of the head (|x-direction displacement of the wrist relative to the head| < 0.1, and 0.1 < y-direction displacement < 0.3), and the ipsilateral elbow angle < 100 degrees; Making a call: The hand is close to the ear position (|x-direction displacement of the wrist relative to the head ∓ 0.15| < 0.05, and |y-direction displacement - 0.05| < 0.05); Eating or drinking in the car: The distance from the hand to the mouth < 0.1, and this state needs to last for more than 0.5 seconds (maintained by a counter); Drowsy: The head is drooping (y-component of the vector of the head relative to the neck < -0.15) and the moving speed < 0.02, and this state needs to last for more than 1 second; Covering the mouth and yawning: The distance from the hand to the mouth < 0.05, and the head is tilted back (y-component of the vector of the head relative to the neck > 0.1).

[0050] Deep learning model-based method: When the buffer collects 30 frames of data, the system enables a pre-trained deep learning time series model. This model receives a tensor with dimensions [1, 30, 48], corresponding to the batch size, time step, and feature dimension respectively.

[0051] The tensor is first fed into a bidirectional LSTM network for temporal modeling. This network is preferably set to 2 layers with 128 hidden layer units, and it ultimately outputs the hidden states in two directions at each time step.

[0052] ; splicing in two directions This yields a feature tensor with dimensions [1, 30, 256]. Next, an attention network is used to calculate the weights for this sequence. This network consists of fully connected layers and a softmax function, receiving the output of an LSTM and calculating the importance weights for each time step, with dimensions [1, 30, 1]. The LSTM output features and attention weights are then weighted and summed to obtain a fixed context vector with dimensions [1, 256]. This vector aggregates the global information of the entire time series. Finally, this context vector is fed into a fully connected layer classifier, which outputs predicted scores for six categories: "focused driving," "using a mobile phone," "making and receiving phone calls," "eating in the car," "drowsy," and "yawning while covering one's mouth." The highest score is taken as the final behavior classification result.

[0053] This invention also provides an environment-adaptive multimodal driver behavior monitoring system for implementing the above-described method. The system includes: an image acquisition module, a viewpoint alignment module, a feature encoding module, a cross-viewpoint fusion module, an illumination-adaptive gating module, a pose estimation module, and a behavior classification module. Specifically, the image acquisition module acquires frontal RGB, frontal infrared, and oblique side infrared images; the feature extraction module extracts features from the RGB images; the viewpoint alignment module, based on deformable convolution, performs geometric alignment on the oblique side infrared images; the encoder module extracts features from the aligned dual-view infrared images; the cross-viewpoint attention fusion module fuses dual-view infrared features using a bidirectional cross-attention mechanism; the illumination-adaptive gating module dynamically fuses infrared and visible light features based on ambient light intensity; the pose estimation module predicts keypoint coordinates based on the fused features; and the behavior classification module identifies driving behavior based on the keypoint coordinates.

[0054] The embodiments described in this specification are merely examples of implementations of the inventive concept and are for illustrative purposes only. The scope of protection of this invention should not be considered limited to the specific forms described in these embodiments; rather, it extends to equivalent technical means conceived by those skilled in the art based on the inventive concept.

Claims

1. An environmentally adaptive multi-modal driver behavior monitoring method, characterized in that, The method comprises the following steps: S1 respectively collecting a front RGB driving image, a front infrared driving image and an oblique side infrared driving image; S2 using a view alignment module based on deformable convolution to perform geometric transformation on the oblique side infrared driving image to align it with the front infrared driving image in space; S3 using an encoder network with shared weights to respectively extract features of the aligned oblique side infrared driving image and the front infrared driving image to generate consistent semantic representation of the dual-view feature map, i.e. the front-view feature map and the side-view feature map; S4 inputting the extracted dual-view features into a cross-view attention fusion module to calculate and fuse the attention map of the front view to the side view and the attention map of the side view to the front view through a bidirectional cross attention mechanism; S5 inputting the fused infrared fusion feature map into a pre-trained first pose estimation network to realize driver pose estimation in the infrared fusion mode; S6 inputting the front RGB driving image into a pre-trained second pose estimation network to realize driver pose estimation in the RGB mode; S7 calculating the illumination intensity scalar of the current environment based on the front RGB driving image, dynamically generating infrared dominant gating weight coefficients and visible light dominant gating weight coefficients according to the illumination intensity scalar, and outputting a set of optimal driver key point coordinates that are robust to illumination; S8 inputting the key point coordinates into a pre-trained time sequence behavior classification model to classify the behavior of the driver according to pre-defined rules, and the behavior categories include: focused driving, using mobile phone and eating.

2. The method of claim 1, wherein, The S2 comprises the following steps: The front infrared driving image and the oblique side infrared driving image are spliced into a feature tensor along the channel dimension; An offset prediction network composed of three convolutional layers is used to predict the offset field required by deformable convolution; A deformable transformation is applied to the oblique side infrared driving image through deformable convolution to realize non-rigid spatial alignment, so that it is consistent with the front infrared driving image in spatial structure.

3. The method of claim 1, wherein, The S3 comprises the following steps: A shared weight backbone network is used as an encoder to process the features of the aligned side-view feature map and the front-view feature map to generate a consistent semantic representation of the dual-view feature map. The encoder network structure is adapted to single-channel infrared image input, comprising a 7x7 convolutional layer followed by batch normalization and ReLU activation function, a 3x3 max pooling layer, two residual blocks, and a 3x3 convolutional layer followed by batch normalization and ReLU activation function.

4. The method of claim 1, wherein, The S4 comprises the following steps: A bidirectional cross attention mechanism is used, which includes two symmetric CrossAttention sub-modules, each CrossAttention sub-module including a query convolutional layer Query, a key convolutional layer Key and a value convolutional layer Value, all of which are composed of three 1x1 convolutional layers: The query convolutional layer projects the input features to the corresponding dimension key channel, and the key convolutional layer and the value convolutional layer project the features of the other view to the same dimension. The Query and the Key are multiplied to obtain an Energy matrix; the Energy is normalized by Softmax in the last dimension to obtain attention weights; then, the attention weights are multiplied with the Value to obtain context features; The dimensions of the features are adjusted, and finally added to the original query features and multiplied by a learnable scaling factor gamma to obtain the attention output of the branch; the attention outputs of the front-to-side and side-to-front directions are concatenated with the original dual-view features in the channel dimension to form a fused feature, which is finally output by a 1x1 convolutional layer.

5. The method of claim 1, wherein, The S5 includes the following steps: The fused feature is input into a backbone feature extraction network based on the HRNet structure, which includes an initial layer and four residual layers, wherein the initial layer includes a 7x7 convolution, a batch normalization, a ReLU activation function and a maximum pooling, and each residual layer includes a plurality of Bottleneck blocks; The initial layer of the backbone feature extraction network is used to process the fused feature, which adopts a 7x7 convolution kernel for preliminary feature transformation and down-sampling, and is connected with a batch normalization layer and a ReLU activation function after the 7x7 convolution kernel; The high-dimensional feature extracted by the backbone network is input into a feature adaptation layer, which is composed of a 1x1 convolution kernel and is connected with a batch normalization layer and a ReLU activation function after the 1x1 convolution kernel; A lightweight Transformer module based on the Swin Transformer architecture is used to enhance the feature processed by the feature adaptation layer, which has a depth configuration of 1, contains 4 attention heads inside, and uses a 4x4 local window for self-attention calculation to recalibrate and enhance the spatial feature relationship; The feature enhanced by the Transformer module is decoded into a keypoint heat map through a prediction head, which is sequentially composed of two convolutional layers, the first layer adopts a 3x3 convolution kernel, and the second layer adopts a 1x1 convolution kernel, and the output channel number corresponds to the predefined keypoint; A post-processing operation based on differentiable softargmax is used to convert the heat map into keypoint coordinates and visibility confidence.

6. The method of claim 1, wherein, The S6 includes the following steps: The front RGB driving image is used as input; The initial convolutional layer of the network encoder adopts a 7x7 convolution kernel; The backbone network is used to extract multi-level features from the image processed by the initial convolutional layer to obtain a high-dimensional feature representation; The backbone network includes an initial layer and four residual layers, wherein the initial layer includes a 7x7 convolution, a batch normalization, a ReLU activation function and a maximum pooling, and each residual layer includes a plurality of Bottleneck blocks; The feature adaptation layer is a 1x1 convolution, which is used to adjust and fuse the channel dimension of the high-dimensional feature representation. The lightweight Transformer module is connected with the feature adaptation layer, and is used for enhancing global context awareness of the feature, wherein the depth, the number of headers, and the window size of the super parameter are the same as the parameters of the corresponding module in the first pose estimation network; The prediction head processes the feature processed by the feature adaptation layer or the lightweight Transformer module through a series of convolution operations, and outputs a key point heat map; A decoding method based on softargmax is used to convert the key point heat map into key point coordinates and visibility confidence.

7. The method of claim 1, wherein, The S7 includes the following steps: The light condition of the current driving environment is quantitatively evaluated: a light intensity scalar is calculated from the front RGB driving image, and the value range is [0, 1]; After obtaining the light intensity scalar, the preset piecewise linear function is used to dynamically calculate the visible light modal weight rgb_weight and the infrared fusion modal weight ir_weight; The piecewise linear function is defined by a low light threshold and a high light threshold. The obtained weight values are fused through a weighted average algorithm to realize the fusion of coordinate data: ； Wherein fused_coords is the fused key point coordinate value, kp_rgb is the key point coordinate obtained in the RGB image, and kp_ir is the key point coordinate obtained in the infrared image; The maximum value of the visibility predicted by the RGB and infrared models is taken as the fused visibility, so that the key points can still be detected in the occlusion case.

8. The method of claim 1, wherein, The S7 includes the following steps: For each frame of key point data in the input sequence, first, the feature is normalized, taking the neck key point coordinate as the origin, and subtracting all key point coordinates from the neck coordinate to obtain the normalized position relative to the neck; A set of predefined multi-modal time sequence feature vectors are calculated, including head position and speed features, hand relative position features, hand and mouth distance features, elbow angle features, head posture features, and visibility features; The feature vectors are sent to a first-in-first-out buffer; according to the filling state of the buffer, one of the following two methods is adaptively selected: A heuristic rule-based method: when the number of frames in the buffer is insufficient, a predefined expert rule is used to judge the single-frame feature calculated in real time, and the rule includes: judging rules for playing a mobile phone, making and receiving a call, eating and drinking in the car, dozing off, and covering the mouth to yawn; A deep learning model-based method: when the number of collected frames in the buffer meets the requirement, a pre-trained deep learning time sequence model is enabled, input into a bidirectional LSTM network for time sequence modeling, and the hidden states of two directions at each time step are output and spliced to obtain a feature tensor, and an attention network is used to calculate the weight of the sequence, The attention network is composed of a full connection layer and a Softmax function, receives the output of the LSTM, and calculates the importance weight of each time step; the LSTM output feature and the attention weight are weighted and summed to obtain a fixed context vector Context Vector; the context vector Context Vector is sent to a full connection layer classifier to output the prediction scores of six categories [focused driving, playing mobile phones, making and receiving calls, eating and drinking in the car, dozing off, covering the mouth and yawning], and the maximum score is taken as the final behavior classification result.

9. An environment-adaptive based multi-modal driver behavior monitoring system implementing the method of claims 1-8, comprising: The image acquisition module, the view alignment module, the feature encoding module, the cross-view fusion module, the light adaptive gating module, the pose estimation module, and the behavior classification module; The image acquisition module is used for acquiring front RGB, front infrared, and oblique side infrared images; The feature extraction module is used for extracting RGB image features; The view alignment module is realized based on deformable convolution and is used for geometric alignment of the oblique side infrared image; The encoder module is used for extracting features of the aligned dual-view infrared image; The cross-view attention fusion module is used for fusing dual-view infrared features through a bidirectional cross-attention mechanism; The light adaptive gating module is used for dynamically fusing infrared and visible light features according to the environmental light intensity; The pose estimation module is used for predicting key point coordinates according to the fused features; The behavior classification module is used for driving behavior recognition based on the key point coordinates.

Images