A method for detecting and reminding fatigue driving of a car recorder based on multi-modal data fusion

Abstract

The application discloses a driving recorder fatigue driving detection and reminding method based on multi-modal data fusion, comprising the following steps: S1, a vehicle-mounted camera collects a face image and pre-processes, carries out face detection and region positioning; S2, a dynamic image block technology is used, and the size and position of the image block are dynamically adjusted; S3, a face image feature is extracted by using a visual transformer; S4, vehicle multi-modal data is acquired and is fused with the face feature; S5, a wolf swarm optimization algorithm is used to optimize the network structure of the visual transformer and train the hyperparameters; S6, a fatigue state is analyzed based on the optimized visual transformer; S7, when the fatigue state is detected, a sound and light alarm is used to remind the driver to rest. Through multi-modal data fusion and an optimized deep learning algorithm, the application provides a driving recorder fatigue driving detection and reminding method with high precision and high real-time performance, and effectively improves the recognition accuracy of fatigue driving and the robustness of the system.

Description

Technical Field

[0001] This invention relates to the field of traffic safety technology, and in particular to a method for fatigue driving detection and alerting based on multimodal data fusion using a dashcam. Background Technology

[0002] With the rapid development of intelligent technologies, intelligent transportation systems have become an important component of modern traffic management and safety assurance. Against this backdrop, fatigued driving, as one of the main causes of traffic accidents, has attracted widespread attention and research. Fatigued driving leads to a significant decline in a driver's attention, judgment, and reaction speed, thereby increasing the risk of traffic accidents. To address this issue, numerous scholars and companies have proposed methods for fatigued driving detection and early warning based on in-vehicle equipment.

[0003] Traditional methods for detecting driver fatigue primarily rely on vehicle state data (such as speed and acceleration) and the driver's physiological and behavioral characteristics. Dynamic features such as vehicle speed, braking frequency, and steering wheel angle are the main basis for early-stage driver fatigue detection. While these methods can monitor the driver's state to some extent, their accuracy and real-time performance are relatively low, especially in complex road conditions and when the driver's behavior is not readily apparent, significantly limiting their effectiveness.

[0004] With the rise of deep learning technology, fatigue driving detection methods based on computer vision and facial features have gradually gained widespread application. Utilizing cameras to acquire facial images of drivers and analyzing changes in areas such as the eyes and mouth (e.g., eye closure, yawning, blinking frequency) through facial feature extraction algorithms has become a hot research direction in fatigue driving detection in recent years. Facial recognition technology, based on methods such as deep neural networks (DNNs) and convolutional neural networks (CNNs), achieves accurate identification of driver fatigue states through large-scale data training.

[0005] However, existing computer vision-based methods still have some significant shortcomings. First, traditional image segmentation techniques use fixed-size image blocks to process the driver's facial region, which has significant limitations when processing changes in facial expressions. For example, the size and shape of the driver's eyes, mouth, and other areas in the image change with facial expressions, and fixed-size image blocks cannot accurately capture these subtle changes, resulting in low accuracy in extracting fatigue features. Furthermore, traditional image processing techniques rely heavily on preset algorithms and rules, and cannot adaptively adjust the size or position of the blocks, thus failing to efficiently extract driver fatigue features.

[0006] Secondly, fatigue detection methods based on facial images often rely on single image data, neglecting other dynamic vehicle data such as speed, acceleration, and steering angle. These factors can influence driver fatigue to some extent. For example, during high-speed driving or prolonged driving, driver fatigue may change due to the vehicle's motion. Traditional methods typically rely solely on image data to determine fatigue levels, failing to comprehensively consider multiple factors, thus affecting detection accuracy and robustness.

[0007] Furthermore, traditional fatigue driving detection methods have certain shortcomings in their optimization algorithms. Many current methods use conventional optimization algorithms (such as gradient descent and genetic algorithms) for network training and model tuning, but these methods are prone to getting trapped in local optima, preventing the model from achieving global optimization and thus affecting the final detection results. Especially when facing complex driving scenarios, existing optimization methods often fail to effectively address issues such as low training efficiency and low accuracy, resulting in poor robustness and stability of fatigue driving detection systems.

[0008] Therefore, how to provide a method for fatigue driving detection and alerting based on multimodal data fusion in dashcams is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0009] One objective of this invention is to propose a fatigue driving detection and alert method for dashcams based on multimodal data fusion. This invention fully utilizes computer vision technology, deep learning algorithms, natural heuristic optimization algorithms, and multimodal data fusion technology. It details the process of extracting driver facial features using dynamic image segmentation combined with a Visual Transformer (ViT), and optimizing the training process of the neural network model using a Wolf Pack Algorithm (WOA), thereby achieving high-precision and robust fatigue driving detection and real-time warning. This method has the advantages of high detection accuracy, strong real-time performance, good system stability, and the ability to comprehensively consider various driver behaviors and vehicle dynamic information.

[0010] The fatigue driving detection and alert method for a vehicle recorder based on multimodal data fusion according to an embodiment of the present invention includes the following steps:

[0011] S1. Collect facial image data of the driver through the vehicle camera, perform preprocessing, perform face detection and region localization, and obtain the facial feature region of the driver.

[0012] S2. Dynamic image segmentation technology is adopted to dynamically adjust the size and position of image blocks according to different features of the driver's facial area, and to segment the facial image.

[0013] S3. Use a visual transformer to extract features from the segmented facial images to identify early signs of driver fatigue.

[0014] S4. Acquire multimodal data of the vehicle and fuse the multimodal data of the vehicle with facial features to form a fatigue driving judgment model;

[0015] S5. Optimize the network structure and training hyperparameters of the visual transformer using the wolf pack optimization algorithm;

[0016] S6. Based on the optimized visual transformer, the driver's fatigue state is analyzed in real time, and the driver's fatigue level is judged by combining facial features and multimodal data from multiple vehicles.

[0017] S7. When the system detects that the driver is fatigued, it will issue an alarm with sound and light to remind the driver to take a rest.

[0018] Optionally, S2 specifically includes:

[0019] S21. Extract specific facial regions from the driver's facial image. The facial regions include the eye, mouth, and eyebrow regions. Using a deep learning-based facial key point detection model, automatically locate and mark the positions of each key point on the face to form the outline of the facial region and accurately determine the boundaries of the facial region.

[0020] S22. An adaptive block-segmentation method based on facial key points is used to process facial images. Combining facial muscle movement patterns and spatial features, the size and position of image blocks in each facial region are dynamically adjusted. The local spatial distribution of each facial region is calculated through the positions of facial key points and the geometric relationships between adjacent key points. Then, the size and shape of the image blocks are dynamically determined according to the movement patterns and texture characteristics of each region.

[0021]

[0022] Among them, P i Let K be the size of the i-th image patch. i Let D be the location of the i-th facial key point. ij Let T be the Euclidean distance between the i-th keypoint and the j-th keypoint. i M represents the texture complexity of the facial region. i For the movement patterns of the facial region, α i ,β i ,γ i ,θ i ρ is a coefficient, γ, δ, ζ are weight adjustment factors, and ρ is a weight adjustment factor. i denoted as the rate of dynamic texture change in the facial region, and log2 is a logarithmic function.

[0023] S23. Image patch sliding is performed using the temporal dimension features of the facial region. The step size and shape of each sliding window dynamically change according to the movement frequency of the facial region and the rate of expression changes.

[0024]

[0025] Among them, D x (K x ,K y ) and D y (K x ,K y ) represent facial key points K respectively x and K y The relative distances on the x and y axes, γ1 and γ2 are the basic adjustment factors for the sliding step size, λ1 and λ2 are the dynamic expansion factors in the time dimension, and M is the relative distance on the x and y axes. x and M y T represents the intensity of changes in facial movement patterns in the x and y directions, respectively. x and T y Δx represents the temporal rate of change of facial features, and Δy represents the offset of the sliding window in the image space along the x and y directions during image processing, respectively.

[0026] S24. Enhance the extraction of facial fatigue features by employing a fusion model that combines dynamic adaptive local correlation tensors and hierarchical time-scale adjustment:

[0027]

[0028] Among them, A i For the enhancement features of the i-th image patch, I(P) i ) and I(P j ) represent image blocks P respectively i and P j The local eigenvectors, where σ is the kernel function width, ||I(P) j )-I(P i )|| 2 For the feature differences between image patches, A j For image block P j Characteristic weighting coefficients, φ i ω is a local dynamic adjustment factor. i Dynamic weights for each image patch, λ i μ is the expansion coefficient related to the time scale of facial features. i T is the decay factor in the time dimension. i exp() is the rate of change over time, N is the total number of image patches, and e is the base of the natural logarithm.

[0029] S25. By weighted summation of the enhanced features of the image patches, a global fatigue feature representation is generated:

[0030]

[0031] Among them, A global As a representation of global fatigue characteristics, A i For the enhancement features of the i-th image patch, α i C represents the attention weights for image patches. i T represents the local feature complexity of an image patch. i λ represents the temporal scale features of the image patch, λ3 is the global feature enhancement factor, and δ1,μ,ν are adaptive adjustment factors.

[0032] Optionally, S3 specifically includes:

[0033] S31. Divide the segmented facial image block P into blocks. i Each image patch P is input into the visual transformer model. i ∈R H×W×C Where H and W are the height and width of the image patch, respectively, and C is the number of channels. Each image patch is converted into a one-dimensional vector V. i ∈R D Where D is the feature embedding dimension and V is the embedding vector. i Represents the high-dimensional features of each image patch;

[0034] S32, For each image block P i A nonlinear adaptive positional coding mechanism for facial regions is introduced, combining the spatial location of facial regions with dynamic changes in facial expressions for positional coding. It depends not only on the geometric structure of facial expressions, but also on the temporal information of each image patch:

[0035]

[0036] Among them, K i Let ρ be the size of the facial region in the i-th image patch. i f is the rate of dynamic texture change in the region where the image patch is located. i For regional frequency characteristics, For the temporal characteristics of facial expressions, η i α is the scaling factor, α is the time dimension exponent, log2 is the logarithmic function, and sin is the sine function;

[0037] S33. The image patch with position encoding is embedded into a vector and input to the self-attention mechanism module of the visual transformer for feature extraction, and the self-attention feature representation of each image patch is calculated.

[0038]

[0039] Among them, Q i For querying the matrix, K i Let B be the key matrix. i For the facial expression-based bias term, d k λ is the dimension of the key. i and μ i M is the adjustment factor. i For the movement patterns of the facial region, T i τ is the time scale for changes in facial expressions. i V is the expansion coefficient, softmax() is the normalization function, and V is the normalization coefficient. i Let be the embedding vector of the i-th image patch;

[0040] S34. Introduce a spatiotemporal fusion module into a multi-layer self-attention network to fuse spatiotemporal features of different scales through a multi-head self-attention mechanism based on multi-scale attention:

[0041]

[0042] in, Q represents the feature representation of the i-th image patch. i and K j For the query and key matrix, β j For the temporal offset of the image patch, ρ i δ is the spatiotemporal scale factor. i and γ i The dynamic adjustment factor is exp(), where exp() is the exponential function and N is the total number of image patches. Embedded vectors for image patches;

[0043] S35. Through spatiotemporal self-attention mechanism and multi-scale feature fusion, a global feature representation of each image patch is generated. The global feature representations of all image patches are weighted and summed to obtain the final fatigue feature representation.

[0044]

[0045] in, This represents the final facial fatigue characteristics. For the global features of the i-th image patch, α i C represents the weighting coefficients for the image patch. i For the detailed features of the image patch, T i λ is the scale of the time feature. i and δ i κ is the adaptive adjustment factor. i It is a comprehensive adjustment factor for overall fatigue characteristics.

[0046] Optionally, S4 specifically includes:

[0047] S41. Extract vehicle speed data V from the vehicle's sensor system. speed Acceleration data a acc Steering angle data θ turn And convert it into time series data D vehicle (t)=[V speed (t),a acc (t),θ turn [(t)], where t represents a time point;

[0048] S42. Represent the driver's final fatigue characteristics. As input for the fusion of vehicle multimodal data, a multimodal fusion network is constructed;

[0049] S43. Integrate the time series data and the final fatigue feature representation into a comprehensive feature representation F. fusion (t):

[0050]

[0051] Where ω1, ω2, ω3, ω4 are weighting coefficients;

[0052] S44. Input the comprehensive feature representation into the fatigue driving judgment model based on an adaptive deep generative adversarial network. The fatigue driving judgment model uses the predicted fatigue driving value... Assess fatigue status:

[0053]

[0054] Among them, W i Let be the weight of the i-th feature dimension. Let be the value of the i-th feature, b be the bias term, τ() be the activation function, γ be the adjustment coefficient, and α be the nonlinear influence factor of multimodal features on fatigue prediction. The information represents the change in each modality data over time, where ∈ is the noise factor, b is the bias term, tanh is the hyperbolic tangent activation function, n is the number of features, and m is the number of time-varying features.

[0055] S45, Predicted values ​​based on fatigue driving Driver fatigue is alerted via audible and visual alarms. If the warning threshold is reached or exceeded, the system will trigger an alarm mechanism to remind the driver to take a break.

[0056] Optionally, S5 specifically includes:

[0057] S51. Initialize the hyperparameters of the visual transformer model, including the image patch size P. iThe number of Transformer layers L, the learning rate η, and the number of hidden units H in each layer;

[0058] S52. Define an objective function F(θ) to evaluate the performance of the visual transformer model. The objective function combines the training loss and a regularization term:

[0059]

[0060] in, For predicted values The loss function between the true value y(t) and the actual value λ r θ is the regularization coefficient. i Let be the hyperparameter of the i-th model, T be the number of training epochs, n be the number of hyperparameters, and K be the number of hyperparameters.

[0061] S53. Optimize hyperparameters using a wolf pack optimization algorithm, simulate the wolf pack foraging process, and find the optimal solution in the parameter space:

[0062]

[0063] in, Let X be the update value of the i-th hyperparameter in round t+1, A and B be scaling factors, D be the distance factor, and X be the distance factor. best and X worst These are the hyperparameter values ​​for the best and worst solutions in the current wolf pack, respectively. Let be the update value of the i-th hyperparameter in round t;

[0064] S54. In each round of optimization, the model validation set is used. Evaluate the optimization effect of hyperparameters and calculate the validation error. And adjust the wolf pack's search strategy based on the error value:

[0065]

[0066] Among them, F WOA (θ) is the objective function optimized using the wolf pack optimization algorithm, γ o As an adjustment factor, α i Here, θ is the set of all hyperparameters of the model, p is the intensity factor of the penalty term, and e is the base of the natural logarithm.

[0067] S55. Repeat steps S53 to S54 until the objective function F(θ) converges to its minimum value, finally obtaining the optimized hyperparameter configuration θ. * These hyperparameters are then used for training the visual converter model and for fatigue driving detection.

[0068] Optionally, S6 specifically includes:

[0069] S61. Combine the optimized visual transformer network model with the optimal hyperparameter θ obtained through the wolf pack optimization algorithm. * This is applied to real-time driver facial features and multimodal data, with the input multimodal data being... Among them, V speed (t) represents the vehicle speed data, a acc (t) Acceleration data, θ turn (t) represents the steering angle data. This represents the final facial fatigue characteristics;

[0070] S62, from the input multimodal data F fusion In (t), the final facial fatigue feature representation and vehicle data are extracted, and fatigue state assessment is performed based on the weighted sum of each feature according to the model. Combining facial features and vehicle multimodal data, a comprehensive feature representation C is defined. fusion (t) is:

[0071]

[0072] Where, ω i Let be the weighting coefficient for the i-th feature. Let ζ be the i-th feature after fusion, ζ be the dynamic weighting factor, Attention() be the weighted feature fusion operation based on the attention mechanism, and n be the number of features;

[0073] S63. Input the comprehensive features into the output layer of the neural network to calculate the driver's fatigue state prediction value, and smooth it by introducing a sliding window mechanism to obtain a stable fatigue state prediction value.

[0074]

[0075] in, These are the smoothed fatigue state prediction values. Let μ be the fatigue state value predicted at time k, μ be the adjustment factor, tanh be the hyperbolic tangent activation function, and T be the value of the fatigue state predicted at time k. cycle For periodic factors, Δt is the sliding window size, and U is the number of time steps;

[0076] S64. Introduce a deep reinforcement learning mechanism based on adaptive dynamic weight adjustment to dynamically adjust the weight coefficients of each modality feature and optimize the weights through continuous environmental feedback:

[0077]

[0078] Where θ is the learning rate. Let ρ be the gradient of the loss function, and ρ be the periodic adjustment factor. Let i be the new weight of the i-th feature at time t+1. Let y(t) be the current weight of the i-th feature at time t, and y(t) be the actual fatigue state. For a periodic function, T period The period length;

[0079] S65. Predicted fatigue state value based on smoothed To determine if a driver is fatigued, if... Reaching or exceeding the fatigue warning threshold will trigger the audible and visual alarm system and visual cues.

[0080] The beneficial effects of this invention are:

[0081] This invention innovatively combines multimodal data fusion with deep vision algorithms to solve several key problems in traditional fatigue driving detection methods. First, it employs dynamic image segmentation technology, which dynamically adjusts the size and position of image blocks based on key points on the driver's face. This innovation significantly improves the accuracy of facial feature extraction, especially in capturing subtle signs of fatigue such as eye closure and yawning. Compared to traditional fixed image block processing methods, this invention's image segmentation method exhibits higher recognition rates and robustness when dealing with complex facial expressions.

[0082] Secondly, traditional fatigue driving detection methods often rely on single facial image data, failing to fully utilize vehicle dynamic information. To address this issue, this invention introduces multimodal data such as vehicle speed, acceleration, and steering angle, and combines this data with facial image data through a fusion algorithm to form a comprehensive judgment basis. Multimodal data fusion can comprehensively reflect the driver's fatigue state, enabling the system to maintain high accuracy and stability under various driving environments. In contrast, existing methods often rely solely on image data, resulting in limitations in detection accuracy and reliability.

[0083] Furthermore, this invention employs the Wolf Pack Optimization (WOA) algorithm to optimize the neural network training process, addressing the problem of traditional training methods easily getting trapped in local optima. By optimizing the network's hyperparameters, this invention improves the model's training efficiency and final detection accuracy, avoiding inefficiencies during training, especially when dealing with large-scale and diverse data, where the optimization effect is particularly significant. This optimization not only enhances system performance but also ensures its real-time performance and stability in practical applications.

[0084] In summary, this invention, by combining advanced computer vision, deep learning, and natural heuristic optimization algorithms, significantly improves the accuracy, robustness, and real-time performance of the fatigue driving detection system, providing a more intelligent and precise solution for traffic safety. Furthermore, the introduction of multimodal data fusion technology enables this invention to more comprehensively assess driver fatigue levels, thereby reducing the risk of traffic accidents caused by fatigue driving. Attached Figure Description

[0085] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0086] Figure 1 This is a flowchart of the fatigue driving detection and alert method for a dashcam based on multimodal data fusion proposed in this invention;

[0087] Figure 2 This is a schematic diagram of multimodal data fusion in the vehicle recorder fatigue driving detection and alert method based on multimodal data fusion proposed in this invention. Detailed Implementation

[0088] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0089] refer to Figure 1 and Figure 2 A method for fatigue driving detection and alerting based on multimodal data fusion in dashcams includes the following steps:

[0090] S1. Collect facial image data of the driver through the vehicle camera, perform preprocessing, perform face detection and region localization, and obtain the facial feature region of the driver.

[0091] S2. Dynamic image segmentation technology is adopted to dynamically adjust the size and position of image blocks according to different features of the driver's facial area, and to segment the facial image.

[0092] S3. Use a visual transformer to extract features from the segmented facial images to identify early signs of driver fatigue.

[0093] S4. Acquire multimodal data of the vehicle and fuse the multimodal data of the vehicle with facial features to form a fatigue driving judgment model;

[0094] S5. Optimize the network structure and training hyperparameters of the visual transformer using the wolf pack optimization algorithm;

[0095] S6. Based on the optimized visual transformer, the driver's fatigue state is analyzed in real time, and the driver's fatigue level is judged by combining facial features and multimodal data from multiple vehicles.

[0096] S7. When the system detects that the driver is fatigued, it will issue an alarm with sound and light to remind the driver to take a rest.

[0097] In this embodiment, S2 specifically includes:

[0098] S21. Extract specific facial regions from the driver's facial image. The facial regions include the eye, mouth, and eyebrow regions. Using a deep learning-based facial key point detection model, automatically locate and mark the positions of each key point on the face to form the outline of the facial region and accurately determine the boundaries of the facial region.

[0099] S22. An adaptive block-segmentation method based on facial key points is used to process facial images. Combining facial muscle movement patterns and spatial features, the size and position of image blocks in each facial region are dynamically adjusted. The local spatial distribution of each facial region is calculated through the positions of facial key points and the geometric relationships between adjacent key points. Then, the size and shape of the image blocks are dynamically determined according to the movement patterns and texture characteristics of each region.

[0100]

[0101] Among them, P i Let K be the size of the i-th image patch. i Let D be the location of the i-th facial key point. ij Let T be the Euclidean distance between the i-th keypoint and the j-th keypoint. i M represents the texture complexity of the facial region. i For the movement patterns of the facial region, α i ,β i ,γ i ,θ i ρ is a coefficient, γ, δ, ζ are weight adjustment factors, and ρ is a weight adjustment factor. i denoted as the rate of dynamic texture change in the facial region, and log2 is a logarithmic function.

[0102] S23. Image patch sliding is performed using the temporal dimension features of the facial region. The step size and shape of each sliding window dynamically change according to the movement frequency of the facial region and the rate of expression changes.

[0103]

[0104] Among them, D x (K x ,K y ) and Dy (K x ,K y ) represent facial key points K respectively x and K y The relative distances on the x and y axes, γ1 and γ2 are the basic adjustment factors for the sliding step size, λ1 and λ2 are the dynamic expansion factors in the time dimension, and M is the relative distance on the x and y axes. x and M y T represents the intensity of changes in facial movement patterns in the x and y directions, respectively. x and T y Δx represents the temporal rate of change of facial features, and Δy represents the offset of the sliding window in the image space along the x and y directions during image processing, respectively.

[0105] S24. Enhance the extraction of facial fatigue features by employing a fusion model that combines dynamic adaptive local correlation tensors and hierarchical time-scale adjustment:

[0106]

[0107] Among them, A i For the enhancement features of the i-th image patch, I(P) i ) and I(P j ) represent image blocks P respectively i and P j The local eigenvectors, where σ is the kernel function width, ||I(P) j )-I(P i )|| 2 For the feature differences between image patches, A j For image block P j Characteristic weighting coefficients, φ i ω is a local dynamic adjustment factor. i Dynamic weights for each image patch, λ i μ is the expansion coefficient related to the time scale of facial features. i T is the decay factor in the time dimension. i exp() is the rate of change over time, N is the total number of image patches, and e is the base of the natural logarithm.

[0108] S25. By weighted summation of the enhanced features of the image patches, a global fatigue feature representation is generated:

[0109]

[0110] Among them, A global As a representation of global fatigue characteristics, A i For the enhancement features of the i-th image patch, α i C represents the attention weights for image patches. iT represents the local feature complexity of an image patch. i λ represents the temporal scale features of the image patch, λ3 is the global feature enhancement factor, and δ1,μ,ν are adaptive adjustment factors.

[0111] In this embodiment, S3 specifically includes:

[0112] S31. Divide the segmented facial image block P into blocks. i Each image patch P is input into the visual transformer model. i ∈R H×W×C Where H and W are the height and width of the image patch, respectively, and C is the number of channels. Each image patch is converted into a one-dimensional vector V. i ∈R D Where D is the feature embedding dimension and V is the embedding vector. i Represents the high-dimensional features of each image patch;

[0113] S32, For each image block P i A nonlinear adaptive positional coding mechanism for facial regions is introduced, combining the spatial location of facial regions with dynamic changes in facial expressions for positional coding. It depends not only on the geometric structure of facial expressions, but also on the temporal information of each image patch:

[0114]

[0115] Among them, K i Let ρ be the size of the facial region in the i-th image patch. i f is the rate of dynamic texture change in the region where the image patch is located. i For regional frequency characteristics, For the temporal characteristics of facial expressions, η i α is the scaling factor, α is the time dimension exponent, log2 is the logarithmic function, and sin is the sine function;

[0116] S33. The image patch with position encoding is embedded into a vector and input to the self-attention mechanism module of the visual transformer for feature extraction, and the self-attention feature representation of each image patch is calculated.

[0117]

[0118] Among them, Q i For querying the matrix, K i Let B be the key matrix. i For the facial expression-based bias term, d k λ is the dimension of the key. i and μ i M is the adjustment factor. i For the movement patterns of the facial region, T iτ is the time scale for changes in facial expressions. i V is the expansion coefficient, softmax() is the normalization function, and V is the normalization coefficient. i Let be the embedding vector of the i-th image patch;

[0119] S34. Introduce a spatiotemporal fusion module into a multi-layer self-attention network to fuse spatiotemporal features of different scales through a multi-head self-attention mechanism based on multi-scale attention:

[0120]

[0121] in, Q represents the feature representation of the i-th image patch. i and K j For the query and key matrix, β j For the temporal offset of the image patch, ρ i δ is the spatiotemporal scale factor. i and γ i The dynamic adjustment factor is exp(), where exp() is the exponential function and N is the total number of image patches. Embedded vectors for image patches;

[0122] S35. Through spatiotemporal self-attention mechanism and multi-scale feature fusion, a global feature representation of each image patch is generated. The global feature representations of all image patches are weighted and summed to obtain the final fatigue feature representation.

[0123]

[0124] in, This represents the final facial fatigue characteristics. For the global features of the i-th image patch, α i C represents the weighting coefficients for the image patch. i For the detailed features of the image patch, T i λ is the scale of the time feature. i and δ i κ is the adaptive adjustment factor. i It is a comprehensive adjustment factor for overall fatigue characteristics.

[0125] In this embodiment, S4 specifically includes:

[0126] S41. Extract vehicle speed data V from the vehicle's sensor system. speed Acceleration data a acc Steering angle data θ turn And convert it into time series data D vehicle (t)=[V speed (t),a acc (t),θ turn[(t)], where t represents a time point;

[0127] S42. Represent the driver's final fatigue characteristics. As input for the fusion of vehicle multimodal data, a multimodal fusion network is constructed;

[0128] S43. Integrate the time series data and the final fatigue feature representation into a comprehensive feature representation F. fusion (t):

[0129]

[0130] Where ω1, ω2, ω3, ω4 are weighting coefficients;

[0131] S44. Input the comprehensive feature representation into the fatigue driving judgment model based on an adaptive deep generative adversarial network. The fatigue driving judgment model uses the predicted fatigue driving value... Assess fatigue status:

[0132]

[0133] Among them, W i Let be the weight of the i-th feature dimension. Let be the value of the i-th feature, b be the bias term, τ() be the activation function, γ be the adjustment coefficient, and α be the nonlinear influence factor of multimodal features on fatigue prediction. The information represents the change in each modality data over time, where ∈ is the noise factor, b is the bias term, tanh is the hyperbolic tangent activation function, n is the number of features, and m is the number of time-varying features.

[0134] S45, Predicted values ​​based on fatigue driving Driver fatigue is alerted via audible and visual alarms. If the warning threshold is reached or exceeded, the system will trigger an alarm mechanism to remind the driver to take a break.

[0135] In this embodiment, S5 specifically includes:

[0136] S51. Initialize the hyperparameters of the visual transformer model, including the image patch size P. i The number of Transformer layers L, the learning rate η, and the number of hidden units H in each layer;

[0137] S52. Define an objective function F(θ) to evaluate the performance of the visual transformer model. The objective function combines the training loss and a regularization term:

[0138]

[0139] in, For predicted values The loss function between the true value y(t) and the actual value λ r θ is the regularization coefficient. i Let be the hyperparameter of the i-th model, T be the number of training epochs, n be the number of hyperparameters, and K be the number of hyperparameters.

[0140] S53. Optimize hyperparameters using a wolf pack optimization algorithm, simulate the wolf pack foraging process, and find the optimal solution in the parameter space:

[0141]

[0142] in, Let X be the update value of the i-th hyperparameter in round t+1, A and B be scaling factors, D be the distance factor, and X be the distance factor. best and X worst These are the hyperparameter values ​​for the best and worst solutions in the current wolf pack, respectively. Let be the update value of the i-th hyperparameter in round t;

[0143] S54. In each round of optimization, the optimization effect of the hyperparameters is evaluated using the model validation set ν, and the validation error is calculated. And adjust the wolf pack's search strategy based on the error value:

[0144]

[0145] Among them, F WOA (θ) is the objective function optimized using the wolf pack optimization algorithm, γ o As an adjustment factor, α i Here, θ is the set of all hyperparameters of the model, p is the intensity factor of the penalty term, and e is the base of the natural logarithm.

[0146] S55. Repeat steps S53 to S54 until the objective function F(θ) converges to its minimum value, finally obtaining the optimized hyperparameter configuration θ. * These hyperparameters are then used for training the visual converter model and for fatigue driving detection.

[0147] In this embodiment, S6 specifically includes:

[0148] S61. Combine the optimized visual transformer network model with the optimal hyperparameter θ obtained through the wolf pack optimization algorithm. * This is applied to real-time driver facial features and multimodal data, with the input multimodal data being... Among them, V speed (t) represents the vehicle speed data, a acc (t) Acceleration data, θ turn(t) represents the steering angle data. This represents the final facial fatigue characteristics;

[0149] S62, from the input multimodal data F fusion In (t), the final facial fatigue feature representation and vehicle data are extracted, and fatigue state assessment is performed based on the weighted sum of each feature according to the model. Combining facial features and vehicle multimodal data, a comprehensive feature representation C is defined. fusion (t) is:

[0150]

[0151] Where, ω i Let be the weighting coefficient for the i-th feature. Let ζ be the i-th feature after fusion, ζ be the dynamic weighting factor, Attention() be the weighted feature fusion operation based on the attention mechanism, and n be the number of features;

[0152] S63. Input the comprehensive features into the output layer of the neural network to calculate the driver's fatigue state prediction value, and smooth it by introducing a sliding window mechanism to obtain a stable fatigue state prediction value.

[0153]

[0154] in, These are the smoothed fatigue state prediction values. Let μ be the fatigue state value predicted at time k, μ be the adjustment factor, tanh be the hyperbolic tangent activation function, and T be the value of the fatigue state predicted at time k. cycle For periodic factors, Δt is the sliding window size, and U is the number of time steps;

[0155] S64. Introduce a deep reinforcement learning mechanism based on adaptive dynamic weight adjustment to dynamically adjust the weight coefficients of each modality feature and optimize the weights through continuous environmental feedback:

[0156]

[0157] Where θ is the learning rate. Let ρ be the gradient of the loss function, and ρ be the periodic adjustment factor. Let i be the new weight of the i-th feature at time t+1. Let y(t) be the current weight of the i-th feature at time t, and y(t) be the actual fatigue state. For a periodic function, T period The period length;

[0158] S65. Predicted fatigue state value based on smoothed To determine if a driver is fatigued, if... Reaching or exceeding the fatigue warning threshold will trigger the audible and visual alarm system and visual cues.

[0159] Example 1:

[0160] To verify the feasibility of this invention in practice, it was applied to a busy highway with high daily traffic volume, especially during late night and early morning hours when driver fatigue is a more serious issue. To reduce the risk of accidents caused by driver fatigue, the local traffic management department collaborated with an intelligent transportation company to install a dashcam system based on the technology of this invention. This system combines multimodal data, including driver facial images, vehicle speed, acceleration, and steering angle, collected by an onboard camera, and performs real-time analysis using the multimodal data fusion algorithm described in this invention. This allows it to effectively determine whether the driver is at risk of driver fatigue and issues warning signals during real-time monitoring.

[0161] The system is equipped with high-performance computing devices, enabling real-time analysis of each driver's fatigue state. In this embodiment, the system collected driving behavior data from multiple different drivers, including normal driving and fatigued driving states, to verify the effectiveness of the method of the present invention.

[0162] In practical applications, the vehicle-mounted camera captures real-time facial images of the driver and performs facial key point detection using a vision transformer. The system processes the facial images using dynamic image segmentation technology, dynamically adjusting the size and position of image blocks based on key point information in areas such as the eyes, mouth, and eyebrows, to accurately capture driver fatigue characteristics, such as closed eyes and yawning.

[0163] The system performs multimodal fusion of facial image data with vehicle speed, acceleration, and steering angle data. Using the fusion algorithm of this invention, it comprehensively assesses whether the driver is at risk of fatigue. When the system detects a risk of driver fatigue, it immediately alerts the driver via an audio-visual warning device to ensure the driver takes timely rest measures.

[0164] To verify the effectiveness of this invention, the system collected fatigue driving behavior data from three drivers at different time periods. First, the vehicle-mounted camera captured real-time facial images of the drivers and automatically located and marked key facial features using a deep learning-based facial landmark detection model. The image data was processed by a visual transformer to extract the driver's fatigue features. Then, the system simultaneously collected dynamic data such as vehicle speed, acceleration, and steering angle, and fused this data with the facial image data. Through an optimized Wolf Pack Optimization (WOA) algorithm, the system can quickly adjust the hyperparameters of the neural network model, thereby improving training efficiency and detection accuracy. Finally, when the system determines that the driver is at risk of fatigue driving, it immediately issues an audible and visual alarm. The system can promptly determine the fatigue state based on vehicle speed and the driver's facial features, and provide multiple warnings if the driver does not respond effectively.

[0165] Table 1 Experimental data of the driver fatigue detection system

[0166]

[0167] Experimental data shows that the system demonstrates high accuracy and timely warning capabilities in detecting driver fatigue at different time periods. Regarding facial fatigue feature recognition accuracy, all drivers achieved over 90% accuracy, with drivers 001, 003, and 004 reaching 95%, 96%, and 94%, respectively. This indicates that the system can effectively identify driver facial fatigue features, especially during nighttime and early morning hours when driver facial changes are more pronounced. The system exhibits good adaptability, validating the effectiveness of the dynamic image segmentation technology and the visual transformer.

[0168] Regarding the impact of changes in vehicle speed, acceleration, and steering angle on fatigue detection, the system can accurately process data from different drivers and driving conditions. Even at higher vehicle speeds (e.g., driver 001, vehicle speed 110 km / h), the system can still accurately detect fatigue. Furthermore, the system also performs stably at lower vehicle speeds, demonstrating that multimodal data fusion can improve the comprehensiveness and robustness of detection.

[0169] Regarding the warning trigger time, all drivers' warning trigger times were within 5 seconds, with driver 003's trigger time being 2 seconds. This indicates that the system can quickly identify fatigue and provide timely warnings, improving traffic safety. In terms of warning accuracy, all drivers maintained a warning accuracy rate above 96%, with the highest reaching 100%, demonstrating the system's stability under different driving environments.

[0170] In summary, the fatigue driving detection system of this invention achieves high precision, high real-time performance, and high robustness under various driving conditions. Whether driving at high speeds, low speeds, or at different times of day, the system can accurately determine the driver's fatigue state and provide timely reminders, thereby effectively improving traffic safety.

[0171] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A driving recorder fatigue driving detection and reminding method based on multi-modal data fusion, characterized in that, Includes the following steps: S1. Collect facial image data of the driver through the vehicle camera, perform preprocessing, perform face detection and region localization, and obtain the facial feature region of the driver. S2. Dynamic image segmentation technology is adopted to dynamically adjust the size and position of image blocks according to different features of the driver's facial area, and to segment the facial image. S3. Use a visual transformer to extract features from the segmented facial images to identify early signs of driver fatigue. S4. Acquire multimodal data of the vehicle and fuse the multimodal data of the vehicle with facial features to form a fatigue driving judgment model; S5. Optimize the network structure and training hyperparameters of the visual transformer using the wolf pack optimization algorithm; S6. Based on the optimized visual transformer, the driver's fatigue state is analyzed in real time, and the driver's fatigue level is judged by combining facial features and multimodal data from multiple vehicles. S7. When the system detects that the driver is fatigued, it will issue an alarm with sound and light to remind the driver to take a rest. S6 specifically includes: S61、the optimized visual transformer network model and the optimal hyperparameters obtained through the wolf swarm optimization algorithm is applied to real-time driver facial features and multi-modal data, the input multi-modal data is , wherein, is vehicle speed data, is acceleration data, is steering angle data, is the final facial fatigue feature representation; S62、from the input multi-modal data In the final facial fatigue feature representation and vehicle data are extracted and a fatigue state assessment is made according to a weighted sum of the individual features according to a model, combining facial features and vehicle multi-modal data to define a comprehensive feature representation is: ； wherein, is a weighting coefficient of the th feature, is a fused th feature, is a dynamic weighting factor, is a weighting feature fusion operation based on an attention mechanism, is a number of features; S63, input the comprehensive features to the output layer of the neural network, calculate the fatigue state prediction value of the driver, and smooth the fatigue state prediction value by introducing a sliding window mechanism to obtain a stable fatigue state prediction value : ； wherein, is the smoothed fatigue state prediction, is the predicted fatigue state value at time k, is the adjustment factor, is the hyperbolic tangent activation function, is the periodicity factor, is the size of the sliding window, is the number of time steps; S64. Introduce a deep reinforcement learning mechanism based on adaptive dynamic weight adjustment to dynamically adjust the weight coefficients of each modality feature and optimize the weights through continuous environmental feedback: ； wherein, is a learning rate, is a loss function gradient, is a periodicity adjustment factor, is a new weight of the i-th feature at time t+1, is a current weight of the i-th feature at time t, is a true fatigue state, is a periodic function, is a period length; S65, predicting the fatigue state based on the smoothed fatigue state , determining whether the driver is in a fatigue state, if reaching or exceeding the fatigue warning threshold, triggering the audible and visual warning system.

2. The method for fatigue driving detection and reminding of the vehicle recorders based on multi-modal data fusion according to claim 1, characterized in that, S2 specifically includes: S21. Extract specific facial regions from the driver's facial image. The facial regions include the eye, mouth, and eyebrow regions. Using a deep learning-based facial key point detection model, automatically locate and mark the positions of each key point on the face to form the outline of the facial region and accurately determine the boundaries of the facial region. S22. An adaptive block-segmentation method based on facial key points is used to process facial images. Combining facial muscle movement patterns and spatial features, the size and position of image blocks in each facial region are dynamically adjusted. The local spatial distribution of each facial region is calculated through the positions of facial key points and the geometric relationships between adjacent key points. Then, the size and shape of the image blocks are dynamically determined according to the movement patterns and texture characteristics of each region. ； wherein, is a size of the th image block, is a position of the th facial landmark, is a Euclidean distance between the th landmark and the th landmark, is a texture complexity of the facial region, is a motion pattern of the facial region, , , , is a coefficient, , , is a weight adjustment factor, is a dynamic texture change rate of the facial region, is a logarithmic function; S23. Image patch sliding is performed using the temporal dimension features of the facial region. The step size and shape of each sliding window dynamically change according to the movement frequency of the facial region and the rate of expression changes. ； ； wherein, and respectively represent the relative distances of the facial landmarks and on the axes, and , and are the base adjustment factors of the sliding step, and are the time-dimension dynamic expansion factors, and respectively represent the variation intensities of the facial motion patterns in and directions, and are the time variation rates of the facial feature variations, and respectively represent the offset amounts of the sliding window in the x and y directions in the image space during the image processing process; S24. Enhance the extraction of facial fatigue features by employing a fusion model that combines dynamic adaptive local correlation tensors and hierarchical time-scale adjustment: ； wherein, is the enhanced feature of the th image patch, and are the local feature vectors of the image patches and respectively, is the kernel function width, is the feature difference between image patches, is the feature weighting coefficient of the image patch , is the local dynamic adjustment factor, is the dynamic weight of each image patch, is the expansion coefficient related to the time scale of facial features, is the decay factor of the time dimension, is the rate of time change, is the exponential function, is the total number of image patches, is the base of the natural logarithm; S25. By weighted summation of the enhanced features of the image patches, a global fatigue feature representation is generated: ； wherein, is a global fatigue feature representation, is an enhanced feature for the th image patch, is an attention weight for the image patch, is a local feature complexity for the image patch, is a temporal scale feature for the image patch, is a global feature enhancement factor, , , is an adaptive adjustment factor. 3.The method for fatigue driving detection and reminding based on multi-modal data fusion of automobile data recorder according to claim 1, characterized in that, S3 specifically includes: S31, the segmented face image blocks are input into the visual transformer model, each image block wherein and are the height and width of the image block, respectively, is the number of channels, each image block is converted into a one-dimensional vector wherein is the feature embedding dimension, the embedding vector represents the high-dimensional features of each image block; S32、for each image block , a nonlinear adaptive position encoding mechanism for the face region is introduced, which combines the spatial position and dynamic changes of the expression of the face region, and the position encoding not only depends on the geometric structure of the facial expression, but also includes the time information of each image block: ； wherein, is the face region size of the th image block, is the dynamic texture variation rate of the region where the image block is located, is the region frequency feature, is the temporal feature of the expression, is the scaling factor, is the time dimension index, is the logarithmic function, and sin is the sine function. S33, embedding the image blocks with position encoding into the vector input to the self-attention mechanism module of the visual transformer for feature extraction, calculating the self-attention feature representation of each image block : ； wherein, is a query matrix, is a key matrix, is a facial-expression-based bias term, is a dimension of the key, and is an adjustment factor, is a motion pattern of the facial region, is a time scale of the expression change, is an expansion coefficient, is a normalization function, is an embedding vector of the i-th image patch; S34. Introduce a spatiotemporal fusion module into a multi-layer self-attention network to fuse spatiotemporal features of different scales through a multi-head self-attention mechanism based on multi-scale attention: ； wherein, is a feature representation of the th image patch, and are a query and a key matrix, is a temporal bias of the image patch, is a spatio-temporal scale factor, and are dynamic adjustment factors, is an exponential function, is a total number of image patches, is an image patch embedding vector; S35, generate a global feature representation of each image block through a spatio-temporal self-attention mechanism and multi-scale feature fusion, and obtain a final fatigue feature representation by weighted summing the global feature representations of all image blocks : ； wherein, is the final facial fatigue feature representation, is the global feature of the th image patch, is the weighting coefficient of the image patch, is the detail feature of the image patch, is the scale of the temporal feature, and is the adaptive adjustment factor, is the comprehensive adjustment factor of the overall fatigue feature.

4. The method for fatigue driving detection and reminding based on multi-modal data fusion of the automobile data recorder according to claim 1, characterized in that, S4 specifically includes: S41, extracting vehicle speed data from the sensor system of the vehicle acceleration data steering angle data and converting into time series data wherein denotes a point in time; S42, representing the final fatigue feature of the driver As an input of the vehicle multi-modal data fusion, a multi-modal fusion network is constructed; S43, fusing the time series data and the final fatigue feature representation into a comprehensive feature representation : ； wherein are weighting coefficients; S44, input the comprehensive feature representation into the fatigue driving judgment model based on the adaptive deep generative adversarial network, the fatigue driving judgment model outputs a prediction value of fatigue driving fatigue state judgment is performed: ； wherein, is the weight of the jth feature dimension, is the value of the jth feature, is the bias term, is the activation function, is the adjustment coefficient, is the nonlinear influence factor of the multi-modal feature pair on fatigue prediction, is the change information of each modality data in the time dimension, is the noise factor, is the bias term, is the hyperbolic tangent activation function, is the number of features, is the number of time-varying features;​​ S45, Predicted values ​​based on fatigue driving The driver is alerted to fatigue through audible and visual alarms. If the warning threshold is reached or exceeded, the system will trigger an alarm mechanism to remind the driver to take a break.

5. The method for fatigue driving detection and alerting based on multimodal data fusion in a vehicle recorder according to claim 1, characterized in that, S5 specifically includes: S51. Initialize the hyperparameters of the visual transformer model, including the image patch size. Transformer layer number Learning rate and the number of hidden units in each layer ; S52. Define the objective function The objective function, used to evaluate the performance of the visual transformer model, combines the training loss and a regularization term: ； in, For predicted values Compared with the true value The loss function between The regularization coefficient is . For the first Each model hyperparameter, For the number of training rounds, The number of hyperparameters, For the number of training rounds, The number of hyperparameters; S53. Optimize hyperparameters using a wolf pack optimization algorithm, simulate the wolf pack foraging process, and find the optimal solution in the parameter space: ； in, For the first The hyperparameter at the th ... The updated value of the wheel, and Scaling factor As a distance factor, and These are the hyperparameter values ​​for the best and worst solutions in the current wolf pack, respectively. For the first The hyperparameter at the th ... The updated value of the wheel; S54. In each round of optimization, the model validation set is used. Evaluate the optimization effect of hyperparameters and calculate the validation error. And adjust the wolf pack's search strategy based on the error value: ； in, It is the objective function optimized using the wolf pack optimization algorithm. To adjust the factor, For each hyperparameter, the adjustment coefficients are... The set of all hyperparameters of the model. The intensity factor of the penalty term. is the base of the natural logarithm; S55. Repeat steps S53 to S54 until the objective function is achieved. The hyperparameters converge to the minimum, ultimately yielding the optimized configuration. These hyperparameters are then used for training the visual converter model and for fatigue driving detection.

Images