Automobile data recorder fatigue driving detection and reminding method based on multi-modal data fusion

Abstract

The invention discloses an automobile data recorder fatigue driving detection and reminding method based on multi-modal data fusion, and the method comprises the following steps: S1, collecting and preprocessing a face image through a vehicle-mounted camera, and carrying out the face detection and region positioning; s2, dynamically adjusting the size and the position of an image block by adopting a dynamic image blocking technology; s3, performing facial image feature extraction by using a visual converter; s4, acquiring vehicle multi-modal data and fusing the vehicle multi-modal data with the facial features; s5, optimizing a visual converter network structure and training hyper-parameters through a wolf pack optimization algorithm; s6, performing fatigue state analysis based on the optimized visual converter; and S7, when the fatigue state is detected, reminding the driver to have a rest through sound-light alarm. The fatigue driving detection and reminding method of the automobile data recorder is high in precision and high in real-time performance through a deep learning algorithm of multi-modal data fusion and optimization, and the recognition accuracy of fatigue driving and the robustness of the system are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of traffic safety, and particularly to a method for detecting and reminding fatigue driving of a driving recorder based on multi-modal data fusion. Background Art

[0002] With the rapid development of intelligent technologies, intelligent transportation systems have become an important part of modern traffic management and safety guarantee. In this context, fatigue driving, as one of the main causes of traffic accidents, has attracted extensive attention and research. Fatigue driving can lead to a significant decline in a driver's attention, judgment, and reaction speed, thus increasing the risk of traffic accidents. To address this issue, many scholars and enterprises have proposed methods for detecting and warning fatigue driving based on in-vehicle devices.

[0003] Traditional fatigue driving detection methods mainly rely on vehicle status data (such as vehicle speed, acceleration, etc.) and the physiological and behavioral characteristics of the driver. Dynamic characteristics such as vehicle speed, braking frequency, and steering wheel angle are the main bases for early fatigue driving detection. Although these methods can monitor the driver's state to a certain extent, their accuracy and real-time performance are relatively low, especially in complex road conditions and when the driver's performance is not obvious, and the detection effect is greatly limited.

[0004] With the rise of deep learning technologies, fatigue driving detection methods based on computer vision and facial features have gradually been widely applied. Using a camera to obtain the driver's facial image and analyzing the changes in areas such as the eyes and mouth through facial feature extraction algorithms, such as eye closure, yawning, and blink frequency, has become a popular research direction for fatigue driving detection in recent years. Facial recognition technology is based on methods such as deep neural networks (DNN) and convolutional neural networks (CNN), and achieves precise recognition of the driver's fatigue state through large-scale data training.

[0005] However, the existing computer vision-based methods still have some obvious deficiencies. First, traditional image block techniques use image blocks of a fixed size to process the driver's facial area, but this has significant limitations in dealing with facial expression changes. For example, the size and shape of areas such as the driver's eyes and mouth change with facial expressions in the image, and fixed-size image blocks cannot accurately capture these subtle changes, resulting in a low extraction accuracy of fatigue features. In addition, traditional image processing techniques mostly rely on preset algorithms and rules and cannot adaptively adjust the block size or position, thus unable to efficiently extract the driver's fatigue features.

[0006] Secondly, fatigue driving detection methods based on facial images often rely on single image data while ignoring other dynamic vehicle data such as vehicle speed, acceleration, steering angle, etc. These factors can affect the driver's fatigue state to a certain extent. For example, in the case of high-speed driving or long-term driving, the driver's fatigue state may change due to the vehicle's motion state. Traditional methods usually only rely on image data to judge the degree of fatigue, resulting in the inability to comprehensively consider multiple factors, thereby affecting the accuracy and robustness of detection.

[0007] In addition, there are certain defects in the optimization algorithms of traditional fatigue driving detection methods. Currently, many methods use conventional optimization algorithms (such as gradient descent method, genetic algorithm, etc.) for network training and model tuning, but these methods are prone to falling into local optimal solutions, resulting in the inability of the model to be globally optimized, thus affecting the final detection effect. Especially in the face of complex driving scenarios, the existing optimization methods often cannot effectively solve problems such as low training efficiency and low accuracy, resulting in poor robustness and stability of the fatigue driving detection system.

[0008] Therefore, how to provide a fatigue driving detection and reminder method for a driving recorder based on multi-modal data fusion is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0009] An object of the present invention is to propose a fatigue driving detection and reminder method for a driving recorder based on multi-modal data fusion. The present invention makes full use of computer vision technology, deep learning algorithms, nature-inspired optimization algorithms, and multi-modal data fusion technology, and details the extraction of driver facial features through dynamic image block technology combined with a vision transformer (ViT), and optimizes the training process of the neural network model through the wolf pack optimization algorithm (WOA), so as to achieve high-precision and high-robustness fatigue driving detection and real-time warning. This method has the advantages of high detection accuracy, strong real-time performance, good system stability, and the ability to comprehensively consider various driver behaviors and vehicle dynamic information.

[0010] According to the fatigue driving detection and reminder method for a driving recorder based on multi-modal data fusion of the embodiments of the present invention, the following steps are included:

[0011] S1. Collect the facial image data of the driver through an in-vehicle camera, perform preprocessing, perform face detection and region localization, and obtain the facial feature region of the driver;

[0012] S2. Adopt dynamic image block technology, dynamically adjust the size and position of the image blocks according to different features of the driver's facial region, and perform block processing on the facial image;

[0013] S3. Use a vision transformer to extract features from the segmented facial images and identify early signals of drowsy driving;

[0014] S4. Obtain the multi-modal data of the vehicle, and fuse the multi-modal data of the vehicle with the facial features to form a drowsy driving judgment model;

[0015] S5. Optimize the network structure and training hyperparameters of the vision transformer through the wolf pack optimization algorithm;

[0016] S6. Based on the optimized vision transformer, perform real-time analysis on the fatigue state of the driver, and comprehensively judge the fatigue degree of the driver based on the facial features and the multi-modal data of multiple vehicles;

[0017] S7. When it is detected that the driver is in a fatigued state, the system issues a warning through an audible and visual alarm to remind the driver to take a break.

[0018] Optionally, the S2 specifically includes:

[0019] S21. Extract specific facial regions from the driver's facial image. The facial regions include the eye region, the mouth region, and the eyebrow region. Use a deep learning-based facial key point detection model to automatically locate and mark the positions of each facial key point, form the contour of the facial region, and accurately determine the boundary of the facial region;

[0020] S22. Process the facial image using an adaptive block method based on facial key points. Combine the facial muscle movement pattern and spatial features to dynamically adjust the size and position of the image blocks in each facial region. Calculate the local spatial distribution of each facial region through the position of the facial key points and the geometric relationship between adjacent key points, and then dynamically determine the size and shape of the image blocks according to the movement pattern of each region and the texture characteristics of the facial region:

[0021]

[0022] where, P i is the size of the i-th image block, K i is the position of the i-th facial key point, D ij is the Euclidean distance between the i-th key point and the j-th key point, T i is the texture complexity of the facial region, M i is the movement pattern of the facial region, α i , β i , γ i , θ i are coefficients, γ, δ, ζ are weight adjustment factors, ρ i is the dynamic texture change rate of the facial region, and log2 is the logarithmic function;

[0023] S23. Slide the image patches using the temporal dimension features of the facial region, and the stride and shape of each sliding window change dynamically according to the movement frequency of the facial region and the rate of expression change:

[0024]

[0025] where D x (K x , K y ) and D y (K x , K y ) respectively represent the relative distances of the facial key points K x and K y on the x and y axes, γ1 and γ2 are the basic adjustment factors of the sliding stride, λ1 and λ2 are the temporal dimension dynamic expansion factors, M x and M y respectively represent the change intensities of the facial movement pattern in the x and y directions, T x and T y are the temporal change rates of facial feature changes, and Δx and Δy respectively represent the offsets of the sliding window in the image space along the x and y directions during the image processing;

[0026] S24. Enhance the extraction of facial fatigue features by adopting a fusion model of dynamic adaptive local correlation tensors and hierarchical time-scale adjustment:

[0027]

[0028] where A i is the enhanced feature of the i-th image patch, I(P i ) and I(P j ) respectively represent the local feature vectors of the image patches P i and P j , σ is the kernel function width, ||I(P j ) - I(P i )|| 2 is the feature difference between image patches, A j is the feature weighting coefficient of the image patch P j , φ i is the local dynamic adjustment factor, ω i is the dynamic weight of each image patch, λ i is the expansion coefficient related to the time scale of facial features, μ i is the decay factor of the temporal dimension, T i is the temporal change rate, exp() is the exponential function, N is the total number of image patches, and e is the base of the natural logarithm;

[0029] S25. Generate a global fatigue feature representation by weighted summation of the enhanced features of the image patches:

[0030]

[0031] where A global is the global fatigue feature representation, A i is the enhanced feature of the i-th image patch, α i is the attention weight of the image patch, C i is the local feature complexity of the image patch, T i is the time-scale feature of the image patch, λ3 is the global feature enhancement factor, and δ1, μ, ν are adaptive adjustment factors.

[0032] Optionally, the S3 specifically includes:

[0033] S31. Input the segmented facial image patch P i into the vision transformer model. Each image patch P i ∈ R H×W×C , where H and W are the height and width of the image patch respectively, C is the number of channels, and each image patch is converted into a one-dimensional vector V i ∈ R D , where D is the feature embedding dimension, and the embedding vector V i represents the high-dimensional feature of each image patch;

[0034] S32. For each image patch P i , introduce a non-linear adaptive position encoding mechanism for the facial region, combine the spatial position and expression dynamic changes of the facial region, and the position encoding depends not only on the geometric structure of the facial expression but also includes the time information of each image patch:

[0035]

[0036] where K i is the facial region size of the i-th image patch, ρ i is the dynamic texture change rate of the region where the image patch is located, f i is the regional frequency feature, is the time feature of the expression, η i is the scaling factor, α is the time dimension exponent, log2 is the logarithmic function, and sin is the sine function;

[0037] S33. Input the image patch embedding vector with position encoding into the self-attention mechanism module of the vision transformer for feature extraction, and calculate the self-attention feature representation of each image patch

[0038]

[0039] Among them, Q i is the query matrix, K i is the key matrix, B i is the bias term based on facial expressions, d k is the dimension of the key, λ i and μ i are adjustment factors, M i is the motion pattern of the facial region, T i is the time scale of expression change, τ i is the expansion coefficient, softmax() is the normalization function, V i is the embedding vector of the i-th image patch;

[0040] S34. Introduce a spatio-temporal fusion module in the multi-layer self-attention network, and fuse spatio-temporal features of different scales through a multi-head self-attention mechanism based on multi-scale attention:

[0041]

[0042] Among them, is the feature representation of the i-th image patch, Q i and K j are the query and key matrices, β j is the time bias of the image patch, ρ i is the spatio-temporal scale factor, δ i and γ i are dynamic adjustment factors, exp() is the exponential function, N is the total number of image patches, is the image patch embedding vector;

[0043] S35. Generate the global feature representation of each image patch through the spatio-temporal self-attention mechanism and multi-scale feature fusion, and perform weighted summation on the global feature representations of all image patches to obtain the final fatigue feature representation

[0044]

[0045] Among them, is the final facial fatigue feature representation, is the global feature of the i-th image patch, α i is the weighting coefficient of the image patch, C i is the detailed feature of the image patch, T i is the scale of the time feature, λ i and δ i are adaptive adjustment factors, κ i is the comprehensive adjustment factor of the overall fatigue feature.

[0046] Optionally, the S4 specifically includes:

[0047] S41. Extract the vehicle speed data V from the vehicle's sensor system speed , the acceleration data a acc , and the steering angle data θ turn , and convert them into time series data D vehicle (t) = [V speed (t), a acc (t), θ turn (t)], where t represents the time point;

[0048] S42. Use the driver's final fatigue characteristic representation as the input for fusing with the vehicle's multimodal data, and construct a multimodal fusion network;

[0049] S43. Fuse the time series data and the final fatigue characteristic representation into a comprehensive characteristic representation F fusion (t):

[0050]

[0051] where ω1, ω2, ω3, ω4 are weighting coefficients;

[0052] S44. Input the comprehensive characteristic representation into the fatigue driving judgment model based on the adaptive deep generative adversarial network. The fatigue driving judgment model judges the fatigue state through the predicted value of fatigue driving :

[0053]

[0054] where, W i is the weight of the i-th feature dimension, is the value of the i-th feature, b is the bias term, τ() is the activation function, γ is the adjustment coefficient, α is the non-linear influence factor of multimodal features on fatigue prediction, is the change information of each modal data in the time dimension, ∈ is the noise factor, b is the bias term, tanh is the hyperbolic tangent activation function, n is the number of features, and m is the number of time-varying features;

[0055] S45. Based on the predicted value of fatigue driving give a fatigue warning to the driver through an audible and visual alarm. If reaches or exceeds the warning threshold, the system will trigger an alarm mechanism to remind the driver to rest.

[0056] Optionally, the specific content of S5 includes:

[0057] S51. Initialize the hyperparameters of the vision transformer model, including the image patch size P i, the number of Transformer layers L, the learning rate η, and the number of hidden units H in each layer;

[0058] S52. Define the objective function F(θ) for evaluating the performance of the vision transformer model. The objective function combines the training loss and the regularization term:

[0059]

[0060] where, is the loss function between the predicted value and the true value y(t), λ r is the regularization coefficient, θ i is the i-th model hyperparameter, T is the number of training epochs, n is the number of hyperparameters, T is the number of training epochs, and K is the number of hyperparameters;

[0061] S53. Optimize the hyperparameters through the wolf pack optimization algorithm, simulate the foraging process of the wolf pack, and find the optimal solution in the parameter space:

[0062]

[0063] where, is the updated value of the i-th hyperparameter in the (t + 1)-th round, A and B are scaling factors, D is the distance factor, X best and X worst are the hyperparameter values of the optimal solution and the worst solution in the current wolf pack respectively, is the updated value of the i-th hyperparameter in the t-th round;

[0064] S54. In each round of optimization, evaluate the optimization effect of the hyperparameters through the model validation set calculate the validation error and adjust the search strategy of the wolf pack according to the error value:

[0065]

[0066] where, F WOA (θ) is the objective function optimized by the wolf pack optimization algorithm, γ o is the adjustment factor, α i is the adjustment coefficient related to each hyperparameter, θ is the set of all hyperparameters of the model, p is the intensity factor of the penalty term, and e is the base of the natural logarithm;

[0067] S55. Repeat steps S53 to S54 until the objective function F(θ) converges to the minimum value, and finally obtain the optimized hyperparameter configuration θ * , and use these hyperparameters for the training of the vision transformer model and fatigue driving detection.

[0068] Optionally, S6 specifically includes:

[0069] S61. Apply the optimized vision transformer network model and the optimal hyperparameters θ obtained by the wolf pack optimization algorithm * to real-time driver facial features and multimodal data. The input multimodal data is where V speed (t) is vehicle speed data, a acc (t) is acceleration data, and θ turn (t) is the steering angle data, which is the final facial fatigue feature representation;

[0070] S62. Extract the final facial fatigue feature representation and vehicle data from the input multimodal data F fusion (t), and evaluate the fatigue state according to the weighted sum of each feature by the model. Combining facial features and vehicle multimodal data, define the comprehensive feature representation C fusion (t) as:

[0071]

[0072] where ω i is the weighting coefficient of the i-th feature, is the i-th feature after fusion, ζ is the dynamic weighting factor, Attention() is the weighted feature fusion operation based on the attention mechanism, and n is the number of features;

[0073] S63. Input the comprehensive feature into the output layer of the neural network, calculate the predicted value of the driver's fatigue state, and perform smoothing processing by introducing a sliding window mechanism to obtain a stable predicted value of the fatigue state

[0074]

[0075] where is the smoothed predicted value of the fatigue state, is the predicted fatigue state value at time k, μ is the adjustment factor, tanh is the hyperbolic tangent activation function, T cycle is the periodic factor, Δt is the sliding window size, and U is the number of time steps;

[0076] S64. Introduce a deep reinforcement learning mechanism based on adaptive dynamic weight adjustment to dynamically adjust the weight coefficients of each modal feature, and optimize the weights through continuous environmental feedback:

[0077]

[0078] where θ is the learning rate, is the gradient of the loss function, and ρ is the periodic adjustment factor. is the new weight of the i-th feature at time t+1. is the current weight of the i-th feature at time t, and y(t) is the true fatigue state. is a periodic function, and T period is the period length;

[0079] S65. Based on the smoothed predicted value of the fatigue state judge whether the driver is in a fatigued state. If reaches or exceeds the fatigue warning threshold, the audible and visual alarm system and visual prompt will be triggered.

[0080] The beneficial effects of the present invention are:

[0081] By innovatively combining multi-modal data fusion with deep vision algorithms, the present invention solves several key problems in traditional fatigue driving detection methods. First, the dynamic image block technology is adopted, which can dynamically adjust the size and position of the image blocks according to the key points on the driver's face. This innovation significantly improves the accuracy of driver face feature extraction, especially when facing subtle fatigue manifestations such as eye closure and yawning, and can capture changes more precisely. Compared with the traditional fixed image block processing method, the image block method of the present invention has a higher recognition rate and robustness when dealing with complex facial expressions.

[0082] Secondly, traditional fatigue driving detection methods mostly rely on single facial image data and fail to make full use of vehicle dynamic information. To solve this problem, the present invention introduces multi-modal data such as vehicle speed, acceleration, and steering angle, and combines these data with facial image data through a fusion algorithm to form a comprehensive judgment basis. Multi-modal data fusion can comprehensively reflect the driver's fatigue state, enabling the system to maintain high accuracy and stability in various driving environments. In contrast, existing methods often rely only on image data, resulting in limited detection accuracy and reliability.

[0083] In addition, the present invention uses the wolf optimization algorithm (WOA) to optimize the training process of the neural network, solving the problem of being easily trapped in local optimal solutions in traditional training methods. By optimizing the hyperparameters of the network, the present invention improves the training efficiency and final detection accuracy of the model, avoiding inefficiencies in the training process, especially when facing large-scale and diverse data, and the optimization effect is particularly significant. This optimization not only improves the performance of the system but also ensures its real-time performance and stability in practical applications.

[0084] In summary, by combining advanced computer vision, deep learning, and nature-inspired optimization algorithms, the present invention significantly improves the accuracy, robustness, and real-time performance of the fatigue driving detection system, providing a more intelligent and precise solution for traffic safety. At the same time, the introduction of multi-modal data fusion technology also enables the present invention to more comprehensively evaluate the fatigue state of the driver, thereby reducing the risk of traffic accidents caused by fatigue driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] The accompanying drawings are used to provide a further understanding of the present invention and form a part of the specification. They are used together with the embodiments of the present invention to explain the present invention and do not constitute a limitation to the present invention. In the drawings:

[0086] Figure 1 is a flowchart of a fatigue driving detection and reminder method for a driving recorder based on multi-modal data fusion proposed by the present invention;

[0087] Figure 2 is a schematic diagram of multi-modal data fusion of a fatigue driving detection and reminder method for a driving recorder based on multi-modal data fusion proposed by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0088] Now, the present invention will be further described in detail with reference to the accompanying drawings. These drawings are all simplified schematic diagrams, only showing the basic structure of the present invention in a schematic manner, so they only show the components related to the present invention.

[0089] Referring to Figure 1 and Figure 2 , a fatigue driving detection and reminder method for a driving recorder based on multi-modal data fusion includes the following steps:

[0090] S1. Collect facial image data of the driver through an in-vehicle camera, perform preprocessing, perform face detection and region localization, and obtain the facial feature region of the driver;

[0091] S2. Adopt the dynamic image block technology, dynamically adjust the size and position of the image blocks according to different features of the driver's facial region, and perform block processing on the facial image;

[0092] S3. Use a vision transformer to extract features from the block-processed facial image and identify early signals of fatigue driving;

[0093] S4. Obtain multi-modal data of the vehicle, and fuse the multi-modal data of the vehicle with the facial features to form a fatigue driving judgment model;

[0094] S5. Optimize the network structure and training hyperparameters of the vision transformer through the wolf pack optimization algorithm;

[0095] S6. Based on the optimized vision transformer, analyze the driver's fatigue state in real time, and comprehensively judge the driver's fatigue level by combining facial features and multi-modal data of multiple vehicles;

[0096] S7. When it is detected that the driver is in a fatigued state, the system issues a warning through an audible and visual alarm to remind the driver to take a rest.

[0097] In this embodiment, the specific steps of S2 are as follows:

[0098] S21. Extract a specific facial area from the driver's facial image. The facial area includes the eye area, mouth area, and eyebrow area. Use a deep learning-based facial key point detection model to automatically locate and mark the positions of each facial key point, form the contour of the facial area, and accurately determine the boundary of the facial area;

[0099] S22. Process the facial image using an adaptive block method based on facial key points. Combine the facial muscle movement pattern and spatial features to dynamically adjust the size and position of the image blocks in each facial area. Calculate the local spatial distribution of each facial area through the geometric relationship between the positions of facial key points and adjacent key points. Then, dynamically determine the size and shape of the image blocks according to the movement pattern of each area and the texture characteristics of the facial area:

[0100]

[0101] Among them, P i is the size of the i-th image block, K i is the position of the i-th facial key point, D ij is the Euclidean distance between the i-th key point and the j-th key point, T i is the texture complexity of the facial area, M i is the movement pattern of the facial area, α i , β i , γ i , θ i are coefficients, γ, δ, ζ are weight adjustment factors, ρ i is the dynamic texture change rate of the facial area, and log2 is the logarithmic function;

[0102] S23. Use the time dimension features of the facial area to slide the image blocks. The step size and shape of each sliding window change dynamically according to the movement frequency of the facial area and the rate of expression change:

[0103]

[0104] Among them, D x (K x , K y ) and Dy (K x , K y ) respectively represent the relative distances of facial key points K x and K y on the x and y axes. γ1 and γ2 are the basic adjustment factors for the sliding step, λ1 and λ2 are the dynamic expansion factors in the time dimension, M x and M y respectively represent the change intensities of the facial motion pattern in the x and y directions, T x and T y are the time change rates of the facial feature changes, and Δx and Δy respectively represent the offsets of the sliding window in the image space along the x and y directions during the image processing;

[0105] S24. By adopting a fusion model of dynamic adaptive local correlation tensors and hierarchical time-scale adjustment, enhance the extraction of facial fatigue features:

[0106]

[0107] Among them, A i is the enhanced feature of the i-th image block, I(P i ) and I(P j ) respectively represent the local feature vectors of the image blocks P i and P j , σ is the kernel function width, ||I(P j ) - I(P i )|| 2 is the feature difference between the image blocks, A j is the feature weighting coefficient of the image block P j , φ i is the local dynamic adjustment factor, ω i is the dynamic weight of each image block, λ i is the expansion coefficient related to the time scale of the facial features, μ i is the decay factor in the time dimension, T i is the time change rate, exp() is the exponential function, N is the total number of image blocks, and e is the base of the natural logarithm;

[0108] S25. By performing weighted summation on the enhanced features of the image blocks, generate the global fatigue feature representation:

[0109]

[0110] Among them, A global is the global fatigue feature representation, A i is the enhanced feature of the i-th image block, α i is the attention weight of the image block, C iis the local feature complexity of the image block, T i is the temporal scale feature of the image block, λ3 is the global feature enhancement factor, and δ1, μ, ν are the adaptive adjustment factors.

[0111] In this embodiment, S3 specifically includes:

[0112] S31. Input the segmented facial image block P i into the vision transformer model. Each image block P i ∈R H×W×C , where H and W are the height and width of the image block respectively, C is the number of channels, and each image block is converted into a one-dimensional vector V i ∈R D , where D is the feature embedding dimension, and the embedding vector V i represents the high-dimensional features of each image block;

[0113] S32. For each image block P i , introduce the non-linear adaptive position encoding mechanism of the facial region, combine the spatial position and expression dynamic changes of the facial region, and the position encoding not only depends on the geometric structure of the facial expression, but also includes the temporal information of each image block:

[0114]

[0115] where, K i is the facial region size of the i-th image block, ρ i is the dynamic texture change rate of the region where the image block is located, f i is the regional frequency feature, is the temporal feature of the expression, η i is the scaling factor, α is the temporal dimension exponent, log2 is the logarithmic function, and sin is the sine function;

[0116] S33. Input the image block embedding vector with position encoding into the self-attention mechanism module of the vision transformer for feature extraction, and calculate the self-attention feature representation of each image block

[0117]

[0118] where, Q i is the query matrix, K i is the key matrix, B i is the bias term based on the facial expression, d k is the dimension of the key, λ i and μ i are the adjustment factors, M i is the motion pattern of the facial region, T iis the time scale of the facial expression change, τ i is the expansion coefficient, softmax() is the normalization function, V i is the embedding vector of the i-th image patch;

[0119] S34. Introduce a spatio-temporal fusion module in the multi-layer self-attention network, and fuse spatio-temporal features of different scales through a multi-head self-attention mechanism based on multi-scale attention:

[0120]

[0121] Among them, is the feature representation of the i-th image patch, Q i and K j are the query and key matrices, β j is the time bias of the image patch, ρ i is the spatio-temporal scale factor, δ i and γ i are the dynamic adjustment factors, exp() is the exponential function, N is the total number of image patches, is the image patch embedding vector;

[0122] S35. Generate the global feature representation of each image patch through the spatio-temporal self-attention mechanism and multi-scale feature fusion, and perform weighted summation on the global feature representations of all image patches to obtain the final fatigue feature representation

[0123]

[0124] Among them, is the final facial fatigue feature representation, is the global feature of the i-th image patch, α i is the weighting coefficient of the image patch, C i is the detailed feature of the image patch, T i is the scale of the time feature, λ i and δ i are the adaptive adjustment factors, κ i is the comprehensive adjustment factor of the overall fatigue feature.

[0125] In this embodiment, the specific content of S4 includes:

[0126] S41. Extract the vehicle speed data V speed , the acceleration data a acc , the steering angle data θ turn from the vehicle's sensor system, and convert them into time series data D vehicle (t) = [V speed (t), a acc (t), θ turn(t), where t represents the time point;

[0127] S42. Represent the final fatigue characteristics of the driver as the input for fusing with vehicle multimodal data, and construct a multimodal fusion network;

[0128] S43. Fuse the time series data and the final fatigue characteristic representation into a comprehensive characteristic representation F fusion (t):

[0129]

[0130] where ω1, ω2, ω3, ω4 are weighting coefficients;

[0131] S44. Input the comprehensive characteristic representation into the fatigue driving judgment model based on the adaptive deep generative adversarial network. The fatigue driving judgment model determines the fatigue state through the predicted value of fatigue driving :

[0132]

[0133] where W i is the weight of the i-th feature dimension, is the value of the i-th feature, b is the bias term, τ() is the activation function, γ is the adjustment coefficient, α is the non-linear influence factor of multimodal features on fatigue prediction, is the change information of each modal data in the time dimension, ∈ is the noise factor, b is the bias term, tanh is the hyperbolic tangent activation function, n is the number of features, and m is the number of time-varying features;

[0134] S45. Based on the predicted value of fatigue driving give a fatigue warning to the driver through an acoustic and optical alarm. If reaches or exceeds the warning threshold, the system will trigger an alarm mechanism to remind the driver to rest.

[0135] In this embodiment, the specific content of S5 includes:

[0136] S51. Initialize the hyperparameters of the vision transformer model, including the image patch size P i , the number of Transformer layers L, the learning rate η, and the number of hidden units H in each layer;

[0137] S52. Define the objective function F(θ) to evaluate the performance of the vision transformer model. The objective function combines the training loss and the regularization term:

[0138]

[0139] where, is the predicted value The loss function between and the true value y(t), λ r is the regularization coefficient, θ i is the i-th model hyperparameter, T is the number of training rounds, n is the number of hyperparameters, T is the number of training rounds, and K is the number of hyperparameters;

[0140] S53. Optimize the hyperparameters through the wolf pack optimization algorithm, simulate the foraging process of the wolf pack, and find the optimal solution in the parameter space:

[0141]

[0142] Among them, is the updated value of the i-th hyperparameter in the (t + 1)-th round, A and B are scaling factors, D is the distance factor, X best and X worst are the hyperparameter values of the optimal solution and the worst solution in the current wolf pack respectively, is the updated value of the i-th hyperparameter in the t-th round;

[0143] S54. In each round of optimization, evaluate the optimization effect of the hyperparameters through the model validation set ν, calculate the validation error and adjust the search strategy of the wolf pack according to the error value:

[0144]

[0145] Among them, F WOA (θ) is the objective function optimized by the wolf pack optimization algorithm, γ o is the adjustment factor, α i is the adjustment coefficient related to each hyperparameter, θ is the set of all hyperparameters of the model, p is the intensity factor of the penalty term, and e is the base of the natural logarithm;

[0146] S55. Repeat steps S53 to S54 until the objective function F(θ) converges to the minimum value, and finally obtain the optimized hyperparameter configuration θ * , and use these hyperparameters for the training of the vision transformer model and fatigue driving detection.

[0147] In this embodiment, the specific content of S6 includes:

[0148] S61. Apply the optimized vision transformer network model and the optimal hyperparameters θ * obtained by the wolf pack optimization algorithm to the real-time driver facial features and multimodal data. The input multimodal data is Among them, V speed (t) is the vehicle speed data, a acc (t) is the acceleration data, θ turn(t) is the steering angle data, is the final facial fatigue feature representation;

[0149] S62. From the input multi-modal data F fusion (t), extract the final facial fatigue feature representation and vehicle data, and evaluate the fatigue state according to the weighted sum of each feature of the model. Combining the facial features and vehicle multi-modal data, define the comprehensive feature representation C fusion (t) as:

[0150]

[0151] where ω i is the weighting coefficient of the i-th feature, is the i-th feature after fusion, ζ is the dynamic weighting factor, Attention() is the weighted feature fusion operation based on the attention mechanism, and n is the number of features;

[0152] S63. Input the comprehensive feature into the output layer of the neural network, calculate the predicted value of the driver's fatigue state, and perform smoothing processing by introducing a sliding window mechanism to obtain a stable predicted value of the fatigue state

[0153]

[0154] where, is the smoothed predicted value of the fatigue state, is the predicted fatigue state value at time k, μ is the adjustment factor, tanh is the hyperbolic tangent activation function, T cycle is the periodic factor, Δt is the sliding window size, and U is the number of time steps;

[0155] S64. Introduce a deep reinforcement learning mechanism based on adaptive dynamic weight adjustment to dynamically adjust the weight coefficients of each modal feature, and optimize the weights through continuous environmental feedback:

[0156]

[0157] where θ is the learning rate, is the gradient of the loss function, ρ is the periodic adjustment factor, is the new weight of the i-th feature at time t + 1, is the current weight of the i-th feature at time t, y(t) is the true fatigue state, is the periodic function, T period is the period length;

[0158] S65. Based on the smoothed predicted value of the fatigue state judge whether the driver is in a fatigue state. If Reaching or exceeding the fatigue warning threshold will trigger the audible and visual alarm system and visual cues.

[0159] Example 1:

[0160] To verify the feasibility of the present invention in implementation, the present invention is applied to a busy highway. The daily traffic flow of this highway is large, especially during late night and early morning hours when the phenomenon of fatigued driving by drivers is more serious. To reduce the accident risk caused by fatigued driving, the local traffic management department cooperates with an intelligent transportation company to install a driving recorder system based on the technology of the present invention. This system combines multi-modal data such as driver facial image data, vehicle speed, acceleration, steering angle, etc. collected by in-vehicle cameras, and performs real-time analysis through the multi-modal data fusion algorithm in the present invention, so as to effectively judge whether there is a risk of fatigued driving by the driver and issue a warning signal during real-time monitoring.

[0161] This system is equipped with high-performance computing devices and can analyze the fatigue state of each driver in real time. In this example, the system collected driving behavior data of multiple different drivers, including normal driving states and fatigued driving states, to verify the effectiveness of the method of the present invention.

[0162] In practical applications, the in-vehicle camera will collect the facial images of the driver in real time and perform facial key point detection based on the vision transformer. The system processes the facial images through dynamic image block technology, and dynamically adjusts the size and position of the image blocks according to the key point information in areas such as eyes, mouth, eyebrows, etc., to accurately capture the fatigue characteristics of the driver, such as eye closure, yawning, etc.

[0163] The system performs multi-modal fusion of the facial image data with data such as the vehicle speed, acceleration, steering angle, etc. of the vehicle, and comprehensively judges whether there is a risk of fatigued driving by the driver through the fusion algorithm of the present invention. When the system detects that there is a risk of fatigued driving by the driver, it will immediately remind through an audible and visual warning device to ensure that the driver takes rest measures in time.

[0164] To verify the effectiveness of the present invention, the system collected fatigue driving behavior data of 3 drivers at different time periods. First, the in-vehicle camera collected the driver's facial images in real time, and through a facial key point detection model based on deep learning, automatically located and marked the key points on the face. The image data was processed by a vision transformer to extract the driver's fatigue features. Then, the system simultaneously collected dynamic data such as the vehicle speed, acceleration, and steering angle, and fused these data with the facial image data. Through the optimized wolf pack optimization algorithm (WOA), the system could quickly adjust the hyperparameters of the neural network model, thereby improving the training efficiency and detection accuracy. Finally, when the system analyzed that the driver was at risk of fatigue driving, it immediately issued an audible and visual alarm. The system could timely judge the fatigue state based on the vehicle speed and the driver's facial features, and give multiple reminders when the driver did not make an effective response.

[0165] Table 1 Experimental data of the driver fatigue detection system

[0166]

[0167] The experimental data shows that the system exhibits high recognition accuracy and timely warning ability in the detection of driver fatigue states at different time periods. From the perspective of the recognition accuracy of facial fatigue features, the accuracy of all drivers exceeded 90%, among which drivers 001, 003, and 004 reached 95%, 96%, and 94% respectively. This indicates that the system can effectively identify the fatigue features on the driver's face. Especially during the night and early morning hours when the driver's face changes significantly, the system shows good adaptability, verifying the effectiveness of the dynamic image segmentation technology and the vision transformer.

[0168] Regarding the impact of changes in vehicle speed, acceleration, and steering angle on fatigue detection, the system can accurately process data under different drivers and driving conditions. Even at a relatively high vehicle speed (such as driver 001 with a vehicle speed of 110 km / h), the system can still accurately detect the fatigue state. In addition, at a lower vehicle speed, the system also performs stably, proving that multi-modal data fusion can improve the comprehensiveness and robustness of detection.

[0169] Regarding the warning trigger time, the warning trigger time of all drivers was within 5 seconds, and the warning trigger time of driver 003 was 2 seconds, indicating that the system can quickly identify the fatigue state and give timely warnings, improving traffic safety. In terms of warning accuracy, the warning accuracy of all drivers remained above 96%, with the highest being 100%, showing the stability of the system in different driving environments.

[0170] In summary, the fatigue driving detection system of the present invention can achieve high precision, high real-time performance, and high robustness under various driving conditions. Whether driving at high speeds, low speeds, or at different times, the system can accurately determine the fatigue state of the driver and give timely reminders, thus effectively improving traffic safety.

[0171] As described above, the above are only the preferred specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, making equivalent substitutions or changes should be covered within the protection scope of the present invention.

Claims

1. A driving recorder fatigue driving detection and reminder method based on multimodal data fusion, characterized in that: The steps include: S1, collecting the driver's facial image data through the vehicle camera, and pre-processing, performing face detection and area positioning, and obtaining the driver's facial feature area; S2, using dynamic image segmentation technology, dynamically adjusting the size and position of the image block according to the different features of the driver's facial area, and performing segmentation processing on the facial image; S3, using a visual transformer to extract features from the segmented facial images to identify early signs of fatigue driving; S4, obtaining multimodal data of the vehicle, and fusing the multimodal data of the vehicle with facial features to form a fatigue driving judgment model; S5. Optimize the network structure and training hyperparameters of the visual transformer by using the wolf pack optimization algorithm; S6. Based on the optimized visual converter, the driver's fatigue state is analyzed in real time, and the driver's fatigue level is judged by combining facial features and multi-modal data of multiple vehicles; S7. When the system detects that the driver is in a fatigued state, it will issue an audible and visual alarm to remind the driver to take a rest.

2. The driving recorder fatigue driving detection and reminder method based on multimodal data fusion according to claim 1 is characterized in that: The S2 specifically includes: S21. Extracting specific facial regions from the driver's facial image, the facial regions include the eye, mouth and eyebrow regions, and using a facial key point detection model based on deep learning to automatically locate and mark the positions of the facial key points, form the contour of the facial region, and accurately determine the boundaries of the facial region; S22, using an adaptive block method based on facial key points to process facial images, combining facial muscle movement patterns and spatial features, dynamically adjusting the image block size and position of each facial region, calculating the local spatial distribution of each facial region through the geometric relationship between the facial key point positions and adjacent key points, and then dynamically determining the size and shape of the image block according to the movement pattern of each region and the texture characteristics of the facial region: Among them, P i is the size of the i-th image block, K i is the position of the ith facial key point, D ij is the Euclidean distance between the i-th key point and the j-th key point, T i is the texture complexity of the facial area, M i is the motion pattern of the facial area, α i ,β i ,γ i ,θ i is the coefficient, γ, δ, ζ are weight adjustment factors, ρ i is the dynamic texture change rate of the facial area, and log2 is the logarithmic function; S23, using the time dimension features of the facial region to slide the image block, the step size and shape of each sliding window dynamically changes according to the movement frequency of the facial region and the rate of expression change: Among them, D x (K x ,K y ) and D y (K x ,K y ) represent facial key points K x and K y The relative distance on the x and y axes, γ1 and γ2 are the basic adjustment factors of the sliding step, λ1 and λ2 are the dynamic expansion factors of the time dimension, M x and M y represent the intensity of facial motion pattern changes in the x and y directions, T x and T y is the temporal rate of change of facial features, Δx and Δy represent the displacement of the sliding window along the x and y directions in the image space during image processing, respectively; S24. Enhanced extraction of facial fatigue features by adopting a fusion model of dynamic adaptive local correlation tensor and hierarchical time scale adjustment: Among them, A i is the enhanced feature of the i-th image block, I(P i ) and I(P j ) represent image blocks P i and P j The local eigenvector of ,σ is the kernel function width, ||I(P j )-I(P i )|| 2 is the feature difference between image blocks, A j is the image block P j The characteristic weighting coefficient of i is the local dynamic adjustment factor, ω i For each image block, dynamic weight, λ i is the expansion coefficient related to the time scale of facial features, μ i is the attenuation factor of the time dimension, T i is the time change rate, exp() is the exponential function, N is the total number of image blocks, and e is the base of the natural logarithm; S25, by performing weighted summation on the enhanced features of the image block, a global fatigue feature representation is generated: Among them, A global is the global fatigue characteristic representation, A i is the enhanced feature of the i-th image block, α i is the attention weight of the image block, C i is the local feature complexity of the image block, T i is the time scale feature of the image block, λ3 is the global feature enhancement factor, and δ1,μ,ν are adaptive adjustment factors.

3. The driving recorder fatigue driving detection and reminder method based on multimodal data fusion according to claim 1 is characterized in that: The S3 specifically includes: S31, the facial image block P after segmentation i Input to the visual transformer model, each image block P i ∈R H×W×C , where H and W are the height and width of the image block, respectively, and C is the number of channels. Each image block is converted into a one-dimensional vector V i ∈R D , where D is the feature embedding dimension and the embedding vector V i Represent the high-dimensional features of each image patch; S32, for each image block P i , introduces a nonlinear adaptive position encoding mechanism for the facial region, combines the spatial position of the facial region and the dynamic changes of expression, and position encoding It not only relies on the geometric structure of facial expressions, but also includes the temporal information of each image patch: Among them, K i is the facial area size of the i-th image block, ρ i is the dynamic texture change rate of the area where the image block is located, f i is the regional frequency characteristic, is the temporal feature of expression, η i is the scaling factor, α is the time dimension index, log2 is the logarithmic function, and sin is the sine function; S33, input the image block embedding vector with position encoding into the self-attention mechanism module of the visual transformer for feature extraction, and calculate the self-attention feature representation of each image block Among them, Q i is the query matrix, K i is the bond matrix, B i is the bias term based on facial expression, d k is the dimension of the key, λ i and μ i is the adjustment factor, M i is the motion pattern of the facial area, T i is the time scale of expression change, τ i is the expansion coefficient, softmax() is the normalization function, V i is the embedding vector of the i-th image block; S34. Introduce a spatiotemporal fusion module in the multi-layer self-attention network to fuse spatiotemporal features of different scales through a multi-head self-attention mechanism based on multi-scale attention: in, is the feature representation of the i-th image block, Q i and K j are the query and key matrices, β j is the temporal offset of the image block, ρ i is the spatiotemporal scale factor, δ i and γ i is the dynamic adjustment factor, exp() is the exponential function, N is the total number of image blocks, Embedding vector for image patch; S35. Generate the global feature representation of each image block through the spatiotemporal self-attention mechanism and multi-scale feature fusion, and perform the weighted summation of the global feature representations of all image blocks to obtain the final fatigue feature representation. in, is the final facial fatigue feature representation, is the global feature of the i-th image block, α i is the weight coefficient of the image block, C i is the detail feature of the image block, T i is the scale of the time feature, λ i and δ i is the adaptive adjustment factor, κ i It is a comprehensive adjustment factor for the overall fatigue characteristics.

4. The driving recorder fatigue driving detection and reminder method based on multimodal data fusion according to claim 1 is characterized in that: The S4 specifically includes: S41, extracting vehicle speed data V from the vehicle's sensor system speed , acceleration data a acc , steering angle data θ turn , and converted into time series data D vehicle (t)=[V speed (t),a acc (t),θ turn (t)], where t represents the time point; S42, the driver's final fatigue characteristics are expressed as As the input for fusion with vehicle multimodal data, a multimodal fusion network is constructed; S43, merging the time series data and the final fatigue feature representation into a comprehensive feature representation F fusion (t): Among them, ω1, ω2, ω3, ω4 are weighting coefficients; S44, inputting the comprehensive feature representation into a fatigue driving judgment model based on an adaptive deep generative adversarial network, and the fatigue driving judgment model uses the predicted value of fatigue driving To judge the fatigue status: Among them, W i is the weight of the i-th feature dimension, is the value of the i-th feature, b is the bias term, τ() is the activation function, γ is the adjustment coefficient, α is the nonlinear influence factor of multimodal features on fatigue prediction, is the change information of each modal data in the time dimension, ∈ is the noise factor, b is the bias term, tanh is the hyperbolic tangent activation function, n is the number of features, and m is the number of time-varying features; S45, Prediction value based on fatigue driving The driver is warned of fatigue through sound and light alarms. If the warning threshold is reached or exceeded, the system will trigger the alarm mechanism to remind the driver to take a rest.

5. The driving recorder fatigue driving detection and reminder method based on multimodal data fusion according to claim 1 is characterized in that: The S5 specifically includes: S51. Initialize the hyperparameters of the visual transformer model, including the image patch size P i , the number of Transformer layers L, the learning rate η, and the number of hidden units in each layer H; S52. Define the objective function F(θ) to evaluate the performance of the visual transformer model. The objective function combines the training loss and the regularization term: in, For the predicted value The loss function between the true value y(t), λ r is the regularization coefficient, θ i is the i-th model hyperparameter, T is the number of training rounds, n is the number of hyperparameters, T is the number of training rounds, and K is the number of hyperparameters; S53. Optimize the hyperparameters through the wolf pack optimization algorithm, simulate the wolf pack foraging process, and find the optimal solution in the parameter space: in, is the updated value of the i-th hyperparameter in the t+1th round, A and B are scaling factors, D is the distance factor, and X best and X worst are the hyperparameter values ​​of the best and worst solutions in the current wolf pack, respectively. is the updated value of the i-th hyperparameter in the t-th round; S54. In each round of optimization, the optimization effect of the hyperparameters is evaluated through the model validation set ν, and the validation error is calculated And adjust the wolf pack's search strategy according to the error value: Among them, F WOA (θ) is the objective function optimized by the wolf pack optimization algorithm, γ o is the adjustment factor, α i is the adjustment coefficient associated with each hyperparameter, θ is the set of all hyperparameters of the model, p is the intensity factor of the penalty term, and e is the base of the natural logarithm; S55, repeat steps S53 to S54 until the objective function F(θ) converges to the minimum value, and finally obtains the optimized hyperparameter configuration θ * , and use these hyperparameters for training the visual transformer model and fatigue driving detection.

6. The driving recorder fatigue driving detection and reminder method based on multimodal data fusion according to claim 1 is characterized in that: The S6 specifically includes: S61, the optimized visual transformer network model and the optimal hyperparameter θ obtained by the wolf pack optimization algorithm * Applied to real-time driver facial features and multimodal data, the input multimodal data is Among them, V speed (t) is the vehicle speed data, a acc (t) Acceleration data, θ turn (t) is the steering angle data, The final facial fatigue feature representation; S62, from the input multimodal data F fusion In (t), the final facial fatigue feature representation and vehicle data are extracted, and the fatigue state is evaluated based on the weighted sum of each feature according to the model. The comprehensive feature representation C is defined by combining facial features and vehicle multimodal data. fusion (t) is: Among them, ω i is the weight coefficient of the i-th feature, is the i-th feature after fusion, ζ is the dynamic weighting factor, Attention() is the weighted feature fusion operation based on the attention mechanism, and n is the number of features; S63, input the comprehensive features into the output layer of the neural network, calculate the driver's fatigue state prediction value, and smooth it by introducing a sliding window mechanism to obtain a stable fatigue state prediction value in, is the smoothed fatigue state prediction value, is the fatigue state value predicted at time k, μ is the adjustment factor, tanh is the hyperbolic tangent activation function, T cycle is the periodic factor, Δt is the sliding window size, and U is the number of time steps; S64. Introduce a deep reinforcement learning mechanism based on adaptive dynamic weight adjustment to dynamically adjust the weight coefficients of each modal feature and optimize the weights through continuous environmental feedback: in, is the learning rate, is the gradient of the loss function, ρ is the periodic adjustment factor, is the new weight of the i-th feature at time t+1, is the current weight of the i-th feature at time t, y(t) is the actual fatigue state, is a periodic function, T period is the cycle length; S65, fatigue state prediction value based on smoothing Determine whether the driver is in a fatigue state. When the fatigue warning threshold is reached or exceeded, the sound and light alarm system and visual prompts will be triggered.

Images