Fatigue driving behavior feature extraction and analysis method based on image recognition

Abstract

The present application relates to the field of image recognition fatigue driving behavior analysis, specifically to the fatigue driving behavior feature extraction and analysis method based on image recognition, the method comprises: through IMU, RGB camera and MEMS vibration sensor, the initial state set of driver is collected in real time, it is converted into four kinds of images such as head posture angular velocity waveform chart, the image features are extracted and input corresponding preset model to obtain fusion feature data, then the weight is dynamically adjusted through scene context feature, the cognitive load index is calculated and the processing mode is divided;The present application realizes multi-modal data fusion and dynamic weight adjustment, can accurately evaluate the cognitive load of driver and graded intervention, improves driving safety.

Description

Technical Field

[0001] This invention relates to the field of fatigue driving behavior analysis based on image recognition, specifically a method for extracting and analyzing fatigue driving behavior features based on image recognition. Background Technology

[0002] Fatigue driving is one of the main causes of traffic accidents, and the core challenge lies in how to quantify the driver's cognitive load in real time and accurately. Traditional fatigue detection methods are mostly based on single-modal data. Existing algorithms often use fixed weights to fuse multimodal data, which cannot dynamically adapt to different driving scenarios, such as changes in feature importance under urban congestion, highway cruising, and nighttime driving. In nighttime driving, micro-expression tension and heart rate variability are more indicative of fatigue, while head posture and muscle control are more critical in urban congestion scenarios. Traditional methods may lead to misjudgments due to fixed weights, and single-modal approaches cannot cover the complete closed loop of "behavior-expression-physiology." Head posture fluctuations can reflect distraction, facial micro-expressions correspond to mental tension, heart rate variability reveals autonomic nervous system stress, and muscle vibration features reflect operational stability. The lack of multi-dimensional feature fusion leads to one-sided load assessment. Therefore, image recognition-based fatigue driving behavior feature extraction and analysis methods are needed to solve the above problems. Summary of the Invention

[0003] To address the technical problems mentioned in the background, this invention provides a method for extracting and analyzing fatigue driving behavior features based on image recognition.

[0004] The objective of this invention can be achieved through the following technical solutions: This invention relates to a method for extracting and analyzing fatigue driving behavior features based on image recognition, the specific steps of which are as follows: Step 1: Real-time acquisition of the driver's initial state set using IMU, RGB camera, and MEMS vibration sensor, specifically: A miniature six-axis inertial measurement unit (IMU) is embedded inside the headrest of the driver's seat. The IMU includes a three-axis accelerometer and a three-axis gyroscope, which detect the acceleration and rotational angular velocity of the head, respectively. The direction of the vehicle's front is the x-axis, the driver's left side is the y-axis, and vertically upward is the z-axis. The rotational angular velocity of the head around the x, y, and z axes is collected at a preset frequency. ,in The data is represented by the acquisition time point; the angular displacement of the head is calculated by integrating the angular velocity. This indicates the object's state from the initial moment to... The cumulative rotation angle at time is calculated using the following logic: ,in This is the initial angular displacement, set when the driver starts the vehicle. Represents from the initial time 0 to time 1 The definite integral of angular velocity; A high-definition RGB camera is installed above the dashboard, and computer vision algorithms are used to identify 68 facial landmarks. , The landmarks are numbered and include the corner of the eye, corner of the mouth, and brow peak. The RGB camera tracks the positional changes of the landmarks in real time. The facial region is automatically segmented and regions of interest (ROIs) for the forehead and cheeks are extracted. The green channel signal is obtained by averaging the pixel values ​​of the ROIs. Then, empirical mode decomposition is used to remove low-frequency drift and high-frequency noise from the green channel signal. The interval between adjacent R waves is calculated to obtain the heart rate variability time sequence. The heart rate variability signal is then obtained by mean filtering and normalization. Heart rate variability signals include low-frequency components. and high-frequency components The component ratio is obtained by dividing the low-frequency component by the high-frequency component. ; A MEMS vibration sensor is deployed under the steering wheel to collect hand muscle vibration signals in real time according to a preset sampling frequency; Step 2: Convert the driver's initial state set into a head posture angular velocity waveform, a heart rate time-frequency spectrum, a facial micro-expression AU heatmap, and a muscle vibration time-frequency feature map, specifically: Specifically, the angular velocities of the x / y / z axes are mapped to the red / green / blue RGB channels of the image, respectively. The normalized magnitude of the angular velocity corresponding to the grayscale value of each channel is obtained. The calculation logic is as follows: , and ,in For channel indexing, This represents the baseline mean of the angular velocities of each axis. The standard deviation of the angular velocity of each axis is used to generate the head attitude angular velocity waveform; The spatiotemporal distribution of facial muscle movement units (AUs) is transformed into a visual heatmap. Intensity set to the center of the forehead. With the brow peak rate of change of vertical distance Its calculation logic is as follows: The baseline distance is the average distance while the driver is awake; 68 landmarks are mapped onto a 100×100 pixel facial grid, each grid... thermal value It is obtained by weighted sum of AU intensities within the corresponding region, and its calculation logic is as follows: ,in The Euclidean distance from the landmark point to the grid point. The width of the Gaussian kernel. It is an exponential function; the thermodynamic value is normalized to... Generate a grayscale image; The time-frequency characteristics of the heart rate variability signal are transformed into a heart rate time-spectrum graph. Perform short-time Fourier transform to generate time-frequency matrix Set the attention weight matrix as follows Weight the time-frequency matrix. For low-frequency components and high-frequency components The attention weight matrix is ​​constructed by setting a weight of 0.8 for each frequency band and 0.2 for the others; the amplitude of the time-frequency matrix is ​​then normalized to... , spectral graph generated; The vibration signal is decomposed into frequency band coefficients using a three-level wavelet packet decomposition method. The vibration energy value of the corresponding sensor is then obtained from these frequency band coefficients. The vibration energy value is normalized according to the baseline energy at the sensor location. The baseline energy is the average energy when the driver is awake. The calculation logic is as follows: ,in and Here are the baseline mean and standard deviation of sensor m. The normalized vibrational energy value is... Through linear mapping to This generates a vibrational energy thermogram, which... The grayscale value corresponding to each pixel.

[0005] Step 3: Extract features from the head posture angular velocity waveform, heart rate time-frequency spectrum, facial micro-expression AU heatmap, and muscle vibration time-frequency feature map. Input each feature into the corresponding preset model, and the preset model outputs the driver's fused feature data, specifically: Specifically, a gray-level co-occurrence matrix is ​​constructed based on the heart rate time-frequency spectrogram, and the number of gray levels is set to [value missing]. The direction is horizontal, the pixel distance is 1, and the grayscale values ​​are counted at the specified direction and distance. and Number of times they occur simultaneously Then normalize it to obtain the gray-level co-occurrence matrix. Its calculation logic is as follows: This is used to extract texture features from the gray-level co-occurrence matrix. Texture features include contrast and entropy. The contrast calculation logic is as follows: The entropy calculation logic is as follows: , It is the natural logarithm; Set the high frequency range in The optimal threshold was calculated using the Otsu method. Its calculation logic is as follows: ,in and The ratio of the two types of pixels after threshold segmentation. and The average grayscale value of the two types of pixels; the time-frequency matrix is ​​binarized, and the calculation logic is as follows: The calculation logic for extracting the area ratio of high-frequency regions is as follows: ,in m1 represents the total number of time points in the time-frequency matrix, and m2 represents the time point number. Support vector machines were chosen as the architecture for the stress index prediction model. Component ratios, contrast, entropy, and the proportion of high-frequency regions were input into the stress index prediction model, and the model outputs the driver's predicted stress index. Its model formula is: ,in For Lagrange multipliers, For kernel function, The training sample feature vector includes component ratios, contrast, entropy, and the proportion of high-frequency region area. For bias terms; Extract the high-intensity area ratio and intensity rise slope of the AU in the facial micro-expression heatmap, and input them into the facial tension index model to obtain the normalized grayscale mean of all pixels within the ROI between the eyebrows. The proportion of pixels with a normalized grayscale mean > 0.6 in the total area of ​​the Region of Interest (ROI) is calculated to obtain the proportion of high-intensity areas (AU). Its calculation logic is as follows: ,in This represents the total number of pixels within the ROI region between the eyebrows. Number the pixels. This is an indicator function, set to 1 if the condition is met, and 0 otherwise; the average AU intensity is taken from three consecutive heatmap frames. slope The calculation logic is as follows: Where Δt is the frame interval; a multilayer perceptron is set as the architecture for the facial tension index model. The multilayer perceptron includes an input layer, a hidden layer, and an output layer. Historical heatmap data of multiple drivers are collected and labeled with tension scores. The model is trained using a loss function and an optimizer; the latest driver facial expression features are obtained and input into the input layer, and the output layer obtains the facial tension index. ; The head posture angular velocity waveform is input into a preset 3D convolutional neural network. The time axis is divided into short and long windows. A fast Fourier transform is performed on the angular velocity of the blue channel within each short window to extract the driver's corresponding rapid head nodding frequency. Calculate the linear regression slope of the red channel angular velocity within a long window. Its calculation logic is as follows: ,in The slope The intercept is... The error term is represented by the linear regression slope, which reflects the driver's continuous head-turning tendency. The slope is obtained by minimizing the sum of squared errors. The rapid head-nodding frequency and the linear regression slope are concatenated using a feature pyramid to output a head posture feature vector. ; Extract the energy distribution entropy and high-frequency energy proportion from the time-frequency feature map of muscle vibration, as well as the vibration energy value. The input is fed into a random forest regression model, which outputs a muscle control stability index of 0-1. The time-frequency feature map of muscle vibration is divided into 36 pixel sub-blocks, and the energy proportion of each pixel sub-block is obtained. Then calculate the energy distribution entropy. Its calculation logic is as follows: ,in Let be the pixel sub-block number, and log be the logarithmic function; then calculate the proportion of high-frequency sub-band energy to the total energy to obtain the high-frequency energy percentage. Its calculation logic is as follows: ,in In frequency The vibration energy value, and These represent the start and end frequencies of the high-frequency sub-band, respectively. and The lowest and highest frequencies of the muscle vibration time-frequency feature map are respectively used. These are then input into a random forest regression model, along with the energy distribution entropy, high-frequency energy proportion, and vibration energy value. Simulated data from multiple drivers are extracted, and the model is trained with 90 trees, a maximum depth of 5, and a minimum sample leaf node of 5. The latest time-point feature values ​​are then input into the model until the muscle control stability index is output. ; Step 4: Dynamically adjust the weights based on contextual features of different scenarios, specifically as follows: Specifically, scene context features include light intensity, vehicle speed, road type, distance to the vehicle ahead, and task complexity. An encoder associates the scene with the features; the scene embedding layer maps the scene context features to scene embeddings through a fully connected layer; the feature embedding layer maps multimodal features to feature embeddings through a fully connected layer; and the association fusion layer calculates the association degree between the scene and the features using dot products, generating an association vector. ;Will Input a preset LSTM model, acquire scene temporal changes, set a context-aware layer, and its computation logic is as follows: ,in This is the hidden state from the previous moment. To output a time-aware vector. For memory networks; time-aware vectors are mapped to weight vectors through two fully connected layers. , These are the weight vectors for the corresponding data; after Softmax normalization, the calculation logic is as follows: ,in It is a 32×4 weight matrix. It is a 64×32 weight matrix. for The bias vector; the normalization calculation logic is as follows: ; Step 5: Calculate the cognitive load index using the obtained data features and corresponding dynamic weights, and then use this index to determine the processing method, specifically: Specifically, the stress index will be predicted. Tension level Head pose feature vector and muscle control stability index The cognitive load index is obtained by normalizing and then weighting the results with dynamic weights. A higher cognitive load index indicates a heavier cognitive load on the driver. Its calculation logic is as follows: ;If a cognitive load index-based processing strategy is adopted, If the value is less than 0.3, it is considered a low load, and the instrument panel sends a green icon indicating a good status, maintaining automatic driving assistance intervention; if 0.3 ≤ If the value is less than 0.7, it is considered a medium load, and a yellow warning icon will be displayed on the dashboard with a text message indicating that caution is needed; if the value is 0.7 or greater... If the load is deemed dangerous, the dashboard will flash a red warning light and emit a high-frequency buzzer, and the seat cushion and backrest will vibrate synchronously at a high frequency.

[0006] Compared with existing technologies, the beneficial effects of this invention are: deep feature extraction, improving detection accuracy. Advanced algorithms are employed to refine the data across various modalities: head posture analysis uses a 3D convolutional neural network to capture motion patterns in both time and space, effectively distinguishing between normal operation and fatigue-related abnormal swaying; facial micro-expressions use graph neural networks to model the coordinated movement of facial feature points, accurately identifying the mental tension behind subtle changes in facial expressions such as frowning and pursing lips; heart rate variability signals are enhanced through time-frequency analysis and attention mechanisms to strengthen the feature response in stress-related frequency bands; muscle vibration features are mined for abnormal patterns in unlabeled data through self-supervised learning; these technologies collectively improve the discriminative power and anti-interference ability of the features, maintaining stable detection performance even under complex conditions such as changes in lighting and drivers wearing obstructions. The system automatically adjusts the evaluation weights of each modality through a scene perception mechanism, enabling the model to flexibly respond to different driving environments. In urban congestion scenarios, the system automatically increases the weights of head posture and muscle vibration features, focusing on analyzing frequent steering and grip force changes. When driving at night, the weights of micro-expressions and heart rate features are significantly increased, focusing on visual fatigue and autonomic nervous system stress. Attached Figure Description

[0007] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. The following drawings are not drawn to scale according to the actual size, but are intended to illustrate the main idea of ​​the present invention.

[0008] Figure 1 This is a flowchart of the method steps of the present invention.

[0009] Figure 2 A flowchart for dynamic weight adjustment logic. Detailed Implementation

[0010] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are also within the scope of protection of the present invention.

[0011] Please refer to Figure 1 As shown, one embodiment of the fatigue driving behavior feature extraction and analysis method based on image recognition in this application includes: Step F10: Real-time acquisition of the driver's initial state set using IMU, RGB camera and MEMS vibration sensor; Specifically, a miniature six-axis inertial measurement unit (IMU) is embedded inside the headrest of the driver's seat. The IMU captures the head's motion in real time without contacting the driver's body. The IMU includes a three-axis accelerometer and a three-axis gyroscope, which detect the head's acceleration and rotational angular velocity, respectively. Its coordinate system is consistent with the vehicle, with the direction of the vehicle's front as the x-axis, the driver's left side as the y-axis, and vertically upward as the z-axis. The rotational angular velocity of the head around the x, y, and z axes is collected at a preset frequency. ,in This is represented by the time point of data collection, such as the change in angular velocity along the z-axis corresponding to a nodding motion, and the change in angular velocity along the y-axis corresponding to turning the head to look in the rearview mirror; the angular displacement of the head is calculated by integrating the angular velocities. This indicates the object's state from the initial moment to... The cumulative rotation angle at time is calculated using the following logic: ,in This is the initial angular displacement, set when the driver starts the vehicle. Represents from the initial time 0 to time 1 The definite integral of angular velocity has the physical meaning of the cumulative effect of angular velocity over time, which completely records the trajectory of the head in three-dimensional space. A high-definition RGB camera is installed above the dashboard, and computer vision algorithms are used to identify 68 facial landmarks. , The landmarks are numbered and include the corner of the eye, corner of the mouth, and brow peak. The RGB camera tracks the positional changes of the landmarks in real time. For example, when frowning, the center of the brow will move closer to the center, and when smiling, the corner of the mouth will stretch to the sides. This method automatically segments and extracts Regions of Interest (ROIs) from the forehead and cheeks, based on photoplethysmography (PPG). It acquires heart rate variability (HRV) signals without the need for electrodes. The forehead and cheeks are specifically chosen as the monitoring areas because these areas have thinner skin and are rich in blood vessels. The green channel signal is obtained by averaging the pixel values ​​of the ROIs. Empirical mode decomposition (EMD) is used to remove low-frequency drift and high-frequency noise from the green channel signal, retaining the 5–25 Hz pulse wave component. This aims to separate the subtle color fluctuations caused by heartbeats. The intervals between adjacent R waves are calculated to obtain the HRV time sequence, which is then mean-filtered and normalized to obtain the final HRV signal. Heart rate variability signals include low-frequency components. and high-frequency components The component ratio is obtained by dividing the low-frequency component by the high-frequency component. The higher the component ratio, the more excited the sympathetic nervous system is, and the greater the risk of mental overload. A MEMS vibration sensor is deployed under the steering wheel to collect hand muscle vibration signals in real time according to a preset sampling frequency; Step F11: Convert the driver's initial state set into a head posture angular velocity waveform, heart rate time spectrum, facial micro-expression AU heat map, and muscle vibration time-frequency feature map; Specifically, the three-dimensional angular velocity is converted into a two-dimensional image that can characterize motion stability, highlighting the abnormal fluctuation characteristics during fatigue. The x / y / z axis angular velocities are mapped to the red / green / blue RGB channels of the image, respectively. The normalized amplitude of the angular velocity corresponding to the grayscale value of each channel is obtained. The calculation logic is as follows: , and ,in For channel indexing, This represents the baseline mean of the angular velocities of each axis. The standard deviation of the angular velocity for each axis is used to generate a waveform of the head attitude angular velocity. The horizontal axis represents time, the vertical axis represents the channel, and the color intensity reflects the degree to which the angular velocity deviates from the baseline. The spatiotemporal distribution of facial muscle action units (AUs) is transformed into a visual heatmap, highlighting muscle activation patterns related to attentional stress, using the corrugator supercilii AU4 as an example. Intensity set to the center of the forehead. With the brow peak rate of change of vertical distance Its calculation logic is as follows: The baseline distance is the average distance while the driver is awake; 68 landmarks are mapped onto a 100×100 pixel facial grid, each grid... thermal value It is obtained by weighted sum of AU intensities within the corresponding region, and its calculation logic is as follows: ,in The Euclidean distance from the landmark point to the grid point. The width of the Gaussian kernel. The function is an exponential function, the purpose of which is to nonlinearly amplify or weight the AU intensity to highlight the contribution of high-intensity AUs or suppress low-intensity interferences; the thermodynamic value is normalized to... Generate a grayscale image; the higher the brightness, the stronger the muscle tension. For example, a brighter area in the center of the forehead indicates severe frowning. The time-frequency characteristics of heart rate variability signals are transformed into a heart rate time-spectrum graph. Blood flow pulse-related frequency components are enhanced through an attention mechanism. Perform short-time Fourier transform to generate time-frequency matrix Set the attention weight matrix as follows Weight the time-frequency matrix. For low-frequency components and high-frequency components The attention weight matrix is ​​constructed by setting a weight of 0.8 for each frequency band and 0.2 for the others; the amplitude of the time-frequency matrix is ​​then normalized to... The spectrum is generated; the horizontal axis represents time, the vertical axis represents frequency, and the color represents energy intensity. The vibration signal is decomposed into frequency band coefficients using a three-level wavelet packet decomposition method. The vibration energy value of the corresponding sensor is then obtained from these frequency band coefficients. The vibration energy value is normalized according to the baseline energy at the sensor location. The baseline energy is the average energy when the driver is awake. The calculation logic is as follows: ,in and Here are the baseline mean and standard deviation of sensor m. The normalized vibrational energy value is... Through linear mapping to This generates a vibrational energy thermogram, which... The grayscale value corresponding to each pixel; Step F12: Extract the features of head posture angular velocity waveform, heart rate time spectrum, facial micro-expression AU heat map and muscle vibration time frequency feature map, input each feature into the corresponding preset model, and output the driver's fused feature data from the preset model; Specifically, a gray-level co-occurrence matrix is ​​constructed based on the heart rate time-frequency spectrogram, and the number of gray levels is set to [value missing]. The direction is horizontal, the pixel distance is 1, and the grayscale values ​​are counted at the specified direction and distance. and Number of times they occur simultaneously Then normalize it to obtain the gray-level co-occurrence matrix. Its calculation logic is as follows: This is used to extract texture features from the gray-level co-occurrence matrix. Texture features include contrast and entropy. The contrast calculation logic is as follows: The entropy calculation logic is as follows: , It is the natural logarithm. It should be noted that in the time-spectrum graph, if the contrast is high, it means that the energy fluctuation in the high-frequency band is obvious, such as the presence of obvious respiratory sinus arrhythmia. Entropy means that under stress, the entropy value increases as the texture complexity of the heart rate time-spectrum graph increases; under deep relaxation, the regularity of the texture of the spectrum graph increases, and the entropy value decreases. Set the high frequency range in The optimal threshold was calculated using the Otsu method. This method determines the threshold by maximizing the inter-class variance of the two classes of pixels. Its calculation logic is as follows: ,in and The ratio of the two types of pixels after threshold segmentation. and The average grayscale value of the two types of pixels; the time-frequency matrix is ​​binarized, and the calculation logic is as follows: The calculation logic for extracting the area ratio of high-frequency regions is as follows: ,in m1 represents the total number of time points in the time-frequency matrix, and m2 represents the time point number. It should be noted that... A higher value indicates that vagal nerve activity is dominant, such as in a resting state; a lower value indicates that sympathetic nerve activity is enhanced, such as in a stress state. Support vector machines were chosen as the architecture for the stress index prediction model. Component ratios, contrast, entropy, and the proportion of high-frequency regions were input into the stress index prediction model, and the model outputs the driver's predicted stress index. Its model formula is: ,in For Lagrange multipliers, For kernel function, The training sample feature vector includes component ratios, contrast, entropy, and the proportion of high-frequency region area. For bias terms; Extract the high-intensity area ratio and intensity rise slope of the AU in the facial micro-expression heatmap, and input them into the facial tension index model to obtain the normalized grayscale mean of all pixels within the ROI between the eyebrows. The proportion of pixels with a normalized grayscale mean > 0.6 in the total area of ​​the Region of Interest (ROI) is calculated to obtain the proportion of high-intensity areas (AU). Its calculation logic is as follows: ,in This represents the total number of pixels within the ROI region between the eyebrows. Number the pixels. This is an indicator function; it returns 1 when the condition is met and 0 otherwise. It should be noted that, normally... The smaller the value, the greater the fatigue; take the average AU intensity of three consecutive heatmap frames. slope The calculation logic is as follows: Where Δt is the frame interval; it should be noted that the slope is smaller under normal conditions, and larger under fatigue conditions; a multilayer perceptron is set as the architecture for the facial tension index model. The multilayer perceptron includes an input layer, a hidden layer, and an output layer. Historical heatmap data of multiple drivers are collected and labeled with tension scores. The model is trained using a loss function and an optimizer; the latest driver facial expression features are obtained and input into the input layer, and the output layer obtains the facial tension index. ; The head posture angular velocity waveform is input into a preset 3D convolutional neural network. The time axis is divided into short and long windows. A fast Fourier transform is performed on the angular velocity of the blue channel within each short window to extract the driver's corresponding rapid head nodding frequency. Calculate the linear regression slope of the red channel angular velocity within a long window. Its calculation logic is as follows: ,in The slope The intercept is... The error term is represented by the linear regression slope, which reflects the driver's continuous head-turning tendency. The slope is obtained by minimizing the sum of squared errors. The rapid head-nodding frequency and the linear regression slope are concatenated using a feature pyramid to output a head posture feature vector. ; Extract the energy distribution entropy and high-frequency energy proportion from the time-frequency feature map of muscle vibration, as well as the vibration energy value. The input is fed into a random forest regression model, which outputs a muscle control stability index of 0-1. The time-frequency feature map of muscle vibration is divided into 36 pixel sub-blocks, and the energy proportion of each pixel sub-block is obtained. Then calculate the energy distribution entropy. Its calculation logic is as follows: ,in Let be the pixel sub-block number, and log be the logarithmic function. A higher value indicates greater driver fatigue; then, the proportion of high-frequency subband energy to total energy is calculated to obtain the high-frequency energy percentage. When fatigued, muscle tremors intensify, and high-frequency energy increases significantly. The calculation logic is as follows: ,in In frequency The vibration energy value, and These represent the start and end frequencies of the high-frequency sub-band, respectively. and The lowest and highest frequencies of the muscle vibration time-frequency feature map are respectively used. These are then input into a random forest regression model along with the energy distribution entropy, high-frequency energy proportion, and vibration energy value. A lower output muscle control stability index indicates greater fatigue. Simulated data from multiple drivers are extracted, and the model is trained with 90 trees, a maximum depth of 5, and a minimum sample leaf node of 5. The feature values ​​at the latest time point are then input into the model until the muscle control stability index is output. ; Step F13: Dynamically adjust the weights based on contextual features of different scenarios; Specifically, scene context features include illumination intensity (0-1), vehicle speed (0-1, normalized to 0-120km / h), road type (0=highway, 1=city, 2=rural), distance to the vehicle ahead (0-1, normalized to 0-50m), and task complexity (0-1, 0=no task, 1=navigation + call). An encoder associates the scene and features. The scene embedding layer maps the scene context features to scene embeddings through a fully connected layer. The feature embedding layer maps multimodal features to feature embeddings through a fully connected layer. The association fusion layer calculates the association degree between the scene and features using dot products, generating an association vector. ;Will Input a preset LSTM model, acquire scene temporal changes, set a context-aware layer, and its computation logic is as follows: ,in This is the hidden state from the previous moment. To output a time-aware vector. For memory networks; time-aware vectors are mapped to weight vectors through two fully connected layers. , These are the weight vectors for the corresponding data; after Softmax normalization, the calculation logic is as follows: ,in It is a 32×4 weight matrix. It is a 64×32 weight matrix. for The bias vector; the normalization calculation logic is as follows: ; Step F14: Calculate the cognitive load index using the obtained data features and corresponding dynamic weights, and use this index to classify the processing methods; Specifically, the stress index will be predicted. Tension level Head pose feature vector and muscle control stability index The cognitive load index is obtained by normalizing and then weighting the results with dynamic weights. A higher cognitive load index indicates a heavier cognitive load on the driver. Its calculation logic is as follows: ;If a cognitive load index-based processing strategy is adopted, If the value is less than 0.3, it is considered a low load, and the instrument panel sends a green icon indicating a good status, maintaining automatic driving assistance intervention; if 0.3 ≤ If the value is less than 0.7, it is considered a medium load, and a yellow warning icon will be displayed on the dashboard with a text message indicating that caution is needed; if the value is 0.7 or greater... If the load is deemed dangerous, the dashboard will flash a red warning light and emit a high-frequency buzzer, and the seat cushion and backrest will vibrate synchronously at a high frequency.

[0012] The foregoing description is illustrative of the invention and should not be construed as limiting it. Although several exemplary embodiments of the invention have been described, those skilled in the art will readily understand that many modifications can be made to the exemplary embodiments without departing from the novel teachings and advantages of the invention. Therefore, all such modifications are intended to be included within the scope of the invention as defined in the claims. It should be understood that the foregoing description is illustrative of the invention and should not be construed as limiting it to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The invention is defined by the claims and their equivalents.

Claims

1. A method for extracting and analyzing fatigue driving behavior features based on image recognition, characterized in that, include: The driver's initial state set is collected in real time using an IMU, RGB camera and MEMS vibration sensor; The driver's initial state set is converted into a head posture angular velocity waveform, a heart rate time spectrum, a facial micro-expression AU heat map, and a muscle vibration time-frequency feature map. Features are extracted from head posture angular velocity waveform, heart rate time spectrum, facial micro-expression AU heat map and muscle vibration time frequency feature map. Each feature is input into the corresponding preset model. The preset model outputs the driver's predicted stress index, facial tension index, head posture feature vector and muscle control stability index, which together constitute the fused feature data. The weights are dynamically adjusted based on the contextual features of different scenarios; The obtained fused feature data and corresponding dynamic weights are used to calculate the cognitive load index, and the processing methods are divided accordingly. Features are extracted from the heart rate spectrogram and facial micro-expression AU heatmap. These features are then input into the corresponding preset models, which output the driver's fused feature data, including: A gray-level co-occurrence matrix was constructed based on the heart rate time-frequency spectrogram, and the number of gray levels was set to [value missing]. Given a horizontal direction and a pixel distance of 1, count the number of times grayscale values ​​m1 and j appear simultaneously at a specified direction and distance. Then normalize it to obtain the gray-level co-occurrence matrix. Its calculation logic is as follows: This is used to extract texture features from the gray-level co-occurrence matrix. Texture features include contrast and entropy. The contrast calculation logic is as follows: The entropy calculation logic is as follows: , It is the natural logarithm; Set the high frequency range in The optimal threshold was calculated using the Otsu method. Its calculation logic is as follows: ,in and The ratio of the two types of pixels after threshold segmentation. and The average grayscale value of the two types of pixels; the time-frequency matrix is ​​binarized, and the calculation logic is as follows: The calculation logic for extracting the area ratio of high-frequency regions is as follows: , where M is the total number of time points in the time-frequency matrix, and t is the time point number; Support Vector Machine (SVM) is chosen as the architecture for the stress index prediction model. The component ratios, contrast, entropy, and high-frequency region area proportions are input into the stress index prediction model. The model outputs the driver's predicted stress index (SI), and its formula is as follows: , where δ is a Lagrange multiplier, Here, rf is the kernel function, and rf is the feature vector of the training samples, including component ratios, contrast, entropy, and the area ratio of high-frequency regions. For bias terms; The high-intensity area ratio and intensity rise slope of AUs in the facial micro-expression heatmap are extracted and input into the facial tension index model. The normalized grayscale mean FS of all pixels within the brow region ROI is obtained. The proportion of pixels with a normalized grayscale mean greater than 0.6 in the total ROI area is calculated to obtain the AU high-intensity area ratio AQ. The calculation logic is as follows: Where N is the total number of pixels in the ROI region between the eyebrows, and m3 is the pixel number. This is an indicator function, set to 1 if the condition is met, and 0 otherwise; the average AU intensity is taken from three consecutive heatmap frames. The slope YG is calculated using the following logic: ,in The frame interval is set; a multilayer perceptron is set as the architecture for the facial tension index model. The multilayer perceptron includes an input layer, a hidden layer and an output layer. Historical heatmap data of multiple drivers are collected and labeled with tension scores. The model is trained through a loss function and an optimizer; the latest driver facial expression features are obtained and input into the input layer, and the output layer obtains the facial tension index YV. It also includes head posture angular velocity waveforms and muscle vibration time-frequency characteristic maps, specifically: The head posture angular velocity waveform is input into a preset 3D convolutional neural network. The time axis is divided into short and long windows. A fast Fourier transform is performed on the angular velocity of the blue channel within each short window to extract the driver's corresponding rapid head nodding frequency. Calculate the linear regression slope of the red channel angular velocity within a long window. Its calculation logic is as follows: ,in η is the slope, and η is the intercept. The error term is represented by the linear regression slope, which reflects the driver's continuous head tilting tendency. The slope is obtained by minimizing the sum of squared errors. The rapid head nodding frequency and the linear regression slope are concatenated through a feature pyramid to output the head posture feature vector HY. Extract the energy distribution entropy and high-frequency energy proportion from the time-frequency feature map of muscle vibration, as well as the vibration energy value. The input is fed into a random forest regression model, which outputs a muscle control stability index of 0-1. The muscle vibration time-frequency feature map is divided into 36 pixel sub-blocks, and the energy proportion value Er of each pixel sub-block is obtained. Then, the energy distribution entropy Ej is calculated. The calculation logic is as follows: ,in Let be the pixel sub-block number, and log be the logarithmic function; then calculate the proportion of high-frequency sub-band energy to the total energy to obtain the high-frequency energy proportion Ec. The calculation logic is as follows: ,in In frequency The vibration energy value, and These represent the start and end frequencies of the high-frequency sub-band, respectively. and The lowest and highest frequencies of the muscle vibration time-frequency feature map are respectively; the energy distribution entropy, the proportion of high-frequency energy, and the vibration energy value are input into the random forest regression model, and the simulation data of multiple drivers are extracted. The number of trees is set to 90, the maximum depth is 5, and the minimum sample leaf node is 5 to train the model; the feature values ​​of the latest time point are obtained and input into the model until the muscle control stability index JH is output. The weights are dynamically adjusted based on contextual features from different scenarios, specifically as follows: An encoder associates the scene with features. Scene context features include light intensity, vehicle speed, road type, distance to the vehicle ahead, and task complexity. The scene embedding layer maps the scene context features to scene embeddings through a fully connected layer, and the feature embedding layer maps multimodal features to feature embeddings through a fully connected layer. The association fusion layer calculates the association degree between the scene and features using dot products, generating an association vector. ;Will Input a preset BLSTM model, obtain scene temporal changes, set a context-aware layer, and its calculation logic is as follows: ,in This is the hidden state from the previous moment. To output the time-aware vector, BLSTM is used as a memory network; the time-aware vector is mapped to LOG through two fully connected layers, and then the LOG is normalized by Softmax to obtain the weight vector of the corresponding data. Its calculation logic is as follows: ,in It is a 4×32 weight matrix. It is a 32×64 weight matrix. for The bias vector; the normalization calculation logic is as follows: ; The driver's initial state set is collected in real time using an IMU, RGB camera, and MEMS vibration sensor, specifically: A miniature six-axis inertial measurement unit (IMU) is embedded inside the headrest of the driver's seat. The IMU includes a three-axis accelerometer and a three-axis gyroscope, which detect the acceleration and rotational angular velocity of the head, respectively. The direction of the vehicle's front is the x-axis, the driver's left side is the y-axis, and vertically upward is the z-axis. The rotational angular velocity of the head around the x, y, and z axes is collected at a preset frequency. Where t represents the acquisition time point; the angular displacement of the head is calculated by integrating the angular velocity. , representing the cumulative rotation angle of the object from the initial time to time t, is calculated as follows: ,in This is the initial angular displacement, set when the driver starts the vehicle. This represents the definite integral of the angular velocity from the initial time 0 to time t. A high-definition RGB camera is installed above the dashboard, and computer vision algorithms are used to identify 68 facial landmarks. , where i represents the landmark number, and the RGB camera tracks the position changes of the landmark in real time; The facial region is automatically segmented and the Region of Interest (ROI) of the forehead and cheeks is extracted. The green channel signal is obtained by averaging the pixel values ​​of the ROI region. Then, low-frequency drift and high-frequency noise in the green channel signal are removed by empirical mode decomposition. The interval between adjacent R waves is calculated to obtain the heart rate variability time sequence. After mean filtering and normalization, the heart rate variability signal W(t) is obtained. The heart rate variability signal includes a low-frequency component LF and a high-frequency component HF. The component ratio DF is obtained by dividing the low-frequency component by the high-frequency component. A MEMS vibration sensor is deployed under the steering wheel to collect hand muscle vibration signals in real time according to a preset sampling frequency; The obtained angular velocity and heart rate variability signals are converted into head posture angular velocity waveforms and heart rate time-frequency spectra, specifically: The angular velocities along the x, y, and z axes are mapped to the red, green, and blue RGB channels of the image, respectively. The normalized magnitude of the angular velocity corresponding to the grayscale value of each channel is then obtained. The calculation logic is as follows: , and , where j2 is the channel index, μ is the baseline mean of the angular velocity of each axis, and σ is the standard deviation of the angular velocity of each axis, thus generating the head attitude angular velocity waveform; The time-frequency characteristics of the heart rate variability signal are transformed into a heart rate time-frequency spectrum. A short-time Fourier transform is performed on W(t) to generate a time-frequency matrix WE(t). The attention weight matrix AR is set to weight the time-frequency matrix. The low-frequency component LF and the high-frequency component HF are assigned a weight of 0.8, and other frequency bands are assigned a weight of 0.2 to form an attention weight matrix; the amplitude of the time-frequency matrix is ​​normalized to [0,255] to generate a time-frequency spectrum.

2. The method for extracting and analyzing fatigue driving behavior features based on image recognition according to claim 1, characterized in that, The cognitive load index is calculated using the obtained data features and corresponding dynamic weights, and the processing methods are then categorized accordingly. The cognitive load index TFG is obtained by normalizing the predicted stress index SI, facial tension index YV, head posture feature vector HY, and muscle control stability index JH, and then weighting them with dynamic weights. A higher cognitive load index indicates a heavier cognitive load on the driver. The calculation logic is as follows: The system uses a cognitive load index-based processing strategy. If TFG < 0.3, it is considered a low load, and the instrument panel sends a green icon indicating a good status, maintaining automatic driving assistance intervention. If 0.3 ≤ TFG < 0.7, it is considered a medium load, and the instrument panel displays a yellow warning icon with a text message indicating that the current status requires vigilance. If 0.7 ≥ TFG, it is considered a dangerous load, and the instrument panel flashes a red warning and emits a high-frequency buzzer, while the seat cushion and backrest vibrate synchronously at a high frequency.

3. The method for extracting and analyzing fatigue driving behavior features based on image recognition according to claim 1, characterized in that, The driver's facial muscle movement units and vibration signals are converted into facial micro-expression AU heatmaps and muscle vibration time-frequency characteristic maps, specifically: The spatiotemporal distribution of facial muscle movement units (AUs) is transformed into a visual heatmap. Intensity set to the center of the forehead. With the brow peak rate of change of vertical distance Its calculation logic is as follows: The baseline distance is the average distance while the driver is awake; 68 landmarks are mapped onto a 100×100 pixel facial grid, each grid... thermal value It is obtained by weighted sum of AU intensities within the corresponding region, and its calculation logic is as follows: ,in ω is the Euclidean distance from the landmark point to the grid point, ω is the Gaussian kernel width, and exp is the exponential function; the thermal values ​​are normalized to [0,255] to generate a grayscale image; The vibration signal is decomposed into frequency band coefficients using a three-level wavelet packet decomposition method. The vibration energy value of the corresponding sensor is then obtained from these frequency band coefficients. The vibration energy value is normalized according to the baseline energy at the sensor location. The baseline energy is the average energy when the driver is awake. The calculation logic is as follows: ,in and Here are the baseline mean and standard deviation of sensor m. The normalized vibrational energy value is... By linearly mapping to [0,255], a vibrational energy heatmap is generated. The grayscale value corresponding to each pixel.

Images