A driver fatigue recognition method and system based on multi-domain driving behavior feature fusion

Abstract

The present application relates to the technical field of driver state monitoring, in particular to a driver fatigue recognition method and system based on multi-domain driving behavior feature fusion. The method acquires a steering control signal and a longitudinal control signal in a vehicle control domain, and a vehicle lateral driving signal in an auxiliary driving domain, and performs synchronization and preprocessing; performs frequency domain analysis on the preprocessed steering control signal, analyzes the continuous change of the longitudinal control signal, and analyzes the position change and yaw change of the vehicle relative to the lane according to the vehicle lateral driving signal, to obtain steering control features, longitudinal control features and lateral driving features respectively; the above features are fused to obtain multi-domain driving behavior features, according to which the driver fatigue state is recognized and fatigue warning is triggered. The present application does not need to increase vehicle-mounted hardware, does not rely on driver face monitoring cameras or physiological data acquisition, and takes into account low cost, all-weather applicability and privacy friendliness, and improves the recognition accuracy.

Description

Technical Field

[0001] This invention relates to the field of driver condition monitoring technology, specifically to a driver fatigue recognition method and system based on multi-domain driving behavior feature fusion. Background Technology

[0002] Driver fatigue monitoring requires first acquiring signals that reflect fatigue levels, and then using algorithms to determine the driver's condition. Using in-vehicle cameras to recognize facial features such as eyelid opening and yawning was an early method, but the images are easily affected by changes in lighting, and misjudgments are possible when the driver is wearing glasses or a mask. Furthermore, the camera module and corresponding computing power incur hardware costs, and collecting facial images raises privacy concerns.

[0003] To circumvent the limitations of cameras, some solutions have shifted to collecting the driver's physiological signals, obtaining indicators such as heart rate and skin conductance through steering wheel capacitive sensors, seat ECG electrodes, or wearable devices. However, this still requires additional sensors, so the cost has not been reduced, and the continuous vibration of the vehicle will interfere with the stability of non-contact measurement.

[0004] Both visual and physiological signal-based solutions are subject to hardware limitations. Therefore, some solutions utilize existing driving operation signals such as steering wheel angle, accelerator, and brake pedals in the vehicle's bus, eliminating the need for additional sensors. However, current solutions utilize signals only superficially, often calculating only the standard deviation of steering wheel angle or zero-speed ratio, failing to extract deeper characteristics that better represent fatigue from signal changes. Furthermore, they frequently rely solely on vehicle control domain signals, neglecting to incorporate lane-keeping assist and other driver assistance domain information, resulting in limited recognition accuracy.

[0005] Therefore, existing technologies have not yet been able to integrate vehicle control domain and driver assistance domain signals to form a driver fatigue recognition solution that is unaffected by lighting and occlusion conditions without increasing hardware costs. Summary of the Invention

[0006] In view of the above-mentioned problems, the technical problem to be solved by the present invention is to provide a driver fatigue recognition method and system based on multi-domain driving behavior feature fusion, which is used to solve the problem that the existing technology is difficult to achieve high-precision fatigue recognition by fusing multiple types of driving behavior signals from the vehicle control domain and the driver assistance domain without increasing hardware costs, and the recognition effect is easily affected by lighting and occlusion conditions.

[0007] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: In a first aspect, the present invention provides a low-cost driver fatigue recognition method based on multi-domain driving behavior signal fusion, comprising: Acquire steering and longitudinal control signals from the vehicle control domain, as well as lateral driving signals from the driver assistance domain, and synchronize and preprocess the acquired signals. Frequency domain analysis was performed on the preprocessed steering control signal to obtain steering control characteristics; the continuous variation of the preprocessed longitudinal control signal was analyzed to obtain longitudinal control characteristics. Based on the preprocessed vehicle lateral travel signal, the changes in the vehicle's position relative to the lane and the changes in the vehicle's yaw are analyzed to obtain the lateral travel characteristics. The steering control features, longitudinal control features, and lateral driving features are fused to obtain multi-domain driving behavior features; The driver's fatigue state is identified based on the multi-domain driving behavior characteristics, and a fatigue warning is triggered based on the driver's fatigue state.

[0008] Secondly, the present invention provides a low-cost driver fatigue recognition system based on multi-domain driving behavior signal fusion, comprising a signal acquisition unit, an edge computing unit, and an early warning unit; The signal acquisition unit is connected to the vehicle bus and is used to acquire steering and longitudinal control signals in the vehicle control domain and lateral driving signals in the driver assistance domain, and send the acquired signals to the edge computing unit. The edge computing unit is used to synchronize and preprocess various received signals, perform frequency domain analysis on the preprocessed steering control signal to obtain steering control characteristics, analyze the continuous changes of the preprocessed longitudinal control signal to obtain longitudinal control characteristics, and analyze the changes in the vehicle's position relative to the lane and the changes in the vehicle's yaw based on the preprocessed vehicle lateral travel signal to obtain lateral travel characteristics. The edge computing unit is also used to fuse the steering control features, longitudinal control features and lateral driving features to obtain multi-domain driving behavior features, identify driver fatigue state based on the multi-domain driving behavior features, and generate fatigue warning trigger information based on the driver fatigue state. The early warning unit is connected to the edge computing unit and is used to execute fatigue early warning based on the fatigue early warning trigger information.

[0009] The beneficial effects of this invention are: fatigue recognition can be completed by reusing the existing steering, longitudinal and lateral driving signals in the vehicle bus without adding any vehicle hardware. This avoids to some extent the problems of visual solutions being affected by lighting and occlusion, and physiological signal solutions requiring additional sensors. At the same time, it does not collect driver images or physiological data, thus balancing all-weather working capability and privacy protection. The recognition process requires low computing resources and is easy to deploy and run on existing vehicle computing units.

[0010] Based on the above technical solution, the present invention can be further improved as follows.

[0011] Furthermore, the step of performing frequency domain analysis on the preprocessed steering control signal to obtain steering control features includes: extracting high-frequency signal components corresponding to the driver's micro-correction of steering from the preprocessed steering control signal, and performing power spectrum analysis on the high-frequency signal components to obtain the energy proportion of the high-frequency signal components in the steering control signal. By comparing the changes in the energy percentage over a continuous time window, the attenuation of the steering micro-correction energy can be obtained. The steering control characteristics are formed by statistically analyzing the torque zero-crossing and low-torque duration of the steering control signal within each time window, and combining the energy ratio, the attenuation of steering micro-correction energy, the torque zero-crossing and low-torque duration.

[0012] The advantage of adopting the above-mentioned further scheme is that it can extract the changing patterns of the driver's high-frequency micro-correction actions from the frequency domain energy distribution of the steering control signal. Compared with simply calculating shallow statistics such as the standard deviation of the steering angle, it can more sensitively reflect the changes in the control behavior that occur in the early stage of fatigue.

[0013] Furthermore, the analysis of the continuous changes in the preprocessed longitudinal control signal to obtain longitudinal control characteristics includes: calculating the change between adjacent sampled values ​​in the longitudinal control signal according to the sampling order to form a longitudinal control change rate sequence; Calculate the sample entropy and variance of the longitudinal manipulation rate of change sequence, and determine the magnitude of the change in the longitudinal manipulation rate of change sequence based on the difference between adjacent rates of change; The longitudinal manipulation feature is formed by combining the sample entropy, the variance of the rate of change, and the magnitude of change.

[0014] The beneficial effect of adopting the above-mentioned further scheme is that it can characterize the irregularity of longitudinal operation by using nonlinear indicators such as sample entropy, and can identify the subtle behavioral feature that pedal operation becomes unstable under fatigue.

[0015] Furthermore, the step of analyzing the vehicle's position change relative to the lane and the vehicle's yaw change based on the preprocessed vehicle lateral driving signal to obtain lateral driving characteristics includes: the vehicle lateral driving signal includes the distance between the left and right lane lines, the yaw rate signal, and the lane departure signal. The lateral offset of the vehicle relative to the center of the lane is determined based on the distance between the left and right lane lines, and the standard deviation of the lateral offset is calculated. The variance of the yaw rate is calculated, and the number of lane departure events is determined based on the lane departure signal. The lateral driving characteristics are formed by combining the standard deviation of the lateral offset, the variance of the yaw rate, and the number of lane departure events.

[0016] The beneficial effect of adopting the above-mentioned further solution is that it introduces information on lane keeping and yaw motion in the assisted driving domain, which helps to compensate for the blind spots caused by relying solely on signals from the vehicle control domain and improves the accuracy of fatigue recognition.

[0017] Furthermore, the process of fusing steering control features, longitudinal control features, and lateral driving features to obtain multi-domain driving behavior features includes: acquiring the steering control features, longitudinal control features, and lateral driving features corresponding to the same time window; standardizing the steering control features, longitudinal control features, and lateral driving features according to the mean and standard deviation of each feature in the training samples; and combining the standardized features according to the time window to form the multi-domain driving behavior features containing driver steering control, longitudinal control, and vehicle lateral driving information.

[0018] The beneficial effect of adopting the above-mentioned further scheme is that it can eliminate the influence of differences in the dimensions and numerical ranges of different features to a certain extent, enabling multi-domain features to participate in subsequent fatigue state identification at the same scale and improving the reliability of the identification results.

[0019] Furthermore, the step of triggering a fatigue warning based on the driver's fatigue state includes: acquiring the driver's fatigue state corresponding to multiple consecutive time windows in chronological order, and counting the number of time windows that reach a state of severe fatigue. The number of time windows that reach a state of severe fatigue is compared with the number of consecutive time windows. When the number of time windows that reach a state of severe fatigue is the majority, the driver is determined to be in a state of severe fatigue and a fatigue warning is triggered.

[0020] The advantage of adopting the above-mentioned further solutions is that it can help avoid triggering unnecessary warnings due to momentary misjudgments in a single time window, and improve the stability of warning results.

[0021] Furthermore, after obtaining the multi-domain driving behavior features, the method further includes: establishing a driver's personal driving behavior baseline, adjusting the identification of the driver's fatigue state based on the personal driving behavior baseline, and updating the personal driving behavior baseline during subsequent driving. The method involves establishing a baseline for the driver's personal driving behavior, continuously acquiring multi-domain driving behavior features corresponding to multiple time windows during the initial driving phase after vehicle startup, and taking the driving state corresponding to the initial driving phase as the conscious driving state. The mean and standard deviation of each driving behavior characteristic within the multiple time windows are calculated respectively; The baseline of the driver's personal driving behavior under conscious driving conditions is established using the mean and standard deviation of each driving behavior characteristic.

[0022] The beneficial effect of adopting the above-mentioned further scheme is the introduction of a personal driving behavior baseline mechanism, which enables the fatigue judgment threshold to be adjusted according to individual driver differences, and helps to reduce recognition bias caused by different driving styles.

[0023] Furthermore, the identification of driver fatigue state based on personal driving behavior baseline includes: obtaining driver fatigue probability based on multi-domain driving behavior features corresponding to the current time window, and obtaining personal baseline value corresponding to steering operation features from the personal driving behavior baseline; The individual baseline value is compared with the pre-obtained mean value of crowd steering and maneuvering characteristics, and the fatigue probability judgment threshold is adjusted according to the ratio of the difference between the individual baseline value and the mean value of crowd steering and maneuvering characteristics to the standard deviation of crowd steering and maneuvering characteristics. The driver's fatigue probability is determined according to the adjusted fatigue probability determination threshold to obtain the driver's fatigue state.

[0024] The beneficial effect of adopting the above-mentioned further scheme is that it dynamically adjusts the fatigue probability judgment threshold according to the differences in individual driving style and group level, thereby reducing fatigue misjudgment caused by different individual driving habits.

[0025] Furthermore, updating the individual driving behavior baseline includes: continuously acquiring driving behavior characteristics corresponding to each time window during subsequent driving; For each driving behavior feature in the individual driving behavior baseline, the individual baseline value corresponding to the previous time window is weighted with the corresponding driving behavior feature value obtained in the current time window to obtain the individual baseline value corresponding to the current time window; The personal driving behavior baseline is updated using the personal baseline value corresponding to the current time window, and the updated personal driving behavior baseline is used for subsequent identification of driver fatigue status.

[0026] The beneficial effect of adopting the above-mentioned further solutions is that the individual driving behavior baseline is continuously updated during the driving process, adapting to changes in the driver's long-term driving habits and maintaining the adaptability of the fatigue recognition threshold. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the system architecture provided in an embodiment of the present invention; Figure 2 This is an overall flowchart of the method provided in the embodiments of the present invention; Figure 3 This is a schematic diagram of the multi-domain feature fusion and fatigue recognition results provided in an embodiment of the present invention. Detailed Implementation

[0028] The principles and features of the present invention are described below. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0029] Example 1 Combination Figure 2 As shown, this embodiment of the invention provides a low-cost driver fatigue recognition method based on multi-domain driving behavior signal fusion, which can identify and warn of driver fatigue state by relying on existing signals of the vehicle bus without increasing vehicle hardware.

[0030] The vehicle bus already contains steering wheel torque signals. Steering wheel angular velocity signal These types of steering control signals, accelerator pedal opening signals This longitudinal steering signal, and the distance between the left and right lane lines. (t), yaw rate Lane departure warning signal The system continuously reads these vehicle lateral movement signals at a frequency of no less than 50Hz, and divides the time window into 60-second lengths with a 1-second sliding step, so that each window exactly covers 3000 sampling points.

[0031] Since these signals themselves contain impulse noise, the signals within the window are first subjected to median filtering to remove impulse noise using 5 sampling points as the window size. The calculation method is as follows: in, The original signal value corresponding to the nth sampling point. This represents the signal value corresponding to the nth sampling point after median filtering, where n is the index of the sampling point within the window. Median filtering replaces the original sampling point with the median value of the five nearest sampling points, which can eliminate instantaneous pulse interference on a single sampling point, and does not blur the signal edges like mean filtering.

[0032] Furthermore, median filtering can only remove isolated impulse noise; the signal also contains high-frequency interference components such as vehicle vibration. Therefore, it is further filtered by a second-order Butterworth low-pass filter with a cutoff frequency of 10Hz and a transfer function of: Where s is the Laplace operator, , The cutoff frequency is set to 10Hz. After converting the transfer function into a digital filter, the signal is filtered to effectively attenuate high-frequency interference components such as vehicle vibration above 10Hz, creating conditions for subsequent extraction of components related to driving behavior in the 0.5–5Hz range.

[0033] Different drivers exhibit individual differences in the force and amplitude of their operations. To eliminate the impact of these differences on subsequent feature extraction, the filtered signal is further normalized using Z-score, calculated as follows: Where μ is the mean of the signal within the window, and σ is the standard deviation of the signal within the window. Represents discrete sequence The version obtained after some kind of processing.

[0034] After this transformation, the signal has a mean of 0 and a variance of 1, so that subsequent feature extraction is not affected by the differences in individual driver operation amplitude, and a standardized signal that eliminates the influence of individual differences is obtained.

[0035] The names, symbols, units, and physical meanings of the above signals are shown in the table below: The above signals are all signals that originally exist in the vehicle bus and can be directly acquired without the need to install any sensors. The system then continuously reads the above signals at a frequency of no less than 50Hz and divides the time window into 60-second lengths and 1-second sliding steps, so that each window exactly covers 3,000 sampling points.

[0036] Because these signals themselves contain impulse noise and high-frequency vibration interference, and there are individual differences in the force and amplitude of different drivers' operations, after the signals in the window are aligned in time, the median filter is first used with 5 sampling points as the window to remove impulse noise. Then, a second-order Butterworth low-pass filter with a cutoff frequency of 10Hz is used to attenuate high-frequency components such as vehicle vibration to below -40dB, retaining only the part that is truly related to driving behavior in the range of 0.5 to 5Hz. Finally, the mean is subtracted and the standard deviation is divided to complete the normalization, resulting in a standardized signal that eliminates the influence of individual differences.

[0037] The standardized steering signal is first bandpass filtered in the 0.5–3 Hz frequency band to separate the high-frequency components unique to the driver's micro-corrections to the steering wheel. The bandpass filter uses a fourth-order Butterworth filter with the following transfer function: in, Using the geometric center frequency, the fourth-order Butterworth bandpass filter has a low cutoff frequency of 0.5Hz and a high cutoff frequency of 3Hz. The corresponding digital filter coefficients are determined by the bilinear transform method. After filtering, the power spectral density of this component is obtained by fast Fourier transform from zero-fill to 4096 points. The ratio of this component to the total energy of 0-10Hz can be calculated. The change of the calculated ratio relative to the previous time window reflects the decay trend of this micro-correction behavior over time. At the same time, the number of times the steering wheel torque crosses zero within the window and the proportion of sampling points with an absolute torque value of less than 0.5Nm further characterize the driver's control state of the steering wheel from the perspectives of operation frequency and hand relaxation. The four statistics together constitute a 4-dimensional steering control feature.

[0038] After performing first-order difference on the standardized accelerator pedal opening signal to obtain the rate of change sequence, the sample entropy of the sequence reflects the irregularity of pedal operation, while the variance of the sequence itself and the average amplitude of the difference between adjacent rates of change reflect the overall fluctuation and instantaneous change of the operation amplitude, respectively. The three statistics together constitute a 3-dimensional longitudinal control feature.

[0039] The lateral offset of a vehicle relative to the center of the lane can be calculated based on the distance between the left and right lane lines. The standard deviation of this offset within the window and the number of times it crosses the center of the lane reflect whether the vehicle is swaying. The variance of the yaw rate signal reflects whether the steering correction has become harsh. The number of occurrences and the duration of lane departure warning events are the most direct evidence of lane departure. These statistics together constitute a 3D lateral driving characteristic.

[0040] Combination Figure 3 As shown, after the steering control features, longitudinal control features, and lateral driving features are standardized according to the mean and standard deviation of the training samples, they are concatenated into a 10-dimensional multi-domain driving behavior feature vector within the same time window. This vector is then fed into a pre-trained LightGBM classifier. The LightGBM classifier consists of multiple decision trees. Each decision tree divides the input multi-domain driving behavior feature vector into different leaf regions and assigns a weight to each leaf region. The final raw score output by the classifier is obtained by weighted summation of the output values ​​of each decision tree. in, This is the standardized multi-domain driving behavior feature vector. The number of decision trees is taken as [number] in a specific embodiment. ; The learning rate is used to control the contribution of each decision tree to the final output; in a specific implementation, it is set to... ; For the first Each decision tree provides an output value for the input feature vector. .

[0041] During the classifier training phase, the splitting method and leaf weights for each decision tree are determined by minimizing the difference between the predicted results and the true labels. The true labels of the training samples... Where 0 represents non-fatigue and 1 represents fatigue, the classifier gives the raw score for each training sample. The predicted probability is obtained by using the sigmoid function. The binary cross-entropy loss function used in training is: in, This represents the number of samples in each batch during training. For the first The true labels of each training sample This represents the predicted probability given by the classifier for the i-th training sample, where i refers to the index of the training sample.

[0042] During training, gradient boosting is used, and each new decision tree aims to fit the negative gradient of the current model to the loss function, thereby gradually reducing the loss function value.

[0043] The data used for classifier training was obtained by collecting vehicle bus signals from more than 50 drivers driving on real roads. Data was collected for no less than 30 minutes for each driver in three states: alert, mild fatigue, and severe fatigue. The fatigue level labels for each time window were labeled by experts based on video playback of the driving process and subjective fatigue scales filled out by the drivers themselves. On this basis, a 5-fold cross-validation method was used to optimize hyperparameters such as the number of decision trees and the learning rate. The trained LightGBM classifier was converted into C code and integrated into the domain controller software for operation.

[0044] After training, the classifier outputs predicted probabilities for the multi-domain driving behavior feature vectors of the current time window in the same way during actual use. That is, the probability of driver fatigue. .

[0045] in accordance with Depending on which interval the driver falls into, the current driving status within the time window is determined as one of three levels: alert, mildly fatigued, or severely fatigued, thus obtaining the driver's fatigue status.

[0046] Considering that there may be instantaneous errors in the judgment of a single window, the driver is finally determined to be in a state of severe fatigue and a fatigue warning is triggered to remind the driver only when the number of windows that are judged as severely fatigued in multiple consecutive time windows is in the majority.

[0047] Multi-domain driving behavior feature recognition of driver fatigue state is achieved by inputting the multi-domain driving behavior features into a pre-trained LightGBM classifier.

[0048] The LightGBM classifier takes standardized multi-domain driving behavior features as input and outputs a fatigue probability score between 0 and 1. The closer the score is to 1, the closer the driver's current driving behavior pattern is to the driving behavior pattern under fatigue.

[0049] Based on the fatigue probability score range, the driver's fatigue state is divided into three levels: alert, mild fatigue, and severe fatigue. In a specific embodiment, when the fatigue probability score is less than 0.3, the driver is determined to be alert. When the fatigue probability score is greater than or equal to 0.3 and less than 0.7, the driver is determined to be in a state of mild fatigue. When the fatigue probability score is greater than or equal to 0.7, the driver is determined to be in a state of severe fatigue.

[0050] The two boundary thresholds of 0.3 and 0.7 were determined based on the statistical distribution of fatigue probability scores corresponding to the samples labeled as mild and severe fatigue by experts in the training samples, so that the three intervals after division maintain a high degree of consistency with the clear, mild fatigue and severe fatigue samples labeled by experts.

[0051] The fatigue probability score is output by the classifier by integrating multi-domain driving behavior features, avoiding the one-sidedness of judging fatigue state based on a single feature or a single threshold. The three-level classification method also provides a basis for distinguishing different fatigue levels and taking different countermeasures. Compared with the existing scheme that only outputs the binary classification result of "fatigue / non-fatigue", it can reflect the development level of driver fatigue state in more detail.

[0052] This embodiment does not require any additional onboard hardware. It utilizes existing steering, longitudinal, and lateral driving signals from the vehicle's bus to perform fatigue recognition, facilitating large-scale vehicle deployment. Furthermore, it does not rely on cameras to capture driver facial images during the recognition process, thus remaining unaffected by lighting conditions or occlusion. It can operate stably even in complex conditions such as nighttime, rain, and fog. The absence of facial image or physiological data collection also mitigates driver privacy concerns to some extent. In addition, features such as the high-frequency micro-correction energy spectrum of steering, the sample entropy of throttle change rate, and the variance of lane departure and yaw integrate information from both the vehicle control domain and the driver assistance domain. Compared to existing solutions that rely solely on shallow statistics such as the standard deviation of a single steering angle, this approach more accurately captures subtle changes in driving behavior under fatigue. The LightGBM classifier used in the recognition process is lightweight and can be directly deployed on existing onboard domain controllers for real-time operation.

[0053] Preferably, in the embodiment, the extraction of steering control features is achieved by performing frequency domain analysis on the preprocessed steering control signal: the signal is bandpass filtered in the 0.5-3Hz frequency band to extract the high-frequency signal components corresponding to the driver's micro-steering corrections, and then the components are subjected to a fast Fourier transform and zero-padded to 4096 points to improve frequency resolution, thus obtaining the transform result. However, due to It is only a frequency domain amplitude, which cannot directly reflect the energy distribution. It needs to be converted into a one-sided power spectral density first. The conversion relationship is as follows: in, Sampling frequency, The total number of points used for the Fast Fourier Transform after zero-padding of the signal is 4096; k is the index of the k-th spectral line in the Fast Fourier Transform result, and k increments sequentially from 0; f is the actual frequency corresponding to the k-th spectral line. It is obtained by converting from the serial number k.

[0054] In the calculation Then, by integrating over the 0.5–3 Hz band and the 0–10 Hz band respectively, and then dividing the former by the latter, we can obtain the ratio of the high-frequency signal component to the total energy. : in, This reflects the energy distribution within a single time window. To further observe the trend of this distribution over time, it is necessary to use the current window's... Subtract the previous window , and then remove the previous window The decay of steering micro-correction energy was obtained. Specifically: in, Let t represent the time corresponding to the previous time window, t represent the time corresponding to the current time window, and ε represent a small quantity to avoid division by zero. A negative value and the larger the absolute value, the faster the fatigue intensifies.

[0055] Besides frequency domain energy, the time domain changes of the steering wheel torque signal itself can also provide a basis for fatigue judgment. Each time the torque signal changes from positive to negative or from negative to positive within the window, it is counted as a zero crossing, and the judgment condition is as follows: The number of zero-crossings is obtained by counting the total number of sampling points within the window that meet this condition. n is the index of the sampling point within the window; In addition to the frequency of zero-torque crossings, the prolonged low torque also reflects driver relaxation, as indicated by the percentage of zero torque duration. To measure, the expression is: in, The zero torque threshold is set to 0.5 Nm, where N is the total number of sampling points within the window. This represents the proportion of sampling points whose absolute torque value is less than this threshold out of the total number of sampling points.

[0056] Based on energy percentage Attenuation Zero crossing times and low torque percentage The four statistical measures together constitute the steering control characteristics.

[0057] Furthermore, when a driver is awake, the frequency of steering wheel corrections typically falls within the 0.5–3 Hz range with small amplitudes. However, after fatigue, these high-frequency corrections decrease and are replaced by significant low-frequency corrections. The power spectrum energy ratio and its attenuation directly reflect this trend. The number of torque zero crossings typically reaches 70–100 times per minute when the driver is awake, but decreases significantly after fatigue. The increase in the proportion of low torque corresponds to situations where the driver's hands relax or briefly leave the steering wheel. These four statistics, from the perspectives of frequency domain energy, time domain rate of change, and operation frequency, jointly characterize the driver's control quality over the steering wheel. Compared to existing schemes that only calculate shallow statistics such as the standard deviation of the steering angle, this approach can more sensitively reflect changes in driving behavior that occur in the early stages of fatigue.

[0058] Preferably, in this embodiment, the extraction of longitudinal control features is achieved by performing continuous change analysis on the preprocessed longitudinal control signal. The irregularity of throttle operation is primarily reflected in the rate of change of throttle opening. Therefore, by subtracting adjacent sample values ​​in the preprocessed signal one by one, a longitudinal control change rate sequence a'(n) reflecting the rate of change of throttle opening is obtained, defined as: a′ in, Let be the throttle opening value of the nth sampling point after preprocessing, N be the total number of sampling points in the window, and a'(n) be the change in throttle opening between two adjacent sampling points.

[0059] After obtaining the rate of change series, the first step is to measure the overall volatility of the series from the perspective of amplitude, i.e., the variance of the rate of change. This is used to reflect whether the throttle operation is stable, and it is measured as follows: in, This is the mean of the sequence.

[0060] Since variance only reflects the overall fluctuation of the rate of change and cannot reveal whether there is a sudden acceleration or deceleration between two adjacent operations, the jerk factor (Jerk) is further calculated to reflect the degree of instantaneous abrupt change between adjacent rates of change. This is statistically analyzed using the following formula: Although variance and jerkiness can reflect the fluctuations in throttle operation from the perspective of amplitude, both are essentially linear statistics and cannot reflect whether the rate of change sequence itself has regularity. In the case of fatigue, the driver's pedal operation often shows a deeper irregularity. Therefore, it is necessary to introduce sample entropy, a non-linear indicator, to further characterize the degree of irregularity of operation from the perspective of sequence complexity.

[0061] When calculating sample entropy, the rate of change sequence needs to be transformed into a comparable vector form. To do this, with an embedding dimension m=2, the rate of change sequence is traversed point by point along the same N to construct an m-dimensional vector, where m is the embedding dimension of the vector. Starting from the i-th sampling point in the rate of change sequence, the construction method is as follows: Having these vectors, we need to measure the similarity between any two vectors. To do this, we take the maximum difference between their corresponding components as the distance between them, i.e., the Chebyshev distance: Where j = 1, 2, ..., N−m, and j ≠ i, j is another m-dimensional vector in the rate of change sequence. The starting sampling point number is k, and the component number inside the vector is k=0,1,...,m-1. It is used to compare the component differences at corresponding positions of two vectors in turn, and take the maximum value as the Chebyshev distance between the two vectors.

[0062] After establishing a distance metric, a standard for judging similarity is needed. Therefore, a similarity tolerance *r* is given, which is taken as the standard deviation of the rate of change sequence. 0.2 times, for each vector, count the number of other vectors whose distance to it does not exceed r. Then, by averaging over all i, we obtain the probability that any two vectors in m-dimensional space are similar. : in, This refers to the i-th vector in the rate of change sequence. The corresponding number of similar vectors, that is, excluding those within the window. The number of other m-dimensional vectors whose Chebyshev distance does not exceed the similarity tolerance r, besides itself.

[0063] Since the similarity probability under a single dimension is insufficient to reflect the trend of sequence complexity changes, the embedding dimension needs to be increased to [missing value]. Repeat the above process of constructing vectors, calculating distances, and counting similarities in the same way to obtain... Then, by comparing the similarity probabilities in the two dimensions, the sample entropy can be obtained. The calculation formula is: and Let represent the probabilities of similarity between any two vectors in m-dimensional and m+1-dimensional spaces, respectively. If the sequence itself has a regularity, then two similar vectors in m-dimensional space will still be highly likely to remain similar after taking an additional sampling point and expanding to m+1 dimensions. and The ratio will then approach 1. After calculation Approaching 0; conversely, if the sequence itself is nearly random, two similar vectors in m-dimensional space, when extended to m+1 dimensions, will become dissimilar because the newly added sampling points no longer follow the original pattern and are more likely to show greater differences. relatively The decline was greater, and the ratio was significantly less than 1. After calculation The value increases.

[0064] From sample entropy Variance of rate of change Together with the three Jerk statistics, the longitudinal manipulation characteristics are constituted.

[0065] Because driver control of the accelerator pedal becomes unstable under fatigue, irregularities and abrupt changes in operation increase. These irregular disturbances increase the randomness of the rate of change sequence. Sample entropy itself is not sensitive to periodic components but is very sensitive to such random disturbances. The closer the sequence is to randomness, the greater the sample entropy. Therefore, the increase in sample entropy corresponds to the intensification of fatigue. This can capture changes in operational irregularities earlier than linear statistics such as variance and jerkiness, which only reflect the magnitude of amplitude. In addition, variance and jerkiness supplement this judgment from the perspectives of overall fluctuation amplitude and instantaneous change amplitude, respectively. The combination of the three statistics can reflect changes in the complexity of the sequence and cover anomalies at the amplitude level. Compared with the processing method of relying on a single simple statistic in the existing scheme, it can more comprehensively characterize the accelerator operation characteristics under fatigue.

[0066] Preferably, in this embodiment, the extraction of lateral driving features is achieved by analyzing the changes in the vehicle's position relative to the lane and the vehicle's yaw rate on the preprocessed vehicle lateral driving signal. The vehicle lateral driving signal includes the distance to the left lane line, the distance to the right lane line, and the yaw rate signal, as well as the lane departure warning signal.

[0067] Fatigued drivers experience a decrease in lateral control precision, which directly manifests as the vehicle no longer maintaining stable centering within the lane. Therefore, the first step is to determine the vehicle's lateral offset relative to the lane center. It is calculated from the distance between the left and right lane lines: in, The distance to the right lane line corresponding to the nth sampling point. is the distance to the left lane line corresponding to the nth sampling point, and N is the total number of sampling points within the window.

[0068] exist A value greater than 0 indicates that the vehicle is veering to the right of the lane; exist A value less than 0 indicates that the vehicle is veering to the left of the lane; exist When the value is 0, it means that the vehicle is in the center of the lane.

[0069] Furthermore, since the lateral offset is only an instantaneous reading at each point and cannot directly indicate the degree of vehicle sway over the entire time window, the standard deviation is required. This measures the overall fluctuation of the offset within the window: in, Horizontal offset The mean within this window, The larger the value, the greater the lateral sway of the vehicle within the lane; this value usually increases when the vehicle is fatigued.

[0070] Furthermore, lateral offset reflects the vehicle's position change within the lane, but fatigue also manifests in the smoothness of steering correction actions. Therefore, further statistical analysis of yaw rate is necessary. Variance within the window : in, yaw rate The mean within this window.

[0071] Furthermore, a sober driver's steering corrections are typically more continuous and smooth, with no dramatic fluctuations in yaw rate. However, a fatigued driver's corrections tend to become stiff and sluggish, with more sharp peaks in yaw rate. It also increases accordingly.

[0072] Standard deviation and variance These are continuous quantities directly calculated from signals from the lane keeping camera and yaw rate sensor, and can only reflect indirect changes in the vehicle's motion state. The lane departure warning system itself already provides a clear judgment on whether the vehicle has deviated from its lane. This judgment also needs to incorporate lateral driving characteristics. The lane departure warning signal LD(n) is a 0 / 1 Boolean value; the frequency of deviation is obtained by counting the number of times LD(n) changes from 0 to 1 within the statistical window. This directly reflects the frequency of deviation events; furthermore, deviation frequency This is a direct result from the lane departure warning system. Compared to the statistics indirectly calculated from lane distance and yaw rate mentioned earlier, this is the most direct evidence of a vehicle deviating from its lane, and it measures the standard deviation of lateral deviation. yaw rate variance and deviation frequency The three statistics together constitute the characteristics of lateral driving.

[0073] Preferably, in this embodiment, the lateral offset The number of zero-crossings within the window, changing from positive to negative or from negative to positive. And the percentage of time that the lane departure warning signal is in a deviating state within the window. They can also be counted together.

[0074] Zero crossings Combined with standard deviation This allows for further differentiation between two different signs of fatigue: frequent minor corrections by the vehicle and a gradual loss of the will to correct. The duration of deviation is also important. The calculation formula is: and As a supplementary statistic to lateral offset and lane departure status, it can assist in verifying the results of fatigue judgment during specific identification.

[0075] Because fatigued drivers have reduced precision in lateral control of the vehicle, the vehicle is prone to swaying in a serpentine manner within the lane. The standard deviation of lateral deviation and the variance of yaw rate can directly reflect the changes in the amplitude of this swaying and the smoothness of steering. The frequency of deviation, as a direct judgment result given by the lane departure warning system, provides strong auxiliary evidence for fatigue status. The combination of these three statistics with information on both lane keeping and yaw motion can, to some extent, compensate for the blind spots that may occur when relying solely on vehicle control domain signals. Compared with the processing method of using only a single domain signal in existing solutions, this method can more comprehensively characterize the changes in the driver's lateral control status.

[0076] Preferably, in the embodiments, the formation of multi-domain driving behavior features is achieved by fusing steering control features, longitudinal control features, and lateral driving features. The steering control features, longitudinal control features, and lateral driving features come from three different signal sources: steering, longitudinal, and lateral, respectively. Their dimensions and numerical ranges are different. If they are directly spliced ​​together, features with larger numerical ranges will dominate in subsequent recognition and mask features with smaller numerical ranges but equally important features. Therefore, it is necessary to standardize each feature separately first.

[0077] Obtain the energy percentage of steering control features within the same time window. Attenuation Zero crossing times Low torque percentage Sample entropy in vertical manipulation features Variance of rate of change Jerk's agility and the standard deviation of lateral offset in lateral driving characteristics. Deviation frequency yaw rate variance The ten features mentioned above are arranged in sequence to form the feature vector F before fusion: Each feature in the feature vector F (i=1,2,...,10, corresponding to the ten components in the F vector) are all first calculated according to their mean in the training samples. and standard deviation To standardize, the calculation formula is as follows: in , and are the mean and standard deviation of the i-th feature in the training samples, respectively, which are fixed values ​​obtained in advance during the model training phase.

[0078] After this transformation, the standardized features It follows a distribution with a mean of 0 and a variance of 1, and the differences between different features are no longer affected by their respective absolute magnitude differences.

[0079] The standardized ten features By combining them according to the same time window, we can obtain multi-domain driving behavior features that include driver steering, longitudinal control, and vehicle lateral movement information.

[0080] Among them, steering control features, longitudinal control features, and lateral driving features reflect changes in driving behavior in three different dimensions. Each feature alone can only reflect one aspect of fatigue state. However, the standardization process eliminates the interference caused by the different dimensions and numerical ranges of each feature, allowing all ten features to participate in the subsequent fatigue state identification on the same scale. Compared with existing schemes that rely on single-domain features, the fused multi-domain driving behavior features can more comprehensively and evenly reflect the driver's comprehensive behavioral changes in the three dimensions of steering, longitudinal control, and lateral driving, thereby improving the accuracy of fatigue identification.

[0081] Preferably, in the embodiments, triggering a fatigue warning based on the driver's fatigue state is achieved by majority voting on the judgment results of multiple consecutive time windows. The fatigue state judgment given by a single time window is easily affected by accidental factors and may produce momentary misjudgments. For example, if the driver makes a large steering correction in a short period of time, it may be misjudged as severe fatigue by a single window. Therefore, the warning cannot be triggered directly based on the judgment result of a single time window, but needs to be confirmed by combining the judgment results of multiple windows before and after.

[0082] After obtaining the driver's fatigue status corresponding to multiple consecutive time windows in chronological order, the number of time windows identified as severely fatigued is first counted. This number is then compared with the total number of time windows included in the statistics. Only when the number of time windows corresponding to severely fatigued states is the driver finally determined to be severely fatigued, and a fatigue warning is triggered. In a specific embodiment, three consecutive time windows (since the windows are divided into 1-second sliding steps, three consecutive windows correspond to approximately 62 seconds of driving behavior data) can be taken as a group for statistics. If two or more of these time windows are identified as severely fatigued, i.e., a majority is reached, the driver is determined to be severely fatigued, and a fatigue warning is triggered.

[0083] The majority voting mechanism avoids triggering warnings based solely on the instantaneous judgment of a single time window by comprehensively considering the judgment results of multiple consecutive time windows. It can filter out the interference caused by occasional fluctuations in driving behavior on fatigue judgment, making the judgment results on which the warning is triggered closer to the driver's actual fatigue state over a period of time. Compared with the method of independently judging and directly triggering warnings window by window, the warning results are more stable and reliable, and reduce unnecessary false alarms to a certain extent.

[0084] Preferably, in the embodiments, establishing a driver's personal driving behavior baseline is achieved by statistically analyzing the driver's driving behavior characteristics during the initial driving phase after vehicle startup. Different drivers have individual differences in their driving habits; for example, some drivers tend to make more steering corrections when awake. If a universally applicable judgment standard is used, it is easy to misjudge these drivers. Therefore, it is necessary to first establish the driver's personal behavior baseline as a reference for subsequent identification.

[0085] After the vehicle starts, the initial driving phase is assumed to be the driver's conscious driving state. During this phase, multi-domain driving behavior features corresponding to multiple time windows are continuously acquired. In a specific embodiment, the initial driving phase takes the first 10 minutes after the vehicle starts. With a window length of 60 seconds and a sliding step of 1 second, starting from the formation of the first complete time window 60 seconds after the vehicle starts, one window is added every second thereafter, corresponding to approximately 540 consecutive time windows.

[0086] For each driving behavior feature in the multi-domain driving behavior feature set, its mean value within the above-mentioned multiple time windows is calculated and used as the central level of that feature in the conscious state: The mean alone is insufficient to characterize the normal fluctuation range of this feature under conscious conditions. Therefore, it is also necessary to calculate the standard deviation of this feature within the same period to reflect the stability of the driver's driving behavior under conscious conditions. in, (w) represents the i-th driving behavior feature value corresponding to the w-th time window, where w refers to the sequence number of the time window within the initial driving phase, w=1,2,...,600. , and are the mean and standard deviation of the i-th driving behavior feature in the initial driving phase, respectively.

[0087] By combining the mean and standard deviation of each driving behavior characteristic, a baseline of the driver's personal driving behavior under conscious driving conditions can be established.

[0088] Once a baseline of individual driving behavior is established, it is used to adjust for driver fatigue and is continuously updated during subsequent driving.

[0089] Using the driver's own behavioral characteristics in a conscious state as a baseline, compared to directly using a fixed, universal standard for the population, can more accurately reflect the driver's own driving behavior habits. This provides an individualized reference for distinguishing between the driver's fatigue state and normal driving habits, and to a certain extent reduces misjudgments caused by individual differences.

[0090] Preferably, in this embodiment, the driver fatigue status identification based on an individual's driving behavior baseline is achieved by individually adjusting the fatigue probability judgment threshold. The aforementioned fatigue probability is a score between 0 and 1 output by the LightGBM classifier based on the multi-domain driving behavior features corresponding to the current time window. This score is used to determine whether the driver is awake, mildly fatigued, or severely fatigued. The boundary threshold between awake and mildly fatigued is 0.3, and the boundary threshold between mildly fatigued and severely fatigued is 0.7. Since different drivers exhibit different driving behaviors when awake, applying the same judgment threshold to all drivers could easily lead to misjudgments of drivers whose driving behaviors fluctuate significantly or minimally. Therefore, it is necessary to adjust the judgment threshold based on the driver's individual baseline level.

[0091] The individual baseline value corresponding to the steering handling characteristics is obtained from the driver's individual driving behavior baseline. In a specific embodiment, the proportion of high-frequency steering energy is taken. Corresponding individual baseline mean As an individual baseline, this individual baseline is compared with the pre-obtained mean of population steering manipulation characteristics; that is, the difference between the individual baseline and the population mean is calculated. The proportion of this difference relative to the population standard deviation is used as the adjustment range to adjust the boundary threshold between mild and severe fatigue, while the boundary threshold between alertness and mild fatigue remains unchanged. The adjustment method is as follows: in, This is the default value for the boundary threshold between mild and severe fatigue, with a value of 0.7. , These represent the mean and standard deviation of the high-frequency energy proportion of pedestrian turning, respectively. These are fixed values ​​pre-calculated based on historical data from multiple drivers, and are used in specific embodiments. α is an adjustment coefficient used to control the degree of influence of individual differences on the adjustment range of the judgment threshold. In the specific embodiment, α is taken as 0.2.

[0092] If a driver's individual baseline value is higher than the population average, it indicates that the driver tends to make more steering corrections when awake, and the corresponding characteristic may show a smaller decrease when fatigued. In this case, the adjusted judgment threshold... The threshold was adjusted accordingly to avoid misjudging a driver's normal driving habits as severely fatigued, and the adjusted boundary threshold was applied. The driver fatigue probability is determined based on the multi-domain driving behavior characteristics corresponding to the current time window, thus obtaining the driver fatigue state.

[0093] Replacing a uniform fixed threshold with an individualized judgment threshold allows the driver's individual driving behavior habits in a conscious state to be incorporated into the fatigue judgment process. Compared with the existing scheme that uses the same judgment standard for all drivers, this helps to reduce false alarms and false negatives caused by differences in driving style, and also helps to improve the applicability of fatigue recognition results to different drivers.

[0094] Preferably, in this embodiment, the personal driving behavior baseline is updated by continuously correcting the baseline value using an exponentially weighted moving average. A driver's driving habits are not static; they may slowly change with increased driving time, road conditions, or improved driving proficiency. If the personal driving behavior baseline remains fixed at the value obtained during the initial driving phase, it will gradually deviate from the driver's current actual driving level. Therefore, it is necessary to continuously update the personal driving behavior baseline during subsequent driving.

[0095] During subsequent driving, driving behavior characteristics corresponding to each time window are continuously acquired. For each driving behavior characteristic in the individual driving behavior baseline, the individual baseline value corresponding to the previous time window and the corresponding driving behavior characteristic value obtained in the current time window are weighted and summed according to a fixed weight to obtain the individual baseline value corresponding to the current time window. The update method is as follows: in, Let i be the individual baseline value corresponding to the i-th driving behavior feature in the previous time window. This refers to the actual feature value obtained for this feature within the current time window. β is the weighted personal baseline value corresponding to the current time window; β is the weighting coefficient used to control the speed of updating the personal baseline value. In the specific embodiment, β=0.99, that is, the actual feature value of the current time window only accounts for a small weight, so that the personal baseline value changes slowly with driving habits, rather than changing drastically due to the accidental fluctuation of a single driving behavior.

[0096] The individual driving behavior baseline is updated using the weighted individual baseline values ​​obtained above, and the updated individual driving behavior baseline is used for subsequent identification of driver fatigue status.

[0097] By continuously updating the individual driving behavior baseline using an exponentially weighted moving average, compared to always using the baseline established in the initial driving phase, the individual driving behavior baseline can be gradually adjusted with the long-term changes in the driver's driving habits. At the same time, the larger weighting coefficient suppresses the impact of fluctuations in single driving behaviors on the baseline, thus balancing the adaptability and stability of the baseline update.

[0098] Example 2 Combination Figure 1 As shown, this embodiment of the invention provides a low-cost driver fatigue recognition system based on multi-domain driving behavior signal fusion. The system as a whole includes a signal acquisition unit, an edge computing unit, and an early warning unit.

[0099] The signal acquisition unit includes a steering wheel torque sensor, an accelerator pedal sensor, a lane-keeping camera, and a yaw rate sensor. These sensors are already configured in the vehicle and do not require additional installation. They are connected to the edge computing unit via the vehicle bus and are responsible for acquiring steering and longitudinal control signals in the vehicle control domain, as well as lateral movement signals in the driver assistance domain. Specifically, the steering wheel torque sensor acquires the steering wheel torque signal T(t) and the steering wheel angular velocity signal ω(t), the accelerator pedal sensor acquires the accelerator pedal opening signal A(t), and the lane-keeping camera acquires the distances between the left and right lane lines. (t), The yaw rate sensor collects the yaw rate signal γ(t), which together constitute the steering control signal, longitudinal control signal, and vehicle lateral travel signal. These signals are then uniformly transmitted to the edge computing unit by the signal acquisition unit. The system continuously reads these signals at a frequency of no less than 50Hz and divides the time window into 60-second lengths with a 1-second sliding step, ensuring that each window covers exactly 3000 sampling points.

[0100] The edge computing unit is deployed within the vehicle's domain controller. After receiving various signals from the signal acquisition unit, the signal preprocessing module first performs synchronization and preprocessing. Since these signals themselves contain impulse noise and high-frequency vibration interference, and there are individual differences in the force and amplitude of different drivers' operations, the signal preprocessing module first performs median filtering to remove impulse noise using 5 sampling points as a window. Then, it uses a second-order Butterworth low-pass filter with a cutoff frequency of 10Hz to attenuate high-frequency components such as vehicle vibration to below -40dB, retaining only the part that is truly related to driving behavior in the range of 0.5 to 5Hz. Finally, it subtracts the mean and divides by the standard deviation to complete normalization, obtaining a standardized signal that eliminates the influence of individual differences.

[0101] The preprocessed signal is then processed by the feature extraction module. For the steering control signal, the feature extraction module first performs a bandpass filter in the 0.5–3 Hz frequency band to separate the high-frequency component unique to the driver's micro-corrections to the steering wheel. This component is then subjected to a zero-filled Fast Fourier Transform up to 4096 points and converted into a one-sided power spectral density. The ratio of this high-frequency component to the total energy in the 0–10 Hz range is then calculated. ,as well as The magnitude of change relative to the previous time window Simultaneously, count the number of times the steering wheel torque crosses zero within the window. And the percentage of sampling points with an absolute torque value of less than 0.5 Nm. ,Depend on , , , The four statistical measures together constitute a 4-dimensional steering control feature.

[0102] For the longitudinal manipulation signal, the feature extraction module first performs a first-order difference to obtain the rate of change sequence, and then calculates the sample entropy of this sequence. ,variance The average magnitude of the difference between the rate of change and the adjacent rate of change is called the jerk (Jerk). These three statistics together constitute a 3-dimensional longitudinal manipulation feature.

[0103] For lateral travel signals, the feature extraction module calculates the lateral offset of the vehicle relative to the lane center based on the distances between the left and right lane lines, and then calculates the standard deviation of this offset within a window. Simultaneously, the variance of the yaw rate signal within the window is calculated. And the number of lane departure warning events occurring in the lane departure warning signal. ,Depend on , , The three statistical measures together constitute a 3-dimensional lateral driving characteristic.

[0104] After feature extraction, the feature fusion module standardizes the steering control features, longitudinal control features, and lateral driving features according to the mean and standard deviation of the training samples, and then concatenates them into a 10-dimensional multi-domain driving behavior feature vector within the same time window. The LightGBM classifier then processes this feature vector, accumulating the raw scores from multiple decision trees and mapping them to the driver fatigue probability using a sigmoid function. Based on the probability range, the driving state is classified into one of three levels: alert, mild fatigue, or severe fatigue. Considering the potential for instantaneous errors in the judgment of a single window, the edge computing unit further counts the number of windows classified as severely fatigued within multiple consecutive time windows in chronological order. Only when this number is the majority is the driver confirmed to be in a severely fatigued state, and a fatigue warning trigger message is generated and sent to the warning unit. The four functional modules—signal preprocessing, feature extraction, feature fusion, and the LightGBM classifier—are sequentially implemented within the edge computing unit, jointly completing the processing from the raw signal to the fatigue warning trigger message. Figure 1 As shown.

[0105] The warning unit consists of a vehicle dashboard and a speaker. After receiving the fatigue warning trigger information sent by the edge computing unit, it displays a warning prompt on the dashboard and simultaneously plays a voice reminder through the speaker, informing the driver that they are currently in a state of severe fatigue and prompting them to rest in time or take appropriate measures.

[0106] The signal acquisition unit directly reuses the vehicle's existing sensors to acquire raw signals. The edge computing unit is deployed within the existing domain controller to complete all calculations for signal processing, feature extraction, feature fusion, and fatigue recognition. The warning unit provides alerts through the vehicle's existing dashboard and speakers. The three units can work together by connecting via the vehicle bus or Ethernet. The entire system can be deployed without adding any hardware. Compared to existing solutions that require additional cameras, physiological sensors, or dedicated computing platforms, the deployment cost and modification difficulty of this system are significantly reduced, making it easy to promote and apply on a large scale in existing vehicle models.

[0107] The modules described in the embodiments of the present invention can be implemented in software or hardware. The names of the modules are not, in some cases, intended to limit the functionality of the module itself.

[0108] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned driver fatigue recognition methods. That is, an electronic device according to an embodiment of the present invention may include, but is not limited to: a processor and a memory; the memory is used to store the computer program; the processor is used to execute the driver fatigue recognition method shown in any embodiment of the present invention by calling the computer program.

[0109] In one optional embodiment, an electronic device is provided, comprising a processor and a memory. The processor and memory are connected, for example, via a bus. Optionally, the electronic device may further include a transceiver, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver is not limited to one unit, and the structure of this electronic device does not constitute a limitation on the embodiments of the present invention.

[0110] The processor can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. The processor can also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0111] A bus can include a pathway for transmitting information between the aforementioned components. The bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc.

[0112] The memory may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0113] The memory stores application code (computer program) that executes the present invention, and its execution is controlled by a processor. The processor executes the application code stored in the memory to implement the content shown in the foregoing method embodiments.

[0114] Among them, electronic devices can also be terminal devices, which can be any device that can install applications, including at least one of smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, smart TVs, and smart in-vehicle devices.

[0115] An embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-described driver fatigue recognition methods.

[0116] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0117] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the driver fatigue recognition method described above.

[0118] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0119] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation that may be implemented in the methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0120] The computer-readable storage medium provided in this invention can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EEPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0121] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0122] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

[0123] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and represent a limitation on a specific order or sequence. Where appropriate, the order of use for similar objects can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that shown or described.

[0124] Those skilled in the art will recognize that this invention can be implemented as a system, method, or computer program product. Therefore, this invention can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0125] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A low-cost driver fatigue recognition method based on multi-domain driving behavior signal fusion, characterized in that, include: Acquire steering and longitudinal control signals from the vehicle control domain, as well as lateral driving signals from the driver assistance domain, and synchronize and preprocess the acquired signals. Frequency domain analysis was performed on the preprocessed steering control signal to obtain steering control characteristics; The continuous changes of the preprocessed longitudinal control signal are analyzed to obtain the longitudinal control characteristics; Based on the preprocessed vehicle lateral travel signal, the changes in the vehicle's position relative to the lane and the changes in the vehicle's yaw are analyzed to obtain the lateral travel characteristics. The steering control features, longitudinal control features, and lateral driving features are fused to obtain multi-domain driving behavior features; The driver's fatigue state is identified based on the multi-domain driving behavior characteristics, and a fatigue warning is triggered based on the driver's fatigue state.

2. The method according to claim 1, characterized in that, The aforementioned frequency domain analysis of the preprocessed steering signal to obtain steering characteristics includes: High-frequency signal components corresponding to the driver's micro-correction of steering are extracted from the preprocessed steering control signal, and power spectrum analysis is performed on the high-frequency signal components to obtain the energy ratio of the high-frequency signal components in the steering control signal. By comparing the changes in the energy percentage over a continuous time window, the attenuation of the steering micro-correction energy can be obtained. The steering control characteristics are formed by statistically analyzing the torque zero-crossing and low-torque duration of the steering control signal within each time window, and combining the energy ratio, the attenuation of steering micro-correction energy, the torque zero-crossing and low-torque duration.

3. The method according to claim 1, characterized in that, The analysis of the continuous changes in the preprocessed longitudinal control signal yields longitudinal control characteristics, including: The change between adjacent sampled values ​​in the longitudinal manipulation signal is calculated according to the sampling order to form a longitudinal manipulation change rate sequence; Calculate the sample entropy and variance of the longitudinal manipulation rate of change sequence, and determine the magnitude of the change in the longitudinal manipulation rate of change sequence based on the difference between adjacent rates of change; The longitudinal manipulation feature is formed by combining the sample entropy, the variance of the rate of change, and the magnitude of change.

4. The method according to claim 1, characterized in that, The method of analyzing the vehicle's position changes relative to the lane and its yaw changes based on the preprocessed vehicle lateral travel signal to obtain lateral travel characteristics includes: The vehicle lateral travel signals include the distance between the left and right lane lines, the yaw rate signal, and the lane departure signal; The lateral offset of the vehicle relative to the center of the lane is determined based on the distance between the left and right lane lines, and the standard deviation of the lateral offset is calculated. The variance of the yaw rate is calculated, and the number of lane departure events is determined based on the lane departure signal. The lateral driving characteristics are formed by combining the standard deviation of the lateral offset, the variance of the yaw rate, and the number of lane departure events.

5. The method according to claim 1, characterized in that, The fusion of steering characteristics, longitudinal control characteristics, and lateral driving characteristics to obtain multi-domain driving behavior characteristics includes: Acquire the steering control characteristics, longitudinal control characteristics, and lateral driving characteristics corresponding to the same time window; The steering control feature, longitudinal control feature, and lateral driving feature are standardized based on the mean and standard deviation of each feature in the training samples. Based on the standardized features corresponding to the time window, the multi-domain driving behavior features, which include driver steering, longitudinal control, and vehicle lateral movement information, are formed.

6. The method according to claim 1, characterized in that, The aforementioned method of triggering fatigue warnings based on driver fatigue status includes: The driver fatigue status is obtained in chronological order for multiple consecutive time windows, and the number of time windows in which the driver reaches a state of severe fatigue is counted. The number of time windows that reach a state of severe fatigue is compared with the number of consecutive time windows. When the number of time windows that reach a state of severe fatigue is the majority, the driver is determined to be in a state of severe fatigue and a fatigue warning is triggered.

7. The method according to claim 1, characterized in that, After obtaining the multi-domain driving behavior characteristics, the following is also included: Establish a driver's personal driving behavior baseline, adjust the identification of driver fatigue state based on the personal driving behavior baseline, and update the personal driving behavior baseline during subsequent driving. The method involves establishing a baseline for the driver's personal driving behavior, continuously acquiring multi-domain driving behavior features corresponding to multiple time windows during the initial driving phase after vehicle startup, and taking the driving state corresponding to the initial driving phase as the conscious driving state. The mean and standard deviation of each driving behavior characteristic within the multiple time windows are calculated respectively; The baseline of the driver's personal driving behavior under conscious driving conditions is established using the mean and standard deviation of each driving behavior characteristic.

8. The method according to claim 7, characterized in that, The identification of driver fatigue state based on individual driving behavior baseline includes: The driver fatigue probability is obtained based on the multi-domain driving behavior characteristics corresponding to the current time window, and the personal baseline value corresponding to the steering control characteristics is obtained from the personal driving behavior baseline. The individual baseline value is compared with the pre-obtained mean value of crowd steering and maneuvering characteristics, and the fatigue probability judgment threshold is adjusted according to the ratio of the difference between the individual baseline value and the mean value of crowd steering and maneuvering characteristics to the standard deviation of crowd steering and maneuvering characteristics. The driver's fatigue probability is determined according to the adjusted fatigue probability determination threshold to obtain the driver's fatigue state.

9. The method according to claim 7, characterized in that, The update of the individual driving behavior baseline includes: During subsequent driving, driving behavior characteristics corresponding to each time window are continuously acquired; For each driving behavior feature in the individual driving behavior baseline, the individual baseline value corresponding to the previous time window is weighted with the corresponding driving behavior feature value obtained in the current time window to obtain the individual baseline value corresponding to the current time window; The personal driving behavior baseline is updated using the personal baseline value corresponding to the current time window, and the updated personal driving behavior baseline is used for subsequent identification of driver fatigue status.

10. A low-cost driver fatigue recognition system based on multi-domain driving behavior signal fusion, characterized in that, It includes a signal acquisition unit, an edge computing unit, and an early warning unit; The signal acquisition unit is connected to the vehicle bus and is used to acquire steering and longitudinal control signals in the vehicle control domain and lateral driving signals in the driver assistance domain, and send the acquired signals to the edge computing unit. The edge computing unit is used to synchronize and preprocess various received signals, perform frequency domain analysis on the preprocessed steering control signal to obtain steering control characteristics, analyze the continuous changes of the preprocessed longitudinal control signal to obtain longitudinal control characteristics, and analyze the changes in the vehicle's position relative to the lane and the changes in the vehicle's yaw based on the preprocessed vehicle lateral travel signal to obtain lateral travel characteristics. The edge computing unit is also used to fuse the steering control features, longitudinal control features and lateral driving features to obtain multi-domain driving behavior features, identify driver fatigue state based on the multi-domain driving behavior features, and generate fatigue warning trigger information based on the driver fatigue state. The early warning unit is connected to the edge computing unit and is used to execute fatigue early warning based on the fatigue early warning trigger information.

Images