Multi-source Signal Fatigue Driving Detection Method, Device and Storage Medium

Abstract

The present application provides a multi-source signal fatigue driving detection method, device and storage medium. The method includes: obtaining the time series data of multiple types of information during driving; segmenting the time series data of multiple types of information to obtain multiple time series data segments; respectively performing quality assessment on the multiple time series data segments, and then determining, according to the data quality assessment level, the target determination models respectively matching the multiple time series data segments from multiple determination models; and respectively passing the multiple time series data segments through the respective determined target determination models for determination to finally obtain the total fatigue degree prediction result; comparing the total fatigue degree prediction result with the warning threshold to determine whether to give a warning prompt to the driver. Using a variety of data fusion methods to effectively mine the characteristics of driver data, matching determination models for data of different qualities, achieving higher detection accuracy, realizing an all-round measurement of the driver's fatigue state, and reducing driving risks through reminders.

Description

Technical Field

[0001] The present application relates to the technical field of driving state detection, and particularly to a multi-source signal fatigue driving detection method, device and storage medium. Background Art

[0002] In recent years, China's road network has been further extended and improved, and the number of motor vehicles in possession has continued to increase. Consequently, the number of motor vehicle driving accidents has also continued to rise. Among driving accidents, accidents caused by fatigue driving are not uncommon. How to effectively monitor the state change of the driver during motor vehicle driving in a timely manner and issue reminders in a timely manner has become an important safety issue.

[0003] In related technologies, the existing fatigue driving detection technologies mainly process the following three types of data: 1) Driver physiological information characteristics, such as the driver's electrocardiogram data, galvanic skin response data, etc.; 2) Driver visual information characteristics, such as the driver's facial expression, blink frequency, etc.; 3) Driver behavior information characteristics, such as the driver's steering wheel usage, etc. The above three types of data can all contain the state information of the driver during driving to a certain extent, but there are certain limitations in using a single piece of information for prediction and simulation, and the detection accuracy of fatigue driving is relatively low. Especially when using visual information characteristics that are greatly affected by light for detection, the data quality is prone to instability and the detection error is relatively large.

[0004] In related technologies, there are also solutions that use multiple pieces of information for detection, but the technology for fusing multiple types of data is still relatively simple, and the effective information in the data cannot be fully mined, which affects the prediction accuracy. Summary of the Invention

[0005] The present application provides a multi-source signal fatigue driving detection method and device to improve the fatigue driving detection accuracy. The technical solution of the present application is as follows:

[0006] In a first aspect, an embodiment of the present application provides a multi-source signal fatigue driving detection method, including:

[0007] Obtaining the time-series data of multiple types of information of the driver during driving within a preset time period;

[0008] Aligning the time axes of the time-series data of the multiple types of information and then segmenting them to obtain multiple time-series data segments;

[0009] Respectively performing quality evaluation on the multiple time-series data segments to obtain the data quality evaluation levels of the multiple time-series data segments;

[0010] According to the data quality evaluation level, determine a target determination model that matches each of the multiple time-series data segments from multiple determination models; and respectively determine the multiple time-series data segments through their respective determined target determination models to obtain multiple fatigue degree prediction results, and perform weighted calculation on the multiple fatigue degree prediction results to obtain a total fatigue degree prediction result;

[0011] Compare the total fatigue degree prediction result with a warning threshold to obtain a comparison result, and determine whether to give a warning prompt to the driver according to the comparison result.

[0012] In some implementation manners, the multiple determination models include a time-series model, a fused data feature model, and a single data feature model, and the data quality evaluation level includes high-quality data, medium-quality data, and low-quality data; the determining, according to the data quality evaluation level, a target determination model that matches each of the multiple time-series data segments from the multiple determination models includes:

[0013] If the data quality evaluation level is high-quality data, the target determination models that match each of the multiple time-series data segments determined from the multiple determination models are one of the time-series model, the fused data feature model, and the single data feature model;

[0014] If the data quality evaluation level is medium-quality data, the target determination models that match each of the multiple time-series data segments determined from the multiple determination models are the fused data feature model or the single data feature model;

[0015] If the data quality evaluation level is low-quality data, the target determination models that match each of the multiple time-series data segments determined from the multiple determination models are the single data feature model.

[0016] In some implementation manners, the time-series model is obtained by training a time-series machine learning model with a first training sample, and the first training sample includes multiple time-series data segments.

[0017] In some implementation manners, the fused data feature model is obtained by training a support vector machine model with a second training sample, and the second training sample includes effective data features extracted from the multiple time-series data segments.

[0018] In some implementations, the single - data feature model includes multiple independent single - data feature models corresponding to the multiple types of information respectively. The independent single - data feature model corresponding to each type of information is obtained by training a support vector machine model with the training samples of this type of information. Among them, the training samples of this type of information include the effective data features extracted from the time - series data corresponding to this type of information in the multiple time - series data segments. The decision output of the single - data feature model is the weighted sum of the outputs of the multiple independent single - data feature models.

[0019] In some implementations, before aligning the time axes of the time - series data of the multiple types of information and then splitting them, it further includes:

[0020] Performing Butterworth low - pass filtering on the time - series data corresponding to the steering wheel data in the multiple types of information to obtain the time - series data of the filtered steering wheel data;

[0021] Performing wavelet filtering on the time - series data corresponding to the electrocardiogram data in the multiple types of information to obtain the time - series data of the filtered electrocardiogram data;

[0022] Performing low - pass filtering or smoothing filtering on the time - series data corresponding to the galvanic skin response data in the multiple types of information to obtain the time - series data of the filtered galvanic skin response data.

[0023] In some implementations, after determining whether to give a warning prompt to the driver according to the comparison result, it further includes:

[0024] Receiving the driver's fatigue feedback information, where the fatigue feedback information is used to adjust the parameters of the corresponding decision model in the update training of the multiple decision models.

[0025] In a second aspect, an embodiment of the present application provides a multi - source signal fatigue driving detection device, including:

[0026] A real - time data acquisition module, configured to acquire the time - series data of multiple types of information of a driver during driving within a preset time period;

[0027] A data pre - processing module, configured to align the time axes of the time - series data of the multiple types of information and then split them to obtain multiple time - series data segments;

[0028] A data quality evaluation module, configured to respectively evaluate the quality of the multiple time - series data segments to obtain the data quality evaluation levels of the multiple time - series data segments respectively.

[0029] A fatigue level determination module, configured to determine, according to the data quality evaluation level, target determination models respectively matching the multiple time-series data segments from multiple determination models; and determine the multiple time-series data segments through the respective target determination models determined thereby, to obtain multiple fatigue level prediction results, and perform weighted calculation on the multiple fatigue level prediction results to obtain an overall fatigue level prediction result;

[0030] An early warning prompt module, configured to compare the overall fatigue level prediction result with an early warning threshold to obtain a comparison result, and determine whether to give an early warning prompt to the driver according to the comparison result.

[0031] In some implementation manners, the multiple determination models include a time-series model, a fused data feature model, and a single data feature model, and the data quality evaluation level includes high-quality data, medium-quality data, and low-quality data; the fatigue level determination module is specifically configured to:

[0032] If the data quality evaluation level is high-quality data, determine, from the multiple determination models, a target determination model respectively matching the multiple time-series data segments as one of the time-series model, the fused data feature model, and the single data feature model;

[0033] If the data quality evaluation level is medium-quality data, determine, from the multiple determination models, a target determination model respectively matching the multiple time-series data segments as the fused data feature model or the single data feature model;

[0034] If the data quality evaluation level is low-quality data, determine, from the multiple determination models, a target determination model respectively matching the multiple time-series data segments as the single data feature model.

[0035] In some implementation manners, the time-series model is obtained by training a time-series machine learning model with a first training sample, and the first training sample includes multiple time-series data segments.

[0036] In some implementation manners, the fused data feature model is obtained by training a support vector machine model with a second training sample, and the second training sample includes valid data features extracted from the multiple time-series data segments.

[0037] In some implementations, the single-data feature model includes a plurality of independent single-data feature models corresponding to the multiple types of information respectively, and the independent single-data feature model corresponding to each type of information is obtained by training a support vector machine model with the training samples of this type of information; wherein, the training samples of this type of information include the effective data features extracted from the time-series data corresponding to this type of information among the multiple time-series data segments; the determination output of the single-data feature model is the weighted sum of the outputs of the multiple independent single-data feature models.

[0038] In some implementations, before the data preprocessing module aligns the time axes of the time-series data of the multiple types of information and then performs segmentation, it is further configured to:

[0039] Perform Butterworth low-pass filtering on the time-series data corresponding to the steering wheel data in the multiple types of information to obtain the time-series data of the filtered steering wheel data;

[0040] Perform wavelet filtering on the time-series data corresponding to the electrocardiogram data in the multiple types of information to obtain the time-series data of the filtered electrocardiogram data;

[0041] Perform low-pass filtering or smoothing filtering on the time-series data corresponding to the galvanic skin response data in the multiple types of information to obtain the time-series data of the filtered galvanic skin response data.

[0042] In some implementations, after the warning prompt module determines whether to give a warning prompt to the driver according to the comparison result, it is further configured to:

[0043] Receive the driver's fatigue feedback information, and the fatigue feedback information is used to adjust the parameters of the corresponding determination model in the update training of the multiple determination models.

[0044] In a third aspect, an embodiment of the present application provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multi-source signal fatigue driving detection method described in the first aspect embodiment of the present application.

[0045] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium storing computer instructions, and the computer instructions are used to make a computer execute the multi-source signal fatigue driving detection method described in the first aspect embodiment of the present application.

[0046] In a fifth aspect, an embodiment of the present application provides a computer program product, including computer instructions, and when the computer instructions are executed by a processor, the steps of the multi-source signal fatigue driving detection method described in the first aspect embodiment of the present application are implemented.

[0047] The technical solutions provided by the embodiments of the present application at least bring the following beneficial effects:

[0048] By detecting the driving state through various types of data of the driver obtained in real time during driving, the detection accuracy is improved. And the obtained various types of data are segmented to obtain multiple time-series data segments, and the data quality of each of the multiple time-series data segments is evaluated respectively. According to the data quality evaluation results, different determination models are matched to determine the degree of fatigue. The multiple determination models use various data fusion methods to effectively mine the characteristics of the collected driver data, enabling the determination models to more comprehensively reflect the driving state of the driver; for data with different data qualities, the determination models are dynamically matched to make them more compatible with the data state, so as to achieve higher detection accuracy, thereby realizing a comprehensive measurement of the driver's fatigue state, timely and effectively detecting whether the driver is fatigued and giving a reminder, thereby reducing the driving risk.

[0049] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application and do not constitute an improper limitation to the present application.

[0051] Figure 1 is a flowchart of a multi-source signal fatigue driving detection method shown according to an exemplary embodiment.

[0052] Figure 2 is a flowchart of a multi-source signal fatigue driving detection method shown according to another exemplary embodiment.

[0053] Figure 3 is a block diagram of a multi-source signal fatigue driving detection device shown according to an exemplary embodiment.

[0054] Figure 4 is a block diagram of an electronic device shown according to an exemplary embodiment. DETAILED DESCRIPTION

[0055] In order to enable those of ordinary skill in the art to better understand the technical solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0056] It should be noted that the terms "first", "second", etc. in this application are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0057] Figure 1 It is a flowchart of a multi-source signal fatigue driving detection method according to an embodiment of the present application. It should be noted that the multi-source signal fatigue driving detection method of the embodiment of the present application can be applied to the multi-source signal fatigue driving detection device of the embodiment of the present application. The multi-source signal fatigue driving detection device can be configured on an electronic device. As Figure 1 shown, the multi-source signal fatigue driving detection method may include the following steps.

[0058] Step S101, obtain the time-series data of multiple types of information of the driver during driving within a preset time period.

[0059] In this embodiment, the multiple types of information may include, but are not limited to, steering wheel data, electrocardiogram data, and galvanic skin response data.

[0060] The steering wheel data, electrocardiogram data, and galvanic skin response data are collected by devices such as sensors on the vehicle. For example, based on the IMU sensor installed on the steering wheel, the steering wheel data (i.e., the steering wheel rotation data) during vehicle driving can be collected; based on the galvanic skin sensor on the steering wheel cover, the galvanic skin response data of the driver during vehicle driving can be collected; based on the electrocardiogram sensor on the steering wheel cover, the electrocardiogram data of the driver during vehicle driving can be collected.

[0061] When obtaining the data, data recording can start from the beginning of the driving behavior and continue to be collected to provide real-time and full-cycle time-series data of multiple types of information.

[0062] In this embodiment, by collecting the electrocardiogram data, galvanic skin response data, and steering wheel operation data (steering wheel angle and vehicle acceleration) of the driver during driving, multiple types of data are used to detect the driving state, improving the detection accuracy.

[0063] Step S102, align the time axes of the time-series data of multiple types of information and then segment them to obtain multiple time-series data segments.

[0064] In this embodiment, the time-series data of multiple types of information within a preset time period obtained can be segmented into equal time lengths or unequal time lengths to obtain multiple time-series data segments, which is not limited here.

[0065] It should be noted that since the sampling frequencies of the time-series data of multiple types of information are different, it is necessary to match the time-series data according to the timestamps recorded in the time-series data of each type of information, and thin out the high-frequency data to complete the accurate matching of the time points of the time-series data of multiple types of information. After aligning the time-series data of multiple types of information along the time axis, data merging can be achieved. For example, the time-series data corresponding to the aligned steering wheel data, electrocardiogram data, and galvanic skin response data are used to construct three-dimensional time-series data segments.

[0066] Step S103: Respectively perform quality evaluation on multiple time-series data segments to obtain the data quality evaluation levels of each of the multiple time-series data segments.

[0067] As a possible implementation, the data quality of the obtained time-series data segments is evaluated according to factors such as the degree of data loss and numerical rationality.

[0068] As an example, the data quality of the time-series data segments corresponding to the steering wheel data, electrocardiogram data, and galvanic skin response data is divided into three categories:

[0069] ① High-quality data refers to the data values from the three sources being all reasonable, without data loss, and forming complete time-series data. In this case, the driver holds the steering wheel with both hands in a standard grip, and the sensor can collect valid data.

[0070] ② Medium-quality data means that there are partial losses and other situations in the data from the three sources. It may be due to the driver's non-standard grip for a short time, resulting in intermittent time-series data.

[0071] ③ Low-quality data refers to situations such as large segments of data being missing or large segments of recorded values being unreasonable or invalid. In this case, the driver may have non-standard driving situations, such as holding the steering wheel with one hand, resulting in the electrocardiogram data not being recorded for a long time.

[0072] Step S104: According to the data quality evaluation levels, determine the target determination models that match each of the multiple time-series data segments from multiple determination models; and respectively pass the multiple time-series data segments through their respective determined target determination models for determination to obtain multiple fatigue degree prediction results, and perform weighted calculation on the multiple fatigue degree prediction results to obtain the total fatigue degree prediction result.

[0073] Optionally, the multiple determination models include a time-series model, a fusion data feature model, and a single data feature model.

[0074] If the data quality assessment level is high-quality data, determine, from multiple determination models, the target determination model that respectively matches multiple time-series data segments as one of a time-series model, a fused data feature model, and a single data feature model;

[0075] If the data quality assessment level is medium-quality data, determine, from multiple determination models, the target determination model that respectively matches multiple time-series data segments as a fused data feature model or a single data feature model;

[0076] If the data quality assessment level is low-quality data, determine, from multiple determination models, the target determination model that respectively matches multiple time-series data segments as a single data feature model.

[0077] That is to say, for high-quality data, since there is little data loss and high quality, any one of the three models of time-series model, fused feature model, and single data feature model can be used for evaluation. For medium-quality data, to avoid the impact of data loss on the time-series model, a fused feature model or a single data model can be used for evaluation. For low-quality data, due to large-area data loss, prediction is performed separately for each type of data, and a single data feature model can be used.

[0078] It should be noted that when determining the fatigue state of a driver within a certain time interval (i.e., within a preset time period), the data quality of different time periods within the time interval may not be the same, that is, the data quality of multiple time-series data segments may be the same or different. When they are different, multiple determination models will be used simultaneously.

[0079] It should also be noted that when calculating the weighted sum of multiple fatigue degree prediction results to obtain the total fatigue degree prediction result, if unequal-duration segmentation is used when obtaining multiple time-series data segments in step S102, for the total fatigue degree prediction result within the preset time period, weighting should be performed according to the duration ratio of the time-series data segments to obtain the total fatigue degree prediction result.

[0080] As an example, the fatigue degree prediction result may be a score value.

[0081] Step S105: Compare the total fatigue degree prediction result with the warning threshold to obtain a comparison result, and determine whether to give a warning prompt to the driver according to the comparison result.

[0082] It can be understood that by setting the warning threshold, the current fatigue state of the driver is evaluated according to the fatigue degree prediction result calculated by the determination model. According to different set warning thresholds, the sensitivity of the determination model to the driver's fatigue degree is adjusted.

[0083] Optionally, when the total fatigue level prediction result exceeds the warning threshold, it is determined that the driver is in a fatigued state, and an alarm is issued to the driver and necessary measures are taken. If the total fatigue level prediction result does not exceed the warning threshold, it is determined that the driver is in a normal driving state.

[0084] Preferably, after determining whether to give a warning prompt to the driver according to the comparison result, it further includes:

[0085] Receiving the fatigue feedback information of the driver, where the fatigue feedback information is used to adjust the parameters of the corresponding determination model in the update training of multiple determination models.

[0086] That is to say, if a misjudgment occurs, the driver can feedback relevant information, and the determination model will be iterated and adjusted according to the fatigue confirmation information feedback by the driver to achieve higher detection accuracy.

[0087] The multi-source signal fatigue driving detection method according to the embodiments of the present application detects the driving state by real-time tracking and recording multiple types of data of the driver during driving, improving the detection accuracy. And the obtained multiple types of data are segmented to obtain multiple time-series data segments, and the data quality of the multiple time-series data segments is evaluated respectively. According to the data quality evaluation results, different determination models are matched to determine the fatigue level. The multiple determination models use a variety of data fusion methods to effectively mine the characteristics of the collected driver data, enabling the determination model to more comprehensively reflect the driving state of the driver; for data with different data qualities, the determination model is dynamically matched to make it more compatible with the data state to achieve higher detection accuracy, so as to comprehensively measure the fatigue state of the driver, timely and effectively detect whether the driver is fatigued and give a reminder, thereby reducing the driving risk. The detection method of the present application can be applied to vehicles equipped with relevant sensors, or intelligent cockpit scenarios such as ships and aircraft with similar driving scenarios.

[0088] Based on the above embodiments, as Figure 2 shown, before segmenting after aligning the time axes of the time-series data of multiple types of information in step S102, it further includes:

[0089] Step S1020, filtering and denoising the time-series data of multiple types of information.

[0090] Preferably, perform Butterworth low-pass filtering on the time-series data corresponding to the steering wheel data in multiple types of information to obtain the time-series data of the filtered steering wheel data; perform wavelet filtering on the time-series data corresponding to the electrocardiogram data in multiple types of information to obtain the time-series data of the filtered electrocardiogram data; perform low-pass filtering or smoothing filtering on the time-series data corresponding to the galvanic skin response data in multiple types of information to obtain the time-series data of the filtered galvanic skin response data.

[0091] In this embodiment, a suitable filtering method is selected to filter and denoise the time-series data corresponding to multiple types of information, so as to improve the accuracy of the data.

[0092] Based on any of the above embodiments, before performing multi-source signal fatigue driving detection, multiple determination models need to be obtained first. Among them, the time-series model is obtained by training a time-series machine learning model with a first training sample, and the first training sample includes multiple time-series data segments. The training method of the time-series model includes the following steps:

[0093] (1a) Obtain the first training sample.

[0094] Obtain the historical data during vehicle driving, which includes time-series data related to the steering wheel, galvanic skin response, and electrocardiogram. Align the time-series data related to the steering wheel, galvanic skin response, and electrocardiogram along the time axis. Complete the precise matching of data time points. Combine the three types of data aligned along the time axis to construct multi-dimensional time-series data segments. As an example, the time-series data is segmented at different time granularities such as 10s, 30s, and 1min to form time-series data segments with different time granularities.

[0095] (1b) Train the time-series machine learning model with the first training sample to obtain a trained time-series model.

[0096] Input the multi-dimensional time-series data segments into a time-series machine learning model such as an RNN for training and modeling, learn the data differences between the normal state and the fatigued state of the driver, and evaluate the fatigue degree of the driver.

[0097] Among them, the fused data feature model is obtained by training a support vector machine model with a second training sample, and the second training sample includes effective data features extracted from multiple time-series data segments. The training method of the fused data feature model includes the following steps:

[0098] (2a) Obtain the second training sample.

[0099] Extract effective data features from the time-series segments related to the steering wheel data, galvanic skin response data, and electrocardiogram data respectively to form the second training sample.

[0100] For the steering wheel data, index feature values such as the mean acceleration, standard deviation of acceleration, sample entropy of acceleration, the number of large corrections of the steering wheel angle, and the mean value of the absolute value of the steering wheel angle greater than the 75th percentile can be extracted.

[0101] For the electrocardiogram data, relevant feature values of heart rate variability can be calculated, such as the mean value of NNI, the median value of NNI, and the variance of NNI.

[0102] For skin conductance data, eigenvalues such as root mean square value, mean power frequency, and median frequency can be calculated.

[0103] (2b) Train the SVM using the second training sample to obtain a trained fusion data feature model.

[0104] Based on all the valid data features within the three types of data obtained, use the SVM to train a determination model, that is, a fusion data feature model, to measure the driver's fatigue level.

[0105] Among them, the single data feature model includes multiple independent single data feature models corresponding to multiple types of information respectively. The independent single data feature model corresponding to each type of information is obtained by training a support vector machine model using the training sample of this type of information; among them, the training sample of this type of information includes the valid data features extracted from the time series data corresponding to this type of information in multiple time series data segments; the determination output of the single data feature model is the weighted sum of the outputs of multiple independent single data feature models. The training method of the single data feature model includes the following steps:

[0106] (3a) Obtain multiple training samples corresponding to multiple types of information.

[0107] Perform separate processing on each type of data, and extract valid data features, that is, extract valid data features from the time series segments related to steering wheel data, skin conductance data, and electrocardiogram data respectively, to form three training samples.

[0108] (3b) Train the SVM using multiple training samples respectively to obtain multiple trained independent single data feature models.

[0109] Train three SVMs using the three training samples corresponding to the steering wheel data, skin conductance data, and electrocardiogram data respectively to obtain three independent single data feature models. That is, construct a determination model for fatigue for each type of data respectively to comprehensively evaluate the driver's fatigue level.

[0110] After completing the construction and training of multiple determination models, during the actual driving process, continuously collect driving data and perform determination through the determination model.

[0111] Corresponding to the above detection method embodiment, Figure 3 is a block diagram of a multi-source signal fatigue driving detection device shown according to an exemplary embodiment. Refer to Figure 3 , the multi-source signal fatigue driving detection device may include: a real-time data acquisition module 301, a data preprocessing module 302, a data quality assessment module 303, a fatigue level determination module 304, and a warning prompt module 305.

[0112] Specifically, the real-time data acquisition module 301 is used to acquire the time-series data of multiple types of information of the driver during driving within a preset time period;

[0113] The data preprocessing module 302 is used to align the time axes of the time-series data of multiple types of information and then segment them to obtain multiple time-series data segments;

[0114] The data quality assessment module 303 is used to respectively perform quality assessment on multiple time-series data segments to obtain the data quality assessment levels of the multiple time-series data segments;

[0115] The fatigue degree determination module 304 is used to determine the target determination models respectively matching the multiple time-series data segments from multiple determination models according to the data quality assessment levels; and respectively determine the multiple time-series data segments through the determined target determination models to obtain multiple fatigue degree prediction results, and perform weighted calculation on the multiple fatigue degree prediction results to obtain the total fatigue degree prediction result;

[0116] The warning prompt module 305 is used to compare the total fatigue degree prediction result with the warning threshold to obtain a comparison result, and judge whether to give a warning prompt to the driver according to the comparison result.

[0117] In some implementation manners, the multiple determination models include a time-series model, a fusion data feature model, and a single data feature model, and the data quality assessment levels include high-quality data, medium-quality data, and low-quality data; the fatigue degree determination module 304 is specifically used for:

[0118] If the data quality assessment level is high-quality data, the target determination models respectively matching the multiple time-series data segments determined from the multiple determination models are one of the time-series model, the fusion data feature model, and the single data feature model;

[0119] If the data quality assessment level is medium-quality data, the target determination models respectively matching the multiple time-series data segments determined from the multiple determination models are the fusion data feature model or the single data feature model;

[0120] If the data quality assessment level is low-quality data, the target determination models respectively matching the multiple time-series data segments determined from the multiple determination models are the single data feature model.

[0121] In some implementation manners, the time-series model is obtained by training a time-series machine learning model with a first training sample, and the first training sample includes multiple time-series data segments.

[0122] In some implementations, the fused data feature model is obtained by training a support vector machine model with second training samples, where the second training samples include valid data features extracted from multiple time-series data segments.

[0123] In some implementations, the single data feature model includes multiple independent single data feature models corresponding to multiple types of information respectively. Each independent single data feature model corresponding to a type of information is obtained by training a support vector machine model with the training samples of that type of information. Among them, the training samples of that type of information include valid data features extracted from the time-series data corresponding to that type of information in multiple time-series data segments. The decision output of the single data feature model is the weighted sum of the outputs of multiple independent single data feature models.

[0124] In some implementations, before splitting the time-series data of multiple types of information after aligning the time axes, the data preprocessing module 302 is further configured to:

[0125] Perform Butterworth low-pass filtering on the time-series data corresponding to the steering wheel data in multiple types of information to obtain the time-series data of the filtered steering wheel data;

[0126] Perform wavelet filtering on the time-series data corresponding to the electrocardiogram data in multiple types of information to obtain the time-series data of the filtered electrocardiogram data;

[0127] Perform low-pass filtering or smoothing filtering on the time-series data corresponding to the galvanic skin response data in multiple types of information to obtain the time-series data of the filtered galvanic skin response data.

[0128] In some implementations, after determining whether to give a warning prompt to the driver according to the comparison result, the warning prompt module 305 is further configured to:

[0129] Receive the fatigue feedback information of the driver, where the fatigue feedback information is used to adjust the parameters of the corresponding decision model in the updated training of multiple decision models.

[0130] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0131] The multi-source signal fatigue driving detection device according to the embodiments of the present application detects the driving state by tracking and recording multiple types of data of the driver in real time during driving, improving the detection accuracy. And the multiple types of acquired data are segmented to obtain multiple time-series data segments, and the data quality of each of the multiple time-series data segments is evaluated. According to the data quality evaluation results, different determination models are matched to determine the fatigue degree. The multiple determination models use a variety of data fusion methods to effectively mine the characteristics of the collected driver data, enabling the determination models to more comprehensively reflect the driving state of the driver; for data with different data qualities, the determination models are dynamically matched to make them more compatible with the data state, so as to achieve higher detection accuracy, thereby realizing an all-round measurement of the driver's fatigue state, timely and effectively detecting whether the driver is fatigued and giving a reminder, thereby reducing the driving risk.

[0132] According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.

[0133] As Figure 4 shown, it is a block diagram of an electronic device for implementing a method for multi-source signal fatigue driving detection according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described herein and / or claimed.

[0134] As Figure 4 shown, the electronic device includes: one or more processors 401, a memory 402, and an interface for connecting each component, including a high-speed interface and a low-speed interface. Each component is interconnected using different buses and can be installed on a common motherboard or in other ways as needed. The processor can process instructions executed within the electronic device, including instructions stored in the memory or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (such as, as a server array, a group of blade servers, or a multi-processor system). Figure 4 One processor 401 is taken as an example in

[0135] The memory 402 is the non-transitory computer-readable storage medium provided by this application. Among them, the memory stores instructions executable by at least one processor, so that the at least one processor executes the multi-source signal fatigue driving detection method provided by this application. The non-transitory computer-readable storage medium of this application stores computer instructions, and these computer instructions are used to make a computer execute the multi-source signal fatigue driving detection method provided by this application.

[0136] As a non-transitory computer-readable storage medium, the memory 402 can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the multi-source signal fatigue driving detection method in the embodiments of this application (for example, Figure 3 the real-time data acquisition module 301, data preprocessing module 302, data quality evaluation module 303, fatigue degree determination module 304, and warning prompt module 305 shown in the appendix). By running the non-transitory software programs, instructions, and modules stored in the memory 402, the processor 401 executes various functional applications and data processing of the server, that is, implements the multi-source signal fatigue driving detection method in the above method embodiments.

[0137] The memory 402 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the electronic device for multi-source signal fatigue driving detection, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include non-transitory memories, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory 402 may optionally include memories remotely provided relative to the processor 401, and these remote memories can be connected to the electronic device for multi-source signal fatigue driving detection through a network. Examples of the above networks include, but are not limited to, the Internet, enterprise intranets, local area networks, mobile communication networks, and their combinations.

[0138] The electronic device for the multi-source signal fatigue driving detection method may further include: an input device 403 and an output device 404. The processor 401, the memory 402, the input device 403, and the output device 404 can be connected through a bus or other means, Figure 4 taking the connection through the bus as an example.

[0139] The input device 403 can receive input digital or character information and generate key signal inputs related to user settings and function controls of the electronic device for multi-source signal fatigue driving detection, such as input devices like touchscreens, keypads, mice, trackpads, touchpads, pointing sticks, one or more mouse buttons, trackballs, joysticks, etc. The output device 404 can include display devices, auxiliary lighting devices (e.g., LEDs), and tactile feedback devices (e.g., vibration motors), etc. The display device can include, but is not limited to, liquid crystal displays (LCDs), light-emitting diode (LED) displays, and plasma displays. In some embodiments, the display device can be a touchscreen.

[0140] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuitry, integrated circuit systems, application specific ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special or general programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] These computing programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disks, optical disks, memories, programmable logic devices (PLDs)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0142] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user; and a keyboard and a pointing device through which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback; and input from the user can be received in any form.

[0143] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area networks, wide area networks, and the Internet.

[0144] A computer system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The client - server relationship is created by computer programs running on respective computers and having a client - server relationship with each other.

[0145] In an exemplary embodiment, a computer program product is also provided, which, when the instructions in the computer program product are executed by a processor of an electronic device, enables the electronic device to execute the above - mentioned method.

[0146] It should also be noted that in the exemplary embodiments of the present invention, some methods or systems are described based on a series of steps or devices. However, the present invention is not limited to the order of the above - mentioned steps, that is, the steps can be executed in the order mentioned in the embodiments, or different from the order in the embodiments, or several steps can be executed simultaneously.

[0147] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include known common knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and embodiments are only regarded as exemplary.

[0148] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. A multi-source signal fatigue driving detection method, characterized in that, it includes: Obtaining the time-series data of multiple types of information of the driver during driving within a preset time period; Aligning the time axes of the time-series data of the multiple types of information and then segmenting them to obtain multiple time-series data segments; Respectively performing quality assessment on the multiple time-series data segments to obtain the data quality assessment levels of the multiple time-series data segments respectively; According to the data quality assessment levels, determining the target determination models respectively matching the multiple time-series data segments from multiple determination models; And respectively determining the multiple time-series data segments through the determined target determination models respectively to obtain multiple fatigue degree prediction results, and performing weighted calculation on the multiple fatigue degree prediction results to obtain the total fatigue degree prediction result; Comparing the total fatigue degree prediction result with a warning threshold to obtain a comparison result, and judging whether to give a warning prompt to the driver according to the comparison result; Wherein, the multiple determination models include a time-series model, a fused data feature model and a single data feature model, and the data quality assessment levels include high-quality data, medium-quality data and low-quality data; the determining the target determination models respectively matching the multiple time-series data segments from multiple determination models according to the data quality assessment levels includes: If the data quality assessment level is high-quality data, determining the target determination models respectively matching the multiple time-series data segments from multiple determination models as one of the time-series model, the fused data feature model and the single data feature model; If the data quality assessment level is medium-quality data, determining the target determination models respectively matching the multiple time-series data segments from multiple determination models as the fused data feature model or the single data feature model; If the data quality assessment level is low-quality data, determining the target determination models respectively matching the multiple time-series data segments from multiple determination models as the single data feature model.

2. The method according to claim 1, characterized in that, The time-series model is obtained by training a time-series machine learning model with a first training sample, and the first training sample includes multiple time-series data segments.

3. The method according to claim 1, characterized in that, The fused data feature model is obtained by training a support vector machine model with a second training sample, and the second training sample includes effective data features extracted from the multiple time-series data segments.

4. The method according to claim 1, characterized in that, The single data feature model includes multiple independent single data feature models respectively corresponding to the multiple types of information. Each independent single data feature model corresponding to a type of information is obtained by training a support vector machine model with the training sample of this type of information; wherein, the training sample of this type of information includes effective data features extracted from the time-series data corresponding to this type of information in the multiple time-series data segments; the determination output of the single data feature model is the weighted sum of the outputs of the multiple independent single data feature models.

5. The method according to claim 1, characterized in that, Before aligning the time axes of the time series data of the multiple types of information and then performing segmentation, the following steps are further included: Performing Butterworth low-pass filtering on the time series data corresponding to the steering wheel data in the multiple types of information to obtain the time series data of the filtered steering wheel data; Performing wavelet filtering on the time series data corresponding to the electrocardiogram data in the multiple types of information to obtain the time series data of the filtered electrocardiogram data; Performing low-pass filtering or smoothing filtering on the time series data corresponding to the galvanic skin response data in the multiple types of information to obtain the time series data of the filtered galvanic skin response data.

6. The method according to claim 1, wherein, after determining whether to give a warning prompt to the driver according to the comparison result, the following steps are further included: Receiving the fatigue feedback information of the driver, and the fatigue feedback information is used to adjust the parameters of the corresponding determination model in the update training of the multiple determination models.

7. A multi-source signal fatigue driving detection device, wherein, it includes: A real-time data acquisition module, configured to acquire the time series data of multiple types of information of the driver during driving within a preset time period; A data preprocessing module, configured to align the time axes of the time series data of the multiple types of information and then perform segmentation to obtain multiple time series data segments; A data quality evaluation module, configured to respectively evaluate the quality of the multiple time series data segments to obtain the data quality evaluation levels of the multiple time series data segments; A fatigue degree determination module, configured to determine, according to the data quality evaluation level, the target determination models respectively matching the multiple time series data segments from multiple determination models; And respectively passing the multiple time series data segments through the determined target determination models for determination to obtain multiple fatigue degree prediction results, and performing weighted calculation on the multiple fatigue degree prediction results to obtain a total fatigue degree prediction result; A warning prompt module, configured to compare the total fatigue degree prediction result with a warning threshold to obtain a comparison result, and determine whether to give a warning prompt to the driver according to the comparison result; wherein, the multiple determination models include a time series model, a fusion data feature model, and a single data feature model, and the data quality evaluation levels include high-quality data, medium-quality data, and low-quality data; when the fatigue degree determination module determines, according to the data quality evaluation level, the target determination models respectively matching the multiple time series data segments from multiple determination models, it is configured to: If the data quality evaluation level is high-quality data, determine, from the multiple determination models, the target determination models respectively matching the multiple time series data segments as one of the time series model, the fusion data feature model, and the single data feature model; If the data quality evaluation level is medium-quality data, determine, from the multiple determination models, the target determination models respectively matching the multiple time series data segments as the fusion data feature model or the single data feature model; If the data quality evaluation level is low-quality data, determine, from the multiple determination models, the target determination models respectively matching the multiple time series data segments as the single data feature model.

8. An electronic device, wherein, it includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute the multi-source signal fatigue driving detection method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions characterized in that the computer instructions are used to cause the computer to execute the multi-source signal fatigue driving detection method according to any one of claims 1 to 6.

Images