Fatigue driving detection method and device and storage medium

Abstract

The invention discloses a fatigue driving detection method and device and a storage medium, and the method comprises the steps: carrying out the data feature extraction of the sober driving data of a driver in a sober driving time period, and obtaining a sober driving feature; based on the similarity between the reference judgment feature and the sober driving feature, the credibility of the reference judgment feature is set; performing data feature extraction on the suspicious driving data of the driver in the suspicious driving time period to obtain suspicious driving features; the reliability of the reference judgment features is utilized, the similarity between the reference judgment features and the suspicious driving features is calculated in a weighted mode to obtain the similarity of the suspicious features, the fatigue detection result corresponding to the driver is obtained based on the similarity of the suspicious features, and the individual difference of the driver is considered. The problem that a fixed fatigue driving detection strategy is insufficient in generalization ability in different scenes is solved, the adaptation degree of the fatigue driving detection strategy and the current scene is improved, and the accuracy of fatigue driving detection in different scenes is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of state detection, in particular to a fatigue driving detection method, device and storage medium. BACKGROUND

[0002] With the development of intelligent driving technology, driver fatigue state detection has become a key link to ensure driving safety.

[0003] In the prior art, a state detection neural network model is generally trained using sample data, and then the state detection neural network model is used to detect whether a driver is in a fatigue state. However, the actual driving scene is complex and variable, and if a fixed fatigue state judgment strategy is used, the accuracy of fatigue driving judgment will be seriously affected. SUMMARY

[0004] To solve the above technical problems, the present application at least provides a fatigue driving detection method, device and storage medium.

[0005] The first aspect of the present application provides a fatigue driving detection method, which comprises: collecting behavior data of a driver in a sober driving time period to obtain sober driving data, performing data feature extraction on the sober driving data to obtain sober driving features; obtaining a preset reference judgment feature for fatigue judgment, calculating the similarity between the reference judgment feature and the sober driving features to obtain a sober feature similarity, and setting the credibility of the reference judgment feature based on the sober feature similarity; wherein the sober feature similarity and the credibility of the reference judgment feature are inversely related; collecting behavior data of the driver in a suspicious driving time period to obtain suspicious driving data, performing data feature extraction on the suspicious driving data to obtain suspicious driving features; calculating the similarity between the reference judgment feature and the suspicious driving features using the credibility of the reference judgment feature to obtain a suspicious feature similarity, and obtaining a fatigue detection result corresponding to the driver based on the suspicious feature similarity.

[0006] In an embodiment, the behavior data includes data of multiple modalities; the method further comprises: analyzing the data quality of the suspicious driving data of each modality; based on the data quality of the suspicious driving data of each modality, setting the credibility of the suspicious driving features extracted from the suspicious driving data of each modality respectively; calculating the similarity between the reference judgment feature and the suspicious driving features using the credibility of the reference judgment feature to obtain a suspicious feature similarity, and obtaining a fatigue detection result corresponding to the driver based on the suspicious feature similarity, including: calculating the similarity between the reference judgment feature and the suspicious driving features using the credibility of the reference judgment feature and the credibility of the suspicious driving features to obtain a suspicious feature similarity, and obtaining a fatigue detection result corresponding to the driver based on the suspicious feature similarity.

[0007] In an embodiment, the behavior data comprises image data and audio data, and the suspicious driving data comprises suspicious image data and suspicious audio data; the data quality of the suspicious driving data of each modality is analyzed, including: extracting the ambient light intensity from the suspicious image data, obtaining the data quality of the suspicious image data based on the ambient light intensity; calculating the speech signal-to-noise ratio of the suspicious audio data, and obtaining the data quality of the suspicious audio data based on the speech signal-to-noise ratio.

[0008] In an embodiment, the suspicious feature similarity is obtained by weighting calculation of the similarity between the benchmark judgment features and the suspicious driving features, using the confidence of the benchmark judgment features and the confidence of the suspicious driving features, including: setting the weighting weight of the benchmark judgment features based on the confidence of the benchmark judgment features; setting the weighting weight of the suspicious driving features based on the confidence of the suspicious driving features; weighting fusion is performed on each benchmark judgment feature and each suspicious driving feature respectively, using the weighting weight of each benchmark judgment feature and the weighting weight of each suspicious driving feature, to obtain the fused judgment features and the fused driving features; similarity calculation is performed on the fused judgment features and the fused driving features to obtain the suspicious feature similarity.

[0009] In an embodiment, the suspicious feature similarity is obtained by weighting calculation of the similarity between the benchmark judgment features and the suspicious driving features, using the confidence of the benchmark judgment features and the confidence of the suspicious driving features, including: calculating the similarity between the benchmark judgment features and the suspicious driving features to obtain the initial similarity corresponding to the benchmark judgment features; setting the weighting weight corresponding to the initial similarity based on the confidence of the benchmark judgment features and the confidence of the suspicious driving features; wherein the confidence of the benchmark judgment features and the confidence of the suspicious driving features are positively correlated with the weighting weight of the initial similarity; weighting sum is performed on each initial similarity using the weighting weight corresponding to each initial similarity to obtain the suspicious feature similarity.

[0010] In an embodiment, different levels of reference judgment features are preset, and the different levels of reference judgment features correspond to different feature types and different fatigue degree levels; data feature extraction is performed on the suspicious driving data to obtain suspicious driving features, including: respectively extracting data features of the feature types corresponding to the reference judgment features of each level from the suspicious driving data to respectively obtain suspicious driving features corresponding to the reference judgment features of each level; based on the credibility of the reference judgment features, weighting calculation is performed on the similarity between the reference judgment features and the suspicious driving features to obtain suspicious feature similarity, and based on the suspicious feature similarity, a fatigue detection result corresponding to the driver is obtained, including: based on the credibility of the reference judgment features of each level, weighting calculation is performed on the similarity between the reference judgment features and the suspicious driving features to obtain suspicious feature similarity; based on the suspicious feature similarity corresponding to the reference judgment features of each level, a fatigue probability corresponding to the reference judgment features of each level is obtained; the fatigue degree corresponding to the driver is obtained by combining the fatigue probability corresponding to the reference judgment features of each level and the fatigue degree level, and the fatigue degree is taken as the fatigue detection result.

[0011] In an embodiment, the suspicious driving data is composed of a plurality of data frames sorted in time sequence; data feature extraction is performed on the suspicious driving data to obtain suspicious driving features, including: using a sliding window, a preset number of data frames are extracted from the suspicious driving data to obtain a window data set; differences between adjacent data frames in the window data set are calculated to obtain time sequence features; and the time sequence features are taken as the suspicious driving features.

[0012] In an embodiment, an initial time period in which the driver drives the vehicle is taken as a sober driving time period; and a subsequent time period after the initial time period is taken as a suspicious driving time period.

[0013] The second aspect of the present application provides a fatigue driving detection device, the device comprising: a sober feature extraction module, configured to collect behavior data of a driver in a sober driving time period to obtain sober driving data, and perform data feature extraction on the sober driving data to obtain sober driving features; a credibility calculation module, configured to obtain preset reference judgment features for fatigue judgment, calculate similarity between the reference judgment features and the sober driving features to obtain sober feature similarity, and set credibility of the reference judgment features based on the sober feature similarity; wherein the sober feature similarity and the credibility of the reference judgment features are inversely related; a suspicious feature extraction module, configured to collect behavior data of the driver in a suspicious driving time period to obtain suspicious driving data, and perform data feature extraction on the suspicious driving data to obtain suspicious driving features; and a fatigue detection module, configured to use the credibility of the reference judgment features to weight calculate similarity between the reference judgment features and the suspicious driving features to obtain suspicious feature similarity, and obtain a fatigue detection result corresponding to the driver based on the suspicious feature similarity.

[0014] The third aspect of the present application provides an electronic device, comprising a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the fatigue driving detection method.

[0015] The fourth aspect of the present application provides a computer readable storage medium, having program instructions stored thereon, the program instructions being executed by a processor to implement the fatigue driving detection method.

[0016] The above scheme, by extracting the data features of the sober driving data of the driver in the sober driving time period, obtains the sober driving features; based on the similarity between the reference judgment features and the sober driving features, the reliability of the reference judgment features is set; the data features of the suspicious driving data of the driver in the suspicious driving time period are extracted to obtain the suspicious driving features; the suspicious feature similarity is obtained by using the reliability of the reference judgment features to weight the similarity between the reference judgment features and the suspicious driving features, and the fatigue detection result corresponding to the driver is obtained based on the suspicious feature similarity, considering the individual differences of the driver, solving the problem of insufficient generalization ability of the fixed fatigue driving detection strategy in different scenes, improving the adaptability of the fatigue driving detection strategy and the current scene, and improving the accuracy of fatigue driving detection in different scenes.

[0017] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, but not limiting the present application. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings incorporated in and forming a part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.

[0019] Figure 1 is a schematic diagram of a scheme implementation environment shown by an exemplary embodiment of the present application;

[0020] Figure 2 is a flowchart of a fatigue driving detection method shown by an exemplary embodiment of the present application;

[0021] Figure 3 is a schematic diagram of a fatigue driving detection method shown by an exemplary embodiment of the present application;

[0022] Figure 4 is a block diagram of a fatigue driving detection device shown by an exemplary embodiment of the present application;

[0023] Figure 5 is a structural schematic diagram of an electronic device shown by an exemplary embodiment of the present application;

[0024] Figure 6FIG. 1 is a structural schematic diagram of a computer readable storage medium according to an example embodiment of the present application. DETAILED DESCRIPTION

[0025] The scheme of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0026] In the following description, specific details are set forth in order to provide a thorough understanding of the present application. The present application may, however, be practiced without these details. In other instances, well-known methods have not been described in detail in order not to unnecessarily obscure the present application.

[0027] The term "and / or", merely describes an associated relationship between associated objects, and means that there can be three relationships, for example, A and / or B, which can mean that A exists alone, A and B exist together, and B exists alone. In addition, the character " / " in this paper generally means that the front and rear associated objects are an "or" relationship. In addition, "multiple" in this paper means two or more than two. In addition, the term "at least one" in this paper means any one of multiple or any combination of at least two of multiple, for example, including at least one of A, B and C, which can mean including any one or more elements selected from the set consisting of A, B and C.

[0028] The applicant found that different drivers have individual differences, and if a fixed fatigue driving detection strategy is used, the driver with large individual differences will have fatigue driving recognition errors.

[0029] In order to solve the above technical problems, the present application at least provides a fatigue driving detection method, device and storage medium.

[0030] The fatigue driving detection method provided by the embodiments of the present application will be described below.

[0031] Please refer to Figure 1 , Figure 1 FIG. 1 is a schematic diagram of a scheme implementation environment according to an example embodiment of the present application. The scheme implementation environment can include a vehicle 110 and a server 120, which are connected in communication with each other.

[0032] The server 120 can be a stand-alone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and big data and artificial intelligence platforms, etc. Basic cloud computing services.

[0033] In one example, the behavior data collection device deployed in the vehicle 110 collects behavior data of the driver, and sends the collected driving data (clear driving data or suspicious driving data) to the server 120. The server 120 can perform fatigue driving detection on the driving data collected from the vehicle 110 to obtain a fatigue detection result.

[0034] In one example, the vehicle 110 is installed with a client running a target application, such as an application providing fatigue driving detection function. The vehicle 110 uses the target application to perform fatigue driving detection on the collected driving data to obtain a fatigue detection result. The server 120 can be a background server of the target application, and is configured to provide background service for the client of the target application.

[0035] The fatigue driving detection method provided by the embodiments of the present application can be performed by the vehicle 110, such as the client of the target application installed and running in the vehicle 110, or the server 120, or by the vehicle 110 and the server 120 in cooperation.

[0036] Please refer to Figure 2 , Figure 2 is a flowchart of the fatigue driving detection method according to an example embodiment of the present application. The fatigue driving detection method can be applied to the implementation environment shown in Figure 1 , and is specifically performed by the server in the implementation environment. It should be understood that the method can also be applied to other example implementation environments, and is specifically performed by devices in other implementation environments. The present embodiment does not limit the implementation environment to which the method is applied.

[0037] As shown in Figure 2 , the fatigue driving detection method at least includes steps S210 to S240, which are described in detail as follows:

[0038] Step S210: Collect behavior data of the driver in the clear driving time period to obtain clear driving data, and perform data feature extraction on the clear driving data to obtain clear driving features.

[0039] The vehicle is deployed with a behavior data collection device, and the behavior data collection device is configured to collect behavior data of the driver in the clear driving time period.

[0040] The behavior data collection device includes but is not limited to an image collection device (such as a camera), and / or an audio collection device (such as a microphone array, and / or an automatic speech recognition (ASR) module, etc.), and / or a physiological data collection device, etc. The behavior data includes but is not limited to images, and / or videos, and / or audio, and / or heart rate, and / or respiratory rate, etc. The present application does not limit this.

[0041] The sober driving time period refers to a time period in which the driver is in a sober state.

[0042] For example, the initial time period in which the driver drives the vehicle can be taken as the sober driving time period. For example, the first 30 minutes in which the driver drives the vehicle are taken as the sober driving time period.

[0043] For example, a state question can be initiated to the driver at the current time, such as asking the driver whether he / she is in a sober state through voice or text, obtaining feedback information of the driver in response to the state question, determining whether the driver is in a sober state, and if it is determined according to the feedback information that the probability of the driver being in a sober state is greater than a preset probability threshold, then the current time is divided into a sober driving time period. For example, the first 15 minutes and the last 15 minutes of the current time are divided into a sober driving time period.

[0044] The behavior data of the driver collected in the sober driving time period is taken as sober driving data.

[0045] Data features of the sober driving data are extracted to obtain sober driving features.

[0046] For example, image feature extraction is performed on image data, and / or audio data extraction is performed on audio data, and / or numerical statistics are performed on heart rate, etc. to obtain sober driving features.

[0047] Step S220: Obtain a preset reference judgment feature for fatigue judgment, calculate the similarity between the reference judgment feature and the sober driving feature to obtain a sober feature similarity, and set the credibility of the reference judgment feature based on the sober feature similarity; wherein the sober feature similarity and the credibility of the reference judgment feature are inversely related.

[0048] The reference judgment feature is obtained, and the reference judgment feature is used to judge whether the driver is in a fatigue state.

[0049] The number of reference judgment features is one or more. For example, the reference judgment features include reference eye features, and / or reference mouth features, and / or reference action posture features, and / or reference voice features, etc.

[0050] Exemplarily, the reference judgment feature can be obtained by performing feature statistics on sample data. For example, the sample data includes sample images and / or sample audios, which are image data and / or audio data of the target person in a fatigue state. By extracting eye features, and / or mouth features, and / or action posture features, and / or voice features, etc. of the target person in the sample data, sample features corresponding to each sample data are obtained. Then, the extracted sample features are weighted and averaged, and the weighted and averaged result is taken as the reference judgment feature. For example, all eye features are weighted and averaged, and the weighted and averaged result is taken as the reference eye feature, and / or all mouth features are weighted and averaged, and the weighted and averaged result is taken as the reference mouth feature.

[0051] A similarity between the reference judgment feature and the sober driving feature is calculated, and the calculated similarity is taken as the sober feature similarity.

[0052] The similarity between the reference judgment feature and the sober driving feature can be a cosine similarity, a Euclidean distance, a Manhattan distance, etc. between the reference judgment feature and the sober driving feature.

[0053] Optionally, in order to facilitate feature similarity calculation, a feature type to which the reference judgment feature belongs can be determined, and when performing data feature extraction on the sober driving data, sober driving features of the same feature type are extracted, so as to ensure that the reference judgment feature and the sober driving feature used for similarity calculation belong to the same feature type. For example, if the reference judgment feature is a reference eye feature, the sober driving feature is an eye feature extracted from the sober driving data, and if the reference judgment feature is a reference mouth feature, the sober driving feature is a mouth feature extracted from the sober driving data.

[0054] Optionally, in order to facilitate feature similarity calculation, the same feature extraction network can also be used to extract features from the sample data and the sober driving data, respectively, to obtain the reference judgment feature and the sober driving feature.

[0055] The reference judgment feature is set according to the sober feature similarity.

[0056] Specifically, the higher the sober feature similarity, the closer the driving behavior feature of the current driver in a sober state is to the reference judgment feature, and the lower the reliability of using the reference judgment feature to detect fatigue driving of the driver; on the contrary, the lower the sober feature similarity, the less close the driving behavior feature of the current driver in a sober state is to the reference judgment feature, and the higher the reliability of using the reference judgment feature to detect fatigue driving of the driver.

[0057] For example, the reference judgment feature includes a reference eye feature, which is obtained by extracting an eye image with high eye closure degree in sample data, and an eye image of the current driver with high eye closure degree is extracted from the sober driving data, and the sober eye feature is obtained by encoding the eye image of the current driver. If the similarity between the reference eye feature and the sober eye feature is high, it indicates that the driver also has high eye closure degree in the sober state, and the reference eye feature is set to be low in reliability relative to the driver.

[0058] Step S230: Collecting the behavior data of the driver in the suspicious driving time period to obtain suspicious driving data, and extracting the data features of the suspicious driving data to obtain suspicious driving features.

[0059] The suspicious driving time period refers to the time period in which the state of the driver needs to be judged.

[0060] For example, the subsequent time period after the initial time period of the driver driving the vehicle can be taken as the suspicious driving time period. For example, after the duration of the driver driving the vehicle exceeds 30 minutes, the subsequent time period is taken as the suspicious driving time period.

[0061] For example, a state question can be initiated to the driver at the current time, such as asking the driver whether he is in a sober state through voice or text, obtaining the feedback information of the driver to the state question, determining whether the driver is in a sober state, and if the probability of the driver being in a sober state is less than a preset probability threshold according to the feedback information, the suspicious driving time period is divided according to the current time. For example, the time after the current time is divided into suspicious driving time periods.

[0062] The behavior data of the driver collected in the suspicious driving time period is taken as the suspicious driving data.

[0063] The data features of the suspicious driving data are extracted to obtain the suspicious driving features.

[0064] For example, image feature extraction is performed on the image data, and / or audio data extraction is performed on the audio data, and / or numerical statistics are performed on the heart rate, etc. to obtain the suspicious driving features.

[0065] Step S240: Using the reliability of the reference judgment feature, the similarity between the reference judgment feature and the suspicious driving feature is weighted to obtain the suspicious feature similarity, and the fatigue detection result corresponding to the driver is obtained based on the suspicious feature similarity.

[0066] According to the credibility of the benchmark judgment feature, a weighted weight of the benchmark judgment feature in the process of similarity calculation with the suspicious driving feature is determined, and the credibility of the benchmark judgment feature is positively correlated with the weighted weight of the benchmark judgment feature.

[0067] Specifically, the higher the credibility of the benchmark judgment feature, the higher the weighted weight of the benchmark judgment feature in the process of similarity calculation with the suspicious driving feature, and vice versa, the lower the credibility of the benchmark judgment feature, the lower the weighted weight of the benchmark judgment feature in the process of similarity calculation with the suspicious driving feature.

[0068] Then, according to the suspicious feature similarity between the benchmark judgment feature and the suspicious driving feature calculated by weighting, a fatigue detection result corresponding to the driver is obtained.

[0069] For example, the fatigue detection result is used to indicate whether the driver is in a fatigue state. If the suspicious feature similarity is greater than a preset similarity threshold, it indicates that the driver is in a fatigue state. If the suspicious feature similarity is not greater than the preset similarity threshold, it indicates that the driver is not in a fatigue state.

[0070] For another example, the fatigue detection result is used to indicate whether the driver is in a fatigue state. The suspicious feature similarity calculated at a historical time is obtained, and an increase value of the suspicious feature similarity at a current time relative to the suspicious feature similarity at the historical time is calculated. If the increase value is greater than a preset increase threshold or the suspicious feature similarity is greater than a preset similarity threshold, it indicates that the driver is in a fatigue state. If the increase value is not greater than the preset increase threshold and the suspicious feature similarity is not greater than the preset similarity threshold, it indicates that the driver is not in a fatigue state.

[0071] For another example, the fatigue detection result is used to indicate the fatigue degree of the driver. The suspicious feature similarity is numerically mapped by using a preset formula to obtain the fatigue degree of the driver, wherein the preset formula is used to define a functional relationship between the suspicious feature similarity and the fatigue degree, and the suspicious feature similarity and the fatigue degree are positively correlated, that is, the higher the suspicious feature similarity, the higher the fatigue degree, and vice versa, the lower the suspicious feature similarity, the lower the fatigue degree.

[0072] The specific calculation method of the fatigue detection result can be flexibly selected according to the actual application scene, and the present application does not limit it.

[0073] Optionally, in order to facilitate the feature similarity calculation, the feature type to which the reference judgment feature belongs can be determined, and when the suspicious driving data is subjected to data feature extraction, the suspicious driving features of the same feature type are extracted to ensure that the reference judgment features and the suspicious driving features belong to the same feature type when the similarity calculation is performed. For example, if the reference judgment feature is a reference eye feature, the suspicious driving feature is an eye feature extracted from the suspicious driving data, and if the reference judgment feature is a reference mouth feature, the suspicious driving feature is a mouth feature extracted from the suspicious driving data.

[0074] Optionally, in order to facilitate the feature similarity calculation, the same feature extraction network can also be used to extract features from the sample data and the suspicious driving data to obtain the reference judgment features and the suspicious driving features, respectively.

[0075] For example, please refer to Figure 3 , Figure 3 is a schematic diagram of a fatigue driving detection method according to an example embodiment of the present application, as shown in Figure 3 , t1 time is detected to start driving the vehicle, t1 time to t2 time is taken as the sober driving time period, the behavior data of the driver in the sober driving time period is collected to obtain the sober driving data. The reference judgment features obtained have N, and the data features of the same feature type as the reference judgment features are extracted from the sober driving data, such as mouth features, eye features, posture features, voice features, etc., to obtain N sober driving features. The reference judgment features and the sober driving features of the same feature type are subjected to similarity calculation respectively to obtain the sober feature similarity, and the confidence of each reference judgment feature is set according to the sober feature similarity.

[0076] The time period after t2 time is taken as the suspicious driving time period, the behavior data of the driver in the suspicious driving time period is collected to obtain the suspicious driving data, and the data features of the same feature type as the reference judgment features are extracted from the suspicious driving data to obtain N suspicious driving features. Then, the similarity between the reference judgment features and the suspicious driving features of the same feature type is calculated according to the confidence of each reference judgment feature to obtain the suspicious feature similarity, and the fatigue detection result corresponding to the driver is obtained according to the suspicious feature similarity.

[0077] The application obtains a sober feature similarity by calculating the similarity between the benchmark judgment feature and the sober driving feature, sets the reliability of the benchmark judgment feature based on the sober feature similarity, obtains a suspicious feature similarity by weighted calculation of the similarity between the benchmark judgment feature and the suspicious driving feature using the reliability of the benchmark judgment feature, and obtains the fatigue detection result of the driver based on the suspicious feature similarity. The individual differences of the driver are considered, the problem of insufficient generalization ability of the fixed fatigue driving detection strategy in different scenes is solved, the adaptation of the fatigue driving detection strategy to the current scene is improved, and the accuracy of fatigue driving detection in different scenes is improved.

[0078] Next, some embodiments of the application will be described in detail.

[0079] In some embodiments, the behavior data includes image data and audio data. A continuous video stream is collected by a camera, and a sequence of RGB image frames collected is taken as image data to be analyzed; an audio stream is collected by a microphone array, and a speech signal output after beamforming and noise reduction is taken as audio data to be analyzed.

[0080] Then, the image data and the audio data timestamps are aligned using a Precision Time Protocol (PTP) protocol to generate image data and audio data with unified timestamps.

[0081] In addition, the image data is preprocessed, and the preprocessing methods include but are not limited to adaptive histogram equalization, face detection and key point positioning, affine transformation alignment of the face region, etc., to obtain a sequence of face regions of interest (ROI).

[0082] In addition, the audio data is preprocessed, and the preprocessing methods include but are not limited to converting the audio to text data using ASR, word segmentation and stop word filtering of the text data output by the ASR, keyword extraction based on a domain dictionary (such as fatigue-related keywords "drowsy" and "yawn"), to obtain a sequence of text keywords.

[0083] Based on the above embodiments, the sober driving data and / or suspicious driving data are preprocessed to obtain a sequence of face regions of interest and a sequence of keywords corresponding to the sober driving data and / or suspicious driving data, and data feature extraction is performed on the sequence of face regions of interest and the sequence of keywords to obtain sober driving features and / or suspicious driving features.

[0084] In some embodiments, the suspicious driving data is composed of a plurality of data frames in chronological order; the data feature extraction on the suspicious driving data in step S230 obtains suspicious driving features, including the following steps S231 to S233.

[0085] Step S231: a sliding window is used to extract a preset number of data frames from the suspicious driving data to obtain a window data set.

[0086] The sliding window is used to limit the number of selected data frames.

[0087] For example, every 5 data frames are taken as a sliding window, and 5 data frames are extracted from the suspicious driving data each time to obtain a window data set.

[0088] For each data frame, the sliding window is used to select adjacent data frames, that is, each data frame corresponds to a window data set.

[0089] The data frame can be an image frame, an audio frame, etc., which is not limited in the present application.

[0090] Step S232: calculate the difference between adjacent data frames in the window data set to obtain a time sequence feature.

[0091] For example, the difference between adjacent data frames in the window data set is directly calculated, and the difference calculation result is taken as the difference between adjacent data frames to obtain the time sequence feature.

[0092] For another example, data feature extraction is performed on each data frame in the window data set to obtain a feature vector of each data frame, the feature vectors of adjacent data frames are calculated, and the difference calculation result is taken as the difference between adjacent data frames to obtain the time sequence feature.

[0093] The above difference calculation can use first-order difference or second-order difference, and the specific calculation method of the time sequence feature can be flexibly selected according to the actual application scene, which is not limited in the present application.

[0094] Step S233: take the time sequence feature as the suspicious driving feature.

[0095] Taking suspicious driving data including image data as an example, the image data is composed of a plurality of image frames sorted in time sequence, and a time-domain difference convolution (TDC) is used to capture the time sequence features corresponding to each image frame, and the difference information between adjacent image frames is particularly focused on, such as eye movement, head posture change, etc., and the fatigue-related actions such as slow blinking and nodding frequency are highlighted through frame difference. For example, for a feature sequence (T frames) of a single image frame, the time sequence feature corresponding to the image frame is output as a matrix (512*T). Wherein, the image frame can be an original image frame, or an image frame obtained after preprocessing, such as a face region of interest obtained after preprocessing, which is not limited in the present application.

[0096] Of course, the time sequence features of other types of data can also be extracted, such as the time sequence features of audio data, and in addition to using TDC to extract time sequence features, other algorithms can also be used to extract time sequence features, such as using a three-dimensional convolutional neural network (3D CNN) to extract time sequence features, which is not limited in the present application.

[0097] The above embodiments analyze the behavior data by mining time sequence features, deeply mine the characteristics of long-time fatigue behavior, and improve the detection accuracy of long-time fatigue behavior. In addition, the time sequence feature mining can distinguish between transient fatigue state and persistent fatigue state, and improve the robustness and practicality of fatigue driving detection.

[0098] In addition to extracting time sequence features in the above embodiments, other types of features can also be extracted, such as spatial features and frequency domain features of data frames.

[0099] For example, a face region of interest sequence is input into an EfficientNetV2-S backbone network, a TDC module, and a non-local attention module to extract visual features, and a 512-dimensional spatio-temporal joint feature vector is output, which contains spatial features and time sequence motion information of image data. The spatio-temporal joint feature vector is used as a sober driving feature or a suspicious driving feature. Wherein, the non-local attention module can enhance the long-range spatial dependence relationship of the key facial region and suppress background noise, and EfficientNetV2-S can be replaced by ResNet50 (Residual Network 50-layer) and a temporal shift module (TSM), which is not limited in the present application.

[0100] For example, the text keyword sequence is input into a DistilBERT (Distilled Bidirectional Encoder Representation from Transformers) pre-training model, a bidirectional long short-term memory network (BiLSTM), a term frequency-inverse document frequency (TF-IDF), and a weighted attention module to extract text features, and a 256-dimensional semantic feature vector containing context-aware fatigue semantic information is output, which is used as a sober driving feature or a suspicious driving feature.

[0101] Specifically, the text keyword sequence is mapped to a 768-dimensional word vector sequence by DistilBERT, the context semantics are captured by BiLSTM according to the word vector sequence, and a 256-dimensional time sequence feature is output. Then, based on the pre-defined TF-IDF weight of the fatigue keyword, the weighted attention module is used to weight and sum the BiLSTM output, and the final semantic feature vector is obtained by normalization.

[0102] In this application, DistilBERT can be replaced by FastText, and TF-IDF weighting can be replaced by self-attention (Self-Attention) mechanism, which is not limited in the application.

[0103] In some embodiments, the behavior data includes data of multiple modalities; the method further includes steps S301 to S302.

[0104] Step S301: Analyze the data quality of suspicious driving data of each modality.

[0105] The quality of behavior data collected in different environments is different. For example, the low light in rainy weather may affect the quality of collected image data, and the high noise in the environment may affect the quality of collected audio data.

[0106] For example, the behavior data includes image data and audio data, and the suspicious driving data includes suspicious image data and suspicious audio data. Analyzing the data quality of suspicious driving data of each modality includes: extracting the environmental light intensity from the suspicious image data, and obtaining the data quality of the suspicious image data based on the environmental light intensity; calculating the speech signal-to-noise ratio of the suspicious audio data, and obtaining the data quality of the suspicious audio data based on the speech signal-to-noise ratio.

[0107] The environmental light intensity is extracted from the suspicious image data, such as analyzing the pixel brightness in the suspicious image data to obtain the environmental light intensity.

[0108] For example, the input is the image feature of the suspicious image data, the first 32 dimensions (including illumination information), and the output is the ambient light intensity light (unit: lux). The fully connected network maps the image feature of the first 32 dimensions to 1 dimension, and the range of the mapped ambient light intensity light is 0-1000 lux.

[0109] The speech signal-to-noise ratio (SNR) is calculated from the suspicious audio data.

[0110] For example, the input is suspicious audio data P, and the speech signal-to-noise ratio SNR is calculated according to the following formula 1.

[0111]

[0112] wherein P signal refers to the speech signal in P, P noise refers to the noise signal in P.

[0113] Of course, in addition to the above-mentioned embodiments for judging the data quality of image data or audio data, other methods can also be used to judge the data quality of image data or audio data, for example, detecting the occlusion degree of the driver in the image data, and / or detecting the clarity of the image data, and / or detecting the distortion degree of the audio data, etc., to obtain the data quality of the image data or the audio data, which is not limited in the present application.

[0114] Step S302: Based on the data quality of the suspicious driving data of each modality, the confidence of the suspicious driving feature corresponding to each modality of the suspicious driving data extracted is set respectively.

[0115] According to the data quality of the suspicious driving data, the confidence of the suspicious driving feature corresponding to the suspicious driving data extracted is set.

[0116] Specifically, the data quality of the suspicious driving data is positively correlated with the confidence of the suspicious driving feature corresponding to the suspicious driving data extracted. The higher the data quality of the suspicious driving data, the higher the confidence of the suspicious driving feature corresponding to the suspicious driving data extracted, and vice versa. The lower the data quality of the suspicious driving data, the lower the confidence of the suspicious driving feature corresponding to the suspicious driving data extracted.

[0117] Based on the above-mentioned embodiments, the confidence of the reference judgment feature, the confidence of the suspicious driving feature, and the similarity between the reference judgment feature and the suspicious driving feature are weighted to obtain the suspicious feature similarity, and the fatigue detection result corresponding to the driver is obtained based on the suspicious feature similarity.

[0118] In some embodiments, in addition to considering the data quality of the suspicious driving data when calculating the suspicious feature similarity, the data quality of the suspicious driving data can also be considered when calculating the sober feature similarity.

[0119] Specifically, based on the data quality of the sober driving data of each modality, the credibility of the sober driving data of each modality corresponding to the extracted sober driving features is respectively set. Then, based on the credibility of the sober driving features, the similarity between the reference judgment features and the sober driving features is calculated to obtain the sober feature similarity.

[0120] For example, the sober driving features whose credibility meet the preset condition are selected to obtain target driving features, the similarity between the target driving features and the reference judgment features belonging to the same feature type is calculated to obtain the sober feature similarity, and the credibility of the reference judgment features is set based on the sober feature similarity. For the sober driving features whose credibility do not meet the preset condition, as fuzzy driving features, the credibility of the reference judgment features belonging to the same feature type as the fuzzy driving features is set as a preset value, which can be set according to experience.

[0121] Among them, the sober driving features whose credibility are greater than a preset threshold can be regarded as the sober driving features whose credibility meet the preset condition, and the sober driving features whose credibility are not greater than the preset threshold can be regarded as the sober driving features whose credibility do not meet the preset condition. The sober driving features whose credibility are in a preset threshold interval can be regarded as the sober driving features whose credibility meet the preset condition, and the sober driving features whose credibility are not in the preset threshold interval can be regarded as the sober driving features whose credibility do not meet the preset condition. The sober driving features can also be sorted in descending order according to the credibility, and the first preset number of sober driving features in the sorted order can be regarded as the sober driving features whose credibility meet the preset condition, and the remaining sober driving features can be regarded as the sober driving features whose credibility do not meet the preset condition, which is not limited in the present application.

[0122] In some embodiments, the similarity between the reference judgment features and the suspicious driving features is weighted calculated to obtain the suspicious feature similarity based on the credibility of the reference judgment features and the credibility of the suspicious driving features, including: setting the weighted weight of the reference judgment features based on the credibility of the reference judgment features; setting the weighted weight of the suspicious driving features based on the credibility of the suspicious driving features; weighting and fusing each reference judgment feature and each suspicious driving feature respectively to obtain fused judgment features and fused driving features by using the weighted weight of each reference judgment feature and the weighted weight of each suspicious driving feature; and calculating the similarity of the fused judgment features and the fused driving features to obtain the suspicious feature similarity.

[0123] The weighting weight of each suspicious driving feature is set according to the credibility of each suspicious driving feature, and the credibility of the suspicious driving feature is positively correlated with the weighting weight of the suspicious driving feature.

[0124] Then, each suspicious driving feature is weighted and fused according to the weighting weight of each suspicious driving feature to obtain a fused driving feature.

[0125] For example, the behavior data includes image data and audio data, the suspicious driving data includes suspicious image data and suspicious audio data, the ambient light intensity of the suspicious image data is light, and the speech signal-to-noise ratio of the suspicious audio data is SNR. The weighting weight of the suspicious driving feature corresponding to the suspicious image data and the suspicious audio data extracted according to the following formula 2 is calculated: α=ω1·Norm(Light)+ω2·Norm(SNR)+ω3·History_Confidence (Formula 2)

[0126] Wherein, α is the weighting weight of the suspicious driving feature corresponding to the suspicious image data, (1-α) is the weighting weight of the suspicious driving feature corresponding to the suspicious audio data, the sum of ω1, ω2 and ω3 is 1, and the specific value can be flexibly set according to the actual application scenario; is the normalization of the light intensity; assuming that the effective range of SNR is 20-60 dB, the speech signal-to-noise ratio is normalized History_Confidence is the average confidence of the fatigue detection results of the last 10 frames.

[0127] Then, the suspicious driving features extracted from the suspicious image data and the suspicious audio data are weighted and fused according to the weighting weight of each suspicious driving feature to obtain the calculation formula of the fused driving feature, which is shown in the following formula 3:

[0128] F fused =α·F visual +(1-α)F test_aligned (Formula 3)

[0129] Wherein, F fused is the fused driving feature, F visual is the suspicious driving feature extracted from the suspicious image data, and F test_aligned is the suspicious driving feature extracted from the suspicious audio data.

[0130] Of course, there can be more types of suspicious driving features in actual application scenarios, which are not limited by the present application.

[0131] Similarly, the weighting weight of each reference judgment feature is set according to the credibility of each reference judgment feature, and the credibility of the reference judgment feature is positively correlated with the weighting weight of the reference judgment feature. Then, each reference judgment feature is weighted and fused according to the weighting weight of each reference judgment feature to obtain a fused judgment feature.

[0132] The suspicious feature similarity is obtained by performing similarity calculation on the fused judgment feature and the fused driving feature.

[0133] In some embodiments, the suspicious feature similarity is obtained by weighting and calculating the similarity between the reference judgment feature and the suspicious driving feature by using the credibility of the reference judgment feature, and includes: setting the weighting weight of the reference judgment feature based on the credibility of the reference judgment feature; weighting and fusing each reference judgment feature by using the weighting weight of each reference judgment feature to obtain a fused judgment feature; fusing each suspicious feature similarity to obtain a fused driving feature; and performing similarity calculation on the fused judgment feature and the fused driving feature to obtain the suspicious feature similarity.

[0134] In some embodiments, the suspicious feature similarity is obtained by weighting and calculating the similarity between the reference judgment feature and the suspicious driving feature by using the credibility of the reference judgment feature and the credibility of the suspicious driving feature, and includes: calculating the similarity between the reference judgment feature and the suspicious driving feature to obtain an initial similarity corresponding to the reference judgment feature; setting the weighting weight corresponding to the initial similarity based on the credibility of the reference judgment feature and the credibility of the suspicious driving feature, wherein the credibility of the reference judgment feature and the credibility of the suspicious driving feature are positively correlated with the weighting weight of the initial similarity; and weighting and summing each initial similarity by using the weighting weight corresponding to each initial similarity to obtain the suspicious feature similarity.

[0135] The similarity between the reference judgment feature and the suspicious driving feature belonging to the same feature type is calculated to obtain an initial similarity corresponding to the reference judgment feature.

[0136] Then, the weighting weight of each initial similarity is set according to the credibility of each reference judgment feature and the credibility of the suspicious driving feature, and the higher the credibility of the reference judgment feature or the credibility of the suspicious driving feature associated with the initial similarity, the higher the weighting weight of the initial similarity, and the lower the credibility of the reference judgment feature or the credibility of the suspicious driving feature associated with the initial similarity, the lower the weighting weight of the initial similarity.

[0137] The weighting weight of each initial similarity is set according to the credibility of each reference judgment feature and the credibility of the suspicious driving feature, and the higher the credibility of the reference judgment feature or the credibility of the suspicious driving feature associated with the initial similarity, the higher the weighting weight of the initial similarity, and the lower the credibility of the reference judgment feature or the credibility of the suspicious driving feature associated with the initial similarity, the lower the weighting weight of the initial similarity.

[0138] In some embodiments, the similarity between the reference judgment features and the suspicious driving features is weighted and calculated based on the credibility of the reference judgment features and the suspicious driving features to obtain a suspicious feature similarity, including: calculating the similarity between each reference judgment feature and the suspicious driving features respectively to obtain an initial similarity corresponding to each reference judgment feature; setting a weighting weight corresponding to each initial similarity based on the credibility of each reference judgment feature; wherein the credibility of the reference judgment features is positively correlated with the weighting weight of the corresponding initial similarity; and weighting and summing each initial similarity based on the weighting weight corresponding to each initial similarity to obtain the suspicious feature similarity.

[0139] The above embodiments adjust the credibility of the suspicious driving data or the sober driving features based on the quality of the behavior data collected in specific environments, which can adapt to complex environments, solve the problem of insufficient generalization ability of fixed fatigue driving detection strategies in different scenarios, and improve the detection accuracy. In addition, the behavior data of multiple models is analyzed, and fatigue detection is comprehensively performed from multiple dimensions, which improves the accuracy and comprehensiveness of fatigue detection.

[0140] In some embodiments, different levels of reference judgment features are preset, and the different levels of reference judgment features correspond to different feature types and different fatigue level grades; and data features of the suspicious driving data are extracted to obtain suspicious driving features, including: respectively extracting data features of the feature types corresponding to each level of reference judgment features from the suspicious driving data to obtain suspicious driving features corresponding to each level of reference judgment features.

[0141] Based on the above embodiments, the similarity between the reference judgment features and the suspicious driving features is weighted and calculated based on the credibility of the reference judgment features to obtain a suspicious feature similarity, and the fatigue detection result corresponding to the driver is obtained based on the suspicious feature similarity, including: weighting and calculating the similarity between the reference judgment features and the suspicious driving features based on the credibility of each level of reference judgment features to obtain the suspicious feature similarity; obtaining a fatigue probability corresponding to each level of reference judgment features based on the suspicious feature similarity corresponding to each level of reference judgment features; and obtaining a fatigue degree corresponding to the driver by combining the fatigue probability corresponding to each level of reference judgment features and the fatigue level grade, and taking the fatigue degree as the fatigue detection result.

[0142] Different levels of reference judgment features are preset, and the different levels of reference judgment features correspond to different feature types and different fatigue level grades.

[0143] For example, the reference judgment features contain three levels, the first level of reference judgment features judges the features of the image data, corresponding to the light fatigue level; the second level of reference judgment features judges the audio data, corresponding to the medium fatigue level; and the third level of reference judgment features judges the multi-modal features of the image data and the audio data, corresponding to the heavy fatigue level.

[0144] From the suspicious driving data, the data features associated with each level are extracted respectively to obtain the suspicious driving features corresponding to the reference judgment features of each level.

[0145] For example, for the first level of reference judgment features, the visual features of the driver such as eye, mouth and head posture are extracted from the image data of the suspicious driving data to obtain the data features associated with the first level of reference judgment features; for the second level of reference judgment features, the speech text keywords such as “yawn”, “sleep”, “tired” and the like of the driver are extracted from the audio data of the suspicious driving data to obtain the data features associated with the second level of reference judgment features; and for the third level of reference judgment features, the visual features of the driver are extracted from the image data of the suspicious driving data, and the speech text keywords of the driver are extracted from the audio data of the suspicious driving data, and the visual features and the speech text keywords are weighted and fused to obtain the data features associated with the third level of reference judgment features.

[0146] Based on the suspicious feature similarity corresponding to the reference judgment features of each level, the fatigue probability corresponding to the reference judgment features of each level is obtained, and the fatigue degree corresponding to the driver is obtained by combining the fatigue probability corresponding to the reference judgment features of each level and the fatigue degree level, and the fatigue degree is taken as the fatigue detection result.

[0147] Wherein, the higher the fatigue degree level is, the higher the corresponding fatigue probability is, and the higher the fatigue degree corresponding to the driver is, and vice versa, the lower the fatigue degree level is, the lower the corresponding fatigue probability is, and the lower the fatigue degree corresponding to the driver is.

[0148] The fatigue probability calculated by the reference judgment features of each level and the fatigue degree level corresponding to the reference judgment features of each level are integrated to obtain the fatigue degree corresponding to the driver, realize multi-granularity fatigue judgment, refine the fatigue state in different scenarios, distinguish different degrees of fatigue state, and effectively reduce the misjudgment rate.

[0149] Optionally, it is judged whether the fatigue probability corresponding to each level of reference judgment features is greater than a preset probability threshold, if so, the corresponding level of alarm strategy is obtained, and the corresponding level of alarm strategy is executed to realize multi-granularity alarm and improve user experience.

[0150] In some embodiments, the reference judgment features of each level can be judged level by level. Specifically, the reference judgment features are traversed from low to high according to the fatigue level, the data features associated with the currently traversed reference judgment feature are extracted from the suspicious driving data, the suspicious driving features corresponding to the currently traversed reference judgment feature are obtained, the similarity between the reference judgment feature and the suspicious driving feature is calculated based on the credibility of the currently traversed reference judgment feature, the suspicious feature similarity is obtained, and the fatigue recognition result corresponding to the driver is obtained based on the suspicious feature similarity. If the fatigue recognition result indicates that the driver is in a fatigue state, the alarm strategy corresponding to the currently traversed reference judgment feature is executed, and the next traversed reference judgment feature is determined, and the fatigue driving detection is continued based on the next traversed reference judgment feature; if the fatigue recognition result indicates that the driver is not in a fatigue state, the fatigue driving detection is ended.

[0151] For example, the image data of the driver is first detected preliminarily, if the fatigue recognition result indicates that the driver is in a fatigue state, a mild alarm is performed, then the audio data is detected for fatigue driving detection, if the fatigue recognition result indicates that the driver is in a fatigue state, a moderate alarm is performed, and the multi-modal fusion result of the image data and the audio data is comprehensively evaluated to accurately assess the fatigue degree.

[0152] The above embodiment selects the reference judgment features level by level for fatigue driving detection, which can save computing resources while ensuring the accuracy of fatigue judgment.

[0153] Figure 4 is a block diagram of a fatigue driving detection device according to an example embodiment of the present application. As shown in Figure 4 the example fatigue driving detection device 400 includes:

[0154] The wake-up feature extraction module 410 is configured to collect the behavior data of the driver in the wake-up driving time period to obtain wake-up driving data, and extract data features from the wake-up driving data to obtain wake-up driving features.

[0155] The credibility calculation module 420 is configured to obtain a preset reference judgment feature for fatigue judgment, calculate the similarity between the reference judgment feature and the wake-up driving feature to obtain a wake-up feature similarity, and set the credibility of the reference judgment feature based on the wake-up feature similarity. The wake-up feature similarity and the credibility of the reference judgment feature are inversely related.

[0156] The suspicious feature extraction module 430 is configured to collect the behavior data of the driver in the suspicious driving time period to obtain suspicious driving data, and extract data features from the suspicious driving data to obtain suspicious driving features.

[0157] The fatigue detection module 440 is configured to determine a suspicious feature similarity by weighting the similarity between the reference judgment feature and the suspicious driving feature according to the reliability of the reference judgment feature, and determine the fatigue detection result of the driver according to the suspicious feature similarity.

[0158] It should be noted that the fatigue driving detection device provided in the above embodiments and the fatigue driving detection method provided in the above embodiments belong to the same concept, and the specific manner in which each module and unit performs operations has been described in detail in the method embodiments, which will not be repeated here. The fatigue driving detection device provided in the above embodiments can be used in actual applications, and the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above described functions, which is not limited here.

[0159] Please refer to Figure 5 , Figure 5 is a structural schematic diagram of an embodiment of an electronic device. The electronic device 500 includes a memory 501 and a processor 502. The processor 502 is configured to execute program instructions stored in the memory 501 to implement the steps in any of the above fatigue driving detection method embodiments. In a specific implementation scenario, the electronic device 500 can include but is not limited to a microcomputer, a server, and in addition, the electronic device 500 can also include a notebook computer, a tablet computer, and the like, without limitation.

[0160] Specifically, the processor 502 is configured to control itself and the memory 501 to implement the steps in any of the above fatigue driving detection method embodiments. The processor 502 can also be referred to as a central processing unit (CPU). The processor 502 can be an integrated circuit chip having a processing capability. The processor 502 can also be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. In addition, the processor 502 can be implemented by an integrated circuit chip together.

[0161] Please refer to Figure 6 , Figure 6FIG. 6 is a structural schematic diagram of an embodiment of the computer readable storage medium of the present application. The computer readable storage medium 600 stores program instructions 610 capable of being executed by a processor, the program instructions 610 being used to implement the steps in any of the fatigue driving detection method embodiments described above.

[0162] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to perform the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, details are not repeated here.

[0163] The above description of each embodiment tends to emphasize the differences between the embodiments, and the same or similar parts can be mutually referred to. For brevity, details are not repeated here.

[0164] In several embodiments provided in the present application, it should be understood that the disclosed methods and apparatuses can be implemented in other ways. For example, the above-described apparatus implementation is only schematic, and the division of the modules or units is only a logical function division, and actual implementation can have another division manner, for example, a unit or component can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed each other can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0165] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit. When the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and various media that can store program codes.

Claims

1. A method of detecting fatigue driving, characterized by, The method comprises: Collecting behavior data of a driver during a sober driving period to obtain sober driving data, performing data feature extraction on the sober driving data to obtain sober driving features; Obtaining a preset reference judgment feature for fatigue judgment, calculating a similarity between the reference judgment feature and the sober driving features to obtain a sober feature similarity, and setting a credibility of the reference judgment feature based on the sober feature similarity; wherein the sober feature similarity and the credibility of the reference judgment feature are inversely related; Collecting behavior data of the driver during a suspicious driving period to obtain suspicious driving data, performing data feature extraction on the suspicious driving data to obtain suspicious driving features; Using the credibility of the reference judgment feature, weighted calculation of a similarity between the reference judgment feature and the suspicious driving features to obtain a suspicious feature similarity, and obtaining a fatigue detection result corresponding to the driver based on the suspicious feature similarity.

2. The method of claim 1, wherein, The behavior data comprises data of multiple modalities; the method further comprises: Analyzing data quality of the suspicious driving data of each modality; Based on the data quality of the suspicious driving data of each modality, the credibility of the suspicious driving features extracted from the suspicious driving data of each modality is set respectively; The use of the credibility of the reference judgment feature, the credibility of the suspicious driving features, and the weighted calculation of the similarity between the reference judgment feature and the suspicious driving features to obtain the suspicious feature similarity comprises: Using the credibility of the reference judgment feature, the credibility of the suspicious driving features, and the weighted calculation of the similarity between the reference judgment feature and the suspicious driving features to obtain the suspicious feature similarity.

3. The method of claim 2, wherein, The behavior data comprises image data and audio data, and the suspicious driving data comprises suspicious image data and suspicious audio data; the analysis of the data quality of the suspicious driving data of each modality comprises: Extracting ambient light intensity from the suspicious image data, and obtaining the data quality of the suspicious image data based on the ambient light intensity; Calculating the speech signal-to-noise ratio of the suspicious audio data, and obtaining the data quality of the suspicious audio data based on the speech signal-to-noise ratio.

4. The method of claim 2, wherein, The use of the credibility of the reference judgment feature, the credibility of the suspicious driving features, and the weighted calculation of the similarity between the reference judgment feature and the suspicious driving features to obtain the suspicious feature similarity comprises: Based on the credibility of the reference judgment feature, the weighted weight of the reference judgment feature is set; based on the credibility of the suspicious driving features, the weighted weight of the suspicious driving features is set; Using the weighted weight of each reference judgment feature and the weighted weight of each suspicious driving feature, the weighted fusion of each reference judgment feature and each suspicious driving feature is performed respectively to obtain fused judgment features and fused driving features; Performing similarity calculation on the fused judgment features and the fused driving features to obtain a suspicious feature similarity.

5. The method of claim 2, wherein, The use of the credibility of the reference judgment feature, the credibility of the suspicious driving features, and the weighted calculation of the similarity between the reference judgment feature and the suspicious driving features to obtain the suspicious feature similarity comprises: Calculate the similarity between the reference judgment feature and the suspicious driving feature, to obtain an initial similarity corresponding to the reference judgment feature; Based on the reliability of the reference judgment feature and the reliability of the suspicious driving feature, set the weighting weight corresponding to the initial similarity; wherein the reliability of the reference judgment feature and the reliability of the suspicious driving feature are positively correlated with the weighting weight of the initial similarity; Using the weighting weight corresponding to each initial similarity, the initial similarity is weighted and summed to obtain a suspicious feature similarity.

6. The method of claim 1, wherein, There are different levels of reference judgment features, and the different levels of reference judgment features correspond to different feature types and different fatigue level grades; the data feature extraction of the suspicious driving data to obtain the suspicious driving feature, comprising: Respectively extracting the data feature of each level of reference judgment feature corresponding feature type from the suspicious driving data, respectively obtaining the suspicious driving feature corresponding to each level of reference judgment feature; Based on the reliability of the reference judgment feature, the similarity between the reference judgment feature and the suspicious driving feature is weighted to obtain a suspicious feature similarity, and the fatigue detection result corresponding to the driver is obtained based on the suspicious feature similarity, comprising: Based on the reliability of each level of reference judgment feature, the similarity between the reference judgment feature and the suspicious driving feature is weighted to obtain a suspicious feature similarity; Based on the suspicious feature similarity corresponding to each level of reference judgment feature, the fatigue probability corresponding to each level of reference judgment feature is obtained; Combining the fatigue probability corresponding to each level of reference judgment feature and the fatigue level grade, the fatigue degree corresponding to the driver is obtained, and the fatigue degree is taken as the fatigue detection result.

7. The method of claim 1, wherein, The suspicious driving data is composed of a plurality of data frames sorted in time sequence; the data feature extraction of the suspicious driving data to obtain the suspicious driving feature, comprising: Using a sliding window, a preset number of data frames are extracted from the suspicious driving data to obtain a window data set; Calculate the difference between adjacent data frames in the window data set to obtain a time sequence feature; The time sequence feature is taken as the suspicious driving feature.

8. The method according to any one of claims 1 to 7, characterized in that, The initial time period of the driver driving the vehicle is taken as the sober driving time period; the subsequent time period after the initial time period is taken as the suspicious driving time period.

9. An electronic device, comprising: The electronic device includes a memory and a processor, and the processor is used to execute the program instructions stored in the memory to realize the steps in the method of any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores program instructions, which can be executed by the processor to realize the steps in the method of any one of claims 1-8.

Images