Fatigue detection method, device and equipment based on wearable equipment and storage medium

Abstract

The invention discloses a fatigue detection method, device and equipment based on wearable equipment and a storage medium, relates to the technical field of wearable equipment, and discloses the fatigue detection method based on the wearable equipment, which is applied to the wearable equipment, and the wearable equipment is provided with a first image acquisition device and a second image acquisition device. The method comprises the following steps: analyzing a current facial image acquired by a first image acquisition device, and determining a current driving state of a user; when the current driving state is not the fatigue state, determining a current weather condition according to an external environment image acquired by a second image acquisition device; and performing fatigue state detection according to the current weather condition and the monitored physiological indexes of the user, and determining a fatigue detection result. Through the above mode, accurate detection of the driving fatigue state through the wearable device is realized, the detection efficiency and the accuracy of the detection result are greatly improved, the driving safety is ensured, and a driver is helped to establish a good driving habit.

Description

Technical Field

[0001] This application relates to the field of wearable device technology, and in particular to fatigue detection methods, apparatus, devices and storage media based on wearable devices. Background Technology

[0002] With the development of wearable devices, such as VR (Virtual Reality) and AR (Augmented Reality) devices, wearable devices can appear in various scenarios in life, improving users' quality of life in many ways. Currently, wearable devices can be used to detect whether a driver is fatigued. The detection method involves using a state evaluation model trained on one driver to detect the driving state of other drivers, which leads to low accuracy. Alternatively, a separate evaluation model can be trained for each driver, which results in low efficiency. Therefore, currently, using wearable devices to detect driver fatigue cannot simultaneously achieve both accuracy and efficiency.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this application is to provide a fatigue detection method, device, equipment, and storage medium based on wearable devices, aiming to solve the technical problem in the prior art that the detection accuracy and efficiency cannot be balanced when using wearable devices to detect driving fatigue.

[0005] To achieve the above objectives, this application proposes a fatigue detection method based on a wearable device. The wearable device is equipped with a first image acquisition device and a second image acquisition device. The first image acquisition device is used to acquire a facial image, and the second image acquisition device is used to acquire an external environment image. The method includes:

[0006] The facial images captured by the first image acquisition device are analyzed to determine the user's current driving status;

[0007] When the current driving state is not a state of fatigue, the current weather conditions are determined based on the external environment image captured by the second image acquisition device;

[0008] Fatigue status is detected based on the current weather conditions and the user's monitored physiological indicators, and the fatigue detection results are determined.

[0009] In one embodiment, the step of detecting fatigue state based on the current weather conditions and the user's monitored physiological indicators, and determining the fatigue detection result, includes:

[0010] Determine the external light intensity and weather type based on the current weather conditions;

[0011] The severity of the weather is determined based on the external light intensity and the weather type.

[0012] When the user's monitored physiological indicators meet the fatigue judgment conditions corresponding to the severity of the weather, the fatigue detection result is determined to be a result of fatigued driving.

[0013] In one embodiment, the fatigue judgment condition includes at least one of the following:

[0014] The user's blinking frequency in the monitored physiological indicators is less than the blinking frequency threshold;

[0015] The user gaze duration in the monitored physiological indicators is greater than the gaze duration threshold.

[0016] The operational response time of the monitored physiological indicators is greater than the response time threshold;

[0017] The monitored heart rate among the physiological indicators is not within the target heart rate range.

[0018] In one embodiment, before the step of determining the fatigue detection result as a result of fatigued driving when the user's monitored physiological indicators meet the fatigue judgment conditions corresponding to the severity of the weather, the method further includes:

[0019] The first image acquisition device acquires multiple consecutive facial images for localization, and determines the user's eye region corresponding to each facial image;

[0020] The user's eye region corresponding to each facial image is analyzed to determine the user's blinking frequency and gaze duration.

[0021] In one embodiment, the wearable device further includes a heart rate sensor for collecting user heart rate data;

[0022] Before the step of determining the fatigue detection result as a result of fatigued driving when the user's monitored physiological indicators meet the fatigue judgment conditions corresponding to the severity of the weather, the method further includes:

[0023] Acquire the user's heart rate data collected by the heart rate sensor;

[0024] The user's heart rate data is processed to obtain a processed heart rate signal;

[0025] Peak detection is performed on the processed heart rate signal to obtain multiple monitoring peak values;

[0026] The user's monitored heart rate is determined based on multiple monitoring peak values.

[0027] In one embodiment, before the step of determining the fatigue detection result as a result of fatigued driving when the user's monitored physiological indicators meet the fatigue judgment conditions corresponding to the severity of the weather, the method further includes:

[0028] Environmental analysis is performed based on the external environment images to determine the driving road environment;

[0029] When a driving change event occurs in the driving road environment, the image acquisition time of the external environment image is obtained;

[0030] Obtain the user response time corresponding to the driving change event;

[0031] The operation response duration is determined based on the user response time and the image acquisition time.

[0032] In one embodiment, the step of analyzing the facial image acquired by the first image acquisition device to determine the user's current driving state includes:

[0033] Obtain sample facial images under various driving weather conditions and sample fatigue labels for each sample facial image;

[0034] Feature extraction is performed on each sample facial image to determine the sample features of each sample facial image;

[0035] A state recognition model is obtained by training a machine learning model using the sample features of multiple sample facial images and the sample fatigue labels of each sample facial image.

[0036] The facial image captured by the first image acquisition device is input into the state recognition model to obtain the user's current driving state.

[0037] In addition, to achieve the above objectives, this application also proposes a fatigue detection device based on a wearable device, the fatigue detection device based on a wearable device comprising: an analysis module, used to analyze the facial image acquired by the first image acquisition device to determine the user's current driving state;

[0038] The processing module is used to determine the current weather conditions based on the external environment image acquired by the second image acquisition device when the current driving state is not a state of fatigue.

[0039] The detection module is used to detect fatigue status based on the current weather conditions and the user's monitored physiological indicators, and to determine the fatigue detection result.

[0040] In addition, to achieve the above objectives, this application also proposes a wearable device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the fatigue detection method based on the wearable device as described above.

[0041] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the fatigue detection method based on wearable devices as described above.

[0042] This application provides a fatigue detection method based on a wearable device. The wearable device is equipped with a first image acquisition device and a second image acquisition device. The first image acquisition device acquires a facial image, and the second image acquisition device acquires an external environment image. The method includes: analyzing the facial image acquired by the first image acquisition device to determine the user's current driving state; when the current driving state is not a fatigue state, determining the current weather conditions based on the external environment image acquired by the second image acquisition device; and performing fatigue state detection based on the current weather conditions and the user's monitored physiological indicators to determine the fatigue detection result. Through this method, accurate detection of driving fatigue state is achieved using a wearable device, significantly improving detection efficiency and accuracy, ensuring driving safety, and helping drivers establish good driving habits. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating an embodiment of the fatigue detection method based on wearable devices provided in this application.

[0046] Figure 2 This is a flowchart illustrating Embodiment 2 of the fatigue detection method based on wearable devices provided in this application;

[0047] Figure 3This is a schematic diagram of the module structure of a fatigue detection device based on a wearable device according to an embodiment of this application;

[0048] Figure 4 This is a schematic diagram of the hardware operating environment involved in the fatigue detection method based on wearable devices in the embodiments of this application.

[0049] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0050] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0051] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0052] The main solution of this application embodiment is: to analyze the facial image acquired by the first image acquisition device to determine the user's current driving state; when the current driving state is not a fatigue state, to determine the current weather conditions based on the external environment image acquired by the second image acquisition device; and to detect fatigue state based on the current weather conditions and the user's monitored physiological indicators to determine the fatigue detection result.

[0053] Wearable devices can be used to detect driver fatigue. One method involves using a state assessment model trained on one driver to detect the driving state of other drivers, which leads to low accuracy. Another method involves training a separate assessment model for each driver, which results in low efficiency. Therefore, current methods for detecting driver fatigue using wearable devices cannot simultaneously achieve both accuracy and efficiency.

[0054] This application analyzes collected facial images and combines them with current weather conditions and monitored physiological indicators to detect driver fatigue. This enables accurate detection of driver fatigue through wearable devices, significantly improving detection efficiency and accuracy, ensuring driving safety, and helping drivers establish good driving habits.

[0055] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or wearable device capable of performing the above functions. The following description uses a wearable device as an example to illustrate this embodiment and the subsequent embodiments.

[0056] Based on this, the embodiments of this application provide a fatigue detection method based on wearable devices, referring to...Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the fatigue detection method based on wearable devices in this application.

[0057] In this embodiment, the fatigue detection method based on wearable devices is applied to a wearable device. The wearable device is equipped with a first image acquisition device and a second image acquisition device. The first image acquisition device is used to acquire facial images, and the second image acquisition device is used to acquire external environment images. The method includes steps S10 to S30:

[0058] Step S10: Analyze the facial image acquired by the first image acquisition device to determine the user's current driving status.

[0059] It should be noted that the wearable device in this embodiment is a VR device, an AR device, or other head-mounted wearable device. This embodiment does not limit this; in this embodiment, an AR device is used as an example for illustration.

[0060] It is understood that the wearable device is equipped with at least one first image acquisition device, a second image acquisition device, and a heart rate acquisition device. The first image acquisition device includes a camera and an infrared light source, and may also include other modules that can achieve the above functions; this embodiment does not limit this. The first image acquisition device is placed inside the wearable device and is used to acquire facial images, which can be used to analyze the driver's facial expressions and eye movement data. The second image acquisition device is placed outside the wearable device and is used to acquire external environment images, which can be used to analyze external weather conditions and the surrounding environment, such as road signs and traffic signals. The heart rate acquisition device is used to collect the user's heart rate data. In this embodiment, the heart rate acquisition device can illuminate the skin with an LED light source and calculate the heart rate by detecting changes in reflected light.

[0061] In specific implementation, when it is detected that the user has correctly worn the wearable device and the driving fatigue detection function has been activated, a facial image captured by the first image acquisition device is obtained. The facial image is then analyzed to determine the current driving state: key point localization is performed on the facial image, and fatigue-related features are extracted through key point localization, such as: the degree of eye opening and closing (which can be used to reflect whether the user has closed their eyes), the degree of mouth opening and closing (which can be used to reflect whether the user is yawning), and head posture (which can be used to reflect whether the user is tilting or nodding as a drowsy action). Based on the extracted features, a pre-trained machine learning model (such as Support Vector Machine (SVM), Random Forest, or Deep Learning model, etc.) is used to determine the user's current driving state. In this embodiment, the current driving state includes both fatigue and wakefulness. In addition to the above analysis method, other methods can also be used to analyze the facial image, and this embodiment does not limit this.

[0062] In one feasible implementation, step S10 may further include steps A11 to A14:

[0063] Step A11: Obtain sample facial images and sample fatigue labels for each sample facial image under various driving weather conditions.

[0064] It should be noted that users' facial expressions reflecting fatigue differ under different driving weather conditions, and each user may exhibit multiple facial expressions when fatigued. Therefore, for different driving weather conditions, a large number of sample facial images of different users in fatigued states are acquired, and each sample facial image is labeled with a fatigue state tag. In this embodiment, the driving weather environment includes light intensity and weather type (e.g., rain, snow, and sunny); the sample fatigue tag refers to the tag corresponding to the fatigue state of the sample facial image.

[0065] Step A12: Extract features from each sample facial image to determine the sample features of each sample facial image.

[0066] It should be noted that for each sample facial image: a facial detection algorithm is used to locate the facial region of the sample facial image, and key point detection is performed on the facial region to determine the positions of key parts such as the eyes, eyebrows, and mouth. Furthermore, feature values ​​related to fatigue state are calculated, such as the degree of eye opening and closing, the degree of mouth opening and closing, head posture, and facial muscle movement features, thereby obtaining the sample features of the sample facial image. In this embodiment, the sample features include, but are not limited to, the degree of eye opening and closing, the degree of mouth opening and closing, head posture, and facial muscle movement features.

[0067] Step A13: The machine learning model is trained using the sample features of multiple sample facial images and the sample fatigue labels of each sample facial image to obtain the state recognition model.

[0068] It should be noted that a machine learning model is trained using sample features and corresponding fatigue labels from multiple sample facial images. This process can be a supervised learning task, and the model can be either a traditional machine learning model (such as Support Vector Machines (SVM), Random Forests, etc.) or a deep learning model (such as Convolutional Neural Networks (CNNs). This embodiment does not impose any restrictions on the machine learning model. The goal of the training process is to minimize the difference between the model's predictions and the actual labels, thereby obtaining the trained machine learning model. In this embodiment, the state recognition model refers to the trained machine learning model, which can be used to output the driving state corresponding to the facial image.

[0069] Step A14: Input the facial image captured by the first image acquisition device into the state recognition model to obtain the user's current driving state.

[0070] It should be noted that when a facial image is input into a state recognition model, the model can output the user's current driving state, which can be used to characterize whether the user is experiencing driving fatigue.

[0071] Step S20: When the current driving state is not a state of fatigue, determine the current weather conditions based on the external environment image acquired by the second image acquisition device.

[0072] It should be noted that if the current driving state is not one of fatigue, it means that the facial image identification indicates the user is not experiencing driving fatigue. To ensure the accuracy of the detection results, the external environment image acquired in the second image acquisition state is analyzed: the average brightness or histogram of the external environment image is calculated to determine the light intensity; the color distribution of the external environment image is analyzed to obtain color information, such as blue (sunny sky), gray (cloudy), white (snow or fog), etc.; texture patterns in the external environment image are analyzed to identify weather features such as clouds and raindrops; the color information and weather features are input into a pre-trained classification model to obtain the current weather type. In this embodiment, the pre-trained classification model is obtained by training a machine learning model with a large number of environmental images labeled with weather types. The machine learning model can be a traditional machine learning model (such as Support Vector Machine (SVM), Random Forest, etc.) or a deep learning model (such as Convolutional Neural Network (CNN). This embodiment does not limit the machine learning model. In this embodiment, the current weather conditions include the light intensity and weather type of the external environment.

[0073] Understandably, when the current driving state is one of fatigue, the fatigue detection result is determined to be a driving fatigue result. This indicates that the user is driving while fatigued, and a corresponding warning message is generated and broadcast to remind the user to maintain safe driving.

[0074] Step S30: Perform fatigue state detection based on the current weather conditions and the user's monitored physiological indicators, and determine the fatigue detection result.

[0075] It should be noted that different weather conditions correspond to different degrees of weather severity. Under different degrees of weather severity, the thresholds for each physiological indicator (such as blink rate, fixation duration, operation response time, and heart rate) when the user is fatigued are different. Therefore, the threshold / range corresponding to each physiological indicator is determined by the current weather conditions, thereby generating corresponding fatigue judgment conditions. It is then determined whether the user's monitored physiological indicators meet the fatigue judgment conditions. If they do, the fatigue detection result is driving fatigue; if not, the fatigue detection result is driving alertness. In this embodiment, the monitored physiological indicators refer to the real-time physiological indicators of the user, including but not limited to blink rate, fixation duration, operation response time, and heart rate.

[0076] This embodiment provides a fatigue detection method based on wearable devices. The wearable device is equipped with a first image acquisition device and a second image acquisition device. The first image acquisition device acquires facial images, and the second image acquisition device acquires external environment images. The method includes: analyzing the facial images acquired by the first image acquisition device to determine the user's current driving state; when the current driving state is not fatigued, determining the current weather conditions based on the external environment images acquired by the second image acquisition device; and performing fatigue state detection based on the current weather conditions and the user's monitored physiological indicators to determine the fatigue detection result. Through the above method, by analyzing the acquired facial images and combining them with current weather conditions and monitored physiological indicators for state detection, accurate detection of driving fatigue state is achieved using wearable devices. This significantly improves detection efficiency and the accuracy of detection results, ensures driving safety, and helps drivers establish good driving habits.

[0077] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 In this embodiment, step S30 includes steps S31 to S33:

[0078] Step S31: Determine the external light intensity and weather type based on the current weather conditions.

[0079] Step S32: Determine the severity of the weather based on the external light intensity and the weather type.

[0080] It should be noted that in this embodiment, multiple weather severity levels are defined, and the severity of weather severity is comprehensively evaluated by combining external light intensity and weather type. For example, the weather severity levels are divided into low severity, moderate severity, medium severity, and severe severity. When the light intensity is 0.001 lux and the weather type is heavy rain, the weather severity level is severe.

[0081] Step S33: When the user's monitored physiological indicators meet the fatigue judgment conditions corresponding to the severity of the weather, the fatigue detection result is determined to be the result of fatigued driving.

[0082] It should be noted that the thresholds / ranges for each physiological indicator differ depending on the severity of the weather, indicating user fatigue. The fatigue assessment criteria are generated based on the fatigue thresholds / ranges of each physiological indicator corresponding to the current level of weather severity. For example, under low severity, the target heart rate range is 60-80 beats / minute, the blink rate threshold is 5 seconds / blink, the fixation duration threshold is 10 seconds, and the response time threshold is 3 seconds.

[0083] In this embodiment, the fatigue judgment condition includes at least one of the following: the user's blink frequency in the monitored physiological indicators is less than the blink frequency threshold; the user's gaze duration in the monitored physiological indicators is greater than the gaze duration threshold; the operation response duration in the monitored physiological indicators is greater than the response duration threshold; and the monitored heart rate in the monitored physiological indicators is not within the target heart rate range.

[0084] It should be noted that in this embodiment, fatigue may cause fluctuations and abnormal changes in heart rate. A sustained high or low heart rate may indicate that the driver is fatigued; a significantly reduced blinking frequency or excessively long fixation time (e.g., more than 5 seconds without blinking) may indicate fatigue; a slowed reaction speed to the surrounding environment or ignoring traffic signals may also indicate fatigue. In summary, if any one or more of the following conditions are met—blinking frequency less than the blinking frequency threshold, fixation duration greater than the fixation duration threshold, operation response time greater than the response time threshold, and the monitored heart rate not within the target heart rate range—the monitored physiological indicators are deemed to meet the fatigue judgment criteria, and the fatigue detection result is considered fatigued driving. The blinking frequency threshold, fixation duration threshold, response time threshold, and target heart rate range can be set by the user according to their physical condition.

[0085] In one feasible implementation, steps B11 to B12 are included before step S33:

[0086] Step B11: The first image acquisition device acquires multiple consecutive facial images for localization, and determines the user's eye region corresponding to each facial image.

[0087] It should be noted that facial detection algorithms (such as OpenCV's Haar cascade classifier, the HOG+SVM detector from the dlib library, and deep learning-based MTCNN) are used to detect facial regions in each facial image. Within the detected facial regions, a dedicated eye detection algorithm is used to locate the eyes. Pre-trained models (such as deep learning-based models) can be used to accurately locate key points of the eyes (such as the corners of the eyes and the center of the pupils). Based on the location of the key points of the eyes corresponding to each facial image, the user's eye region corresponding to each facial image is extracted.

[0088] Step B12: Analyze the user's eye area corresponding to each facial image to determine the user's blinking frequency and gaze duration.

[0089] It's important to note that the eye aspect ratio (EAR) is calculated, and the duration of time the eyes remain open is monitored. EAR is an eye state metric that determines whether the eyes are closed by calculating the ratio of the distance between the upper and lower eyelids to the width of the eye. When the EAR is below a preset threshold, the eyes are considered closed. Changes in EAR are continuously monitored. When the EAR returns to normal after being below the threshold for a period of time, a blink event is considered to have occurred. Blink frequency calculation: The number of blink events occurring within a certain time window (e.g., per minute) is counted to calculate the user's blink frequency.

[0090] Understandably, the process involves detecting the position of the pupil and tracking its movement within the eye socket. Based on the pupil's position and the shape of the eye, the direction of the eyeball is estimated. Then, based on the direction of the eyeball and the position of the eye, the current fixation point is determined through geometric calculations. Finally, the duration of continuous eye fixation on the current fixation point is recorded to obtain the user's fixation duration.

[0091] In one feasible implementation, the wearable device further includes a heart rate sensor for collecting user heart rate data; prior to step S33, steps C11 to C14 are also included:

[0092] Step C11: Obtain the user's heart rate data collected by the heart rate sensor.

[0093] Step C12: Perform signal processing on the user's heart rate data to obtain a processed heart rate signal.

[0094] Step C13: Peak detection is performed on the processed heart rate signal to obtain multiple monitoring peaks.

[0095] Step C14: Determine the user's monitored heart rate based on multiple monitoring peak values.

[0096] It should be noted that in this embodiment, the user's heart rate data collected by the heart rate sensor is preprocessed to remove noise, improve signal quality, and enhance the useful components of the signal, thereby obtaining processed user heart rate data. In this embodiment, processing the heart rate signal refers to the processed user heart rate data; the heart rate sensor in this embodiment is set as a PPG (Photoplethysmography Sensor), but it can also be other sensors, and this embodiment does not limit it.

[0097] Understandably, peak-valley detection algorithms are used to identify peak values ​​in heart rate signals. In this embodiment, a threshold can be set; when the signal exceeds this threshold, a peak value is identified. Alternatively, the local maximum value in the signal can be used as the peak value. False positive peaks are removed, and the remaining valid peak values ​​are used as the monitored peak values. In this embodiment, false positive peaks can be removed through rule-based filtering: unreasonable peak values ​​can be excluded based on typical heart rate intervals (e.g., 60-100 beats / minute), or other methods can be used. This embodiment does not impose any limitations on this method.

[0098] In practice, the time interval (RR interval) between two consecutive monitoring peaks is measured, and then the user's monitored heart rate is calculated using the formula: Heart Rate = 60 / (Average RR Interval × Sampling Frequency). In this embodiment, to improve accuracy, the average of multiple RR intervals can be calculated to obtain the average RR interval, thereby determining the final monitored heart rate.

[0099] In one feasible implementation, steps D11 to D14 are included before step S33:

[0100] Step D11: Perform environmental analysis based on the external environment image to determine the driving road environment.

[0101] Step D12: When a driving change event occurs in the driving road environment, the image acquisition time of the external environment image is obtained.

[0102] Step D13: Obtain the user response time corresponding to the driving change event.

[0103] Step D14: Determine the operation response duration based on the user response time and the image acquisition time.

[0104] It should be noted that the acquired external environment images undergo preprocessing, including image resizing, noise removal, and conversion to grayscale or a specific color space. Then, computer vision analysis technology is used to analyze the adjusted external environment images to identify information such as road signs, traffic lights, pedestrians, and other vehicles, thereby obtaining the driving road environment.

[0105] It is understandable that identifying driving change events, such as changes in traffic lights, the appearance of a pedestrian ahead, or vehicle acceleration or deceleration, involves recording the image acquisition time of the external environment when such events occur in the driving road environment. In this embodiment, the image acquisition time can be used as the event occurrence time.

[0106] In the specific implementation, the user's response time to a driving change event is determined by the operation information fed back from the vehicle's controller to the wearable device. In this embodiment, if the user does not respond to the driving change event, the user's response time is recorded as +∞. For example, in this embodiment, if the user ignores a traffic signal, the user's response time is determined to be recorded as +∞.

[0107] It should be noted that the time difference between the user's response time and the image acquisition time is calculated to obtain the user's operation response time. For example, if the traffic light turns red at time t1 and the user presses the brake at time t2, then the response time is t2-t1.

[0108] This embodiment provides a fatigue detection method based on wearable devices. This embodiment determines the external light intensity and weather type based on the current weather conditions; determines the severity of the weather based on the external light intensity and the weather type; and determines the fatigue detection result as fatigue driving when the user's monitored physiological indicators meet the fatigue judgment conditions corresponding to the severity of the weather. This method ensures the accuracy of the fatigue detection results.

[0109] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the fatigue detection method based on wearable devices in this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0110] This application also provides a fatigue detection device based on a wearable device; please refer to [reference needed]. Figure 3 The fatigue detection device based on wearable devices includes:

[0111] The analysis module 10 is used to analyze the facial images acquired by the first image acquisition device to determine the user's current driving status.

[0112] Processing module 20 is used to determine the current weather conditions based on the external environment image acquired by the second image acquisition device when the current driving state is not a state of fatigue.

[0113] The detection module 30 is used to detect fatigue state based on the current weather conditions and the user's monitored physiological indicators, and to determine the fatigue detection result.

[0114] Optionally, the detection module 30 is further configured to:

[0115] The external light intensity and weather type are determined based on the current weather conditions; the severity of the weather is determined based on the external light intensity and the weather type; when the user's monitored physiological indicators meet the fatigue judgment conditions corresponding to the severity of the weather, the fatigue detection result is determined to be a result of fatigued driving.

[0116] Optionally, the detection module 30 is further configured to:

[0117] The first image acquisition device acquires multiple consecutive facial images for localization, and determines the user's eye region corresponding to each facial image; the user's blinking frequency and user gaze duration are analyzed according to the user's eye region corresponding to each facial image.

[0118] Optionally, the detection module 30 is further configured to:

[0119] The system acquires user heart rate data collected by the heart rate sensor; performs signal processing on the user heart rate data to obtain a processed heart rate signal; performs peak detection on the processed heart rate signal to obtain multiple monitoring peaks; and determines the user's monitored heart rate based on the multiple monitoring peaks.

[0120] Optionally, the detection module 30 is further configured to:

[0121] Environmental analysis is performed based on the external environment image to determine the driving road environment; when a driving change event occurs in the driving road environment, the image acquisition time of the external environment image is obtained; the user response time corresponding to the driving change event is obtained; and the operation response duration is determined based on the user response time and the image acquisition time.

[0122] Optionally, the analysis module 10 is further configured to:

[0123] Acquire sample facial images and fatigue labels for each sample facial image under various driving weather conditions; extract features from each sample facial image to determine the sample features of each sample facial image; train a machine learning model using the sample features and fatigue labels of multiple sample facial images to obtain a state recognition model; input the facial images acquired by the first image acquisition device into the state recognition model to obtain the user's current driving state.

[0124] The fatigue detection device based on wearable devices provided in this application, employing the fatigue detection method based on wearable devices in the above embodiments, can solve the technical problem in the prior art where the detection accuracy and efficiency of driving fatigue state using wearable devices cannot be simultaneously guaranteed. Compared with the prior art, the beneficial effects of the fatigue detection device based on wearable devices provided in this application are the same as those of the fatigue detection method based on wearable devices provided in the above embodiments, and other technical features in the fatigue detection device based on wearable devices are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0125] This application provides a wearable device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the fatigue detection method based on the wearable device in Embodiment 1 above.

[0126] The following is for reference. Figure 4 The diagram illustrates a structural schematic suitable for implementing the embodiments of this application. The wearable devices in the embodiments of this application may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The wearable device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0127] like Figure 4As shown, the wearable device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the wearable device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the wearable device to communicate wirelessly or wiredly with other devices to exchange data. While wearable devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0128] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0129] The wearable device provided in this application, employing the fatigue detection method based on wearable devices described in the above embodiments, can solve the technical problem in the prior art where the detection accuracy and efficiency of driving fatigue state using wearable devices cannot be simultaneously guaranteed. Compared with the prior art, the beneficial effects of the wearable device provided in this application are the same as those of the fatigue detection method based on wearable devices provided in the above embodiments, and other technical features of this wearable device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0130] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0131] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0132] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the fatigue detection method based on a wearable device in the above embodiments.

[0133] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0134] The aforementioned computer-readable storage medium may be included in the wearable device; or it may exist independently and not assembled into the wearable device.

[0135] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a wearable device, cause the wearable device to: analyze the facial image acquired by the first image acquisition device to determine the user's current driving state; when the current driving state is not a fatigue state, determine the current weather conditions based on the external environment image acquired by the second image acquisition device; and perform fatigue state detection based on the current weather conditions and the user's monitored physiological indicators to determine the fatigue detection result.

[0136] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0137] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0138] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0139] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described fatigue detection method based on wearable devices. This solves the technical problem in the prior art where the detection accuracy and efficiency of driving fatigue detection using wearable devices cannot be simultaneously guaranteed. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the fatigue detection method based on wearable devices provided in the above embodiments, and will not be repeated here.

[0140] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the fatigue detection method based on wearable devices as described above.

[0141] The computer program product provided in this application can solve the technical problem in the prior art that the detection accuracy and efficiency of driving fatigue detection using wearable devices cannot be balanced. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the fatigue detection method based on wearable devices provided in the above embodiments, and will not be repeated here.

[0142] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A fatigue detection method based on wearable devices, characterized in that, The fatigue detection method based on wearable devices is applied to a wearable device, which is equipped with a first image acquisition device and a second image acquisition device. The first image acquisition device is used to acquire facial images, and the second image acquisition device is used to acquire external environment images. The method includes: The current facial image captured by the first image acquisition device is analyzed to determine the user's current driving status; When the current driving state is not a state of fatigue, the current weather conditions are determined based on the external environment image captured by the second image acquisition device; Fatigue status is detected based on the current weather conditions and the user's monitored physiological indicators, and the fatigue detection results are determined.

2. The method as described in claim 1, characterized in that, The step of detecting fatigue state based on the current weather conditions and the user's monitored physiological indicators, and determining the fatigue detection result, includes: Determine the external light intensity and weather type based on the current weather conditions; The severity of the weather is determined based on the external light intensity and the weather type. When the user's monitored physiological indicators meet the fatigue judgment conditions corresponding to the severity of the weather, the fatigue detection result is determined to be a result of fatigued driving.

3. The method as described in claim 2, characterized in that, The fatigue judgment criteria include at least one of the following: The user's blinking frequency in the monitored physiological indicators is less than the blinking frequency threshold; The user gaze duration in the monitored physiological indicators is greater than the gaze duration threshold. The operational response time of the monitored physiological indicators is greater than the response time threshold; The monitored heart rate among the physiological indicators is not within the target heart rate range.

4. The method as described in claim 3, characterized in that, Before the step of determining the fatigue detection result as a result of fatigued driving when the user's monitored physiological indicators meet the fatigue judgment conditions corresponding to the severity of the weather, the method further includes: The first image acquisition device acquires multiple consecutive facial images for localization, and determines the user's eye region corresponding to each facial image; The user's eye region corresponding to each facial image is analyzed to determine the user's blinking frequency and gaze duration.

5. The method as described in claim 3, characterized in that, The wearable device also includes a heart rate sensor, which is used to collect the user's heart rate data; Before the step of determining the fatigue detection result as a result of fatigued driving when the user's monitored physiological indicators meet the fatigue judgment conditions corresponding to the severity of the weather, the method further includes: Acquire the user's heart rate data collected by the heart rate sensor; The user's heart rate data is processed to obtain a processed heart rate signal; Peak detection is performed on the processed heart rate signal to obtain multiple monitoring peak values; The user's monitored heart rate is determined based on multiple monitoring peak values.

6. The method as described in claim 3, characterized in that, Before the step of determining the fatigue detection result as a result of fatigued driving when the user's monitored physiological indicators meet the fatigue judgment conditions corresponding to the severity of the weather, the method further includes: Environmental analysis is performed based on the external environment images to determine the driving road environment; When a driving change event occurs in the driving road environment, the image acquisition time of the external environment image is obtained; Obtain the user response time corresponding to the driving change event; The operation response duration is determined based on the user response time and the image acquisition time.

7. The method according to any one of claims 1 to 6, characterized in that, The step of analyzing the current facial image captured by the first image acquisition device to determine the user's current driving state includes: Obtain sample facial images under various driving weather conditions and sample fatigue labels for each sample facial image; Feature extraction is performed on each sample facial image to determine the sample features of each sample facial image; A state recognition model is obtained by training a machine learning model using the sample features of multiple sample facial images and the sample fatigue labels of each sample facial image. The current facial image captured by the first image acquisition device is input into the state recognition model to obtain the user's current driving state.

8. A fatigue detection device based on a wearable device, characterized in that, The fatigue detection device based on wearable devices includes: The analysis module is used to analyze the current facial image captured by the first image acquisition device to determine the user's current driving status; The processing module is used to determine the current weather conditions based on the external environment image acquired by the second image acquisition device when the current driving state is not a state of fatigue. The detection module is used to detect fatigue status based on the current weather conditions and the user's monitored physiological indicators, and to determine the fatigue detection result.

9. A wearable device, characterized in that, The wearable device includes: a memory, a processor, and a fatigue detection program based on the wearable device stored in the memory and executable on the processor, the fatigue detection program being configured to implement the fatigue detection method based on the wearable device as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a fatigue detection program based on a wearable device, which, when executed by a processor, implements the fatigue detection method based on a wearable device as described in any one of claims 1 to 7.

Images