A fatigue driving warning method and related equipment

Abstract

This application discloses a fatigue driving warning method, comprising: acquiring the driving status information of a target vehicle and the driving status information of the target vehicle driver; determining the vehicle's driving fatigue state based on driving speed and pedal position information, and determining the driver's driving fatigue state based on facial and hand position information; determining a target fatigue driving level for the target vehicle from multiple fatigue driving levels based on the vehicle's driving fatigue state and the driver's fatigue state, each of the multiple fatigue driving levels corresponding to different fatigue driving warning schemes; and issuing a fatigue driving warning prompt to the target vehicle based on the fatigue driving warning scheme corresponding to the target fatigue driving level. Based on the above method, the accuracy and efficiency of judging the driver's fatigue driving state can be improved, and personalized warnings can be provided according to the driver's actual fatigue level, thus improving the pertinence and effectiveness of the warning.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a fatigue driving warning method and related equipment. Background Technology

[0002] Fatigue driving refers to the phenomenon that drivers experience inattention, fatigue, drowsiness, and operational errors after driving for a long time, which can easily lead to traffic accidents. Therefore, fatigue driving detection has become a crucial task.

[0003] In related technologies, traditional methods mainly determine whether the driver is fatigued by acquiring facial images, processing them, and then analyzing the images. However, these methods rely on limited data and cannot accurately assess driver fatigue, resulting in poor judgment efficiency and impacting user experience. Summary of the Invention

[0004] The fatigue driving warning method and related equipment provided in this application embodiment at least solve the problem of poor judgment effect and efficiency of driver fatigue driving state in related technologies.

[0005] In a first aspect, embodiments of this application provide a fatigue driving warning method, including:

[0006] The driving status information of the target vehicle and the driving status information of the driver of the target vehicle are obtained. The driving status information includes the driving speed of the target vehicle and the position information of the accelerator pedal. The driving status information includes the facial status information and hand position information of the driver.

[0007] The vehicle fatigue state of the target vehicle is determined based on the driving speed and the pedal position information, and the driver fatigue state is determined based on the facial state information and the hand position information.

[0008] Based on the vehicle's fatigue state and the driver's fatigue state, the target fatigue driving level of the target vehicle is determined from multiple fatigue driving levels, and each of the multiple fatigue driving levels corresponds to a different fatigue driving warning scheme.

[0009] The fatigue driving warning scheme corresponding to the target fatigue driving level is used to issue a fatigue driving warning to the target vehicle.

[0010] Preferably, obtaining the driving status information of the target vehicle and the driving status information of the target vehicle driver includes:

[0011] Based on the driving status information, the driving status change information of the target vehicle within a preset time period is determined, wherein the driving status change information includes a speed change value determined based on the driving speed and a pedal position change value determined based on the pedal position information; and

[0012] The driving state information of the driver within a preset time period is determined based on the driving state information. The driving state change information includes facial state change values ​​determined based on the facial state information and hand position change values ​​determined based on the hand position information.

[0013] Preferably, the vehicle fatigue state of the target vehicle is determined based on the driving speed and the pedal position information;

[0014] The acceleration value of the target vehicle within the preset time period is determined based on the speed change value;

[0015] The acceleration value and the change in pedal position are judged based on the first preset judgment method, and the first judgment result is output.

[0016] If the first determination result indicates that the acceleration value meets the driving speed change condition and the pedal position change value is greater than the preset pedal position change threshold, then the vehicle driving fatigue state is determined to be a vehicle driving fatigue state.

[0017] Preferably, the driving speed change condition includes the acceleration value being less than a preset acceleration threshold, and the acceleration value being negative.

[0018] Preferably, the facial state information includes the facial feature point location information of the driver's facial feature points, and the facial feature point location information is determined based on the RGB image and depth image of the driver's face;

[0019] Determining the driver's fatigue state based on the facial state information and the hand position information includes:

[0020] Based on the facial feature point location information, determine the driver's facial state changes within the preset time period;

[0021] The facial state change information and the hand position change value are judged based on the second preset judgment method, and the second judgment result is output.

[0022] If the second determination result indicates that the facial state change information meets the facial state change condition, or the hand position change value is greater than the preset hand position change threshold, then the driver's driving fatigue state is determined to be a fatigued driving state.

[0023] Preferably, the facial feature point location information includes facial organ feature point location information and facial orientation feature point location information;

[0024] The step of determining the driver's facial state changes within the preset time period based on the facial feature point location information includes:

[0025] Based on the location information of the facial feature points, determine the facial organ change information of the driver's facial organs; and

[0026] The facial orientation change information of the driver's face is determined based on the facial orientation feature point location information.

[0027] Preferably, the facial organ change information includes eye change information and mouth change information;

[0028] The step of determining the facial organ change information of the driver's facial organs based on the facial organ feature point location information includes:

[0029] The aspect ratio of multiple eyes of the driver within the preset time period is determined based on the location information of the driver's eye feature points;

[0030] Determine, from among the plurality of eye aspect ratios, target eye aspect ratios that are lower than a preset eye aspect ratio threshold, and the number of such target eye aspect ratios; and

[0031] Based on the location information of the driver's mouth feature points, determine the ratio of multiple mouth aspect ratios of the driver within the preset time period;

[0032] Among the plurality of mouth aspect ratios, a target mouth aspect ratio higher than a preset mouth aspect ratio threshold and the number of such target mouth aspect ratios are determined.

[0033] Preferably, the facial state change conditions include: the number of the target mouth aspect ratio values ​​is greater than a preset value, or the facial orientation change information is greater than a preset orientation angle change value.

[0034] Secondly, embodiments of this application provide a fatigue driving warning device, comprising:

[0035] The acquisition module is used to acquire the driving status information of the target vehicle and the driving status information of the driver of the target vehicle. The driving status information includes the driving speed of the target vehicle and the pedal position information of the accelerator pedal, and the driving status information includes the driver's facial status information and hand position information.

[0036] The determination module is used to determine the vehicle fatigue state of the target vehicle based on the driving speed and the pedal position information, and to determine the driver fatigue state based on the facial state information and the hand position information.

[0037] The determining module is further configured to determine the target fatigue driving level of the target vehicle from multiple fatigue driving levels based on the vehicle's driving fatigue state and the driver's fatigue state, wherein each of the multiple fatigue driving levels corresponds to a different fatigue driving warning scheme.

[0038] The execution module is used to issue a fatigue driving warning to the target vehicle based on the fatigue driving warning scheme corresponding to the target fatigue driving level.

[0039] Thirdly, embodiments of this application provide an electronic device, including: a processor, and a memory storing a program, wherein the program includes instructions that, when executed by the processor, cause the processor to perform the method according to the first aspect.

[0040] Fourthly, embodiments of this application provide a non-transitory machine-readable medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method according to the first aspect.

[0041] The beneficial effects of the embodiments of this application are as follows:

[0042] The fatigue driving warning method provided in this application can solve the problems of poor judgment effect and efficiency in related technologies regarding the driver's fatigue driving state. In practical applications, the driving state information of the target vehicle and the driving state information of the target vehicle driver can be obtained. In this embodiment, the driving state information includes the target vehicle's driving speed and accelerator pedal position information, and the driving state information includes the driver's facial state information and hand position information. Thus, the vehicle's driving fatigue state can be determined based on the aforementioned driving speed and pedal position information, and the driver's driving fatigue state can be determined based on the aforementioned facial state information and hand position information. Then, the vehicle's driving fatigue state and the driver's driving fatigue state can be combined to perform a comprehensive analysis of the target vehicle and the driver, and determine the target fatigue level of the target vehicle among multiple fatigue driving levels. Since each of the aforementioned multiple fatigue driving levels corresponds to a different fatigue driving warning scheme, fatigue driving warning prompts can be executed on the target vehicle based on the fatigue driving warning scheme corresponding to the target fatigue driving level. Based on the above method, by combining driving data such as vehicle driving status information and driver driving status information, the accuracy and efficiency of judging driver fatigue driving status can be improved. Furthermore, fatigue driving is divided into multiple levels, and different warning schemes are designed for each level. This method can provide personalized warnings based on the actual fatigue level of the driver, avoiding a one-size-fits-all warning approach and improving the pertinence and effectiveness of the warnings.

[0043] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0045] Figure 1 A flowchart of a fatigue driving warning method provided for an exemplary embodiment of this application.

[0046] Figure 2 A flowchart illustrating a method for determining vehicle driving fatigue state, provided as an exemplary embodiment of this application.

[0047] Figure 3 A flowchart of a method for determining driving fatigue state provided as an exemplary embodiment of this application.

[0048] Figure 4 This is a schematic diagram of a target eye feature point provided for an exemplary embodiment of this application.

[0049] Figure 5 This is a schematic diagram of a target mouth feature point provided for an exemplary embodiment of this application.

[0050] Figure 6 This is a schematic diagram of a fatigue driving warning device provided as an exemplary embodiment of this application.

[0051] Figure 7 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this application. Detailed Implementation

[0052] Embodiments of this embodiment will now be described in more detail with reference to the accompanying drawings. While some embodiments of this embodiment are shown in the drawings, it should be understood that this embodiment can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this embodiment. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of this embodiment.

[0053] Fatigue driving refers to a state in which a driver's driving ability is impaired due to physical or mental fatigue. In this state, the driver's attention, reaction speed, judgment, and coordination are all affected, significantly increasing the risk of traffic accidents. Fatigue driving can be caused by a variety of factors, including prolonged driving, lack of sleep, work or life stress, night driving, and certain health problems or medication side effects. The main characteristics of fatigue driving include: inattention, slow reaction time, impaired judgment, blurred vision, and drowsiness.

[0054] Currently, traditional methods in related technologies mainly determine whether a driver is fatigued by acquiring and processing facial images. However, these methods rely on limited data and cannot accurately assess driver fatigue, resulting in poor accuracy and efficiency in determining driver fatigue and negatively impacting user experience.

[0055] To address the problem of poor accuracy and efficiency in judging driver fatigue in related technologies, this application provides a fatigue driving early warning method.

[0056] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0057] Figure 1 A flowchart illustrating a fatigue driving warning method provided as an exemplary embodiment of this application. See also... Figure 1 This method can be applied to automobiles or electronic devices that establish a communication connection with automobiles. The method includes the following steps.

[0058] Step S101: Obtain the driving status information of the target vehicle and the driving status information of the target vehicle driver. The driving status information includes the driving speed of the target vehicle and the pedal position information of the accelerator pedal. The driving status information includes the driver's facial status information and hand position information.

[0059] Step S102: Determine the vehicle fatigue state of the target vehicle based on the driving speed and pedal position information, and determine the driver fatigue state based on facial state information and hand position information.

[0060] Step S103: Based on the vehicle's driving fatigue state and the driver's fatigue state, determine the target fatigue driving level of the target vehicle among multiple fatigue driving levels. Each of the multiple fatigue driving levels corresponds to a different fatigue driving warning scheme.

[0061] Step S104: Execute a fatigue driving warning prompt for the target vehicle based on the fatigue driving warning scheme corresponding to the target fatigue driving level.

[0062] The fatigue driving warning method provided in this application can be applied to automobiles or electronic devices that establish a communication connection with automobiles. Specifically, it can be applied to in-vehicle computers, laptops, mobile phones, or tablets that establish a connection with in-vehicle computers.

[0063] By acquiring the driving status information of the target vehicle and the driving status information of the target vehicle driver, it is possible to obtain the vehicle's speed, accelerator pedal position information, driver's facial and hand position information within a preset time period.

[0064] In practical applications, onboard computers or electronic devices that establish communication connections with the vehicle can obtain real-time data about the car and engine through the On-Board Diagnostics (OBD) interface to obtain information such as the target vehicle's speed and the position of the accelerator pedal; they can also obtain the driver's facial information through onboard cameras, which may include RGB cameras and depth cameras; and they can obtain the driver's hand position information through wearable devices, which may include smartwatches.

[0065] Furthermore, when acquiring the driving status information of the target vehicle and the driving status information of the target vehicle driver, specifically:

[0066] Based on the driving status information, determine the changes in the driving status of the target vehicle within a preset time period, and based on the driving status information, determine the changes in the driving status of the driver within a preset time period.

[0067] In this embodiment, the driving state change information includes speed change value determined based on driving speed and pedal position change value determined based on pedal position information; the driving state change information includes facial state change value determined based on facial state information and hand position change value determined based on hand position information.

[0068] In practical applications, the preset time period can include any time period. Specifically, the duration of the preset time period can include one minute, five minutes, or thirty minutes, etc. The embodiments of this application do not limit the duration of the preset time period.

[0069] After obtaining the driving speed at each moment within the aforementioned preset time period, the speed change value of the target vehicle within the aforementioned preset time period can be determined; correspondingly, based on the same principle, the pedal position change value can be determined based on the pedal position information; the facial state change value can be determined based on the facial state information; and the hand position change value can be determined based on the hand position information.

[0070] After collecting and organizing the data obtained above, the vehicle's driving fatigue state can be determined based on driving speed and pedal position information, and the driver's driving fatigue state can be determined based on facial and hand position information.

[0071] Figure 2 A flowchart illustrating a method for determining vehicle driving fatigue state as provided in an exemplary embodiment of this application. See also... Figure 2 The method includes the following steps.

[0072] Step S201: Determine the acceleration value of the target vehicle within a preset time period based on the speed change value.

[0073] Step S202: Based on the first preset judgment method, judge the acceleration value and the pedal position change value, and output the first judgment result.

[0074] Step S203: If the first judgment result shows that the acceleration value meets the driving speed change condition and the pedal position change value is greater than the preset pedal position change threshold, then the vehicle driving fatigue state is determined to be the vehicle driving fatigue state.

[0075] In this embodiment, the acceleration value can be determined based on the target vehicle's speed at the beginning of the preset time period, the speed at the end of the preset time period, and the duration of the preset time period.

[0076] In practical applications, the first preset judgment method may include judging whether the acceleration value meets the driving speed change condition and whether the pedal position change value is greater than a preset pedal position change threshold. If the first judgment result indicates that the acceleration value meets the driving speed change condition and the pedal position change value is greater than the preset pedal position change threshold, then the vehicle driving fatigue state is determined to be a vehicle driving fatigue state.

[0077] In one optional embodiment, the driving speed change condition may include an acceleration value that is less than a preset acceleration threshold and an acceleration value that is negative.

[0078] In one optional embodiment, the pedal position change threshold may include 0.05m, and the preset acceleration threshold may include 20m / s². 2 .

[0079] Figure 3 A flowchart illustrating a method for determining driver fatigue state, provided as an exemplary embodiment of this application. See also... Figure 3 The method includes the following steps. In this embodiment, it should be noted that the facial state information includes the facial feature point location information of the driver's facial feature points, wherein the facial feature point location information is determined based on the RGB image and depth image of the driver's face.

[0080] Step S301: Determine the driver's facial state changes within a preset time period based on the facial feature point location information.

[0081] Step S302: Based on the second preset judgment method, judge the facial state change information and the hand position change value, and output the second judgment result.

[0082] Step S303: If the second judgment result shows that the facial state change information meets the facial state change condition, or the hand position change value is greater than the preset hand position change threshold, then the driver's driving fatigue state is determined to be a fatigued driving state.

[0083] In this embodiment, in practical applications, facial feature point calibration can be achieved using a cascaded regression algorithm from a cross-platform general library (e.g., the Dlib library). Facial feature point location information can include facial organ feature point location information and facial orientation feature point location information.

[0084] In one optional embodiment, when determining the driver's facial state change information within a preset time period based on the facial feature point location information, the driver's facial organ change information can be determined based on the facial organ feature point location information; and the driver's facial orientation change information can be determined based on the facial orientation feature point location information.

[0085] Furthermore, information on changes in facial organs includes information on changes in the eyes and mouth.

[0086] When determining the facial organ changes of a driver based on the location information of facial organ feature points, multiple aspect ratios of the driver's eyes within a preset time period can be determined based on the location information of the driver's eye feature points. Then, target aspect ratios and the number of target aspect ratios that are lower than a preset aspect ratio threshold can be determined from among the multiple aspect ratios. Similarly, multiple aspect ratios of the driver's mouth can be determined based on the location information of the driver's mouth feature points within a preset time period. Then, target aspect ratios and the number of target mouth aspect ratios that are higher than a preset mouth aspect ratio threshold can be determined from among the multiple mouth aspect ratios.

[0087] In an optional embodiment, the facial state change conditions include: the number of target mouth aspect ratio ratios is greater than a preset value, or the facial orientation change information is greater than a preset orientation angle change value.

[0088] In practical applications, when determining the aspect ratio of multiple eyes of a driver within a preset time period based on the location information of the driver's eye feature points, a preset number of target eye feature points can be identified from the driver's multiple eye feature points. In this embodiment, six eye feature points of the driver are selected as an example. Figure 4 As shown, in this embodiment, the target eye feature points sequentially include a first eye feature point, a second eye feature point, a third eye feature point, a fourth eye feature point, a fifth eye feature point, and a sixth eye feature point. The coordinate information of the first eye feature point, the second eye feature point, the third eye feature point, the fourth eye feature point, the fifth eye feature point, and the sixth eye feature point is represented by P1, P2, P3, P4, P5, and P6, respectively.

[0089] In this embodiment, the ratio of the driver's eye aspect ratio can be calculated based on the following formula (1):

[0090]

[0091] Here, EAR represents the aspect ratio of the driver's eyes.

[0092] In this embodiment, after determining multiple eye aspect ratios of the driver within a preset time period, these multiple eye aspect ratios can be filtered based on a preset eye aspect ratio threshold to identify target eye aspect ratios lower than the preset threshold, and the number of these target eye aspect ratios can be counted. In an optional embodiment, the number of target eye aspect ratios can also be determined by real-time counting. When the number of target eye aspect ratios is greater than a preset value, the driver's driving fatigue state is determined to be a state of fatigued driving.

[0093] Optionally, in this embodiment, the preset eye aspect ratio threshold may include 0.2. When the driver's eye aspect ratio is less than 0.2, it can be determined that the driver is in a closed-eye state.

[0094] When determining the aspect ratio of multiple mouth features within a preset time period based on the location information of the driver's mouth feature points, a preset number of target eye feature points can be identified from the multiple mouth feature points of the driver. In this embodiment, six mouth feature points of the driver are selected as an example. Figure 5 As shown, in this embodiment, the target mouth feature points sequentially include a first mouth feature point, a second mouth feature point, a third mouth feature point, a fourth mouth feature point, a fifth mouth feature point, and a sixth mouth feature point. The coordinate information of the first mouth feature point, the second mouth feature point, the third mouth feature point, the fourth mouth feature point, the fifth mouth feature point, and the sixth mouth feature point is represented by Dot1, Dot2, Dot3, Dot4, Dot5, and Dot6, respectively.

[0095] In this embodiment, the aspect ratio of the driver's mouth can be calculated based on the following formula (2):

[0096]

[0097] MAR represents the aspect ratio of the driver's mouth.

[0098] In this embodiment, after determining multiple mouth aspect ratios for the driver within a preset time period, these ratios can be filtered based on a preset mouth aspect ratio threshold to identify target mouth aspect ratios higher than the threshold, and the number of these target mouth aspect ratios can be counted. In an optional embodiment, the number of target mouth aspect ratios can also be determined by real-time counting. When the number of target mouth aspect ratios exceeds a preset value, the driver's driving fatigue state is determined to be a state of fatigued driving.

[0099] Optionally, in this embodiment, the preset eye aspect ratio threshold may include 0.4. When the aspect ratio of the driver's mouth is greater than 0.4, it can be determined that the driver is yawning.

[0100] When determining the driver's facial orientation change information based on the facial orientation feature point location information, a preset number of target facial orientation feature points can be identified from multiple facial orientation feature points of the driver. In this embodiment, six target facial orientation feature points of the driver are selected as an example. In this embodiment, based on the facial orientation feature point location information of the above-mentioned target facial orientation feature points, the intrinsic parameters (camera matrix) of the vehicle camera, and the distortion coefficient, the rotation vector of the above-mentioned target facial orientation feature points is determined. Then, the above-mentioned rotation vector is converted into a rotation matrix, and then the corresponding Euler angles are extracted from the rotation matrix. Finally, the driver's facial orientation can be determined based on the value of the Euler angles, thereby obtaining the facial orientation change information.

[0101] In practical applications, based on the facial orientation feature point location information, the intrinsic parameters (camera matrix), and distortion coefficients of the vehicle camera, the rotation vector of the facial orientation feature point can be obtained using the `solvePnP` function of OpenCV (a cross-platform computer vision library). Furthermore, the rotation vector can be converted into a rotation matrix using either a rotation matrix-based method or a quaternion-based method.

[0102] In an alternative embodiment, the facial feature point location information is determined based on an RGB image of the driver's face, a depth image, or both an RGB image and a depth image.

[0103] In this embodiment, a fusion strategy of RGB and depth images is employed. During daylight with sufficient illumination, the RGB image model corresponding to the RGB image is used as the primary model for fusion, while the depth image model corresponding to the depth image is used for correction. Under low-light conditions, the depth image model corresponding to the depth image is used as the primary model for fusion. Based on this method, the facial feature point location information of the driver's face can be clearly captured under all conditions, including strong light, low light, direct light, uneven lighting, and complete darkness.

[0104] In the above embodiments, the driver's facial state change information is determined based on multiple facial feature points, which may include facial organ feature points and facial orientation feature points. Specifically, facial organ feature points may further include eye feature points and mouth feature points.

[0105] In one optional embodiment, multiple key facial feature points can be determined based on the driver's RGB image and depth image. Then, eye feature points, mouth feature points, and facial orientation feature points can be determined from these multiple key facial feature points. Prior to this, the driver's RGB image and depth image can be processed using an RGB image model and a depth image model to determine the aforementioned multiple key facial feature points from the driver's RGB image and depth image.

[0106] In this embodiment, when training the RGB image model, the driver's RGB image is first annotated with key facial feature points using methods such as manual annotation to generate training samples. These training samples are then used to train the RGB image model, resulting in a trained RGB image model. This model can be used to identify, detect, and annotate the coordinates of multiple key facial feature points on the driver's RGB facial image. In practical applications, the RGB image model may include a face detector provided by the dlib library. The dlib library is a cross-platform, general-purpose library written using modern C++ technology.

[0107] Furthermore, when training the depth image model, the RGB image can first be converted into a depth image, and edge detection is performed in the depth image. Then, keypoint detection is performed on both the RGB and depth images using a scale-invariant feature transform algorithm or an accelerated robust feature algorithm. Features are then extracted from the region surrounding each keypoint to generate descriptors. Finally, the generated descriptors are used to match the keypoints in the RGB and depth images, thereby achieving registration between the RGB and depth images. The registered depth image and its corresponding facial key feature points are used as training samples. The RGB image model is then trained based on these training samples to obtain the depth image model. This depth image model can be used to identify, detect, and annotate the coordinates of multiple facial key feature points in a driver's facial depth image.

[0108] Based on the above embodiments, the fusion strategy of RGB image and depth image can be executed through RGB image model and depth image model to achieve the technical effect of clearly capturing the facial feature point position information of the driver's facial feature points under all conditions such as strong light, weak light, direct light, uneven lighting and complete darkness.

[0109] In an optional embodiment, the hand position change value may include the hand acceleration value of the hand movement. In this embodiment, the hand acceleration value can be obtained through an electronic device worn by the driver's hand, wherein the electronic device may include an accelerometer, a smartwatch, etc.

[0110] In this embodiment, the hand position change threshold may include a hand acceleration threshold. When the hand acceleration value is greater than the hand acceleration threshold, the driver's driving fatigue state can be determined as a fatigued driving state.

[0111] Once the vehicle's fatigue state and driver fatigue state are determined, the target fatigue driving level of the target vehicle can be determined from multiple fatigue driving levels based on the aforementioned vehicle fatigue state and driver fatigue state.

[0112] In one optional embodiment, driver fatigue can be categorized into facial organ fatigue, facial orientation fatigue, and hand fatigue. Based on the above, vehicle driving fatigue and driver fatigue can specifically include vehicle driving fatigue, facial organ fatigue, facial orientation fatigue, and hand fatigue. Therefore, the target fatigue level of the target vehicle can be determined based on vehicle driving fatigue, facial organ fatigue, facial orientation fatigue, and hand fatigue.

[0113] In practical applications, the target fatigue driving level of a target vehicle can be determined based on the number of fatigue states mentioned above, including vehicle driving fatigue state, facial organ fatigue state, facial orientation fatigue state, and hand fatigue state.

[0114] Specifically, multiple levels of fatigue driving can include Level 1, Level 2, Level 3, and Level 4 fatigue driving. Correspondingly, if the vehicle's fatigue state and the driver's fatigue state satisfy any one of the aforementioned fatigue states of vehicle driving, facial organ fatigue, facial orientation fatigue, and hand fatigue, the target vehicle can be determined to be Level 1 fatigue driving; if the vehicle's fatigue state and the driver's fatigue state satisfy any two of these states, the target vehicle can be determined to be Level 2 fatigue driving; if the vehicle's fatigue state and the driver's fatigue state satisfy any three of these states, the target vehicle can be determined to be Level 3 fatigue driving; and if both the vehicle's fatigue state and the driver's fatigue state satisfy all of these states, the target vehicle can be determined to be Level 4 fatigue driving.

[0115] Once the target fatigue driving level of the target vehicle is determined, a fatigue driving warning can be issued to the target vehicle based on the fatigue driving warning scheme corresponding to the target fatigue driving level.

[0116] In this embodiment, different levels of driver fatigue correspond to different driver fatigue warning schemes. Specifically, driver fatigue warning schemes may include voice reminders, image reminders, and haptic reminders. For example, voice reminders can play warning sounds through the car audio system; image reminders may include displaying warning images; and haptic reminders may include seat vibration, mobile phone vibration, or smartwatch vibration.

[0117] In one optional embodiment, for Level 1 and Level 2 fatigue driving, a fatigue driving warning scheme can be constructed using voice reminders; for Level 3 fatigue driving, a fatigue driving warning scheme can be constructed using haptic reminders; and for Level 4 fatigue driving, a fatigue driving warning scheme can be constructed using a combination of voice reminders, image reminders, and haptic reminders. It should be noted that the above fatigue driving warning schemes are merely illustrative embodiments, and custom settings can be made for fatigue driving warning schemes at different fatigue driving levels according to specific implementation needs.

[0118] Based on the fatigue driving warning method provided in the embodiments of this application, the embodiments of this application also provide a fatigue driving warning device, such as... Figure 6 As shown, the fatigue driving warning device includes: an acquisition module 601, a determination module 602, and an execution module 603.

[0119] The acquisition module 601 is used to acquire the driving status information of the target vehicle and the driving status information of the driver of the target vehicle. The driving status information includes the driving speed of the target vehicle and the pedal position information of the accelerator pedal, and the driving status information includes the driver's facial status information and hand position information.

[0120] The determination module 602 is used to determine the vehicle driving fatigue state of the target vehicle based on the driving speed and pedal position information, and to determine the driver's driving fatigue state based on facial state information and hand position information.

[0121] The determination module 602 is also used to determine the target fatigue driving level of the target vehicle among multiple fatigue driving levels based on the vehicle's driving fatigue state and driver fatigue state. Each of the multiple fatigue driving levels corresponds to a different fatigue driving warning scheme.

[0122] The execution module 603 is used to provide fatigue driving warning prompts to the target vehicle based on the fatigue driving warning scheme corresponding to the target fatigue driving level.

[0123] Optionally, the acquisition module 601 is specifically used to determine the driving status change information of the target vehicle within a preset time period based on the driving status information. The driving status change information includes the speed change value determined based on the driving speed and the pedal position change value determined based on the pedal position information.

[0124] The determining module 602 is also used to determine the driver's driving status change information within a preset time period based on the driving status information. The driving status change information includes facial status change values ​​determined based on facial status information and hand position change values ​​determined based on hand position information.

[0125] Optionally, the determining module 602 is specifically used to determine the acceleration value of the target vehicle within a preset time period based on the speed change value; to judge the acceleration value and the pedal position change value based on a first preset judgment method, and to output a first judgment result; if the first judgment result indicates that the acceleration value meets the driving speed change condition and the pedal position change value is greater than a preset pedal position change threshold, then the vehicle driving fatigue state is determined to be the vehicle driving fatigue state.

[0126] Optionally, the driving speed change conditions include acceleration values ​​being less than a preset acceleration threshold and acceleration values ​​being negative.

[0127] Optionally, the facial state information includes the location information of facial feature points of the driver's face, which is determined based on the RGB image and depth image of the driver's face.

[0128] Accordingly, the determining module 602 is specifically used to determine the driver's facial state change information within a preset time period based on the facial feature point location information; to judge the facial state change information and hand position change value based on the second preset judgment method, and output the second judgment result; if the second judgment result shows that the facial state change information meets the facial state change condition, or the hand position change value is greater than the preset hand position change threshold, then the driver's driving fatigue state is determined to be a fatigued driving state.

[0129] Optionally, the facial feature point location information includes facial organ feature point location information and facial orientation feature point location information.

[0130] Accordingly, the determining module 602 is specifically used to determine the facial organ change information of the driver's facial organs based on the facial organ feature point location information; and to determine the facial orientation change information of the driver's facial orientation based on the facial orientation feature point location information.

[0131] Optionally, facial organ change information includes eye change information and mouth change information.

[0132] Accordingly, the determining module 602 is specifically used to determine multiple aspect ratios of the driver's eyes within a preset time period based on the driver's eye feature point location information; determine target eye aspect ratios and the number of target eye aspect ratios that are lower than a preset eye aspect ratio threshold among the multiple eye aspect ratios; and determine multiple mouth aspect ratios of the driver within a preset time period based on the driver's mouth feature point location information; determine target mouth aspect ratios and the number of target mouth aspect ratios that are higher than a preset mouth aspect ratio threshold among the multiple mouth aspect ratios.

[0133] Optionally, the number of aspect ratios of the target mouth is greater than a preset value, or the facial orientation change information is greater than a preset orientation angle change value.

[0134] This application also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, which, when executed by the at least one processor, causes the electronic device to perform the methods of this application embodiment.

[0135] This application also provides a non-transitory machine-readable medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform the method of this application embodiment.

[0136] This application also provides a computer program product, including a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform the methods of this application embodiment.

[0137] refer to Figure 7 The present invention describes a structural block diagram of an electronic device that can serve as a server or client in embodiments of this application, which is an example of a hardware device that can be applied to various aspects of this application. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the application described and / or claimed herein.

[0138] like Figure 7 As shown, the electronic device includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 702 or a computer program loaded into a random access memory (RAM) 703 from a storage unit 708. The RAM 703 may also store various programs and data required for the operation of the electronic device. The computing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0139] Multiple components in the electronic device are connected to I / O interface 705, including: input unit 706, output unit 707, storage unit 708, and communication unit 709. Input unit 706 can be any type of device capable of inputting information into the electronic device. Input unit 706 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 707 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 708 may include, but is not limited to, disks and optical discs. Communication unit 709 allows the electronic device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0140] The computing unit 701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, CPUs, graphics processing units (GPUs), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above. For example, in some embodiments, the method embodiments of this application can be implemented as computer programs tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be delivered via ROM.

[0141] 702 and / or communication unit 709 are loaded and / or installed on the electronic device. In some embodiments, computing unit 701 may be configured to perform the methods described above by any other suitable means (e.g., by means of firmware).

[0142] Computer programs used to implement the methods of the embodiments of this application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0143] In the context of embodiments of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0144] It should be noted that the term "comprising" and its variations used in the embodiments of this application are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; and the term "some embodiments" means "at least some embodiments". The modifications of "one" and "multiple" mentioned in the embodiments of this application are illustrative and not restrictive. Those skilled in the art should understand that, unless explicitly indicated otherwise in the context, they should be understood as "one or more".

[0145] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0146] The steps described in the method embodiments provided in this application can be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of protection of this application is not limited in this respect.

[0147] The term "embodiment" in this specification refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily imply the same embodiment, nor does it imply independence from or alternative to other embodiments. The various embodiments in this specification are described in a related manner, with reference to each other for similar or identical parts. In particular, for apparatus, device, and system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, and relevant details are referred to in the description of the method embodiments.

[0148] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A method of drowsy driving warning, characterized by, The method comprises: obtaining driving state information of a target vehicle and driving state information of a driver of the target vehicle, wherein the driving state information comprises driving speed and pedal position information of an accelerator pedal of the target vehicle, and the driving state information comprises face state information and hand position information of the driver; determining a vehicle driving fatigue state of the target vehicle according to the driving speed and the pedal position information, and determining a driving fatigue state of the driver according to the face state information and the hand position information; determining a target fatigue driving level of the target vehicle in a plurality of fatigue driving levels according to the vehicle driving fatigue state and the driving fatigue state, wherein each fatigue driving level corresponds to a different fatigue driving warning scheme; performing fatigue driving warning prompting on the target vehicle based on the fatigue driving warning scheme corresponding to the target fatigue driving level; the determination of the vehicle driving fatigue state of the target vehicle according to the driving speed and the pedal position information comprises: determining an acceleration value of the target vehicle in a preset time period according to a speed change value; judging the acceleration value and a pedal position change value based on a first preset judgment method, and outputting a first judgment result, wherein the first preset judgment method comprises judging whether the acceleration value satisfies a driving speed change condition and whether the pedal position change value is greater than a preset pedal position change threshold value; if the first judgment result indicates that the acceleration value satisfies the driving speed change condition and the pedal position change value is greater than the preset pedal position change threshold value, the vehicle driving fatigue state is determined as a vehicle driving fatigue state; the driving speed change condition comprises that the acceleration value is less than a preset acceleration threshold value and the acceleration value is negative; the face state information comprises face feature point position information of face feature points of the driver, and the face feature point position information is determined based on an RGB image and a depth image of the face of the driver; the determination of the driving fatigue state of the driver according to the face state information and the hand position information comprises: determining face state change information of the driver in the preset time period according to the face feature point position information; judging the face state change information and a hand position change value based on a second preset judgment method, and outputting a second judgment result, wherein the second preset judgment method comprises judging whether the face state change information satisfies a face state change condition or whether the hand position change value is greater than a preset hand position change threshold value; if the second judgment result indicates that the face state change information satisfies the face state change condition or the hand position change value is greater than the preset hand position change threshold value, the driving fatigue state of the driver is determined as a fatigue driving state; wherein the face feature point position information comprises face organ feature point position information and face orientation feature point position information; the determination of the face state change information of the driver in the preset time period according to the face feature point position information comprises: determine face organ change information of the driver face organ according to the face organ feature point position information; and determine face orientation change information of the driver face orientation according to the face orientation feature point position information; the hand position change value includes a hand acceleration value of hand movement, the hand acceleration value is obtained by an electronic device worn by the driver hand; the face state change condition includes: a quantity of target mouth aspect ratio ratio is greater than a preset value, or the face orientation change information is greater than a preset orientation angle change value; the target fatigue driving level of the target vehicle is determined according to the fatigue state quantity of the vehicle driving fatigue state, the face organ fatigue state, the face orientation fatigue state and the hand fatigue state; the plurality of fatigue driving levels include a first fatigue driving, a second fatigue driving, a third fatigue driving and a fourth fatigue driving; if the vehicle driving fatigue state and the driving fatigue state satisfy any one of the vehicle driving fatigue state, the face organ fatigue state, the face orientation fatigue state and the hand fatigue state, the target vehicle is determined as the first fatigue driving; if the vehicle driving fatigue state and the driving fatigue state satisfy any two of the vehicle driving fatigue state, the face organ fatigue state, the face orientation fatigue state and the hand fatigue state, the target vehicle is determined as the second fatigue driving; if the vehicle driving fatigue state and the driving fatigue state satisfy any three of the vehicle driving fatigue state, the face organ fatigue state, the face orientation fatigue state and the hand fatigue state, the target vehicle is determined as the third fatigue driving; if the vehicle driving fatigue state and the driving fatigue state satisfy the vehicle driving fatigue state, the face organ fatigue state, the face orientation fatigue state and the hand fatigue state, the target vehicle is determined as the fourth fatigue driving.

2. The fatigue driving warning method according to claim 1, characterized by, the obtained driving state information of the target vehicle and the driving state information of the driver of the target vehicle include: determine driving state change information of the target vehicle in a preset time period according to the driving state information, the driving state change information includes a speed change value determined based on the driving speed and a pedal position change value determined based on the pedal position information; and determine driving state change information of the driver in a preset time period according to the driving state information, the driving state change information includes a face state change value determined based on the face state information and a hand position change value determined based on the hand position information.

3. The method of claim 1, wherein, the face organ change information includes eye change information and mouth change information; the determination of the face organ change information of the driver face organ according to the face organ feature point position information includes: determine a plurality of eye aspect ratio ratios of the driver in the preset time period according to the eye feature point position information of the driver; determining a target eye aspect ratio ratio and a quantity of the target eye aspect ratio ratio from the plurality of eye aspect ratio ratios, which are lower than a preset eye aspect ratio threshold value; and determining a plurality of mouth aspect ratio ratios of the driver within the preset time period according to the mouth feature point position information of the driver; determining a target mouth aspect ratio ratio and a quantity of the target mouth aspect ratio from the plurality of mouth aspect ratio ratios, which are higher than a preset mouth aspect ratio threshold value.

4. The drowsy driving warning method according to claim 3, characterized by, The face state change condition comprises that the quantity of the target mouth aspect ratio ratio is greater than a preset value, or the face orientation change information is greater than a preset orientation angle change value.

5. A fatigue driving warning device for implementing the fatigue driving warning method according to any one of claims 1 to 4, characterized by, The method comprises: an acquisition module, configured to acquire driving state information of a target vehicle and driving state information of a driver of the target vehicle, wherein the driving state information comprises driving speed and pedal position information of an accelerator pedal of the target vehicle, and the driving state information comprises face state information and hand position information of the driver; a determination module, configured to determine a vehicle driving fatigue state of the target vehicle according to the driving speed and the pedal position information, and determine a driving fatigue state of the driver according to the face state information and the hand position information; the determination module is further configured to determine a target fatigue driving level of the target vehicle from a plurality of fatigue driving levels according to the vehicle driving fatigue state and the driving fatigue state, wherein each fatigue driving level corresponds to a different fatigue driving early warning scheme; an execution module, configured to perform fatigue driving early warning prompting on the target vehicle based on the fatigue driving early warning scheme corresponding to the target fatigue driving level.

6. An electronic device comprising: A processor and a memory storing a program, wherein the program includes instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-4.

7. A non-transitory machine-readable medium having stored thereon computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-4.

Images