A control method, apparatus, equipment, medium, product, and vehicle

By acquiring vehicle driving data and driver head images, the system determines the level of abnormality and executes safety controls, thus solving the problem of traffic accidents caused by driver fatigue and distraction, and achieving effective driver monitoring and alerting.

CN123061391APending Publication Date: 2026-11-10BEIJING CO WHEELS TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510585257.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-07
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

How to effectively monitor and remind drivers to prevent traffic accidents caused by subjective factors such as fatigue driving and distracted driving.

Method used

By acquiring vehicle driving data and multiple consecutive frames of driver head images within a safety monitoring period, the driver's abnormality level is determined, and corresponding safety controls, such as alerts and vehicle operation interventions, are implemented based on this level.

Benefits of technology

It enables accurate monitoring and tiered alerts for drivers, effectively preventing traffic accidents.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123061391A_ABST
    Figure CN123061391A_ABST
Patent Text Reader

Abstract

This invention discloses a control method, apparatus, device, medium, product, and vehicle. The method includes: acquiring vehicle driving data and multiple consecutive frames of driver head images within a safety monitoring period; determining a target driver anomaly level based on the vehicle driving data and the driver anomaly type corresponding to each frame of driver head images; and executing corresponding safety controls based on the target driver anomaly level. Through the technical solution of this invention, drivers can be monitored and alerted to effectively prevent traffic accidents.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of vehicle technology, and in particular to a control method, device, equipment, medium, product and vehicle. Background Technology

[0002] With the deepening development of vehicle intelligence and the continuous improvement of people's living standards, cars have gradually entered millions of households. Consequently, traffic accidents are also increasing year by year, with subjective factors such as fatigued driving and distracted driving becoming the main causes of road accidents.

[0003] Therefore, how to monitor and remind drivers to effectively prevent traffic accidents has become an urgent problem to be solved. Summary of the Invention

[0004] This invention provides a control method, device, equipment, medium, product, and vehicle that can monitor and remind drivers to effectively prevent traffic accidents.

[0005] According to one aspect of the present invention, a control method is provided, comprising:

[0006] Acquire vehicle driving data and multiple consecutive frames of driver head images within a safety monitoring period;

[0007] The target driver's abnormality level is determined based on the vehicle's driving data and the driver's abnormality type corresponding to each frame of the driver's head image.

[0008] Based on the target driver's abnormality level, corresponding safety controls are implemented.

[0009] In some embodiments of the present invention, before determining the target driver anomaly level based on the vehicle's driving data and the driver anomaly type corresponding to each frame of the driver's head image, the method further includes:

[0010] Obtain the head deviation angle threshold;

[0011] Based on the head deviation angle threshold, the driver's head images of multiple consecutive frames within the safety monitoring period are classified and identified to obtain the driver's abnormality type corresponding to each frame of driver's head images.

[0012] In some embodiments of the present invention, the target driver anomaly level is determined based on the vehicle's driving data and the driver anomaly type corresponding to each frame of the driver's head image, including:

[0013] The duration of the driver abnormality type is determined based on the driver abnormality type corresponding to each frame of driver head image;

[0014] Obtain the trigger time range for each driver's abnormality level corresponding to the vehicle's driving data;

[0015] The driver abnormality level corresponding to the trigger time range in which the duration of the driver abnormality type falls is taken as the target driver abnormality level.

[0016] In some embodiments of the present invention, the driving data includes: vehicle speed, driving mode, and attention alert mode;

[0017] The trigger time range for each driver anomaly level corresponding to the vehicle's driving data is obtained, including:

[0018] If the vehicle's driving mode is assisted driving mode and the attention reminder mode is standard mode, then obtain the trigger time range of each driver abnormality level corresponding to the vehicle speed.

[0019] If the vehicle's driving mode is assisted driving mode and the attention alert mode is sensitive mode, then obtain the trigger time range for each driver's abnormal level.

[0020] In some embodiments of the present invention, the trigger time range limit for each driver abnormality level corresponding to the standard mode is greater than the trigger time range limit for each driver abnormality level corresponding to the sensitive mode.

[0021] In some embodiments of the present invention, before acquiring the vehicle's driving data, the method further includes:

[0022] Upon detecting a trigger operation on the target attention reminder control, the attention reminder mode is switched to the attention reminder mode corresponding to the target attention reminder control. The attention reminder mode includes: standard mode and sensitive mode.

[0023] or,

[0024] Switch the attention alert mode to the attention alert mode corresponding to time information and / or environmental information, whereby the environmental information includes: weather information and / or traffic information.

[0025] In some embodiments of the present invention, corresponding safety controls are performed based on the target driver's abnormality level, including:

[0026] Based on the target driver's abnormality level, a target alert message is generated, wherein the target alert message includes at least one of: pop-up alert message, voice alert message, and head-up display alert message;

[0027] Security control is performed based on the target alert information.

[0028] In some embodiments of the present invention, it further includes:

[0029] Based on the target driver's abnormality level, generate safety control commands;

[0030] The safety control command is sent to the control system so that the control system controls the vehicle to stop, decelerate, exit the assisted driving mode, and suspend the use of the assisted driving mode at least one of the following:

[0031] In some embodiments of the present invention, the driver abnormality type includes: a distraction type; the target driver abnormality level includes three distraction levels, wherein the trigger time range limit for distraction level one is less than the trigger time range limit for distraction level two, and the trigger time range limit for distraction level two is less than the trigger time range limit for distraction level three; and corresponding safety controls are executed based on the target driver abnormality level, including:

[0032] If the target driver's abnormality level is level one distraction, then a first pop-up reminder and a first voice reminder will be issued;

[0033] If the target driver's abnormality level is level two distraction, then a first pop-up reminder, a first voice reminder, and a head-up display reminder will be issued;

[0034] If the target driver's abnormality level is Distraction Level 3, a second pop-up reminder, a second voice reminder, and a head-up display reminder will be issued, and the target driver's abnormality level will be sent to the control system so that the control system can control the vehicle. The second pop-up reminder includes a first pop-up reminder and a driver assistance exit and disable reminder. The second voice reminder includes a first voice reminder and a driver assistance exit and disable reminder.

[0035] In some embodiments of the present invention, the driver abnormality type includes: fatigue type, and the target driver abnormality level includes two fatigue levels, wherein the trigger time range limit of fatigue level one is smaller than the trigger time range limit of fatigue level two.

[0036] Based on the target driver's abnormality level, corresponding safety controls are implemented, including:

[0037] If the target driver's abnormality level is fatigue level one, then a third pop-up reminder, a third voice reminder, and a head-up display reminder will be issued;

[0038] If the target driver's abnormal level is fatigue level 2, a fourth pop-up reminder, a fourth voice reminder, and a head-up display reminder will be issued, and the target driver's abnormal level will be sent to the control system so that the control system can control the vehicle. The fourth pop-up reminder includes a third pop-up reminder and a driver assistance exit and disable reminder, and the fourth voice reminder includes a third voice reminder and a driver assistance exit and disable reminder.

[0039] In some embodiments of the present invention, the driver abnormality type includes: making a phone call type, and the target driving abnormality level includes making a phone call level one;

[0040] Based on the target driver's abnormality level, corresponding safety controls are implemented, including:

[0041] If the target driver's abnormality level is Level 1 (making a phone call), then a fifth pop-up reminder will be issued.

[0042] In some embodiments of the present invention, the target driver anomaly level is determined based on the vehicle's driving data and the driver anomaly type corresponding to each frame of the driver's head image, including:

[0043] Get the steering wheel grip status;

[0044] The target driver's abnormality level is determined based on the steering wheel grip status, the vehicle's driving data, and the driver's abnormality type corresponding to each frame of the driver's head image.

[0045] In some embodiments of the present invention, obtaining the trigger time range for each driver abnormality level corresponding to the vehicle speed includes:

[0046] Get the trigger time range of each driver abnormality level corresponding to the first speed range and the trigger time range of each driver abnormality level corresponding to the second speed range, wherein the speed in the first speed range is greater than a first value and less than or equal to a second value, and the speed in the second speed range is greater than or equal to the second value.

[0047] If the vehicle speed is within the first vehicle speed range, then the trigger time range of each driver abnormality level corresponding to the first vehicle speed range shall be used as the trigger time range of each driver abnormality level corresponding to the vehicle speed.

[0048] If the vehicle speed is within the second vehicle speed range, then the trigger time range of each driver abnormality level corresponding to the second vehicle speed range shall be used as the trigger time range of each driver abnormality level corresponding to the vehicle speed.

[0049] In some embodiments of the present invention, the limit of the trigger time range for each distraction level corresponding to the first speed range is greater than the limit of the trigger time range for each distraction level corresponding to the second speed range, and the limit of the trigger time range for each fatigue level corresponding to the first speed range is greater than the limit of the trigger time range for each fatigue level corresponding to the second speed range.

[0050] In some embodiments of the present invention, it further includes:

[0051] The accuracy of receiving feedback information and / or classification and recognition results sent by the camera;

[0052] A first signal is generated based on the accuracy of the returned information and / or classification and identification results, wherein the first signal includes: a status information identifier of the driver monitoring system;

[0053] The first signal is sent to the control system so that the control system controls the vehicle based on the first signal.

[0054] In some embodiments of the present invention, it further includes:

[0055] A second signal is generated based on the driver anomaly type corresponding to each frame of the driver's head image, or based on the target driver anomaly level.

[0056] The second signal is sent to the control system so that the control system controls the vehicle based on the second signal.

[0057] In some embodiments of the present invention, the head deviation angle threshold includes: head top deviation angle threshold, head bottom deviation angle threshold, head left deviation angle threshold and head right deviation angle threshold.

[0058] According to another aspect of the present invention, a control device is provided, the device comprising:

[0059] The acquisition module is used to acquire vehicle driving data and multiple consecutive frames of driver head images within a safety monitoring period.

[0060] The target driver anomaly level determination module is used to determine the target driver anomaly level based on the vehicle's driving data and the driver anomaly type corresponding to each frame of the driver's head image;

[0061] The safety control module is used to perform corresponding safety controls based on the abnormality level of the target driver.

[0062] In some embodiments of the present invention, it further includes:

[0063] The head deviation angle threshold acquisition module is used to acquire the head deviation angle threshold before determining the target driver's abnormality level based on the vehicle's driving data and the driver's abnormality type corresponding to each frame of driver's head image.

[0064] The identification module is used to classify and identify multiple consecutive frames of driver head images within the safety monitoring period based on the head deviation angle threshold, and obtain the driver abnormality type corresponding to each frame of driver head image.

[0065] In some embodiments of the present invention, the target driver anomaly level determination module is specifically used for:

[0066] The duration of the driver abnormality type is determined based on the driver abnormality type corresponding to each frame of driver head image;

[0067] Obtain the trigger time range for each driver's abnormality level corresponding to the vehicle's driving data;

[0068] The driver abnormality level corresponding to the trigger time range in which the duration of the driver abnormality type falls is taken as the target driver abnormality level.

[0069] In some embodiments of the present invention, the driving data includes: vehicle speed, driving mode, and attention alert mode;

[0070] The target driver anomaly level determination module is specifically used for:

[0071] If the vehicle's driving mode is assisted driving mode and the attention reminder mode is standard mode, then obtain the trigger time range of each driver abnormality level corresponding to the vehicle speed.

[0072] If the vehicle's driving mode is assisted driving mode and the attention alert mode is sensitive mode, then obtain the trigger time range for each driver's abnormal level.

[0073] In some embodiments of the present invention, the trigger time range limit for each driver abnormality level corresponding to the standard mode is greater than the trigger time range limit for each driver abnormality level corresponding to the sensitive mode.

[0074] In some embodiments of the present invention, the target driver anomaly level determination module is specifically used for:

[0075] Upon detecting a trigger operation on the target attention reminder control, the attention reminder mode is switched to the attention reminder mode corresponding to the target attention reminder control. The attention reminder mode includes: standard mode and sensitive mode.

[0076] or,

[0077] Switch the attention alert mode to the attention alert mode corresponding to time information and / or environmental information, whereby the environmental information includes: weather information and / or traffic information.

[0078] In some embodiments of the present invention, the safety control module is specifically used for:

[0079] Based on the target driver's abnormality level, a target alert message is generated, wherein the target alert message includes at least one of: pop-up alert message, voice alert message, and head-up display alert message;

[0080] Security control is performed based on the target alert information.

[0081] In some embodiments of the present invention, it further includes:

[0082] A safety control command generation module is used to generate safety control commands based on the abnormality level of the target driver.

[0083] The safety control command sending module is used to send the safety control command to the control system so that the control system controls the vehicle to stop, decelerate, exit the assisted driving mode, and suspend the use of the assisted driving mode at least one of the following:

[0084] In some embodiments of the present invention, the driver abnormality type includes: distraction type; the target driver abnormality level includes three distraction levels; the trigger time range limit for distraction level one is less than the trigger time range limit for distraction level two; the trigger time range limit for distraction level two is less than the trigger time range limit for distraction level three; and the safety control module is specifically used for:

[0085] If the target driver's abnormality level is level one distraction, then a first pop-up reminder and a first voice reminder will be issued;

[0086] If the target driver's abnormality level is level two distraction, then a first pop-up reminder, a first voice reminder, and a head-up display reminder will be issued;

[0087] If the target driver's abnormality level is Distraction Level 3, a second pop-up reminder, a second voice reminder, and a head-up display reminder will be issued, and the target driver's abnormality level will be sent to the control system so that the control system can control the vehicle. The second pop-up reminder includes a first pop-up reminder and a driver assistance exit and disable reminder. The second voice reminder includes a first voice reminder and a driver assistance exit and disable reminder.

[0088] In some embodiments of the present invention, the driver abnormality type includes: fatigue type, and the target driver abnormality level includes two fatigue levels, wherein the trigger time range limit for fatigue level one is smaller than the trigger time range limit for fatigue level two; the safety control module is specifically used for:

[0089] If the target driver's abnormality level is fatigue level one, then a third pop-up reminder, a third voice reminder, and a head-up display reminder will be issued;

[0090] If the target driver's abnormal level is fatigue level 2, a fourth pop-up reminder, a fourth voice reminder, and a head-up display reminder will be issued, and the target driver's abnormal level will be sent to the control system so that the control system can control the vehicle. The fourth pop-up reminder includes a third pop-up reminder and a driver assistance exit and disable reminder, and the fourth voice reminder includes a third voice reminder and a driver assistance exit and disable reminder.

[0091] In some embodiments of the present invention, the driver abnormality type includes: making a phone call type, and the target driving abnormality level includes making a phone call level one; the safety control module is specifically used for:

[0092] If the target driver's abnormality level is Level 1 (making a phone call), then a fifth pop-up reminder will be issued.

[0093] In some embodiments of the present invention, the target driver anomaly level determination module is specifically used for:

[0094] Get the steering wheel grip status;

[0095] The target driver's abnormality level is determined based on the steering wheel grip status, the vehicle's driving data, and the driver's abnormality type corresponding to each frame of the driver's head image.

[0096] In some embodiments of the present invention, the target driver anomaly level determination module is specifically used for:

[0097] Get the trigger time range of each driver abnormality level corresponding to the first speed range and the trigger time range of each driver abnormality level corresponding to the second speed range, wherein the speed in the first speed range is greater than a first value and less than or equal to a second value, and the speed in the second speed range is greater than or equal to the second value.

[0098] If the vehicle speed is within the first vehicle speed range, then the trigger time range of each driver abnormality level corresponding to the first vehicle speed range shall be used as the trigger time range of each driver abnormality level corresponding to the vehicle speed.

[0099] If the vehicle speed is within the second vehicle speed range, then the trigger time range of each driver abnormality level corresponding to the second vehicle speed range shall be used as the trigger time range of each driver abnormality level corresponding to the vehicle speed.

[0100] In some embodiments of the present invention, the limit of the trigger time range for each distraction level corresponding to the first speed range is greater than the limit of the trigger time range for each distraction level corresponding to the second speed range, and the limit of the trigger time range for each fatigue level corresponding to the first speed range is greater than the limit of the trigger time range for each fatigue level corresponding to the second speed range.

[0101] In some embodiments of the present invention, the apparatus further includes:

[0102] The receiving module is used to receive the feedback information and / or the accuracy of the classification and recognition results sent by the camera;

[0103] The first signal generation module is used to generate a first signal based on the accuracy of the returned information and / or classification and identification results, wherein the first signal includes: a status information identifier of the driver monitoring system;

[0104] A first signal transmitting module is used to transmit the first signal to the control system so that the control system controls the vehicle based on the first signal.

[0105] In some embodiments of the present invention, the apparatus further includes:

[0106] The second signal generation module is used to generate a second signal based on the driver abnormality type corresponding to each frame of driver head image, or based on the target driver abnormality level.

[0107] The second signal transmitting module is used to transmit the second signal to the control system so that the control system controls the vehicle based on the second signal.

[0108] In some embodiments of the present invention, the head deviation angle threshold includes: head top deviation angle threshold, head bottom deviation angle threshold, head left deviation angle threshold and head right deviation angle threshold.

[0109] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0110] At least one processor; and

[0111] A memory communicatively connected to the at least one processor; wherein,

[0112] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the control method described in any embodiment of the present invention.

[0113] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the control method described in any embodiment of the present invention.

[0114] According to another aspect of the present invention, a computer program product is provided, which, when executed by a processor, implements the control method as described in any of the embodiments of the present invention.

[0115] According to another aspect of the present invention, a vehicle is provided, the vehicle comprising: a driver monitoring system for performing the control method described in any embodiment of the present invention.

[0116] The technical solution of this embodiment first acquires vehicle driving data and multiple consecutive frames of driver head images within a safety monitoring period; then, based on the vehicle driving data and the driver anomaly type corresponding to each frame of driver head images, it determines the target driver anomaly level; finally, it executes corresponding safety controls based on the target driver anomaly level. By combining vehicle driving data and the driver anomaly type corresponding to each frame of driver head images within a multiple consecutive frame of driver head images during the safety monitoring period, the driver anomaly level can be determined more accurately. Executing corresponding safety controls based on the target driver anomaly level enables driver monitoring and tiered alerts, thereby effectively preventing traffic accidents.

[0117] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0118] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0119] Figure 1 This is a flowchart of the first control method in an embodiment of the present invention;

[0120] Figure 2 This is a flowchart of the second control method in an embodiment of the present invention;

[0121] Figure 3 This is a flowchart of the third control method in this embodiment of the invention;

[0122] Figure 4 This is a flowchart of the fourth control method in this embodiment of the invention;

[0123] Figure 5 This is a schematic diagram of the structure of a control device according to an embodiment of the present invention;

[0124] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0125] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0126] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0127] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0128] Example 1

[0129] Figure 1 This is a flowchart of the first control method in this embodiment of the invention. This embodiment is applicable to situations where the driver is in an abnormal condition and the vehicle needs to be controlled. This method can be executed by the control device in this embodiment of the invention, which can be implemented in software and / or hardware, such as... Figure 1 As shown, the method specifically includes the following steps:

[0130] S101. Acquire vehicle driving data and multiple consecutive frames of driver head images within the safety monitoring period.

[0131] In this embodiment, the vehicle's driving data includes: the vehicle's driving mode, and may also include: vehicle speed and / or attention alert mode.

[0132] In this embodiment, the safety monitoring time period can be a time period preset in the program, a time period set by the user, or a time period determined based on time information and / or environmental information. For example, if the current time is night, the safety monitoring time period is determined as a first time period; if the current time is daytime, the safety monitoring time period is determined as a second time period, where the length of the first time period is shorter than the length of the second time period. The environmental information includes weather information and / or traffic information. If it is extreme weather, the safety monitoring time period is determined as the first time period; if it is non-extreme weather, the safety monitoring time period is determined as the second time period. If the traffic situation is complex (e.g., congested road sections or school road sections with complex traffic situations), the safety monitoring time period is determined as the first time period; if the traffic situation is good (e.g., good traffic situations on road sections with fewer vehicles), the safety monitoring time period is determined as the second time period. This embodiment of the invention does not impose any limitations on this.

[0133] In this embodiment, the continuous multiple frames of driver head images during the safety monitoring period can be obtained by a camera mounted on the A-pillar and / or a camera mounted on the rearview mirror inside the vehicle.

[0134] The control method provided in this embodiment is executed by the driver monitoring system (DMS), a vehicle safety system that primarily performs functions such as driver identification, driver fatigue monitoring, driver attention monitoring, and monitoring of dangerous driving behaviors. The DMS can provide alerts to the driver and initiate interventions to help the driver manage the vehicle.

[0135] In this embodiment, the vehicle's driving data can be obtained by the driver monitoring system acquiring the vehicle's driving data via the CAN bus.

[0136] S102. Determine the target driver's abnormality level based on the vehicle's driving data and the driver's abnormality type corresponding to each frame of the driver's head image.

[0137] In this embodiment, the abnormal driver types include: making phone calls, being distracted, and being fatigued.

[0138] In this embodiment, classifying and identifying multiple consecutive frames of driver head images within the safety monitoring period to obtain the driver anomaly type corresponding to each frame can be achieved by: pre-setting a head deviation angle threshold for driver distraction, and inputting multiple consecutive frames of driver head images within the safety monitoring period and the head deviation angle threshold into a pre-trained classification and recognition model to obtain the driver anomaly type corresponding to each frame. Alternatively, classifying and identifying multiple consecutive frames of driver head images within the safety monitoring period to obtain the driver anomaly type corresponding to each frame can be achieved by: sequentially inputting multiple consecutive frames of driver head images within the safety monitoring period into a pre-trained classification and recognition model to obtain the driver anomaly type corresponding to each frame.

[0139] It should be noted that the driver's abnormality type can also be determined based on the driver's line of sight pitch angle. For example, if the driver's line of sight pitch angle is determined to be greater than a preset pitch angle threshold based on the driver's head image, then the driver's abnormality type is determined to be distraction type.

[0140] In a specific example, a large set of pre-labeled images of drivers making and not making phone calls is prepared as a training set to train the classifier. During training, the classifier learns the relationship between different features and phone-making behavior, resulting in a trained classifier. Features are extracted from each frame of the driver's head image, and these extracted features are used as input to the trained classifier to determine the driver's anomaly type (whether it is a phone-making type).

[0141] In another specific example, a large amount of head image data containing drivers in different states (fatigued and not fatigued) is collected. This data should cover various driving scenarios, such as daytime, nighttime, different lighting conditions, and different road conditions. The data is labeled to determine whether the driver is fatigued in each image, resulting in a training set. A classifier is then trained to obtain a trained classifier. Each frame of the driver's head image is input into the trained classifier to obtain the driver's anomaly type (whether it is fatigued).

[0142] It should be noted that when identifying a driver's head image to determine if the driver is fatigued, the driver's head image can be analyzed to obtain the driver's eye features and / or mouth features. If the driver's eye opening is less than the opening threshold and / or the driver's mouth aspect ratio is greater than the ratio threshold, then the driver is determined to be fatigued.

[0143] In another specific example, under different scenarios (such as different lighting conditions, driving environments, etc.), a large number of driver head images are collected using an in-vehicle camera. The deviation angle of the driver's head in the collected images is accurately labeled to obtain a training set. A classifier is then trained to obtain a trained classifier. Each frame of the driver's head image is input into the trained classifier to obtain the driver's anomaly type (whether it is a distraction type).

[0144] In another specific example, under different scenarios (such as different lighting conditions and driving environments), a large number of driver head images are collected using an in-vehicle camera. The deviation angle of the driver's head, the driver's fatigue state, and the driver's phone-calling state in the collected images are labeled to obtain a training set. A classifier is then trained to obtain a trained classifier. Each frame of the driver's head image is input into the trained classifier to obtain the driver's abnormal type (phone-calling type, distraction type, and fatigue type).

[0145] In this embodiment, if the driver's abnormality type is distraction, the target driver's abnormality level includes: Distraction Level 1, Distraction Level 2, and Distraction Level 3. If the driver's abnormality type is fatigue, the target driver's abnormality level includes: Fatigue Level 1 and Fatigue Level 2. If the driver's abnormality type is making a phone call, the target driver's abnormality level includes: Making a phone call Level 1.

[0146] In this embodiment, the method for determining the target driver abnormality level based on the vehicle's driving data and the driver abnormality type corresponding to each frame of the driver's head image can be as follows: when the vehicle speed is greater than zero and the vehicle is in D gear, the target driver abnormality level is determined based on the vehicle's driving data and the driver abnormality type corresponding to each frame of the driver's head image. Alternatively, the method for determining the target driver abnormality level based on the vehicle's driving data and the driver abnormality type corresponding to each frame of the driver's head image can be as follows: the vehicle's driving data includes the driving mode; when the driving mode is assisted driving mode, the trigger time range corresponding to the driver abnormality type is obtained; and the duration of the driver abnormality type is determined based on the driver abnormality type corresponding to multiple consecutive frames of driver's head images within the safety monitoring period. The target driver abnormality level is then determined based on the duration of the driver abnormality type and the trigger time range corresponding to the driver abnormality type. Finally, the method for determining the target driver abnormality level based on the vehicle's driving data and the driver abnormality type corresponding to each frame of the driver's head image can be as follows: the vehicle's driving data includes the driving mode and the vehicle speed; when the driving mode is assisted driving mode, the vehicle speed and the trigger time range corresponding to the driver abnormality type are obtained; and the duration of the driver abnormality type is determined based on the driver abnormality type corresponding to multiple consecutive frames of driver's head images within the safety monitoring period. The target driver abnormality level is determined based on the trigger time range corresponding to the vehicle speed and driver abnormality type, and the duration of the driver abnormality type. Alternatively, the target driver abnormality level can be determined based on the vehicle's driving data and the driver abnormality type corresponding to each frame of the driver's head image: determine the duration of the driver abnormality type based on the driver abnormality type corresponding to each frame of the driver's head image; obtain the trigger time range for each driver abnormality level corresponding to vehicle speed, driving mode, and attention alert mode; and use the driver abnormality level corresponding to the trigger time range where the duration of the driver abnormality type falls as the target driver abnormality level.

[0147] Optionally, based on the vehicle's driving data and the driver anomaly type corresponding to each frame of the driver's head image, the target driver anomaly level is determined, including:

[0148] The duration of the driver abnormality type is determined based on the driver abnormality type corresponding to each frame of the driver's head image.

[0149] In this embodiment, the duration of the driver abnormality type is accumulated based on the driver abnormality type corresponding to each driver head image in a series of consecutive driver head images within a safety monitoring period.

[0150] Obtain the trigger time range for each driver's abnormality level corresponding to the vehicle's driving data.

[0151] In this embodiment, the vehicle's driving data includes: driving mode; if the driving mode is assisted driving mode, then the trigger time range for each driver's abnormality level is obtained. For example, the trigger time range for level one (making a phone call) is greater than 3 seconds; the trigger time range for level one (fatigue) is greater than or equal to 3 seconds and less than 8 seconds; the trigger time range for level two (fatigue) is greater than or equal to 8 seconds. The trigger time range for level one (distraction) is greater than 3 seconds and less than 8 seconds; the trigger time range for level two (distraction) is greater than or equal to 8 seconds and less than 13 seconds; and the trigger time range for level three (distraction) is greater than or equal to 13 seconds.

[0152] The driver abnormality level corresponding to the trigger time range in which the duration of the driver abnormality type falls is taken as the target driver abnormality level.

[0153] In this embodiment, the driver abnormality level corresponding to the trigger time range within which the duration of the driver abnormality type falls is taken as the target driver abnormality level. For example, if the distraction type lasts for 4 seconds, then the target driver abnormality level is determined to be Distraction Level 1.

[0154] The technical solution provided in this embodiment first determines the duration of the driver abnormality type based on the driver abnormality type corresponding to each frame of the driver's head image; then, it obtains the trigger time range of each driver abnormality level corresponding to the vehicle's driving data; finally, it uses the driver abnormality level corresponding to the trigger time range in which the driver abnormality type duration is located as the target driver abnormality level. This allows for a more accurate determination of the driver abnormality level based on the duration of the driver abnormality type, thereby enabling more targeted reminders to the driver.

[0155] Optionally, the driver abnormality type includes: making a phone call.

[0156] In this embodiment, if the driver's abnormality type corresponding to the driver's head image is the "making a phone call" type, then the duration of the "making a phone call" type is obtained; if the vehicle is in assisted driving mode, then the trigger time range corresponding to the first level of "making a phone call" is obtained; if the duration of the "making a phone call" type is within the trigger time range corresponding to the first level of "making a phone call", then the target driver's abnormality level is determined to be the first level of "making a phone call".

[0157] In this embodiment, the driver assistance system includes: Lane Keeping Assist (LKA) system, Lane Keeping Assist Plus (LKA+) system, and Navigate on Autopilot (NOA) system; or, the driver assistance system includes: Lane Centering Control (LCC) system, Lane Centering Control Plus (LCC+) system, and Navigate on Autopilot system. The LKA system monitors whether the vehicle deviates from its lane using cameras, sensors, and other devices. When the vehicle shows a tendency to deviate, it assists the driver in keeping the vehicle within the lane by issuing warnings or automatically adjusting the steering, thereby improving driving safety and comfort. LKA+ is an upgrade to the basic lane keeping assist function.

[0158] S103. Execute corresponding safety controls based on the target driver's abnormality level.

[0159] In this embodiment, the safety control includes at least one of the following: controlling the pop-up reminder on the central control display, controlling the audio system to play voice reminders, controlling the head-up display to show reminders, and sending signals to the control system to cause the control system to perform corresponding controls. The head-up display (HUD) projects relevant vehicle information, such as vehicle speed, navigation instructions, and driver assistance system prompts, onto the windshield or a dedicated display screen in front of the driver using optical reflection. This allows the driver to directly obtain key information without looking down at the instrument panel or other devices. The information is directly displayed in front of the driver, facilitating quick reading and making the driving process more convenient and comfortable, especially at high speeds or in complex road conditions, effectively reducing driving stress.

[0160] In this embodiment, the method of performing corresponding safety control based on the target driver's abnormal level can be as follows: if the target driver's abnormal level is level one distraction, a first pop-up reminder and a first voice reminder are issued; if the target driver's abnormal level is level two distraction, a first pop-up reminder, a first voice reminder, and a head-up display reminder are issued; if the target driver's abnormal level is level three distraction, a second pop-up reminder, a second voice reminder, and a head-up display reminder are issued, and the target driver's abnormal level is sent to the control system so that the control system can control the vehicle. The second pop-up reminder includes: a first pop-up reminder and a reminder to exit and disable driver assistance. The second voice reminder includes: The first voice prompt and the driver assistance system exit and disable prompt are issued; if the target driver's abnormal level is fatigue level one, a third pop-up prompt, a third voice prompt, and a head-up display prompt are issued; if the target driver's abnormal level is fatigue level two, a fourth pop-up prompt, a fourth voice prompt, and a head-up display prompt are issued, and the target driver's abnormal level is sent to the control system so that the control system can control the vehicle. The fourth pop-up prompt includes the third pop-up prompt and the driver assistance system exit and disable prompt, and the fourth voice prompt includes the third voice prompt and the driver assistance system exit and disable prompt; if the target driver's abnormal level is making a phone call level one, a fifth pop-up prompt is issued.

[0161] Optionally, based on the target driver's abnormality level, corresponding safety controls are performed, including:

[0162] Based on the target driver's abnormality level, a target alert message is generated.

[0163] In this embodiment, the target reminder information includes at least one of: pop-up reminder information, voice reminder information, and head-up display reminder information.

[0164] In this embodiment, the pop-up reminder can be a Toast notification. The Toast notification will be displayed briefly on the screen and then automatically disappear, without occupying screen space for an extended period to avoid interfering with the driver's normal vision and operation. The head-up display reminder is displayed on a head-up monitor. The voice reminder is played through an audio system.

[0165] It should be noted that prompts can also be displayed through the Environmental Interpretation and Detection (EID) interface.

[0166] Security control is performed based on the target alert information.

[0167] In this embodiment, the target reminder information is sent to the corresponding device to achieve security control. For example, a pop-up reminder information can be sent to a central control display screen to display the pop-up reminder information. A voice reminder information can be sent to an audio system to play the voice reminder information. A heads-up display reminder information can be sent to a heads-up display to display the heads-up display reminder information.

[0168] It should be noted that the abnormal level of the target driver can also be sent to the control system so that the control system can generate target reminder information and then perform safety control based on the target reminder information. This embodiment of the invention does not limit this.

[0169] The technical solution provided in this embodiment enables safety control based on at least one of pop-up reminder information, voice reminder information, and head-up display reminder information. This allows for monitoring and tiered reminders to the driver, effectively preventing traffic accidents.

[0170] Optional, also includes:

[0171] Based on the target driver's abnormality level, generate safety control commands;

[0172] The safety control command is sent to the control system so that the control system controls the vehicle to stop, decelerate, exit the assisted driving mode, and suspend the use of the assisted driving mode at least one of the following:

[0173] In this embodiment, the safety control commands include at least one of the following: a braking command, a deceleration command, a command to exit the assisted driving mode, and a command to suspend the use of the assisted driving mode.

[0174] In this embodiment, the control system can be an autonomous driving system, a driver assistance system, or the like.

[0175] The technical solution provided in this embodiment first generates a safety control command corresponding to the abnormal level of the target driver, and then sends the above safety control command to the control system, which can further improve driving safety while realizing graded reminders.

[0176] Optionally, the driver abnormality type includes: distraction type, and the target driver abnormality level includes three distraction levels, where the trigger time range limit for distraction level one is less than the trigger time range limit for distraction level two, and the trigger time range limit for distraction level two is less than the trigger time range limit for distraction level three.

[0177] Based on the target driver's abnormality level, corresponding safety controls are implemented, including:

[0178] If the target driver's abnormality level is level one distraction, then a first pop-up reminder and a first voice reminder will be issued.

[0179] In this embodiment, the first pop-up reminder can be a Toast notification, for example, a Toast notification specifically stating: "Please pay attention to road conditions ahead." The first voice reminder includes: "Please pay attention to road conditions ahead." The first voice reminder uses a common sound effect and is only played once.

[0180] If the target driver's abnormality level is level two distraction, then a first pop-up reminder, a first voice reminder, and a head-up display reminder will be issued.

[0181] It should be noted that the first voice prompt corresponding to the second level of distraction uses the first sound effect and is played in a loop once. The frequency of the first sound effect is higher than the frequency of the general sound effect.

[0182] In this embodiment, the head-up display alert includes: a white light alert and a text alert (e.g., "Please pay attention to road conditions ahead"). It should be noted that if the target driver's distraction level is level two, no icon will be displayed on the head-up display interface (if the target driver's distraction level is level three, an icon will be displayed on the head-up display interface; this icon can be a system-defined alert icon). The white light alert adds a white light effect to the head-up display interface.

[0183] If the target driver's abnormality level is Distraction Level 3, a second pop-up reminder, a second voice reminder, and a head-up display reminder will be issued, and the target driver's abnormality level will be sent to the control system so that the control system can control the vehicle. The second pop-up reminder includes a first pop-up reminder and a driver assistance exit and disable reminder. The second voice reminder includes a first voice reminder and a driver assistance exit and disable reminder.

[0184] It should be noted that the second voice prompt corresponding to the third level of distraction uses a second sound effect and is played repeatedly. The frequency of the second sound effect is higher than that of the first sound effect.

[0185] In this embodiment, the head-up display reminder includes: a white light reminder, an icon reminder, and a text reminder (e.g., "Please pay attention to road conditions ahead"). The second pop-up reminder for disabling and activating driver assistance can display "Driver assistance is disabling and activating." The second voice reminder for disabling and activating driver assistance can play a message indicating that driver assistance is disabling and activating. The second pop-up reminder can be a Toast notification, for example, displaying "Main title: Please pay attention to road conditions ahead, Subtitle: Driver assistance is about to be disabled and activated" on the central control interface. The second voice reminder can be "Please pay attention to road conditions ahead, driver assistance is about to be disabled and activated."

[0186] It should be noted that if the target driver's abnormality level is Distraction Level 3, the system will send a notification indicating that the driver's abnormality level is Distraction Level 3. This will cause the system to control the vehicle to brake, decelerate, exit the assisted driving mode, or suspend the use of the assisted driving mode at least one of the following:

[0187] Optionally, the driver abnormality type includes: fatigue type, and the target driver abnormality level includes two fatigue levels, with the trigger time range limit of fatigue level one being smaller than the trigger time range limit of fatigue level two.

[0188] Based on the target driver's abnormality level, corresponding safety controls are implemented, including:

[0189] If the target driver's abnormality level is fatigue level one, then a third pop-up reminder, a third voice reminder, and a head-up display reminder will be issued.

[0190] In this embodiment, the third pop-up reminder can be a Toast notification, for example, displaying "Please concentrate on driving" on the central control interface. The third voice reminder can be the audio system playing "Please concentrate on driving". The third voice reminder can use the first sound effect and loop once.

[0191] If the target driver's abnormal level is fatigue level 2, a fourth pop-up reminder, a fourth voice reminder, and a head-up display reminder will be issued, and the target driver's abnormal level will be sent to the control system so that the control system can control the vehicle. The fourth pop-up reminder includes a third pop-up reminder and a driver assistance exit and disable reminder, and the fourth voice reminder includes a third voice reminder and a driver assistance exit and disable reminder.

[0192] In this embodiment, the fourth pop-up reminder can be a Toast notification, for example, displaying "Main title: Please concentrate on driving, Subtitle: Driver assistance will be discontinued and disabled soon" on the central control interface. The fourth voice reminder can be the audio system playing "Please concentrate on driving, driver assistance will be discontinued and disabled soon." The third voice reminder can use the second sound effect, looping once.

[0193] It should be noted that if the target driver's abnormality level is fatigue level 2, then the system will send a message indicating that the target driver's abnormality level is fatigue level 2, so that the system will control the vehicle to brake, decelerate, exit the assisted driving mode, or suspend the use of the assisted driving mode at least one of the following:

[0194] Optionally, the driver abnormality type includes: making a phone call, and the target driving abnormality level includes making a phone call (Level 1).

[0195] Based on the target driver's abnormality level, corresponding safety controls are implemented, including:

[0196] If the target driver's abnormality level is Level 1 (making a phone call), then a fifth pop-up reminder will be issued.

[0197] In this embodiment, the fifth pop-up reminder can be a Toast notification, for example, displaying "895 Danger Level 1" on the central control interface.

[0198] The technical solution provided in this embodiment can monitor and provide graded reminders to drivers when they are in different levels of driver abnormality, thereby effectively preventing traffic accidents.

[0199] Optional, also includes:

[0200] The accuracy of receiving feedback information and / or classification and recognition results sent by the camera.

[0201] In this embodiment, the camera includes: an A-pillar camera and / or a camera on the rearview mirror inside the vehicle.

[0202] In this embodiment, the accuracy of the classification and recognition result is the accuracy of the classification and recognition result obtained by classifying and recognizing multiple consecutive frames of driver head images within the safety monitoring time period.

[0203] A first signal is generated based on the accuracy of the returned information and / or classification and identification results, wherein the first signal includes: a status information identifier of the driver monitoring system.

[0204] In this embodiment, if the DMS receives feedback information sent by the camera, the method for generating the first signal based on the feedback information can be as follows: If the feedback information determines that the camera is operating normally, a first signal is generated to indicate that the driver monitoring system is available. For example, the first signal may include an identification code and 0X1 (available identifier). If the feedback signal determines that the camera is abnormal, a first signal is generated to indicate that the driver monitoring system is unavailable. For example, the first signal may include an identification code and 0X2 (unavailable identifier).

[0205] In this embodiment, the method for generating the first signal based on the accuracy of the classification and recognition results can be as follows: If the accuracy of the classification and recognition results is less than or equal to an accuracy threshold, a first signal is generated to characterize that the driver monitoring system is unusable. For example, the first signal may include an identification code and 0X2 (unusable identifier). If the accuracy of the classification and recognition results is greater than the accuracy threshold, a first signal is generated to characterize that the driver monitoring system is usable. For example, the first signal may include an identification code and 0X1 (usable identifier).

[0206] In this embodiment, the method for generating the first signal based on the accuracy of the returned information and the classification and recognition results can be as follows: If the returned information determines that the camera is operating normally and the accuracy of the classification and recognition results is greater than the accuracy threshold, then a first signal is generated to characterize the availability of the driver monitoring system. For example, the first signal may include an identification code and 0X1 (availability identifier). If the returned signal determines that the camera is malfunctioning, and / or the accuracy of the classification and recognition results is less than or equal to the accuracy threshold, then a first signal is generated to characterize the unavailability of the driver monitoring system. For example, the first signal may include an identification code and 0X2 (unavailability identifier).

[0207] In this embodiment, the first signal includes: an identification code and a status information identifier for the driver monitoring system. It should be noted that the status information of the driver monitoring system includes: available or unavailable.

[0208] It should be noted that after the driver monitoring system is activated, an initialization signal is sent to the driver assistance system in advance. For example, the initialization signal may include: an identification code and 0X0 (initialization identifier).

[0209] The first signal is sent to the control system so that the control system controls the vehicle based on the first signal.

[0210] In this embodiment, the control system can be an autonomous driving system, a driver assistance system, etc. Controlling the vehicle can include at least one of suspending / exiting the driver assistance mode, braking to a stop, and decelerating.

[0211] The technical solution provided in this embodiment generates a first signal based on the accuracy of the returned information and / or classification and identification results, and sends the first signal to the control system, which can ensure that the control system is aware of the status of the driver monitoring system and further improve driving safety.

[0212] Optional, also includes:

[0213] A second signal is generated based on the driver anomaly type corresponding to each frame of the driver's head image or based on the target driver anomaly level.

[0214] In this embodiment, the method for generating the second signal based on the driver anomaly type corresponding to each frame of the driver's head image can be as follows: classify and identify multiple consecutive frames of driver's head images within the safety monitoring time period to obtain the driver anomaly type corresponding to each frame of the driver's head image, and generate the second signal based on the driver anomaly type. For example, the second signal can include: an identification code and 0y1 (distraction type).

[0215] In this embodiment, the second signal can be generated based on the target driver's abnormality level as follows: if the target driver's abnormality level is distraction level three or fatigue level two, then the second signal is generated based on the target driver's abnormality level. For example, if the target driver's abnormality level is distraction level three, then the second signal includes: an identification code and 0y1 (distraction type).

[0216] The second signal is sent to the control system so that the control system controls the vehicle based on the second signal.

[0217] In this embodiment, the control system can be an autonomous driving system, a driver assistance system, etc. Controlling the vehicle can include at least one of suspending / exiting the driver assistance mode, braking to a stop, and decelerating.

[0218] In a specific example, if the target driver's abnormality level is Distraction Level 3 or Fatigue Level 2, a second signal is generated based on the driver's abnormality level and sent to the control system. This second signal then prompts the control system to brake the vehicle based on the second signal. It should be noted that after receiving the second signal, if the control system does not receive a normal driving signal within the first immediate period, it will control the vehicle to brake to a stop using Motorized Responsive Braking (MRM); if a normal driving signal is received within the first immediate period, the warning will be canceled.

[0219] The technical solution provided in this embodiment generates a second signal based on the driver abnormality type corresponding to each frame of the driver's head image and / or the target driver abnormality level, and then sends the second signal to the control system. This enables the control system to control the vehicle based on the second signal, thereby informing the control system of the driver's abnormal situation and allowing the control system to perform corresponding control based on the driver's abnormal situation, thus further ensuring driving safety.

[0220] Example 2

[0221] Figure 2This is a flowchart of the second control method in this embodiment of the invention, which is an optimization based on the above embodiment. In this embodiment, before determining the target driver abnormality level based on the vehicle's driving data and the driver abnormality type corresponding to each frame of the driver's head image, the method further includes: obtaining a head deviation angle threshold; classifying and identifying multiple consecutive frames of driver's head images within the safety monitoring time period based on the head deviation angle threshold to obtain the driver abnormality type corresponding to each frame of the driver's head image. Figure 2 As shown, the method specifically includes the following steps:

[0222] S201. Acquire vehicle driving data and multiple consecutive frames of driver head images within the safety monitoring period.

[0223] S202, Obtain the head deviation angle threshold.

[0224] In this embodiment, the head deviation angle threshold can be a value preset by the system, a value set by the user, or a value calculated based on the driver's historical data. This embodiment of the invention does not impose any restrictions on this.

[0225] Optionally, the head deviation angle thresholds include: head top deviation angle threshold, head bottom deviation angle threshold, head left deviation angle threshold, and head right deviation angle threshold.

[0226] In this embodiment, the deviation angle threshold above the head can be 15 degrees, the deviation angle threshold below the head can be 10 degrees, the deviation angle threshold to the left of the head can be 20 degrees, and the deviation angle threshold to the right of the head can be 22 degrees.

[0227] S203. Based on the head deviation angle threshold, classify and identify multiple consecutive frames of driver head images within the safety monitoring time period to obtain the driver abnormality type corresponding to each frame of driver head image.

[0228] In this embodiment, the head deviation angle threshold and each frame of the driver's head image are input into the classification and recognition model to obtain the driver abnormality type corresponding to each frame of the driver's head image.

[0229] S204. Determine the target driver's abnormality level based on the vehicle's driving data and the driver's abnormality type corresponding to each frame of the driver's head image.

[0230] In this embodiment, the method for determining the target driver abnormality level based on the vehicle's driving data and the driver abnormality type corresponding to each frame of driver head image can be as follows: determine the duration of the driver abnormality type based on the driver abnormality type corresponding to each frame of driver head image; obtain the trigger time range of each driver abnormality level corresponding to the vehicle's driving data; and take the driver abnormality level corresponding to the trigger time range in which the duration of the driver abnormality type is located as the target driver abnormality level.

[0231] Optionally, based on the vehicle's driving data and the driver anomaly type corresponding to each frame of the driver's head image, the target driver anomaly level is determined, including:

[0232] The duration of the driver abnormality type is determined based on the driver abnormality type corresponding to each frame of the driver's head image.

[0233] Obtain the trigger time range for each driver's abnormality level corresponding to the vehicle's driving data.

[0234] In this embodiment, the vehicle's driving data includes: driving mode, and may also include: vehicle speed and / or attention alert mode.

[0235] In this embodiment, the trigger time range for each driver abnormality level can be obtained as follows: if the vehicle's driving mode is assisted driving mode, then the trigger time range for each driver abnormality level is obtained; alternatively, if the vehicle's driving mode is assisted driving mode, then the trigger time range for each driver abnormality level corresponding to the vehicle speed is obtained. Alternatively, if the vehicle's driving mode is assisted driving mode and the attention alert mode is standard mode, then the trigger time range for each driver abnormality level corresponding to the vehicle speed is obtained; if the vehicle's driving mode is assisted driving mode and the attention alert mode is sensitive mode, then the trigger time range for each driver abnormality level is obtained.

[0236] The driver abnormality level corresponding to the trigger time range in which the duration of the driver abnormality type falls is taken as the target driver abnormality level.

[0237] Optionally, the driving data includes: vehicle speed, driving mode, and attention alert mode;

[0238] The trigger time range for each driver anomaly level corresponding to the vehicle's driving data is obtained, including:

[0239] If the vehicle's driving mode is assisted driving mode and the attention reminder mode is standard mode, then obtain the trigger time range for each driver abnormality level corresponding to the vehicle speed.

[0240] In this embodiment, if the vehicle's driving mode is assisted driving mode and the attention reminder mode is standard mode, and the vehicle speed is within the first speed range, the upper limit of the trigger time range for level 1 distraction is 25 seconds and the lower limit is 20 seconds (the trigger time range is greater than or equal to 20 seconds and less than 25 seconds), the upper limit of the trigger time range for level 2 distraction is 30 seconds and the lower limit is 25 seconds (the trigger time range is greater than or equal to 25 seconds and less than 30 seconds), and the lower limit of the trigger time range for level 3 distraction is 30 seconds (the trigger time range is greater than or equal to 30 seconds). When the vehicle speed is within the second speed range, the upper limit of the trigger time range for Distraction Level 1 is 15 seconds, and the lower limit is 10 seconds (the trigger time range is greater than or equal to 10 seconds and less than 15 seconds). The upper limit of the trigger time range for Distraction Level 2 is 20 seconds, and the lower limit is 15 seconds (the trigger time range is greater than or equal to 15 seconds and less than 20 seconds). The lower limit of the trigger time range for Distraction Level 3 is 20 seconds (the trigger time range is greater than or equal to 20 seconds). When the vehicle speed is within the first speed range, the upper limit of the trigger time range for Fatigue Level 1 is 20 seconds, and the lower limit is 15 seconds (the trigger time range is greater than or equal to 15 seconds and less than 20 seconds). The lower limit of the trigger time range for Fatigue Level 2 is 20 seconds (the trigger time range is greater than or equal to 20 seconds). When the vehicle speed is within the second speed range, the upper limit of the trigger time range for fatigue level one is 13 seconds, and the lower limit is 8 seconds (the trigger time range is greater than or equal to 8 seconds and less than 13 seconds). The lower limit of the trigger time range for fatigue level two is 13 seconds (the trigger time range is greater than or equal to 13 seconds). The vehicle speed within the first speed range is greater than a first value and less than or equal to a second value; the vehicle speed within the second speed range is greater than or equal to a second value; and the first value is less than the second value.

[0241] If the vehicle's driving mode is assisted driving mode and the attention alert mode is sensitive mode, then obtain the trigger time range for each driver's abnormal level.

[0242] In this embodiment, if the vehicle's driving mode is assisted driving mode, the attention alert mode is sensitive mode, and the vehicle speed is greater than zero, then the upper limit of the trigger time range for Distraction Level 1 is 8 seconds, and the lower limit is 3 seconds (the trigger time range is greater than or equal to 3 seconds and less than 8 seconds); the upper limit of the trigger time range for Distraction Level 2 is 13 seconds, and the lower limit is 8 seconds (the trigger time range is greater than or equal to 8 seconds and less than 13 seconds); the lower limit of the trigger time range for Distraction Level 3 is 13 seconds (the trigger time range is greater than or equal to 13 seconds). Similarly, the upper limit of the trigger time range for Fatigue Level 1 is 8 seconds, and the lower limit is 3 seconds (the trigger time range is greater than or equal to 3 seconds and less than 8 seconds); the lower limit of the trigger time range for Fatigue Level 2 is 8 seconds (the trigger time range is greater than or equal to 8 seconds).

[0243] The technical solution provided in this embodiment obtains the trigger time range of each driver abnormality level corresponding to driving mode, attention reminder mode and vehicle speed, which can make the obtained trigger time range match the driver abnormality level more accurately, thus obtaining a more accurate driver abnormality level.

[0244] Optionally, the trigger time range limits for each driver anomaly level corresponding to the standard mode are all greater than the trigger time range limits for each driver anomaly level corresponding to the sensitive mode.

[0245] In this embodiment, the limitations include: upper limit and / or lower limit.

[0246] The technical solution provided in this embodiment has a trigger time range limit for each driver abnormality level corresponding to the sensitive mode that is smaller than the trigger time range limit for each driver abnormality level corresponding to the standard mode. This ensures that the driver is reminded more promptly in the sensitive mode and more gently in the standard mode, thereby improving driving safety.

[0247] Optionally, the trigger time range for each driver abnormality level corresponding to the vehicle speed is obtained, including:

[0248] Obtain the trigger time range for each driver abnormality level corresponding to the first speed range and the trigger time range for each driver abnormality level corresponding to the second speed range.

[0249] In this embodiment, the vehicle speed within the first speed range is greater than a first value and less than or equal to a second value, and the vehicle speed within the second speed range is greater than or equal to the second value. The first value is less than the second value; for example, the first value could be zero and the second value could be 30.

[0250] If the vehicle speed is within the first vehicle speed range, then the trigger time range of each driver abnormality level corresponding to the first vehicle speed range shall be used as the trigger time range of each driver abnormality level corresponding to the vehicle speed.

[0251] In this embodiment, the vehicle speed can be the current vehicle speed.

[0252] In this embodiment, the trigger time ranges for each driver abnormality level include: the trigger time range for distraction level 1, the trigger time range for distraction level 2, the trigger time range for distraction level 3, the trigger time range for fatigue level 1, and the trigger time range for fatigue level 2.

[0253] If the vehicle speed is within the second vehicle speed range, then the trigger time range of each driver abnormality level corresponding to the second vehicle speed range shall be used as the trigger time range of each driver abnormality level corresponding to the vehicle speed.

[0254] In this embodiment, the trigger time range limit for each driver abnormality level corresponding to the first vehicle speed range is greater than the trigger time range limit for each driver abnormality level corresponding to the second vehicle speed range.

[0255] The technical solution provided in this embodiment, by setting different trigger time ranges for different driver abnormality levels for different vehicle speed ranges, can ensure that the driver is promptly reminded at higher vehicle speeds, allowing the driver sufficient reaction time and improving driving safety. At lower vehicle speeds, it can accurately determine the driver's abnormality level and provide the driver with appropriate reminders, thereby improving driving safety.

[0256] Optionally, the trigger time range limit for each distraction level corresponding to the first speed range is greater than the trigger time range limit for each distraction level corresponding to the second speed range, and the trigger time range limit for each fatigue level corresponding to the first speed range is greater than the trigger time range limit for each fatigue level corresponding to the second speed range.

[0257] In this embodiment, the distraction levels include: Distraction Level 1, Distraction Level 2, and Distraction Level 3. The fatigue levels include: Fatigue Level 1 and Fatigue Level 2.

[0258] Optionally, the driver abnormality type includes: distraction type, and the target driver abnormality level includes three distraction levels, wherein the trigger time range limit of distraction level one is less than the trigger time range limit of distraction level two, and the trigger time range limit of distraction level two is less than the trigger time range limit of distraction level three.

[0259] In this embodiment, if the attention reminder mode is the standard mode, the trigger time range limit for level 1 distraction is less than the trigger time range limit for level 2 distraction, and the trigger time range limit for level 2 distraction is less than the trigger time range limit for level 3 distraction. If the attention reminder mode is the sensitive mode, the trigger time range limit for level 1 distraction is less than the trigger time range limit for level 2 distraction, and the trigger time range limit for level 2 distraction is less than the trigger time range limit for level 3 distraction.

[0260] Optionally, the driver abnormality type includes: fatigue type, and the target driver abnormality level includes two fatigue levels, with the trigger time range limit for fatigue level one being smaller than the trigger time range limit for fatigue level two.

[0261] In this embodiment, if the attention reminder mode is the standard mode, the trigger time range limit for fatigue level one is less than the trigger time range limit for fatigue level two; if the attention reminder mode is the sensitive mode, the trigger time range limit for fatigue level one is less than the trigger time range limit for fatigue level two.

[0262] Optionally, before acquiring the vehicle's driving data, the following may also be included:

[0263] Upon detecting a trigger operation on the target attention reminder control, the attention reminder mode is switched to the attention reminder mode corresponding to the target attention reminder control. The attention reminder mode includes: standard mode and sensitive mode.

[0264] In this embodiment, the target attention reminder control can be a physical control or a virtual control. The target attention reminder control can be a control corresponding to either standard mode or sensitive mode. In a specific example, if the target attention reminder control is a control corresponding to standard mode, then clicking the target attention reminder control will switch to standard mode. If the target attention reminder control is a control corresponding to sensitive mode, then clicking the target attention reminder control will switch to sensitive mode.

[0265] In this embodiment, the triggering operation includes, but is not limited to, touch triggering, gesture triggering, and voice triggering.

[0266] or,

[0267] Switch the attention alert mode to the attention alert mode corresponding to time information and / or environmental information, whereby the environmental information includes: weather information and / or traffic information.

[0268] In this embodiment, the time information can be either daytime or nighttime. If it is daytime, the attention reminder mode is switched to standard mode; if it is nighttime, the attention reminder mode is switched to sensitive mode.

[0269] In this embodiment, if the weather information indicates extreme weather (e.g., strong winds, rain, snow, fog, hail, etc.), the attention alert mode is switched to sensitive mode. If the traffic information indicates complex traffic conditions (e.g., congested roads or school zones), the attention alert mode is switched to sensitive mode. If the weather information indicates non-extreme weather (clear weather, visibility greater than the visibility threshold), the attention alert mode is switched to standard mode. If the traffic information indicates good traffic conditions (e.g., few vehicles), the attention alert mode is switched to standard mode. If the weather information indicates non-extreme weather and the traffic information indicates good traffic conditions, the attention alert mode is switched to standard mode.

[0270] In this embodiment, if it is daytime, the weather information is not extreme weather, and the traffic information is good traffic conditions, then the attention reminder mode will be switched to the standard mode.

[0271] The technical solution provided in this embodiment allows the attention reminder mode to be triggered manually or automatically, improving the flexibility of switching between attention reminder modes.

[0272] S205. Perform corresponding safety controls based on the target driver's abnormality level.

[0273] In a specific example, such as Figure 3 As shown, the DMS system is activated, requesting the opening of the A-pillar camera and the rearview mirror camera. Based on the continuous multi-frame driver head images captured by the A-pillar camera and the rearview mirror camera, and the head deflection angle threshold, an algorithm is used to identify the driver abnormality type, including distraction and fatigue. Based on the vehicle's driving data and the driver abnormality type corresponding to each frame of the driver's head image, the target driver abnormality level (distraction level, fatigue level, phone call level) and HUD adjustment information are determined (the position of the HUD is adjusted according to the driver's head deflection angle or the driver's posture, for example, adjusting the height of the HUD according to the driver's head deflection angle or the driver's posture). Based on the target driver abnormality level and HUD adjustment information, DMS signals (reminder signal, HUD adjustment result signal, intelligent driving system signal) are generated and sent to the external system to enable the external system to provide pop-up reminders, adjust the HUD, and perform MRM braking. It should be noted that the intelligent driving system signal is a signal sent to the intelligent driving system itself. For example, if the target driver's abnormality level is level three distraction or level two fatigue, a signal is sent to the intelligent driving system so that the intelligent driving system, upon receiving the signal, enters the Minimum Risk Maneuver (MRM) braking stop. MRM braking stop achieves vehicle braking and stopping by performing the lowest risk maneuver, aiming to minimize traffic risk. It should also be noted that the prerequisite for the DMS to determine the target driver's abnormality level based on the vehicle's driving data and the driver abnormality type corresponding to each frame of the driver's head image is that the vehicle speed is greater than zero and the gear is in Drive (D).

[0274] The technical solution of this embodiment first classifies and identifies multiple consecutive frames of driver head images within the safety monitoring time period based on the head deviation angle threshold to obtain the driver abnormality type corresponding to each frame of driver head image. Then, based on the vehicle's driving data and the driver abnormality type corresponding to each frame of driver head image, the target driver abnormality level is determined. Finally, corresponding safety control is executed based on the target driver abnormality level, which can obtain a more accurate driver abnormality type, so as to realize driver monitoring and graded reminders, thereby effectively preventing traffic accidents from occurring.

[0275] Example 3

[0276] Figure 4 This is a flowchart of the fourth control method in this embodiment of the invention. This embodiment is an optimization based on the above embodiments. In this embodiment, the target driver abnormality level is determined according to the vehicle's driving data and the driver abnormality type corresponding to each frame of the driver's head image, including: obtaining the steering wheel takeover state; and determining the target driver abnormality level according to the steering wheel takeover state, the vehicle's driving data, and the driver abnormality type corresponding to each frame of the driver's head image. Figure 4 As shown, the method specifically includes the following steps:

[0277] S401. Acquire vehicle driving data and multiple consecutive frames of driver head images within the safety monitoring period.

[0278] S402, Obtain the steering wheel grip status.

[0279] In this embodiment, the steering wheel gripping state includes: the driver gripping the steering wheel and the driver not gripping the steering wheel. The driver gripping the steering wheel includes: the driver gripping the steering wheel on one side and the driver gripping the steering wheel on both sides.

[0280] In this embodiment, the methods for obtaining the steering wheel grip state include: through a steering wheel capacitive sensor, through a steering wheel torque sensor, and through camera visual recognition; as long as any one of these methods successfully identifies the steering wheel grip state as the driver gripping the steering wheel, then the steering wheel grip state is determined to be the driver gripping the steering wheel.

[0281] In a specific example, the gripping state of the steering wheel is obtained through a capacitive sensor on the steering wheel. If the change in capacitance of the capacitive sensor is greater than a preset threshold, the gripping state of the steering wheel is determined to be that the driver is gripping the steering wheel.

[0282] S403. Determine the target driver's abnormality level based on the steering wheel grip state, the vehicle's driving data, and the driver's abnormality type corresponding to each frame of the driver's head image.

[0283] Optionally, the method for determining the target driver abnormality level based on the steering wheel grip state, the vehicle's driving data, and the driver abnormality type corresponding to each frame of the driver's head image can be as follows: determine the duration of the driver abnormality type based on the driver abnormality type corresponding to each frame of the driver's head image; obtain the trigger time range of each driver abnormality level corresponding to the vehicle's driving data; and take the driver abnormality level corresponding to the trigger time range in which the duration of the driver abnormality type is located and the driver abnormality level corresponding to the driver not gripping the steering wheel as the target driver abnormality level.

[0284] Optionally, the method for determining the target driver abnormality level based on the steering wheel grip state, the vehicle's driving data, and the driver abnormality type corresponding to each frame of the driver's head image can also be as follows: If the steering wheel grip state is that the driver is not gripping the steering wheel, then obtain the trigger time range of each driver abnormality level corresponding to the driver not gripping the steering wheel and the vehicle's driving data; and use the driver abnormality level corresponding to the trigger time range in which the duration of the driver abnormality type falls as the target driver abnormality level. If the steering wheel grip state is that the driver is gripping the steering wheel, then obtain the trigger time range of each driver abnormality level corresponding to the vehicle's driving data; and use the driver abnormality level corresponding to the trigger time range in which the duration of the driver abnormality type falls as the target driver abnormality level.

[0285] Optionally, the method for determining the target driver abnormality level based on the steering wheel grip state, the vehicle's driving data, and the driver abnormality type corresponding to each frame of the driver's head image can also be as follows: If the steering wheel grip state is that the driver is not gripping the steering wheel, and the vehicle's driving mode is assisted driving mode, then obtain the trigger time range for each driver abnormality level corresponding to the driver not gripping the steering wheel; and use the driver abnormality level corresponding to the trigger time range in which the duration of the driver abnormality type falls as the target driver abnormality level. If the steering wheel grip state is that the driver is gripping the steering wheel, and the vehicle's driving mode is assisted driving mode, then obtain the trigger time range for each driver abnormality level; and use the driver abnormality level corresponding to the trigger time range in which the duration of the driver abnormality type falls as the target driver abnormality level.

[0286] Optionally, the method for determining the target driver abnormality level based on the steering wheel grip state, the vehicle's driving data, and the driver abnormality type corresponding to each frame of the driver's head image can also be as follows: If the steering wheel grip state is that the driver is not holding the steering wheel, the vehicle's driving mode is assisted driving mode, and the attention reminder mode is standard mode, then obtain the trigger time range for each driver abnormality level corresponding to the driver not holding the steering wheel and the vehicle speed; and use the driver abnormality level corresponding to the trigger time range in which the duration of the driver abnormality type falls as the target driver abnormality level. If the steering wheel grip state is that the driver is holding the steering wheel, the vehicle's driving mode is assisted driving mode, and the attention reminder mode is standard mode, then obtain the trigger time range for each driver abnormality level corresponding to the vehicle speed; and use the driver abnormality level corresponding to the trigger time range in which the duration of the driver abnormality type falls as the target driver abnormality level.

[0287] Optionally, the method for determining the target driver's abnormality level based on the steering wheel grip state, the vehicle's driving data, and the driver abnormality type corresponding to each frame of the driver's head image can also be as follows: When the vehicle speed exceeds the speed limit, the target driver's abnormality level is determined based on the steering wheel grip state, the vehicle's driving data, and the driver abnormality type corresponding to each frame of the driver's head image. For example, the speed limit could be 90 km / h. That is, when the vehicle speed is less than or equal to the speed limit, the target driver's abnormality level is determined based on the vehicle's driving data and the driver abnormality type corresponding to each frame of the driver's head image. When the vehicle speed exceeds the speed limit, the target driver's abnormality level is determined based on the steering wheel grip state, the vehicle's driving data, and the driver abnormality type corresponding to each frame of the driver's head image.

[0288] Optionally, the method for determining the target driver abnormality level based on the steering wheel grip state, the vehicle's driving data, and the driver abnormality type corresponding to each frame of the driver's head image can also be as follows: If the steering wheel grip state is that the driver is not holding the steering wheel, the vehicle's driving mode is assisted driving mode, and the attention reminder mode is sensitive mode, then obtain the trigger time range for each driver abnormality level corresponding to the driver not holding the steering wheel; and take the driver abnormality level corresponding to the trigger time range in which the duration of the driver abnormality type falls as the target driver abnormality level. If the steering wheel grip state is that the driver is holding the steering wheel, the vehicle's driving mode is assisted driving mode, and the attention reminder mode is sensitive mode, then obtain the trigger time range for each driver abnormality level; and take the driver abnormality level corresponding to the trigger time range in which the duration of the driver abnormality type falls as the target driver abnormality level.

[0289] Optionally, the method for determining the target driver anomaly level based on the steering wheel grip state, the vehicle's driving data, and the driver anomaly type corresponding to each frame of the driver's head image can also be as follows: If the steering wheel grip state is that the driver is not gripping the steering wheel, then the duration of the driver not gripping the steering wheel is obtained, and the driver anomaly level corresponding to the trigger time range of the driver anomaly type duration and the steering wheel release anomaly level corresponding to the target time range of the driver not gripping the steering wheel duration are used as the target driver anomaly level. It should be noted that the target time range can be a system-set time range, a user-set time range, or a time range determined based on time information and / or environmental information. The time information can be daytime or nighttime. The target time range corresponding to daytime is greater than the target time range corresponding to nighttime. Environmental information includes: weather information and / or traffic information. Weather information includes: extreme weather and non-extreme weather. Extreme weather can be: strong winds, rain, snow, fog, hail, etc. The target time range corresponding to extreme weather is smaller than the target time range corresponding to non-extreme weather. Traffic information includes: complex traffic conditions and good traffic conditions. Traffic conditions can be complex, such as congested roads or school zones, or good, such as light traffic. The target time range for good traffic conditions is longer than that for complex traffic conditions.

[0290] It should be noted that when the driver is not holding the steering wheel, the trigger time limit for each driver abnormality level is less than the trigger time limit for each driver abnormality level when the driver is holding the steering wheel.

[0291] S404. Perform corresponding safety controls based on the target driver's abnormality level.

[0292] The technical solution provided in this embodiment first classifies and identifies multiple consecutive frames of driver head images within the safety monitoring period to obtain the driver anomaly type corresponding to each frame of the driver head image; then, based on the steering wheel grip state, the vehicle's driving data, and the driver anomaly type corresponding to each frame of the driver head image, the target driver anomaly level is determined; finally, corresponding safety controls are executed based on the target driver anomaly level. Combining the steering wheel grip state and the driver anomaly type to comprehensively determine the target driver anomaly level can improve the accuracy of driver monitoring and alerts, thereby enhancing driving safety.

[0293] Example 4

[0294] Figure 5This is a schematic diagram of a control device provided in an embodiment of the present invention. This embodiment is applicable to situations where the driver is in an abnormal situation and the vehicle needs to be controlled. The device can be implemented using software and / or hardware, and can be integrated into any device that provides control functions, such as a driver monitoring system. Figure 5 As shown, the control device specifically includes: an acquisition module 501, a target driver abnormality level determination module 502, and a safety control module 503.

[0295] The acquisition module is used to acquire vehicle driving data and multiple consecutive frames of driver head images within the safety monitoring period.

[0296] The target driver anomaly level determination module is used to determine the target driver anomaly level based on the vehicle's driving data and the driver anomaly type corresponding to each frame of the driver's head image;

[0297] The safety control module is used to perform corresponding safety controls based on the abnormality level of the target driver.

[0298] Optional, also includes:

[0299] The head deviation angle threshold acquisition module is used to acquire the head deviation angle threshold before determining the target driver's abnormality level based on the vehicle's driving data and the driver's abnormality type corresponding to each frame of driver's head image.

[0300] The identification module is used to classify and identify multiple consecutive frames of driver head images within the safety monitoring period based on the head deviation angle threshold, and obtain the driver abnormality type corresponding to each frame of driver head image.

[0301] Optionally, the target driver anomaly level determination module is specifically used for:

[0302] The duration of the driver abnormality type is determined based on the driver abnormality type corresponding to each frame of driver head image;

[0303] Obtain the trigger time range for each driver's abnormality level corresponding to the vehicle's driving data;

[0304] The driver abnormality level corresponding to the trigger time range in which the duration of the driver abnormality type falls is taken as the target driver abnormality level.

[0305] Optionally, the driving data includes: vehicle speed, driving mode, and attention alert mode;

[0306] The target driver anomaly level determination module is specifically used for:

[0307] If the vehicle's driving mode is assisted driving mode and the attention reminder mode is standard mode, then obtain the trigger time range of each driver abnormality level corresponding to the vehicle speed.

[0308] If the vehicle's driving mode is assisted driving mode and the attention alert mode is sensitive mode, then obtain the trigger time range for each driver's abnormal level.

[0309] Optionally, the trigger time range limits for each driver anomaly level corresponding to the standard mode are all greater than the trigger time range limits for each driver anomaly level corresponding to the sensitive mode.

[0310] Optionally, the target driver anomaly level determination module is specifically used for:

[0311] Upon detecting a trigger operation on the target attention reminder control, the attention reminder mode is switched to the attention reminder mode corresponding to the target attention reminder control. The attention reminder mode includes: standard mode and sensitive mode.

[0312] or,

[0313] Switch the attention alert mode to the attention alert mode corresponding to time information and / or environmental information, whereby the environmental information includes: weather information and / or traffic information.

[0314] Optionally, the safety control module is specifically used for:

[0315] Based on the target driver's abnormality level, a target alert message is generated, wherein the target alert message includes at least one of: pop-up alert message, voice alert message, and head-up display alert message;

[0316] Security control is performed based on the target alert information.

[0317] Optional, also includes:

[0318] A safety control command generation module is used to generate safety control commands based on the abnormality level of the target driver.

[0319] The safety control command sending module is used to send the safety control command to the control system so that the control system controls the vehicle to stop, decelerate, exit the assisted driving mode, and suspend the use of the assisted driving mode at least one of the following:

[0320] Optionally, the driver abnormality type includes: distraction type; the target driver abnormality level includes three distraction levels, where the trigger time limit for distraction level one is less than the trigger time limit for distraction level two, and the trigger time limit for distraction level two is less than the trigger time limit for distraction level three. The safety control module is specifically used for:

[0321] If the target driver's abnormality level is level one distraction, then a first pop-up reminder and a first voice reminder will be issued;

[0322] If the target driver's abnormality level is level two distraction, then a first pop-up reminder, a first voice reminder, and a head-up display reminder will be issued;

[0323] If the target driver's abnormality level is Distraction Level 3, a second pop-up reminder, a second voice reminder, and a head-up display reminder will be issued, and the target driver's abnormality level will be sent to the control system so that the control system can control the vehicle. The second pop-up reminder includes a first pop-up reminder and a driver assistance exit and disable reminder. The second voice reminder includes a first voice reminder and a driver assistance exit and disable reminder.

[0324] Optionally, the driver abnormality type includes: fatigue type, and the target driver abnormality level includes two fatigue levels, with the trigger time range limit for fatigue level one being smaller than the trigger time range limit for fatigue level two; the safety control module is specifically used for:

[0325] If the target driver's abnormality level is fatigue level one, then a third pop-up reminder, a third voice reminder, and a head-up display reminder will be issued;

[0326] If the target driver's abnormal level is fatigue level 2, a fourth pop-up reminder, a fourth voice reminder, and a head-up display reminder will be issued, and the target driver's abnormal level will be sent to the control system so that the control system can control the vehicle. The fourth pop-up reminder includes a third pop-up reminder and a driver assistance exit and disable reminder, and the fourth voice reminder includes a third voice reminder and a driver assistance exit and disable reminder.

[0327] Optionally, the driver abnormality type includes: making a phone call, and the target driving abnormality level includes making a phone call (Level 1); the safety control module is specifically used for:

[0328] If the target driver's abnormality level is Level 1 (making a phone call), then a fifth pop-up reminder will be issued.

[0329] Optionally, the target driver anomaly level determination module is specifically used for:

[0330] Get the steering wheel grip status;

[0331] The target driver's abnormality level is determined based on the steering wheel grip status, the vehicle's driving data, and the driver's abnormality type corresponding to each frame of the driver's head image.

[0332] Optionally, the target driver anomaly level determination module is specifically used for:

[0333] Get the trigger time range of each driver abnormality level corresponding to the first speed range and the trigger time range of each driver abnormality level corresponding to the second speed range, wherein the speed in the first speed range is greater than a first value and less than or equal to a second value, and the speed in the second speed range is greater than or equal to the second value.

[0334] If the vehicle speed is within the first vehicle speed range, then the trigger time range of each driver abnormality level corresponding to the first vehicle speed range shall be used as the trigger time range of each driver abnormality level corresponding to the vehicle speed.

[0335] If the vehicle speed is within the second vehicle speed range, then the trigger time range of each driver abnormality level corresponding to the second vehicle speed range shall be used as the trigger time range of each driver abnormality level corresponding to the vehicle speed.

[0336] In some embodiments of the present invention, the limit of the trigger time range for each distraction level corresponding to the first speed range is greater than the limit of the trigger time range for each distraction level corresponding to the second speed range, and the limit of the trigger time range for each fatigue level corresponding to the first speed range is greater than the limit of the trigger time range for each fatigue level corresponding to the second speed range.

[0337] Optionally, the device further includes:

[0338] The receiving module is used to receive the feedback information and / or the accuracy of the classification and recognition results sent by the camera;

[0339] The first signal generation module is used to generate a first signal based on the accuracy of the returned information and / or classification and identification results, wherein the first signal includes: a status information identifier of the driver monitoring system;

[0340] A first signal transmitting module is used to transmit the first signal to the control system so that the control system controls the vehicle based on the first signal.

[0341] Optionally, the device further includes:

[0342] The second signal generation module is used to generate a second signal based on the driver abnormality type corresponding to each frame of driver head image, or based on the target driver abnormality level.

[0343] The second signal transmitting module is used to transmit the second signal to the control system so that the control system controls the vehicle based on the second signal.

[0344] Optionally, the head deviation angle thresholds include: head top deviation angle threshold, head bottom deviation angle threshold, head left deviation angle threshold, and head right deviation angle threshold.

[0345] The above-mentioned products can execute the control methods provided in any embodiment of the present invention, and have the corresponding functional modules and beneficial effects of the execution method.

[0346] Example 5

[0347] Figure 6 A schematic diagram of an electronic device 60 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0348] like Figure 6 As shown, the electronic device 60 includes at least one processor 61 and a memory, such as a read-only memory (ROM) 62 and a random access memory (RAM) 63, communicatively connected to the at least one processor 61. The memory stores computer programs executable by the at least one processor. The processor 61 can perform various appropriate actions and processes based on the computer program stored in the ROM 62 or loaded into the RAM 63 from storage unit 68. The RAM 63 can also store various programs and data required for the operation of the electronic device 60. The processor 61, ROM 62, and RAM 63 are interconnected via a bus 64. An input / output (I / O) interface 65 is also connected to the bus 64.

[0349] Multiple components in electronic device 60 are connected to I / O interface 65, including: input unit 66, such as keyboard, mouse, etc.; output unit 67, such as various types of monitors, speakers, etc.; storage unit 68, such as disk, optical disk, etc.; and communication unit 69, such as network card, modem, wireless transceiver, etc. Communication unit 69 allows electronic device 60 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0350] Processor 61 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 61 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 61 performs the various methods and processes described above, such as control methods.

[0351] In some embodiments, the control method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 68. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 60 via ROM 62 and / or communication unit 69. When the computer program is loaded into RAM 63 and executed by processor 61, one or more steps of the control method described above may be performed. Alternatively, in other embodiments, processor 61 may be configured to execute the control method by any other suitable means (e.g., by means of firmware).

[0352] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0353] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0354] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0355] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0356] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0357] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0358] In one embodiment, the present invention further includes a computer program product, which includes a computer program that, when executed by a processor, implements the control method of any embodiment of the present invention.

[0359] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0360] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0361] Example 6

[0362] This invention provides a vehicle, the vehicle including: a driver monitoring system, the driver monitoring system being used to execute the control method of any embodiment of this invention.

[0363] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A control method, characterized in that, include: Acquire vehicle driving data and multiple consecutive frames of driver head images within a safety monitoring period; The target driver's abnormality level is determined based on the vehicle's driving data and the driver's abnormality type corresponding to each frame of the driver's head image. Based on the target driver's abnormality level, corresponding safety controls are implemented.

2. The method according to claim 1, characterized in that, Before determining the target driver anomaly level based on the vehicle's driving data and the driver anomaly type corresponding to each frame of the driver's head image, the process also includes: Obtain the head deviation angle threshold; Based on the head deviation angle threshold, the driver's head images of multiple consecutive frames within the safety monitoring period are classified and identified to obtain the driver's abnormality type corresponding to each frame of driver's head images.

3. The method according to claim 1 or 2, characterized in that, Based on the vehicle's driving data and the driver anomaly type corresponding to each frame of the driver's head image, the target driver anomaly level is determined, including: The duration of the driver abnormality type is determined based on the driver abnormality type corresponding to each frame of driver head image; Obtain the trigger time range for each driver's abnormality level corresponding to the vehicle's driving data; The driver abnormality level corresponding to the trigger time range in which the duration of the driver abnormality type falls is taken as the target driver abnormality level.

4. The method according to claim 3, characterized in that, The driving data includes: vehicle speed, driving mode, and attention alert mode; The trigger time range for each driver anomaly level corresponding to the vehicle's driving data is obtained, including: If the vehicle's driving mode is assisted driving mode and the attention reminder mode is standard mode, then obtain the trigger time range of each driver abnormality level corresponding to the vehicle speed. If the vehicle's driving mode is assisted driving mode and the attention alert mode is sensitive mode, then obtain the trigger time range for each driver's abnormal level.

5. The method according to claim 4, characterized in that, The trigger time range limits for each driver anomaly level corresponding to the standard mode are all greater than the trigger time range limits for each driver anomaly level corresponding to the sensitive mode.

6. The method according to claim 4, characterized in that, Before acquiring vehicle driving data, the following steps are also included: Upon detecting a trigger operation on the target attention reminder control, the attention reminder mode is switched to the attention reminder mode corresponding to the target attention reminder control. The attention reminder mode includes: standard mode and sensitive mode. or, Switch the attention alert mode to the attention alert mode corresponding to time information and / or environmental information, whereby the environmental information includes: weather information and / or traffic information.

7. The method according to any one of claims 1-6, characterized in that, Based on the target driver's abnormality level, corresponding safety controls are implemented, including: Based on the target driver's abnormality level, a target alert message is generated, wherein the target alert message includes at least one of: pop-up alert message, voice alert message, and head-up display alert message; Security control is performed based on the target alert information.

8. The method according to any one of claims 1-7, characterized in that, Also includes: Based on the target driver's abnormality level, generate safety control commands; The safety control command is sent to the control system so that the control system controls the vehicle to stop, decelerate, exit the assisted driving mode, and suspend the use of the assisted driving mode at least one of the following:

9. The method according to any one of claims 1-6, characterized in that, The driver abnormality type includes: distraction type. The target driver abnormality level includes three distraction levels. The trigger time range limit of distraction level one is less than the trigger time range limit of distraction level two, and the trigger time range limit of distraction level two is less than the trigger time range limit of distraction level three. Based on the target driver's abnormality level, corresponding safety controls are implemented, including: If the target driver's abnormality level is level one distraction, then a first pop-up reminder and a first voice reminder will be issued; If the target driver's abnormality level is level two distraction, then a first pop-up reminder, a first voice reminder, and a head-up display reminder will be issued; If the target driver's abnormality level is Distraction Level 3, a second pop-up reminder, a second voice reminder, and a head-up display reminder will be issued, and the target driver's abnormality level will be sent to the control system so that the control system can control the vehicle. The second pop-up reminder includes a first pop-up reminder and a driver assistance exit and disable reminder. The second voice reminder includes a first voice reminder and a driver assistance exit and disable reminder.

10. The method according to any one of claims 1-6, characterized in that, The driver abnormality type includes fatigue type, and the target driver abnormality level includes two fatigue levels, with the trigger time range limit for fatigue level one being smaller than the trigger time range limit for fatigue level two. Based on the target driver's abnormality level, corresponding safety controls are implemented, including: If the target driver's abnormality level is fatigue level one, then a third pop-up reminder, a third voice reminder, and a head-up display reminder will be issued; If the target driver's abnormal level is fatigue level 2, a fourth pop-up reminder, a fourth voice reminder, and a head-up display reminder will be issued, and the target driver's abnormal level will be sent to the control system so that the control system can control the vehicle. The fourth pop-up reminder includes a third pop-up reminder and a driver assistance exit and disable reminder, and the fourth voice reminder includes a third voice reminder and a driver assistance exit and disable reminder.

11. The method according to any one of claims 1-6, characterized in that, The driver abnormality type includes: making phone calls, and the target driving abnormality level includes making phone calls at level one. Based on the target driver's abnormality level, corresponding safety controls are implemented, including: If the target driver's abnormality level is Level 1 (making a phone call), then a fifth pop-up reminder will be issued.

12. The method according to any one of claims 1-8, characterized in that, Based on the vehicle's driving data and the driver anomaly type corresponding to each frame of the driver's head image, the target driver anomaly level is determined, including: Get the steering wheel grip status; The target driver's abnormality level is determined based on the steering wheel grip status, the vehicle's driving data, and the driver's abnormality type corresponding to each frame of the driver's head image.

13. The method according to claim 4, characterized in that, Obtain the trigger time range for each driver anomaly level corresponding to the vehicle speed, including: Get the trigger time range of each driver abnormality level corresponding to the first speed range and the trigger time range of each driver abnormality level corresponding to the second speed range, wherein the speed in the first speed range is greater than a first value and less than or equal to a second value, and the speed in the second speed range is greater than or equal to the second value. If the vehicle speed is within the first vehicle speed range, then the trigger time range of each driver abnormality level corresponding to the first vehicle speed range shall be used as the trigger time range of each driver abnormality level corresponding to the vehicle speed. If the vehicle speed is within the second vehicle speed range, then the trigger time range of each driver abnormality level corresponding to the second vehicle speed range shall be used as the trigger time range of each driver abnormality level corresponding to the vehicle speed.

14. The method according to claim 13, characterized in that, The trigger time limits for each distraction level corresponding to the first speed range are all greater than the trigger time limits for each distraction level corresponding to the second speed range, and the trigger time limits for each fatigue level corresponding to the first speed range are all greater than the trigger time limits for each fatigue level corresponding to the second speed range.

15. The method according to claim 1, characterized in that, Also includes: The accuracy of receiving feedback information and / or classification and recognition results sent by the camera; A first signal is generated based on the accuracy of the returned information and / or classification and identification results, wherein the first signal includes: a status information identifier of the driver monitoring system; The first signal is sent to the control system so that the control system controls the vehicle based on the first signal.

16. The method according to claim 1, characterized in that, Also includes: A second signal is generated based on the driver anomaly type corresponding to each frame of the driver's head image, or based on the target driver anomaly level. The second signal is sent to the control system so that the control system controls the vehicle based on the second signal.

17. The method according to claim 2, characterized in that, The head deviation angle thresholds include: head top deviation angle threshold, head bottom deviation angle threshold, head left deviation angle threshold, and head right deviation angle threshold.

18. A control device, characterized in that, include: The acquisition module is used to acquire vehicle driving data and multiple consecutive frames of driver head images within a safety monitoring period. The target driver anomaly level determination module is used to determine the target driver anomaly level based on the vehicle's driving data and the driver anomaly type corresponding to each frame of the driver's head image; The safety control module is used to perform corresponding safety controls based on the abnormality level of the target driver.

19. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the control method according to any one of claims 1-17.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are used to cause a processor to execute the control method according to any one of claims 1-17.

21. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the control method according to any one of claims 1-17.

22. A vehicle, characterized in that, The vehicle includes a driver monitoring system, which is used to execute the control method according to any one of claims 1-17.