Fatigue driving warning method, device, terminal and fatigue driving warning system

By adopting a multi-sensory early warning scheme based on the driver's fatigue level, combining the functions of the vehicle machine system and intelligent hardware system, providing a refreshing mode or takeover mode, the problem of single and poor fatigue driving early warning in the existing technology is solved, significantly improving Safety and comfort.

CN114821966BActive Publication Date: 2024-11-12GREAT WALL MOTOR CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110785650.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-07-12
Publication Date
2024-11-12
Estimated Expiration
2041-07-12

AI Technical Summary

Technical Problem

The existing fatigue driving warning method is single, and it cannot effectively reduce the safety hazards caused by fatigue driving.

Method used

According to the driver's fatigue level, different levels of early warning schemes are adopted, including combined early warnings of hearing, vision, smell and touch. Through the coordinated work of the vehicle and machine system and intelligent hardware system, a refreshing mode or takeover mode is provided to relieve fatigue.

Benefits of technology

It improves the effect of fatigue driving warning, reduces the safety hazards brought by fatigue driving, and improves the safety and comfort of the driver.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114821966B_ABST
    Figure CN114821966B_ABST
Patent Text Reader

Abstract

The present invention provides a fatigue driving warning method, device, terminal and fatigue driving warning system. The method comprises: determining the driver's fatigue driving level; if the fatigue driving level is level one, executing a warning scheme based on hearing and vision; if the fatigue driving level is level two, executing a warning scheme based on hearing, vision and smell; if the fatigue driving level is level three, executing a warning scheme based on hearing, vision and touch; wherein the fatigue driving degrees of level one, level two and level three increase step by step. The present invention adopts different warning schemes for different levels of fatigue driving, which can achieve better warning effects and reduce the safety hazards caused by fatigue driving.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of fatigue driving technology, and in particular to a fatigue driving warning method, device, terminal and fatigue driving warning system. Background Art

[0002] Fatigue driving can easily lead to traffic accidents. It refers to the phenomenon that after a long period of continuous driving, the driver's physiological and psychological functions become disordered, and objectively his driving skills decline. Fatigue driving can easily lead to traffic accidents. Therefore, it is very necessary to give drivers fatigue driving warnings.

[0003] At present, fatigue driving warning usually uses a single method to warn when the driver is detected to be driving fatigued, such as voice reminder, etc. However, this warning method is relatively simple and cannot achieve a good warning effect, and still has a high safety hazard caused by fatigue driving. Summary of the invention

[0004] The embodiments of the present invention provide a fatigue driving warning method, device, terminal and fatigue driving warning system to solve the problem that the warning method is relatively single, cannot achieve a good warning effect, and still has a high safety hazard caused by fatigue driving.

[0005] In a first aspect, an embodiment of the present invention provides a fatigue driving warning method, comprising:

[0006] Determine the driver's level of fatigue;

[0007] If the fatigue driving level is level one, the early warning plan based on hearing and vision will be implemented;

[0008] If the fatigue driving level is level 2, a warning plan based on hearing, vision and smell will be implemented;

[0009] If the fatigue driving level is level three, a warning plan based on hearing, vision and touch will be implemented;

[0010] Among them, the degree of fatigue driving increases gradually from level one, level two to level three.

[0011] In a possible implementation, an early warning scheme based on hearing, vision and touch is implemented, including:

[0012] Sending a third-level control instruction to the vehicle system and the intelligent hardware system; the third-level control instruction is used to instruct the intelligent hardware system to control at least one of the seat belt, the steering wheel and the seat for early warning; the third-level control instruction is used to instruct the vehicle system to prompt the driver through voice and / or display screen whether to start the alertness mode or the takeover mode, and start the alertness mode or the takeover mode based on the driver's choice;

[0013] Among them, in the refreshing mode, the car system controls the vehicle air conditioning, windows and vehicle multimedia to relieve the driver's driving fatigue; in the takeover mode, the car system controls the intelligent navigation system and ADAS to navigate to the nearest rest area or the rest area designated by the driver.

[0014] In a possible implementation, in the refreshing mode, the vehicle system controls the target temperature of the vehicle air conditioner to be a preset temperature, the air volume gear to be a preset gear, the air outlet mode to be a face blowing mode, and the circulation mode to be an internal circulation;

[0015] In the refreshment mode, the car system controls the opening ratio of the windows according to the vehicle speed;

[0016] In the refreshing mode, the car system controls the multimedia to increase the music playback volume.

[0017] In a possible implementation, an early warning scheme based on hearing, vision and smell is implemented, including:

[0018] Send secondary control instructions to the vehicle system and the intelligent hardware system; the secondary control instructions are used to instruct the vehicle system to communicate with the driver using a voice that the driver is interested in; the secondary control instructions are used to instruct the intelligent hardware system to control the interior ambient light for light reminders and to control the vehicle fragrance equipment to start working.

[0019] In a possible implementation, an auditory and visual based warning scheme is implemented, including:

[0020] Sending a first-level control command to the vehicle system and the intelligent hardware system; the first-level control command is used to instruct the vehicle system to use the virtual image and voice of interest to the driver for voice reminders; the first-level control command is used to instruct the intelligent hardware system to control the interior ambient light for light reminders.

[0021] In a possible implementation, determining the driver's driving fatigue level includes:

[0022] Acquire a plurality of continuous frames of images of the driver captured within a preset time period before the current moment;

[0023] For each frame of image, determine the eye area of ​​the frame image, and perform binarization processing on the eye area of ​​the frame image to obtain a histogram matrix of the eye area of ​​the frame image; calculate a first difference sum between the histogram matrix of the eye area of ​​the frame image and a predetermined standard histogram matrix of an open-eye state; calculate a second difference sum between the histogram matrix of the eye area of ​​the frame image and a predetermined standard histogram matrix of a closed-eye state; if the first difference sum is less than the second difference sum, determine that the driver is in an open-eye state at the corresponding moment of the frame image, otherwise, determine that the driver is in a closed-eye state at the corresponding moment of the frame image;

[0024] According to the driver's eye status at each moment within the preset time, the PERCLOS value and average eye closure time corresponding to the preset time are calculated;

[0025] If the PERCLOS value corresponding to the preset duration is greater than the preset threshold, the driver's fatigue driving level is determined based on the average eye closure time.

[0026] In a second aspect, an embodiment of the present invention provides a fatigue driving warning device, comprising:

[0027] A fatigue driving level determination module, used to determine the driver's fatigue driving level;

[0028] A first warning module is used to execute a warning scheme based on hearing and vision if the fatigue driving level is level one;

[0029] The second warning module is used to execute a warning scheme based on hearing, vision and smell if the fatigue driving level is level 2;

[0030] A third warning module is used to execute a warning scheme based on hearing, vision and touch if the fatigue driving level is level three;

[0031] Among them, the degree of fatigue driving increases gradually from level one, level two to level three.

[0032] In a third aspect, an embodiment of the present invention provides a terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the fatigue driving warning method as described in the first aspect or any possible implementation method of the first aspect are implemented.

[0033] In a fourth aspect, an embodiment of the present invention provides a fatigue driving warning system, comprising a vehicle system, an intelligent hardware system and a terminal as described in the third aspect;

[0034] The terminal is connected to the vehicle system and the intelligent hardware system respectively.

[0035] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the fatigue driving warning method as described in the first aspect or any possible implementation method of the first aspect are implemented.

[0036] The embodiment of the present invention provides a fatigue driving warning method, device, terminal and fatigue driving warning system, which classifies the driver's fatigue driving degree and adopts different warning schemes for different levels of fatigue driving. When the fatigue driving level is level one, a warning scheme based on hearing and vision is executed; when the fatigue driving level is level two, a warning scheme based on hearing, vision and smell is executed; when the fatigue driving level is level three, a warning scheme based on hearing, vision and touch is executed. As the degree of fatigue driving increases, a more appropriate warning scheme can be adopted to achieve a better warning effect and reduce the safety hazards caused by fatigue driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0038] Figure 1 is a structural schematic diagram of a fatigue driving warning system provided by an embodiment of the present invention;

[0039] Figure 2 is a flowchart of an implementation of a fatigue driving warning method provided by an embodiment of the present invention;

[0040] Figure 3 is a principle curve diagram of PERCLOS provided by an embodiment of the present invention;

[0041] Figure 4 is a schematic diagram of the structure of a fatigue driving warning device provided by an embodiment of the present invention;

[0042] Figure 5 is a schematic diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0044] In order to make the purpose, technical solutions and advantages of the present invention more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0045] Figure 1This is a schematic diagram of the structure of the fatigue driving warning system provided by an embodiment of the present invention. Figure 1 As shown, the fatigue driving warning system may include a vehicle system 5, an intelligent hardware system 6 and a terminal 4; the terminal 4 is connected to the vehicle system 5 and the intelligent hardware system 6 respectively.

[0046] Among them, the terminal 4 can be a vehicle-mounted terminal including a DMS (driver monitoring system), which can obtain the monitoring video of the driver through a camera, and the camera can be set at a position where the driver's facial features can be captured. According to the monitoring video, the real-time monitoring image of the driver is obtained, so that the driver's fatigue driving level can be judged, and different commands are sent to the intelligent hardware system 6 and the vehicle system 5 according to the fatigue driving level, so that the intelligent hardware system 6 and the vehicle system 5 can execute different early warning schemes according to different commands, for example, active voice interaction, intelligent hobby ecological push, refreshing fragrance system, intelligent cruise system, intelligent driving assistance and other AI artificial intelligence algorithm innovative functions can be used, and the five-sense cockpit push method is used to push step by step, and the innovative scene improves the user experience, avoids user complaints caused by hardware accumulation, and is of great significance to improving traffic safety conditions.

[0047] See also Figure 2 , which shows a flowchart of the implementation of the fatigue driving warning method provided by an embodiment of the present invention, which is described in detail as follows:

[0048] In S201 , the driver's fatigue driving level is determined.

[0049] In the embodiment of the present invention, the driver's fatigue driving level is first determined, so that a warning scheme corresponding to the fatigue driving level can be provided in real time.

[0050] The driver's driving fatigue level can be monitored in real time, and the warning plan can be changed at any time according to the changes in the driver's driving fatigue level.

[0051] Among them, the fatigue driving level can indicate the driver's degree of fatigue driving.

[0052] In some embodiments of the present invention, the above S201 may include the following steps:

[0053] Acquire a plurality of continuous frames of images of the driver captured within a preset time period before the current moment;

[0054] For each frame of image, determine the eye area of ​​the frame image, and perform binarization processing on the eye area of ​​the frame image to obtain a histogram matrix of the eye area of ​​the frame image; calculate a first difference sum between the histogram matrix of the eye area of ​​the frame image and a predetermined standard histogram matrix of an open-eye state; calculate a second difference sum between the histogram matrix of the eye area of ​​the frame image and a predetermined standard histogram matrix of a closed-eye state; if the first difference sum is less than the second difference sum, determine that the driver is in an open-eye state at the corresponding moment of the frame image, otherwise, determine that the driver is in a closed-eye state at the corresponding moment of the frame image;

[0055] According to the driver's eye status at each moment within the preset time, the PERCLOS value and average eye closure time corresponding to the preset time are calculated;

[0056] If the PERCLOS value corresponding to the preset duration is greater than the preset threshold, the driver's fatigue driving level is determined based on the average eye closure time.

[0057] In an embodiment of the present invention, a continuous multi-frame image of the driver captured within a preset time period before the current moment can be obtained based on the captured surveillance video of the driver, that is, the surveillance video captured within the preset time period before the current moment is decomposed into a continuous multi-frame image. The preset time period before the current moment may include the current moment. The preset time period can be set according to actual needs, for example, it can be set to several blink cycles.

[0058] PERCLOS is defined as the degree of eye closure within a certain period of time. It has become a scientific and effective method to measure fatigue. When the proportion of time that the eyes are closed within a certain time interval exceeds 15%, it is considered to be a fatigue state. The PERCLOS method determines fatigue driving by the proportion of time that the eyes are closed. However, the size of the eyes varies from person to person, and the area of ​​the eyes also changes dynamically due to the influence of the scene and head movement. The degree of eye openness is relative to the maximum open state of the eyes.

[0059] At present, the PERCLOS method has three different criteria for judging fatigue, namely EM criterion, P70 criterion and P80 criterion. Their specific meanings are as follows:

[0060] EM criteria: The eye is considered closed if more than 50% of the pupil is covered by the eyelids;

[0061] P70 criterion: If the pupil is covered by the eyelids for more than 70%, the eye is considered closed;

[0062] P80 criterion: If more than 80% of the pupil is covered by the eyelid, the eye is considered closed.

[0063] When a person is concentrating or in a state of deep thought, the eyelids may cover more than 50% or even 70% of the pupil, so the P80 criterion is adopted in this embodiment.

[0064] Figure 3 This is the principle curve diagram of PERCLOS, such as Figure 3 As shown in the figure, t0-t5 is a closed eye cycle, t0-t1 is the time from fully open eyes to 80% open eyes, t1-t2 is the time from 80% open eyes to 20% open eyes, t3-t4 is the time from 20% open eyes to 80% open eyes, and t4-t5 is the time from 80% open eyes to fully open eyes. Then, the PERCLOS value PERCLOS(t0-t5) in this closed eye cycle is:

[0065]

[0066] In this eye-closing cycle, the eye-closing duration T (t0-t5) is:

[0067] T(t0-t5)=t3-t2

[0068] Similarly, we can derive the PERCLOS value and average eye closure duration within a duration of 1 to 2 minutes. Assume that there are n eye closure cycles within a duration of T, and the duration in which the proportion of the pupil covered by the eyelid exceeds 80% in the kth eye closure cycle is T. k , then the PERCLOS value in the duration T is Within the duration T, the average eye closure duration t is

[0069] The specific process for determining the driver's fatigue driving level is detailed as follows:

[0070] 1) On-site video input, that is, obtaining the driver's monitoring video shot within a preset time before the current moment.

[0071] 2) Image acquisition: Based on the above monitoring images, real-time frame capture is performed to obtain monitoring images in YUV format.

[0072] 3) Face recognition: match the face area in each frame of the image through the face model. If the match is successful, continue to step 4). Otherwise, return to step 1), adjust the camera angle, and re-shoot the video.

[0073] 4) Eye detection: determine the eye area in each frame of the image by the relative position of the eyes in the face model.

[0074] 5) Human eye state recognition, that is, recognition of whether the eyes are open or closed. Perform binarization on the eye area of ​​each frame image to obtain the histogram matrix of the eye area of ​​the frame image; calculate the first difference and S1 between the histogram matrix of the eye area of ​​the frame image and the predetermined standard histogram matrix of the open eyes state; calculate the second difference and S2 between the histogram matrix of the eye area of ​​the frame image and the predetermined standard histogram matrix of the closed eyes state; if S1 is less than S2, it is determined that the driver is in the open eyes state at the corresponding moment of the frame image, otherwise, it is determined that the driver is in the closed eyes state at the corresponding moment of the frame image.

[0075] The histogram matrix can be expressed as H = [h{x1, f(x1)}, ..., h{x m ,f(x m )}], x1-x m There are m gray levels, f(x m ) is the frequency of occurrence of the mth gray level. The standard histogram matrix for the open eyes state and the standard histogram matrix for the closed eyes state can be pre-calibrated. Calculating the difference sum of the two histogram matrices can be calculating the sum of the frequency differences of each gray level in the two histogram matrices.

[0076] 6) Calculate the PERCLOS value and the average eye-closing duration. The PERCLOS value and the average eye-closing duration corresponding to the preset duration before the current moment are calculated according to the above formula. Specifically, the eye-opening duration and the eye-closing duration within the preset duration can be determined according to the eye state corresponding to each frame image, and then the PERCLOS value and the average eye-closing duration corresponding to the preset duration before the current moment are calculated.

[0077] 7) Determine the driver's fatigue driving level at the current moment. If the PERCLOS value corresponding to the preset duration is greater than the preset threshold, the driver's fatigue driving level is determined based on the average eye-closing duration. For example, if the average eye-closing duration is within the first preset duration range, it is level one. If the average eye-closing duration is within the second preset duration range, it is level two. If the average eye-closing duration is within the third preset duration range, it is level three. Among them, the duration within the third preset duration range is greater than the duration within the second preset duration range, and the duration within the second preset duration range is greater than the duration within the first preset duration range. The specific range can be set according to actual needs and is not specifically limited here. The above preset thresholds can also be set according to actual needs and are not specifically limited here.

[0078] In one possible implementation, the PERCLOS fatigue level evaluation index (PERCLOS value and average eye closure time) can be used in combination with eye blinking frequency, mouth opening degree and head movement trajectory to make a comprehensive judgment on the fatigue level and divide the driving fatigue level into mild fatigue, moderate fatigue and severe fatigue, i.e. level one, level two and level three.

[0079] In this embodiment, the terminal can continuously collect and analyze key facial markers through visual algorithms and artificial intelligence algorithms with sufficient performance, flexibly connect new sensors through SoC solutions and support different system topologies, such as independent single-box design, integrated cluster / infotainment system + DMS solution ECU, long-distance sensors, etc., and issue different commands based on the judgment results to trigger early warning plans of corresponding levels.

[0080] In S202, if the fatigue driving level is level one, an early warning scheme based on hearing and vision is executed.

[0081] In some embodiments of the present invention, the above-mentioned execution of the early warning scheme based on hearing and vision may include:

[0082] Sending a first-level control command to the vehicle system and the intelligent hardware system; the first-level control command is used to instruct the vehicle system to use the virtual image and voice of interest to the driver for voice reminders; the first-level control command is used to instruct the intelligent hardware system to control the interior ambient light for light reminders.

[0083] Among them, the first-level control instruction can be an instruction identifier. The terminal can send control instructions containing different instruction identifiers to the vehicle system and the intelligent hardware system according to different fatigue driving levels. The vehicle system and the intelligent hardware system can decide whether to execute the instruction based on the assigned identifier based on their respective identifier filters. If the identifier exists in the queue, the instruction is executed, otherwise it is not executed; according to application requirements, the best balance is achieved between performance and ease of use, and problems in synchronization and data sampling are minimized. After the terminal makes a level judgment on the current driver status, the judgment result is passed to the vehicle system and the intelligent hardware system through the instruction identifier. The vehicle system and the intelligent hardware system can send the instruction identifier to the corresponding operation system, such as the central operation system, the vehicle operation system, each ECU operation system, etc.

[0084] When the vehicle system receives a first-level control instruction, it passes the first-level control instruction to the corresponding ECU control units to execute the corresponding plan, specifically to the intelligent voice system and the intelligent AI image management system. Based on the first-level control instruction, the intelligent voice system can select the voice that the driver is interested in and play a voice reminder according to the driver's usual habits. Based on the first-level control instruction, the intelligent AI image management system can select the virtual AI image that the user is interested in, display it on the display screen, and provide voice warnings to the driver in combination with the voice. Among them, some celebrity voices can be preset in the cloud corpus for drivers to choose and download, and some family members' voices can also be preset. Some virtual images can also be preset in the virtual image library in the cloud for users to choose.

[0085] When the intelligent hardware system receives the primary control instruction, it transmits the primary control instruction to the interior ambient light, causing the interior ambient light to flash.

[0086] The terminal continuously monitors the driver's fatigue driving level. When the level is reduced, the warning scheme based on hearing and vision is no longer executed; when the level is increased to level 2, S203 can be executed.

[0087] In S203, if the fatigue driving level is level 2, a warning scheme based on hearing, vision and smell is executed.

[0088] In some embodiments of the present invention, the above-mentioned execution of the early warning scheme based on hearing, vision and smell may include:

[0089] Send secondary control instructions to the vehicle system and the intelligent hardware system; the secondary control instructions are used to instruct the vehicle system to communicate with the driver using a voice that the driver is interested in; the secondary control instructions are used to instruct the intelligent hardware system to control the interior ambient light for light reminders and to control the vehicle fragrance equipment to start working.

[0090] When the vehicle system receives the secondary control command, it sends it to the intelligent voice system and the intelligent ecological push system. The intelligent voice system can actively start a conversation with the driver using a voice that the driver is interested in, for example: "Handsome, I detect that you are a little tired, do you need some music to refresh yourself?" Actively and continuously have a continuous conversation with the driver to distract the user's attention, allowing him to stimulate the brain nerves through continuous operation and activity of the brain, thereby distracting the perception of fatigue.

[0091] Based on the driver's answers and the access to the intelligent ecological push system, multimedia information or other relevant information and interfaces can be pushed to the driver, completing the conversation with the user more intelligently, so that the driver's intelligent interactive perception of the car is constantly reshaped and the user experience is improved.

[0092] When the intelligent hardware system receives the secondary control command, it can pass it to the vehicle fragrance device and the interior ambient light. The interior ambient light can be adjusted to a blue-green band based on the command. The vehicle fragrance device can be pre-set with a mint refreshing fragrance, and based on the command, the fragrance is activated.

[0093] The terminal continuously monitors the driver's fatigue driving level. When the level decreases, the warning plan corresponding to the first level is executed; when the level increases to the third level, S204 can be executed.

[0094] In S204, if the fatigue driving level is level three, a warning scheme based on hearing, vision and touch is executed;

[0095] Among them, the degree of fatigue driving increases gradually from level one, level two to level three.

[0096] In some embodiments of the present invention, the above-mentioned execution of the early warning scheme based on hearing, vision and touch may include:

[0097] Sending a third-level control instruction to the vehicle system and the intelligent hardware system; the third-level control instruction is used to instruct the intelligent hardware system to control at least one of the seat belt, the steering wheel and the seat for early warning; the third-level control instruction is used to instruct the vehicle system to prompt the driver through voice and / or display screen whether to start the alertness mode or the takeover mode, and start the alertness mode or the takeover mode based on the driver's choice;

[0098] Among them, in the refreshing mode, the car system controls the vehicle air conditioning, windows and vehicle multimedia to relieve the driver's driving fatigue; in the takeover mode, the car system controls the intelligent navigation system and ADAS to navigate to the nearest rest area or the rest area designated by the driver.

[0099] In some embodiments of the present invention, in the refreshing mode, the vehicle system controls the target temperature of the vehicle air conditioner to be a preset temperature, the air volume gear to be a preset gear, the air outlet mode to be a face blowing mode, and the circulation mode to be an internal circulation;

[0100] In the refreshment mode, the car system controls the opening ratio of the windows according to the vehicle speed;

[0101] In the refreshing mode, the car system controls the multimedia to increase the music playback volume.

[0102] In this embodiment, when the fatigue driving level is level three, a level three control instruction can be sent to the intelligent hardware system first. The intelligent hardware system triggers the vehicle seat belt vibration, steering wheel vibration and / or seat massage according to the instruction. Combined with the vehicle configuration, at least one of the seat belt, steering wheel and seat can be started simultaneously.

[0103] After a period of time, for example, after 30 seconds, the third-level control command is sent to the vehicle system. The vehicle system can ask the driver whether to turn on the refreshment mode / takeover mode through the intelligent voice system, and / or display "refreshment mode or takeover mode" on the display screen. The driver can select one of the modes through voice or display. If the driver does not make a selection within a certain period of time, for example, within 30 seconds, the refreshment mode can be automatically entered.

[0104] Refreshing Mode:

[0105] Triggering prerequisite: Level 3 control command, the air conditioner is in non-front defrost state.

[0106] a) Refreshing mode activated:

[0107] √ The display shows that the refresh mode is on. The driver can select "Cancel" on the display to cancel the refresh mode;

[0108] √ Voice prompt: "We have detected your fatigue and are about to turn on the refreshment mode for you."

[0109] b) Refresh mode execution:

[0110] √ Car air conditioning

[0111] i. The target temperature of the air conditioner is changed to the preset temperature. The preset temperature can be a lower temperature, for example, 18 degrees;

[0112] ii. The air volume gear is changed to a preset gear, for example, gear 3;

[0113] iii. The air outlet mode is switched to the face blowing mode; the circulation mode is switched to the internal circulation mode;

[0114] √ Car windows

[0115] i. Window status:

[0116] The windows are controlled only when the vehicle is in a safe state; otherwise, the windows are not controlled in the refreshment mode.

[0117] When the vehicle speed is not greater than the first preset vehicle speed, it can be considered as a safe state.

[0118] Furthermore, when the vehicle speed is not greater than the first preset vehicle speed and no user is making a phone call in the vehicle, it can be considered to be in a safe state.

[0119] Furthermore, when the vehicle speed is not greater than the first preset speed, and no user is making a phone call in the vehicle, and the window anti-pinch function is turned on, it can be determined to be a safe state.

[0120] If the current vehicle window is in an open state, the vehicle window may not be controlled to remain in the open state.

[0121] ii. In a safe state:

[0122] If the second preset vehicle speed <= vehicle speed < the third preset vehicle speed, then all the vehicle windows are opened, that is, all the vehicle windows are opened;

[0123] If the third preset vehicle speed <= vehicle speed <= first preset vehicle speed, the window is opened 1 / 2, that is, the entire window is opened halfway, half-open.

[0124] Among them, the first preset vehicle speed may be 90 km / h, the second preset vehicle speed may be 30 km / h, and the third preset vehicle speed may be 60 km / h.

[0125] iii. Abnormal weather:

[0126] If the weather in the user's area is blowing sand, floating dust, sandstorm, or strong sandstorm, do not open the car windows;

[0127] When the car window is open and it detects rain, the voice or display prompts: "Is it raining? Remember to close the window";

[0128] When the refresh mode is turned off, close the windows if they are not closed;

[0129] iv. User self-adjustment: If the user adjusts the window by himself while controlling the window in the refresh mode, the voice or display will prompt "Refresh window has been released~", and the window will not be closed after 3 minutes;

[0130] √In-car multimedia

[0131] i. If music is playing in the car, turn up the volume of the music being played;

[0132] ii. If the music is not turned on in the car, this strategy will not be executed;

[0133] iii. Special circumstances:

[0134] If the user chooses to turn off or pause the music during the volume adjustment process, a voice prompt or display screen will appear: "The refreshing music has been turned off."

[0135] c) Refresh mode off:

[0136] √ The windows and air conditioner will be turned off after being turned on for a period of time, for example, after 3 minutes. If the user operates the windows to fully open or to half of the full window, the windows will no longer be closed when the refresh mode is turned off, but the voice or display will remind you normally: "The refresh mode will be turned off for you soon."

[0137] √ Music: If the user is still in the second / third level fatigue state and the user does not adjust the volume, the music will continue to play until the user operates or the fatigue state is changed to the first level or no fatigue.

[0138] Takeover mode:

[0139] Triggering conditions: Level 3 control command, user actively selects takeover mode;

[0140] a) Takeover mode is turned on:

[0141] √ Voice reminder: Actively ask "Please tell me your resting place, no more than 30 kilometers away". If the address is more than 30 kilometers away, the user needs to confirm. During the navigation task, active voice interaction is carried out according to the navigation strategy;

[0142] √ Navigation: Actively start the navigation mode and provide the user with the next step of service based on the address entered by the user;

[0143] If the navigation distance is less than 30km, the user task will be performed normally;

[0144] When the navigation distance is between 30km and 50km, the system will proactively ask the user to confirm the address. If the user chooses to execute the navigation task, the vehicle will still execute the user's navigation task.

[0145] When the navigation distance is >50km, the user's navigation task will not be executed, and the user will be proactively informed that the vehicle cannot perform the assisted driving function in the warning mode due to the long distance, and the user will be proactively asked whether to exit the fatigue driving warning mode;

[0146] √ Interface: During the execution of navigation strategies, various pop-up boxes will be automatically prompted based on the navigation tasks.

[0147] b) Takeover mode execution:

[0148] √ When the vehicle is detected to be driving on a highway, the high-speed assisted driving function is activated to navigate to the nearest service area;

[0149] √ When the vehicle is detected driving on a normal road, the low-speed assisted driving combination function is activated to navigate to the rest area selected by the user.

[0150] c) Takeover mode off:

[0151] When the user reaches the navigation destination, the takeover mode is turned off.

[0152] In the embodiment of the present invention, the fatigue driving level is divided into three levels, and the degree of fatigue driving gradually increases from level 1 to level 3. For example, level 1 may represent mild fatigue, level 2 may represent moderate fatigue, and level 3 may represent severe fatigue. In practical applications, the fatigue driving level can be divided according to actual needs, for example, it can be divided into two levels, four levels, five levels, etc., and no specific limitation is made here.

[0153] As can be seen from the above description, the embodiment of the present invention adopts different warning schemes for different levels of fatigue driving by grading the driver's fatigue driving degree. When the fatigue driving level is level one, a warning scheme based on hearing and vision is executed; when the fatigue driving level is level two, a warning scheme based on hearing, vision and smell is executed; when the fatigue driving level is level three, a warning scheme based on hearing, vision and touch is executed. As the degree of fatigue driving increases, a more appropriate warning scheme can be adopted to achieve a better warning effect and reduce the safety hazards caused by fatigue driving.

[0154] The embodiment of the present invention takes into account the all-round needs of the driver and passengers in the car, and adopts an upgraded perception and active interaction strategy from the perspective of user experience. It follows the principle of usability, ease of use, and good usability, and softly reminds the driver to warn of fatigue and intelligent driving takeover. The interface is what you see is what you get, and the trigger mechanism corresponds to the warning result. This solution can ultimately solve the fundamental problem of fatigue driving. According to the development stage of autonomous driving, the driver can navigate to the nearest rest area for sufficient sleep.

[0155] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0156] The following is an embodiment of the device of the present invention. For details not described in detail, reference may be made to the corresponding method embodiments described above.

[0157] Figure 4 The structure diagram of the fatigue driving warning device provided by the embodiment of the present invention is shown. For the convenience of explanation, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0158] like Figure 4 As shown, the fatigue driving warning device 30 includes: a fatigue driving level determination module 31 , a first warning module 32 , a second warning module 33 and a third warning module 34 .

[0159] A fatigue driving level determination module 31 is used to determine the driver's fatigue driving level;

[0160] A first warning module 32 is used to execute a warning scheme based on hearing and vision if the fatigue driving level is level one;

[0161] The second warning module 33 is used to execute a warning scheme based on hearing, vision and smell if the fatigue driving level is level 2;

[0162] The third warning module 34 is used to execute a warning scheme based on hearing, vision and touch if the fatigue driving level is level three;

[0163] Among them, the degree of fatigue driving increases gradually from level one, level two to level three.

[0164] In a possible implementation, the third early warning module 34 is further configured to:

[0165] Sending a third-level control instruction to the vehicle system and the intelligent hardware system; the third-level control instruction is used to instruct the intelligent hardware system to control at least one of the seat belt, the steering wheel and the seat for early warning; the third-level control instruction is used to instruct the vehicle system to prompt the driver through voice and / or display screen whether to start the alertness mode or the takeover mode, and start the alertness mode or the takeover mode based on the driver's choice;

[0166] Among them, in the refreshing mode, the car system controls the vehicle air conditioning, windows and vehicle multimedia to relieve the driver's driving fatigue; in the takeover mode, the car system controls the intelligent navigation system and ADAS to navigate to the nearest rest area or the rest area designated by the driver.

[0167] In a possible implementation, in the refreshing mode, the vehicle system controls the target temperature of the vehicle air conditioner to be a preset temperature, the air volume gear to be a preset gear, the air outlet mode to be a face blowing mode, and the circulation mode to be an internal circulation;

[0168] In the refreshment mode, the car system controls the opening ratio of the windows according to the vehicle speed;

[0169] In the refreshing mode, the car system controls the multimedia to increase the music playback volume.

[0170] In a possible implementation, the second early warning module 33 is further configured to:

[0171] Send secondary control instructions to the vehicle system and the intelligent hardware system; the secondary control instructions are used to instruct the vehicle system to communicate with the driver using a voice that the driver is interested in; the secondary control instructions are used to instruct the intelligent hardware system to control the interior ambient light for light reminders and to control the vehicle fragrance equipment to start working.

[0172] In a possible implementation, the first early warning module 32 may also be used to:

[0173] Sending a first-level control command to the vehicle system and the intelligent hardware system; the first-level control command is used to instruct the vehicle system to use the virtual image and voice of interest to the driver for voice reminders; the first-level control command is used to instruct the intelligent hardware system to control the interior ambient light for light reminders.

[0174] In a possible implementation, the fatigue driving level determination module 31 may also be used to:

[0175] Acquire a plurality of continuous frames of images of the driver captured within a preset time period before the current moment;

[0176] For each frame of image, determine the eye area of ​​the frame image, and perform binarization processing on the eye area of ​​the frame image to obtain a histogram matrix of the eye area of ​​the frame image; calculate a first difference sum between the histogram matrix of the eye area of ​​the frame image and a predetermined standard histogram matrix of an open-eye state; calculate a second difference sum between the histogram matrix of the eye area of ​​the frame image and a predetermined standard histogram matrix of a closed-eye state; if the first difference sum is less than the second difference sum, determine that the driver is in an open-eye state at the corresponding moment of the frame image, otherwise, determine that the driver is in a closed-eye state at the corresponding moment of the frame image;

[0177] According to the driver's eye status at each moment within the preset time, the PERCLOS value and average eye closure time corresponding to the preset time are calculated;

[0178] If the PERCLOS value corresponding to the preset duration is greater than the preset threshold, the driver's fatigue driving level is determined based on the average eye closure time.

[0179] Figure 5 is a schematic diagram of a terminal provided by an embodiment of the present invention. Figure 5 As shown, the terminal 4 of this embodiment includes: a processor 40, a memory 41, and a computer program 42 stored in the memory 41 and executable on the processor 40. When the processor 40 executes the computer program 42, the steps in the above-mentioned various fatigue driving warning method embodiments are implemented, for example Figure 2 Alternatively, when the processor 40 executes the computer program 42, the functions of each module / unit in the above-mentioned device embodiments are realized, for example, Figure 4 Functionality of the modules / units 31 to 34 shown.

[0180] Exemplarily, the computer program 42 may be divided into one or more modules / units, which are stored in the memory 41 and executed by the processor 40 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program 42 in the terminal 4. For example, the computer program 42 may be divided into Figure 4 Modules / units 31 to 34 are shown.

[0181] The terminal 4 may include, but is not limited to, a processor 40 and a memory 41. Those skilled in the art will appreciate that Figure 5 It is only an example of terminal 4 and does not constitute a limitation on terminal 4. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal may also include input and output devices, network access devices, buses, etc.

[0182] The processor 40 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0183] The memory 41 may be an internal storage unit of the terminal 4, such as a hard disk or memory of the terminal 4. The memory 41 may also be an external storage device of the terminal 4, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal 4. Further, the memory 41 may also include both an internal storage unit and an external storage device of the terminal 4. The memory 41 is used to store the computer program and other programs and data required by the terminal. The memory 41 may also be used to temporarily store data that has been output or is to be output.

[0184] See also Figure 1 , the embodiment of the present invention also provides a fatigue driving warning system, including a vehicle system 5, an intelligent hardware system 6 and the terminal 4 as described above;

[0185] The terminal 4 is connected to the vehicle system 5 and the intelligent hardware system 6 respectively.

[0186] For a detailed description of the fatigue driving warning system, please refer to the specific instructions in the above method / device, which will not be repeated here.

[0187] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0188] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0189] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0190] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0191] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0192] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0193] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned fatigue driving warning method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0194] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A fatigue driving warning method, It is characterized in that include: Determine the driver's level of fatigue; If the fatigue driving level is level one, a warning plan based on hearing and vision is implemented; If the fatigue driving level is level 2, a warning plan based on hearing, vision and smell is implemented; If the fatigue driving level is level three, a warning plan based on hearing, vision and touch is implemented; The first level, the second level and the third level of fatigue driving degrees increase gradually; The implementation of an early warning program based on hearing, vision and touch includes: Sending a third-level control instruction to the vehicle system; the third-level control instruction is used to instruct the vehicle system to prompt the driver through voice and / or display screen whether to start the refreshing mode or the takeover mode, and start the refreshing mode or the takeover mode based on the driver's choice; In the refresh mode, the vehicle system controls the windows in a safe state, and controls the windows to be fully opened when the vehicle speed is greater than or equal to the second preset speed and less than the third preset speed, and controls the windows to be half-open when the vehicle speed is greater than or equal to the third preset speed and less than or equal to the first preset speed; wherein, when the vehicle speed is not greater than the first preset speed, and there is no user in the car making a phone call, and the window anti-pinch function is turned on, it is determined to be in a safe state.

2. The fatigue driving warning method according to claim 1, It is characterized in that The implementation of the early warning scheme based on hearing, vision and touch also includes: Sending a third-level control instruction to the intelligent hardware system; the third-level control instruction is used to instruct the intelligent hardware system to control at least one of the seat belt, the steering wheel and the seat to issue an early warning; Among them, in the refreshing mode, the vehicle system controls the vehicle air conditioning, windows and vehicle multimedia to alleviate the driver's fatigue driving; in the takeover mode, the vehicle system controls the intelligent navigation system and ADAS to navigate to the nearest rest area or the rest area designated by the driver.

3. The fatigue driving warning method according to claim 2, It is characterized in that In the refreshing mode, the vehicle system controls the target temperature of the vehicle air conditioner to be a preset temperature, the air volume gear to be a preset gear, the air outlet mode to be a face blowing mode, and the circulation mode to be an internal circulation; In the refreshment mode, the vehicle system controls the opening ratio of the vehicle window according to the vehicle speed; In the refreshing mode, the vehicle system controls the multimedia to increase the volume of music playback.

4. The fatigue driving warning method according to claim 1, It is characterized in that The implementation of an early warning program based on hearing, vision and smell includes: Sending secondary control instructions to the vehicle system and the intelligent hardware system; the secondary control instructions are used to instruct the vehicle system to communicate with the driver using a voice that the driver is interested in; the secondary control instructions are used to instruct the intelligent hardware system to control the interior ambient light for light reminders and to control the vehicle fragrance equipment to start working.

5. The fatigue driving warning method according to claim 1, It is characterized in that The implementation of an auditory and visual warning program includes: Sending a first-level control instruction to the vehicle system and the intelligent hardware system; the first-level control instruction is used to instruct the vehicle system to use a virtual image and sound that the driver is interested in for voice reminders; the first-level control instruction is used to instruct the intelligent hardware system to control the interior ambient light for light reminders.

6. The fatigue driving warning method according to any one of claims 1 to 5, It is characterized in that Determining the driver's fatigue driving level includes: Acquire a plurality of continuous frames of images of the driver captured within a preset time period before the current moment; For each frame of image, determine the eye area of ​​the frame image, and perform binarization processing on the eye area of ​​the frame image to obtain a histogram matrix of the eye area of ​​the frame image; calculate a first difference sum between the histogram matrix of the eye area of ​​the frame image and a predetermined standard histogram matrix of an open-eye state; calculate a second difference sum between the histogram matrix of the eye area of ​​the frame image and a predetermined standard histogram matrix of a closed-eye state; if the first difference sum is less than the second difference sum, determine that the driver is in an open-eye state at the corresponding moment of the frame image, otherwise, determine that the driver is in a closed-eye state at the corresponding moment of the frame image; According to the driver's eye status at each moment within the preset time, calculating the PERCLOS value and average eye closure time corresponding to the preset time; If the PERCLOS value corresponding to the preset duration is greater than a preset threshold, the driver's fatigue driving level is determined according to the average eye-closing duration.

7. A fatigue driving warning device, It is characterized in that include: A fatigue driving level determination module, used to determine the driver's fatigue driving level; A first warning module, configured to execute a warning scheme based on hearing and vision if the fatigue driving level is level one; A second warning module is used to execute a warning scheme based on hearing, vision and smell if the fatigue driving level is level 2; A third warning module, configured to execute a warning scheme based on hearing, vision and touch if the fatigue driving level is level three; The first level, the second level and the third level of fatigue driving degrees increase gradually; In the third warning module, the execution of the warning scheme based on hearing, vision and touch includes: Sending a third-level control instruction to the vehicle system; the third-level control instruction is used to instruct the vehicle system to prompt the driver through voice and / or display screen whether to start the refreshing mode or the takeover mode, and start the refreshing mode or the takeover mode based on the driver's choice; In the refresh mode, the vehicle system controls the windows in a safe state, and controls the windows to be fully opened when the vehicle speed is greater than or equal to the second preset speed and less than the third preset speed, and controls the windows to be half-open when the vehicle speed is greater than or equal to the third preset speed and less than or equal to the first preset speed; wherein, when the vehicle speed is not greater than the first preset speed, and there is no user in the car making a phone call, and the window anti-pinch function is turned on, it is determined to be in a safe state.

8. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the steps of the fatigue driving warning method as described in any one of claims 1 to 6 are implemented.

9. A fatigue driving warning system, It is characterized in that It comprises a vehicle system, an intelligent hardware system and a terminal as claimed in claim 8; The terminal is connected to the vehicle system and the intelligent hardware system respectively.

10. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the steps of the fatigue driving warning method as described in any one of claims 1 to 6 above are implemented.