Driving fatigue intervention method and system based on multi-modal perception interaction

Abstract

The invention belongs to the technical field of intelligent automobile control, and discloses a driving fatigue intervention method and system based on multi-mode perception interaction, and the method comprises the steps: obtaining the state and control information of a driver after an automobile is started, judging whether the driver is in a fatigue driving state or not according to the state and control information of the driver, and determining the fatigue level; if it is determined that the driver is in the fatigue driving state, corresponding intervention actions are executed according to the fatigue level and the set fatigue intervention measures. According to the method, whether the driver is in the fatigue state or not and the corresponding fatigue level are judged by analyzing the state of the driver and the control information, and corresponding intervention measures are taken according to different fatigue levels, so that the effective fatigue state adjusting effect is achieved, and interference to the driver is avoided.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent automobile control, and in particular relates to a driving fatigue intervention method and system based on multimodal perception interaction. Background Art

[0002] As the number of cars continues to rise, while improving the convenience of life, the increase in traffic accidents caused by it cannot be ignored. Fatigue driving is one of the important causes of traffic accidents. Fatigue driving refers to the phenomenon that the driver's physiological and psychological functions are out of balance after driving for a long time, which objectively leads to a decrease in driving function and reaction. The car cockpit needs to identify potential risks in a timely manner, help the driver adjust his or her physical and mental state, and work together to ensure driving safety.

[0003] The continuous improvement of the level of automobile intelligence today provides favorable conditions for early identification and effective intervention of fatigue driving. Traditional non-contact driving fatigue reminders are less interactive and lack regulatory intervention based on whether the driver is out of fatigue or not. Summary of the invention

[0004] In view of the above problems, the present invention provides a driving fatigue intervention method and system based on multimodal perception interaction, which adopts the following technical solutions:

[0005] A driving fatigue intervention method based on multimodal perception interaction includes the following steps:

[0006] After the vehicle is started, the driver's status and control information are obtained, and based on the driver's status and control information, it is determined whether the driver is in a fatigue driving state and the fatigue level is determined;

[0007] If it is determined that the driver is in a fatigue driving state, corresponding intervention actions are performed according to the fatigue level and the set fatigue intervention measures.

[0008] Further, the interventions included mild fatigue interventions and severe fatigue interventions;

[0009] Among them, mild fatigue intervention measures include turning on refreshing fragrance, blowing cold air from the air conditioner on the face, and playing beta wave music; severe fatigue intervention measures include seat massage, intelligent voice interaction, and making virtual calls to the vehicle's computer.

[0010] Furthermore, obtaining the driver status and control information, and judging whether the driver is in a fatigue driving state according to the driver status and control information, and determining the fatigue level, comprises the following steps:

[0011] Collect the driver's eye movement information and driving control behavior information in real time;

[0012] Based on the driver's eye movements and driving control behavior information, it is judged whether the driver is in a fatigue driving state and the fatigue level is determined.

[0013] Furthermore, eye movement information includes blinking frequency, blinking duration, eye closure state and line of sight direction; driving control behavior information includes vehicle speed change information, driver steering operation information and vehicle road driving information; fatigue level includes mild fatigue and severe fatigue.

[0014] Further, judging whether the driver is in a fatigue driving state and determining the fatigue level according to the driver's eye movements and driving control behavior information includes the following steps:

[0015] If the driver's line of sight is not looking straight ahead or looking left and right intermittently, the driver is determined to be in a state of mild fatigue; if the driver's blinking frequency decreases, the duration of each blink increases, and the driver is in a closed eye state within the set time, the driver is determined to be in a state of severe fatigue;

[0016] If it is determined based on the vehicle speed change information that the vehicle has irregular acceleration and braking behavior, and the vehicle speed suddenly increases or decreases, it is determined that the driver's alertness is reduced and the driver is in a state of mild fatigue; if it is determined based on the driver's steering operation information that the steering force of the vehicle's steering wheel is unstable and discontinuous, the steering angle exceeds the set value, or the steering angle remains unchanged within the set time, it is determined that the driver is in a state of severe fatigue; if it is determined based on the vehicle's road driving information that the vehicle deviates from the lane direction multiple times within the set time and the lane keeping ability is reduced, it is determined that the driver is in a state of severe fatigue.

[0017] Furthermore, if it is determined that the driver is in a fatigue driving state, corresponding intervention actions are performed according to the fatigue level and the set fatigue intervention measures, including the following steps:

[0018] If the driver is determined to be in a state of mild fatigue, the vehicle will make a voice announcement, including that you are detected to be in a state of fatigue driving, and the vehicle will turn on refreshing fragrance, air conditioning and beta wave music for you;

[0019] If the driver is determined to be in a state of severe fatigue and the user selects seat massage in the fatigue intervention measures, the vehicle will make a voice announcement and start the seat massage;

[0020] If the driver is determined to be in a state of severe fatigue and the user selects intelligent voice interaction in the fatigue intervention measures, the vehicle voice will interact with the driver based on driving and user data;

[0021] If it is determined that the driver is still in a state of severe fatigue, and the user chooses to make a virtual call to the vehicle's computer in the fatigue intervention measures, the information that the driver is still in a state of severe fatigue will be sent to the cloud server. The cloud server will make an AI call to the vehicle's computer with a randomly generated virtual number to remind the driver to stop and rest.

[0022] Furthermore, the method further comprises the following steps:

[0023] When performing intervention actions, the driver's status and control information are continuously obtained, and based on the driver's status and control information, it is determined whether the driver is still in a fatigue state. If the driver is still in a fatigue state, corresponding fatigue intervention measures are performed according to the current fatigue level. After it is determined that the driver is out of the fatigue state, the vehicle will return to the state before the fatigue driving intervention is performed.

[0024] Furthermore, when performing the intervention action, the driver's state and control information are continuously obtained, and it is determined whether the driver is still in a fatigue state based on the driver's state and control information. If the driver is still in a fatigue state, corresponding fatigue intervention measures are performed based on the current fatigue level, including the following steps:

[0025] If it is determined that the driver has been in a mild fatigue state for a set time, or has entered a severe fatigue state within the set time, the vehicle will take over the severe fatigue intervention measures. At this time, the mild fatigue intervention measures will continue;

[0026] If the driver is in a state of severe fatigue, a virtual call is made to the vehicle computer and the driver's condition is continuously monitored after the call is hung up. If it is determined that the driver is still in a state of fatigue driving, the driver will be reminded again to stop and rest in the form of a voice broadcast, which is a mild fatigue intervention measure that has been activated.

[0027] The present invention also provides a driving fatigue intervention system based on multimodal perception interaction, comprising an information analysis unit and a fatigue intervention unit:

[0028] The information analysis unit is also used to obtain the driver's state and control information after the vehicle is started, and to determine whether the driver is in a fatigue driving state and the fatigue level according to the driver's state and control information;

[0029] The fatigue intervention unit is used to perform corresponding intervention actions according to the fatigue level and set fatigue intervention measures if it is determined that the driver is in a fatigue driving state.

[0030] Further, the information analysis unit includes an in-cabin camera arranged above the steering wheel, an out-of-vehicle camera arranged outside the vehicle body, a vehicle speed sensor, a torque sensor arranged on the steering wheel, and a central control computer;

[0031] The in-cabin camera is used to collect the driver's eye movement information, the vehicle speed sensor is used to collect the vehicle speed change information, the vehicle exterior camera is used to collect the vehicle road driving information, the torque sensor is used to collect the driver's steering operation information, and the central control computer is used to determine whether the driver is in a fatigue driving state and the fatigue level according to the driver's state and control information;

[0032] The fatigue intervention unit includes an in-vehicle fragrance, an air conditioner, a vehicle computer unit and a seat, wherein the in-vehicle fragrance is configured with a refreshing fragrance; the air conditioner is used to deliver cool air to the driver's face; the vehicle computer unit is used to play beta wave music, and the seat is used to massage the driver. The vehicle computer unit is also used to send information that the driver is in a state of severe fatigue to a cloud server and establish virtual telephone communication with the cloud server.

[0033] Beneficial effects of the present invention:

[0034] 1. The embodiment of the present invention determines whether the driver is in a fatigue state and the corresponding fatigue level by analyzing the driver's state and control information, and takes corresponding intervention measures for different fatigue levels to achieve an effective fatigue state adjustment effect and avoid interference to the driver.

[0035] 2. The embodiment of the present invention links multiple electronic components of the vehicle to take different intervention measures for different fatigue levels from multiple sensory dimensions such as smell, touch, and hearing, so as to help the driver get rid of fatigue and restore to an alert and active state.

[0036] Other features and advantages of the present invention will be described in the following description, and partly become obvious from the description, or be understood by implementing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 A schematic flow chart of a driving fatigue intervention method based on multimodal perception interaction according to an embodiment of the present invention is shown;

[0039] Figure 2 A schematic diagram of a driver status information collection and analysis process according to an embodiment of the present invention is shown;

[0040] Figure 3 A schematic diagram of a driver fatigue state intervention process according to an embodiment of the present invention is shown;

[0041] Figure 4 A structural schematic diagram of a driving fatigue intervention system based on multimodal perception interaction according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] It should be noted that the terms "first", "second", etc. in the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0044] The embodiment of the present invention provides a driving fatigue intervention method and system based on multimodal perception interaction, which stimulates and reminds drivers in a fatigued state from multi-dimensional senses in combination with multimodal intelligent interaction.

[0045] like Figure 1 As shown, a driving fatigue intervention method based on multimodal perception interaction includes the following steps:

[0046] S1. Configuring fatigue intervention measures according to user personal preference settings, including the following steps: configuring all intervention measures by default, and configuring some fatigue intervention measures according to user personal preference.

[0047] Among them, intervention measures include mild fatigue intervention measures and severe fatigue intervention measures. Mild fatigue intervention measures include turning on refreshing fragrance, air conditioning blowing cold air on the face and playing beta wave music. The main purpose of mild fatigue intervention measures is to improve driver alertness and maintain brain activity. Severe fatigue intervention measures include seat massage, intelligent voice interaction and making a virtual call to the vehicle computer. The main purpose of severe fatigue intervention measures is to help drivers get out of a severe fatigue state close to sleep.

[0048] S2. After the vehicle is started, the driver's state and control information are obtained, and the driver is judged whether he is in a fatigue driving state according to the driver's state and control information, and the fatigue level is determined, including the following steps:

[0049] S21. Collect the driver's eye movement information and driving control behavior information in real time, wherein the eye movement information includes blinking frequency, blinking duration, eye closure state and line of sight direction.

[0050] Driving control behavior information includes vehicle speed change information, driver steering operation information and vehicle road driving information.

[0051] S22. Judging whether the driver is in a fatigue driving state based on the driver's eye movements and driving control behavior information, and determining a fatigue level, wherein the fatigue level includes mild fatigue and severe fatigue.

[0052] like Figure 2 As shown, for example, step S22 includes the following steps:

[0053] S221. If the driver's line of sight is not looking straight ahead or looking left and right intermittently, it is determined that the driver is in a state of mild fatigue; if the driver's blinking frequency decreases, the duration of each blink increases, and the driver is in a closed-eye state within a set time, it is determined that the driver is in a state of severe fatigue.

[0054] S222. If it is determined based on the vehicle speed change information that the vehicle has irregular acceleration and braking behavior, and the vehicle speed suddenly increases or decreases, it is determined that the driver's alertness is reduced and the driver is in a state of mild fatigue; if it is determined based on the driver's steering operation information that the steering force of the vehicle steering wheel is unstable and discontinuous, the steering angle exceeds the set value, or the steering angle remains unchanged within a set time, it is determined that the driver is in a state of severe fatigue; if it is determined based on the vehicle road driving information that the vehicle deviates from the lane direction multiple times within a set time and the lane keeping ability is reduced, it is determined that the driver is in a state of severe fatigue.

[0055] S3. If it is determined that the driver is in a fatigue driving state, corresponding intervention actions are performed according to the fatigue level and the set fatigue intervention measures, such as Figure 3 As shown, the following steps are included:

[0056] S31. If it is determined that the driver is currently in a state of mild fatigue, the vehicle will make a voice announcement. The voice announcement includes that it has detected that you are in a state of fatigue driving. The vehicle will turn on refreshing fragrance, air conditioning and beta wave music for you.

[0057] Among them, the refreshing fragrance can be any fresh fragrance with customizable concentration. The air conditioning temperature is set to 22 degrees Celsius by default and can be adjusted. The wind direction is fixed to the driver's face. Beta wave music is music with a frequency of 14-28 cycles. This band is the same frequency as the brain waves when people are thinking; playing this band of music keeps the brain active and alert.

[0058] S32. If it is determined that the driver is in a state of severe fatigue and the user selects seat massage in the fatigue intervention measures, the vehicle makes a voice announcement and starts the seat massage. The massage mode and intensity are freely set by the user.

[0059] S33. If it is determined that the driver is in a state of severe fatigue and the user selects intelligent voice interaction in the fatigue intervention measures, the vehicle voice will interact freely with the driver based on driving and user data, including the following steps: Intelligent voice interaction generates interactive content based on travel destination, geographic location and user interests.

[0060] For example, intelligent voice interaction generates interactive content based on the travel destination, including: inferring the travel destination based on the user's navigation destination, driving road and driver's clothing information; the vehicle voice assistant asks the driver whether the travel destination is consistent with the calculated travel destination. If consistent, a voice conversation is conducted based on the destination's weather and traffic conditions and itinerary.

[0061] For example, intelligent voice interaction generates interactive content based on geographic location, including: when the vehicle detects that its location is out of the daily living area or is on a highway, the driver is explained about interesting attractions around the vehicle's location based on the vehicle's location information, recommended food and leisure places, and asked whether to navigate there or mark it for collection; if the user responds that he is not interested, the user's preferences will be further asked and recommendations will be made again.

[0062] For example, intelligent voice interaction generates interactive content based on user interests, including: the vehicle generates preference tags based on the content that users listen to and browse on a daily basis in the vehicle, and the voice assistant recommends information and program lists that may be of interest to the driver, who then chooses what to listen to; if there is no content that the user is satisfied with, the driver will further inquire about the reasons for dissatisfaction and optimize the recommendation.

[0063] It should be noted that the voice interactions based on the above three methods are all aimed at obtaining driver feedback and awakening the driver's activity. If the vehicle does not receive timely and clear feedback, it will take further action to make a virtual call to the vehicle's computer.

[0064] S34. If it is determined that the driver is still in a state of severe fatigue, and the user chooses to make a virtual call to the vehicle computer in the fatigue intervention measures, the information that the driver is still in a state of severe fatigue will be sent to the cloud server. The cloud server will make an AI call to the vehicle computer with a randomly generated virtual number to remind the driver to increase his attention and stop the car to rest as soon as possible.

[0065] S4. When performing the intervention action, the driver's state and control information are continuously obtained, and it is determined whether the driver is still in a fatigued state based on the driver's state and control information. If the driver is still in a fatigued state, corresponding fatigue intervention measures are performed according to the current fatigue level until it is determined that the driver is out of the fatigue state. The vehicle will return to the state before the fatigue driving intervention is performed, including the following steps:

[0066] S41. If it is determined that the driver has been in a mild fatigue state for a set time (e.g., 30 seconds), or has entered a severe fatigue state within the set time, the vehicle will take over the severe fatigue intervention measures. At this time, the mild fatigue intervention measures will continue.

[0067] S42: If the driver is in a state of severe fatigue, a virtual call is made to the vehicle computer and the driver's state is continuously monitored after the call is hung up. If it is determined that the driver is still in a state of fatigue driving, the driver will be reminded again in the form of a voice broadcast to stop the car and rest as soon as possible. The mild fatigue intervention measures that have been activated, such as turning on refreshing fragrance, air conditioning and beta wave music, will continue to run.

[0068] The embodiment of the present invention divides the fatigue level into two levels, namely, mild fatigue in which the driver's alertness and concentration are reduced, and severe fatigue in which there are great hidden dangers to driving safety. By analyzing the driver's eye movements and driving control behavior information, it is determined whether the driver is in a fatigue state and the corresponding fatigue level, and different intervention measures are taken for mild fatigue and severe fatigue to achieve an effective fatigue state adjustment effect and avoid interference to the driver.

[0069] Based on the above-mentioned driving fatigue intervention method based on multimodal perception interaction, such as Figure 4 As shown, an embodiment of the present invention also provides a multi-modal perception and interactive driving fatigue intervention system, including an information analysis unit and a fatigue intervention unit.

[0070] Among them, the information analysis unit is used to provide a human-computer interaction interface, and the human-computer interaction interface configures fatigue intervention measures according to the user's personal preference settings.

[0071] The information analysis unit is also used to obtain the driver's status and control information after the vehicle is started, and to determine whether the driver is in a fatigue driving state and the fatigue level based on the driver's status and control information.

[0072] The fatigue intervention unit is used to perform corresponding intervention actions according to the fatigue level and set fatigue intervention measures if it determines that the driver is in a fatigue driving state.

[0073] The information analysis unit is also used to continuously obtain the driver's status and control information when performing intervention actions, and determine whether the driver is still in a fatigue state based on the driver's status and control information.

[0074] The fatigue intervention unit is also used to execute corresponding fatigue intervention measures according to the current fatigue level if the driver is still in a fatigue state, until it is determined that the driver is out of the fatigue state, and the vehicle will return to the state before the fatigue driving intervention is executed.

[0075] For example, the information analysis unit includes an in-cabin camera arranged above the steering wheel, an out-of-vehicle camera arranged outside the vehicle body, a vehicle speed sensor, a torque sensor arranged on the steering wheel, and a central control computer.

[0076] like Figure 2 As shown, the in-cabin camera is used to collect the driver's eye movement information, the vehicle speed sensor is used to collect the vehicle speed change information, the external camera is used to collect the vehicle road driving information, the torque sensor is used to collect the driver's steering operation information, and the central control computer is used to determine whether the driver is in a fatigue driving state and the fatigue level according to the driver's status and control information.

[0077] For example, the fatigue intervention unit includes in-vehicle fragrance, air conditioning, a car computer unit and seats. The in-vehicle fragrance is configured with refreshing fragrance, the air conditioning is used to deliver cold air to the driver's face, the car computer unit is used to play beta wave music, and the seat is used to massage the driver. The car computer unit is also used to send information that the driver is in a state of severe fatigue to the cloud server and establish virtual telephone communication with the cloud server.

[0078] The embodiment of the present invention collects the driver's eye movement characteristics and analyzes the driving control behavior, and links multiple electronic components of the vehicle to take different intervention measures for different degrees of driving fatigue from multiple sensory dimensions such as smell, touch, and hearing, so as to help the driver get rid of fatigue and restore to an alert and active state.

[0079] The identification and judgment of fatigue driving are somewhat complex. The single detection method of the existing technology may cause misjudgment and have a negative impact on the driving experience. The present invention combines artificial intelligence, machine learning, and sensor technology to integrate multimodal interaction into the driving scene, obtain driver status information from multiple aspects through sensors inside and outside the cockpit, and improve the reliability of judgment through comprehensive analysis of information. The fatigue driving intervention method of the embodiment of the present invention avoids interference with driving while ensuring a certain degree of effectiveness.

[0080] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A driving fatigue intervention method based on multimodal perception interaction, characterized in that: The following steps are involved: After the vehicle is started, the driver's status and control information are obtained, and based on the driver's status and control information, it is determined whether the driver is in a fatigue driving state and the fatigue level is determined; If it is determined that the driver is in a fatigue driving state, corresponding intervention actions are performed according to the fatigue level and the set fatigue intervention measures.

2. The driving fatigue intervention method based on multimodal perception interaction according to claim 1 is characterized in that: Interventions included mild fatigue interventions and severe fatigue interventions; Among them, mild fatigue intervention measures include turning on refreshing fragrance, blowing cold air from the air conditioner on the face, and playing beta wave music; severe fatigue intervention measures include seat massage, intelligent voice interaction, and making virtual calls to the vehicle's computer.

3. The driving fatigue intervention method based on multimodal perception interaction according to claim 1 is characterized in that: Acquiring the driver's state and control information, and judging whether the driver is in a fatigue driving state according to the driver's state and control information, and determining the fatigue level, including the following steps: Collect the driver's eye movement information and driving control behavior information in real time; Based on the driver's eye movements and driving control behavior information, it is judged whether the driver is in a fatigue driving state and the fatigue level is determined.

4. The driving fatigue intervention method based on multimodal perception interaction according to claim 3 is characterized in that: Eye movement information includes blinking frequency, blinking duration, eye closure state and gaze direction; driving control behavior information includes vehicle speed change information, driver steering operation information and vehicle road driving information; Fatigue levels range from mild fatigue to severe fatigue.

5. The driving fatigue intervention method based on multimodal perception interaction according to claim 4 is characterized in that: Judging whether the driver is in a fatigue driving state and determining the fatigue level according to the driver's eye movements and driving control behavior information includes the following steps: If the driver's line of sight is not looking straight ahead or looking left and right intermittently, the driver is determined to be in a state of mild fatigue; if the driver's blinking frequency decreases, the duration of each blink increases, and the driver is in a closed eye state within the set time, the driver is determined to be in a state of severe fatigue; If it is determined based on the vehicle speed change information that the vehicle has irregular acceleration and braking behavior, and the vehicle speed suddenly increases or decreases, it is determined that the driver's alertness is reduced and the driver is in a state of mild fatigue; if it is determined based on the driver's steering operation information that the steering force of the vehicle's steering wheel is unstable and discontinuous, the steering angle exceeds the set value, or the steering angle remains unchanged within the set time, it is determined that the driver is in a state of severe fatigue; if it is determined based on the vehicle's road driving information that the vehicle deviates from the lane direction multiple times within the set time and the lane keeping ability is reduced, it is determined that the driver is in a state of severe fatigue.

6. The driving fatigue intervention method based on multimodal perception interaction according to claim 4 is characterized in that: If it is determined that the driver is in a fatigue driving state, the corresponding intervention action is performed according to the fatigue level and the set fatigue intervention measures, including the following steps: If the driver is determined to be in a state of mild fatigue, the vehicle will make a voice announcement, including that you are detected to be in a state of fatigue driving, and the vehicle will turn on refreshing fragrance, air conditioning and beta wave music for you; If the driver is determined to be in a state of severe fatigue and the user selects seat massage in the fatigue intervention measures, the vehicle will make a voice announcement and start the seat massage; If the driver is determined to be in a state of severe fatigue and the user selects intelligent voice interaction in the fatigue intervention measures, the vehicle voice will interact with the driver based on driving and user data; If it is determined that the driver is still in a state of severe fatigue, and the user chooses to make a virtual call to the vehicle's computer in the fatigue intervention measures, the information that the driver is still in a state of severe fatigue will be sent to the cloud server. The cloud server will make an AI call to the vehicle's computer with a randomly generated virtual number to remind the driver to stop and rest.

7. The driving fatigue intervention method based on multimodal perception interaction according to any one of claims 1 to 6, characterized in that: The following steps are also included: When performing intervention actions, the driver's status and control information are continuously obtained, and based on the driver's status and control information, it is determined whether the driver is still in a fatigue state. If the driver is still in a fatigue state, corresponding fatigue intervention measures are performed according to the current fatigue level. After it is determined that the driver is out of the fatigue state, the vehicle will return to the state before the fatigue driving intervention is performed.

8. The driving fatigue intervention method based on multimodal perception interaction according to claim 7 is characterized in that: When performing intervention actions, the driver's status and control information are continuously obtained, and whether the driver is still in a fatigue state is determined based on the driver's status and control information. If the driver is still in a fatigue state, the corresponding fatigue intervention measures are performed according to the current fatigue level, including the following steps: If it is determined that the driver has been in a mild fatigue state for a set time, or has entered a severe fatigue state within the set time, the vehicle will take over the severe fatigue intervention measures. At this time, the mild fatigue intervention measures will continue; If the driver is in a state of severe fatigue, a virtual call is made to the vehicle computer and the driver's condition is continuously monitored after the call is hung up. If it is determined that the driver is still in a state of fatigue driving, the driver will be reminded again to stop and rest in the form of a voice broadcast, which is a mild fatigue intervention measure that has been activated.

9. A driving fatigue intervention system based on multimodal perception interaction, characterized in that: Includes information analysis unit and fatigue intervention unit: The information analysis unit is also used to obtain the driver's state and control information after the vehicle is started, and to determine whether the driver is in a fatigue driving state and the fatigue level according to the driver's state and control information; The fatigue intervention unit is used to perform corresponding intervention actions according to the fatigue level and set fatigue intervention measures if it is determined that the driver is in a fatigue driving state.

10. The driving fatigue intervention system based on multimodal perception interaction according to claim 9 is characterized in that: The information analysis unit includes an in-cabin camera arranged above the steering wheel, an out-of-vehicle camera arranged outside the vehicle body, a vehicle speed sensor, a torque sensor arranged on the steering wheel, and a central control computer; The in-cabin camera is used to collect the driver's eye movement information, the vehicle speed sensor is used to collect the vehicle speed change information, the vehicle exterior camera is used to collect the vehicle road driving information, the torque sensor is used to collect the driver's steering operation information, and the central control computer is used to determine whether the driver is in a fatigue driving state and the fatigue level according to the driver's state and control information; The fatigue intervention unit includes an in-vehicle fragrance, an air conditioner, a vehicle computer unit and a seat, wherein the in-vehicle fragrance is configured with a refreshing fragrance; the air conditioner is used to deliver cool air to the driver's face; the vehicle computer unit is used to play beta wave music, and the seat is used to massage the driver. The vehicle computer unit is also used to send information that the driver is in a state of severe fatigue to a cloud server and establish virtual telephone communication with the cloud server.

Images