Driver monitoring method and system based on vehicle terminal and cloud analysis

By combining data analysis methods of on-board terminals and cloud in the driver monitoring system, the problem of insufficient computing power of on-board terminals is solved, and the accuracy and safety of driver status monitoring is improved, ensuring that it can be effectively identified in different network environments. And deal with abnormal driver status.

CN114348017BActive Publication Date: 2024-11-26YINGCHE XINGCHUANG INTELLIGENT TECH (SHANGHAI) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111555459.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-17
Publication Date
2024-11-26
Estimated Expiration
2041-12-17

AI Technical Summary

Technical Problem

Due to the computing power limitation of the on-board terminal, the existing driver monitoring system is insufficient in monitoring accuracy and cannot effectively identify the driver's abnormal state, resulting in safety risks.

Method used

Driver monitoring methods based on vehicle terminals and clouds are adopted. By collecting video data and deciding whether to send data to the cloud for analysis based on network bandwidth conditions. If the bandwidth is insufficient, the identification results of the vehicle terminal are used. If the bandwidth is sufficient, the identification results of the cloud are used. As a result, preset autonomous driving programs are executed, including audio and video prompts, physical prompts, and takeover prompts.

Benefits of technology

It improves the accuracy and accuracy of driver status monitoring, ensures that abnormal driver status can still be effectively identified and deal with when the network environment is poor, and ensures traffic safety.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114348017B_ABST
    Figure CN114348017B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of intelligent driving technology, and provides a driver monitoring method and system based on an on-board terminal and cloud analysis. The method includes: collecting and analyzing the driver's video data to obtain a first driver status recognition result; judging: if the bandwidth meets the video transmission requirements, the driver's video data is sent to the cloud, and the second driver status recognition result obtained by the cloud based on the driver video data analysis is used as the driver status true value; otherwise, the first driver status recognition result is used as the driver status true value; and a preset automatic driving program is executed according to the driver status true value. The present invention monitors and analyzes the driver's video data through the on-board terminal and the cloud. When the network environment is good, cloud monitoring with stronger computing power and higher accuracy is used first to obtain the driver's real-time status; when the network environment is poor, a driver monitoring controller is used. The two complement each other and can greatly improve the accuracy and precision of driver status monitoring.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to a driver monitoring method and system based on a vehicle-mounted terminal and cloud analysis. Background Art

[0002] The driver monitor system (DMS) is part of the human-computer interaction of autonomous driving. It uses images obtained by the camera and data input by other body sensors to monitor the driver's driving behavior and physiological state through visual tracking, motion recognition and other technologies. When it is determined that the driver is not present or is in an abnormal driving state (fatigue, distraction, etc.), the autonomous driving system will issue an alarm to the driver or implement other safety strategies to ensure the safe operation of the vehicle.

[0003] However, due to the limitation of the computing power of the vehicle terminal, the existing DMS has the disadvantage of insufficient monitoring accuracy. How to provide a more reliable driver monitoring method and system has become a technical problem that needs to be solved urgently in the industry. Summary of the invention

[0004] The present invention provides a driver monitoring method and system based on a vehicle-mounted terminal and cloud analysis, which are used to solve the defect of insufficient monitoring accuracy in the prior art and realize more reliable driver status monitoring.

[0005] The present invention provides a driver monitoring method based on an on-board terminal and cloud analysis, which is applied to the on-board terminal and includes:

[0006] Collecting and analyzing driver video data to obtain a first driver state recognition result;

[0007] Execute judgment based on network communication conditions:

[0008] If the bandwidth meets the video transmission requirement, the driver video data is sent to the cloud, and the second driver state recognition result obtained by parsing the driver video data and returned by the cloud is used as the driver state true value;

[0009] If the bandwidth does not meet the video transmission requirement, the first driver state recognition result is used as the driver state true value;

[0010] Execute a preset automatic driving program according to the true value of the driver state.

[0011] According to a driver monitoring method based on an on-board terminal and cloud analysis provided by the present invention, the step of executing a preset automatic driving program according to the true value of the driver state includes:

[0012] Determining that the true value of the driver status is a first value, executing a preset automatic driving program to control the vehicle to travel, and controlling the vehicle-mounted audio and video system to perform audio and video prompts;

[0013] The first value refers to the true value of the driver state when the driver is in an abnormal driving state;

[0014] The video and audio prompts include any one or a combination of the following:

[0015] Display prompt information on the instrument panel;

[0016] Displaying prompt information on the display screen;

[0017] Play the reminder tone via the infotainment controller IVI.

[0018] According to a driver monitoring method based on an in-vehicle terminal and cloud analysis provided by the present invention, after the step of determining that the true value of the driver status is a first value, executing a preset automatic driving program to control the vehicle to travel, and controlling the in-vehicle audio and video system to execute audio and video prompts, the method further includes:

[0019] If it is determined that the driver status true value maintains the first value within a set time period after the in-vehicle audio and video system executes the audio and video prompt, a preset automatic driving program is executed to control the vehicle to decelerate and move to a safe position to stop.

[0020] According to a driver monitoring method based on an in-vehicle terminal and cloud analysis provided by the present invention, after the step of determining that the true value of the driver status is a first value, executing a preset automatic driving program to control the vehicle to travel, and controlling the in-vehicle audio and video system to execute audio and video prompts, the method further includes:

[0021] Determining that the driver state true value maintains the first value within a set period of time after the vehicle-mounted audio and video system executes the audio and video prompt, then controlling the vehicle to execute the physical prompt through a preset automatic driving program;

[0022] The physical prompts include any one or any combination of the following:

[0023] Control the vibration of the driver's seat;

[0024] Control the seat belt worn by the driver to tighten to the set position;

[0025] Control the ambient light to illuminate the set color.

[0026] According to a driver monitoring method based on an in-vehicle terminal and cloud analysis provided by the present invention, after the step of determining that the true value of the driver status is a first value, executing a preset automatic driving program to control the vehicle to travel, and controlling the in-vehicle audio and video system to execute audio and video prompts, the method further includes:

[0027] Determining that the driver state true value maintains the first value within a set period of time after the in-vehicle audio and video system executes the audio and video prompt, then controlling the in-vehicle audio and video system to execute the takeover prompt through a preset automatic driving program until the driver takes over the vehicle control;

[0028] The takeover prompt includes any one or any combination of the following:

[0029] Display takeover information on the dashboard;

[0030] Displaying takeover information on the display screen;

[0031] Playback takes over voice via the infotainment controller IVI.

[0032] According to a driver monitoring method based on an in-vehicle terminal and cloud analysis provided by the present invention, after the step of determining that the driver state true value maintains a first value within a set period after the in-vehicle audio and video system executes the audio and video prompt, executing a preset automatic driving program to control the vehicle to decelerate and move to a safe position to stop, the method further includes:

[0033] If it is determined that the driver has taken over control of the vehicle, the step of executing the preset automatic driving program to control the vehicle to decelerate and move it to a safe position to stop is terminated.

[0034] According to a driver monitoring method based on an on-board terminal and cloud analysis provided by the present invention, the step of executing a preset automatic driving program according to the true value of the driver state includes:

[0035] If it is determined that the driver state true value is a second value, the preset automatic driving program is executed to control the vehicle driving; the second value refers to the driver state true value when the driver is in a normal driving state.

[0036] According to a driver monitoring method based on an in-vehicle terminal and cloud analysis provided by the present invention, after the step of determining that the true value of the driver status is a first value, executing a preset automatic driving program to control the vehicle to travel, and controlling the in-vehicle audio and video system to execute audio and video prompts, the method further includes:

[0037] If it is determined that within a set time period after the in-vehicle audio and video system executes the audio and video prompt, the driver status true value is a second value, then the step of controlling the in-vehicle audio and video system to execute the audio and video prompt is terminated, and the preset automatic driving program is executed to control the vehicle driving; the second value refers to the driver status true value when the driver is in a normal driving state.

[0038] According to a driver monitoring method based on an in-vehicle terminal and cloud analysis provided by the present invention, if the bandwidth meets the video transmission requirements, the driver video data is sent to the cloud, and the second driver state recognition result returned by the cloud and obtained based on the driver video data analysis is used as the driver state true value. The steps include:

[0039] determining that a second driver state recognition result obtained by parsing the driver video data and returned by the cloud is received, and taking the second driver state recognition result as the driver state true value;

[0040] If it is determined that the second driver state recognition result obtained based on the analysis of the driver video data and returned by the cloud is not received, the first driver state recognition result is used as the true value of the driver state.

[0041] A driver monitoring method based on a vehicle terminal and cloud analysis provided by the present invention also includes:

[0042] It is determined that the vehicle is in a non-operating state, and according to the first driver state recognition result and the second driver state recognition result, the driver video data parsing algorithm deployed on the vehicle terminal is adjusted.

[0043] According to a driver monitoring method based on an in-vehicle terminal and cloud analysis provided by the present invention, the step of determining that the vehicle is in a non-operating state and adjusting the driver video data analysis algorithm deployed on the in-vehicle terminal according to the first driver state recognition result and the second driver state recognition result comprises:

[0044] Uploading the first driver status recognition result to the cloud;

[0045] Determine that the vehicle is in a non-operating state, and receive, via OTA, a driver video data parsing algorithm update file deployed on the vehicle terminal and obtained based on the first driver state recognition result and the second driver state recognition result, which is returned from the cloud;

[0046] It is determined that the vehicle is in a non-operating state, the update file is run, and the driver video data parsing algorithm deployed on the vehicle terminal is updated.

[0047] The present invention also provides a driver monitoring system based on an in-vehicle terminal and cloud analysis, which is applied to the in-vehicle terminal and includes:

[0048] An acquisition module, used for acquiring and analyzing driver video data to obtain a first driver state recognition result;

[0049] The judgment module is used to perform judgments based on network communication conditions:

[0050] If the bandwidth meets the video transmission requirement, the driver video data is sent to the cloud, and the second driver state recognition result obtained by parsing the driver video data and returned by the cloud is used as the driver state true value;

[0051] If the bandwidth does not meet the video transmission requirement, the first driver state recognition result is used as the driver state true value;

[0052] The automatic driving module is used to execute a preset automatic driving program according to the true value of the driver state.

[0053] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program on the vehicle terminal, the steps of the driver monitoring method based on the vehicle terminal and cloud analysis as described in any one of the above are implemented.

[0054] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the driver monitoring method based on a vehicle-mounted terminal and cloud analysis as described in any one of the above are implemented.

[0055] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned driver monitoring methods based on a vehicle-mounted terminal and cloud analysis.

[0056] The driver monitoring method and system based on vehicle-mounted terminal and cloud analysis provided by the present invention monitors and analyzes the driver's video data through the vehicle-mounted terminal and the cloud. In the case of a good network environment, cloud monitoring with stronger computing power and higher accuracy is used first to obtain the driver's real-time status; in the case of a poor network environment, the driver monitoring controller is used. The two complement each other and can greatly improve the accuracy and precision of driver status monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1 It is a flow chart of a driver monitoring method based on a vehicle-mounted terminal and cloud analysis provided by the present invention;

[0059] Figure 2 It is a schematic diagram of the hardware architecture provided by an embodiment of the present invention;

[0060] Figure 3 It is a schematic diagram of a workflow provided by an embodiment of the present invention;

[0061] Figure 4 It is a structural schematic diagram of a driver monitoring system based on an on-board terminal and cloud analysis provided by the present invention;

[0062] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention.

[0063] Reference numerals:

[0064] 1: Driver monitoring camera;

[0065] 2: On-board driver monitoring controller;

[0066] 3: Remote communication module (T-Box);

[0067] 4: Cloud processor;

[0068] 5: Autonomous driving domain controller;

[0069] 6: Power unit;

[0070] 7: Steering device;

[0071] 8: Braking device;

[0072] 9: Vibrating seat;

[0073] 10: Pre-tighten seat belts;

[0074] 11: Ambient light;

[0075] 12: Entertainment information controller (IVI);

[0076] 13: Instrument;

[0077] 14: Central control display screen;

[0078] 401: Collection module;

[0079] 402: judgment module;

[0080] 403: Autonomous driving module;

[0081] 510: processor;

[0082] 520: communication interface;

[0083] 530: memory;

[0084] 540: Communication bus. DETAILED DESCRIPTION

[0085] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0086] Combine the following Figure 1-Figure 3 The present invention describes a driver monitoring method based on a vehicle terminal and cloud analysis.

[0087] like Figure 1 As shown, an embodiment of the present invention provides a driver monitoring method based on an in-vehicle terminal and cloud analysis, which is applied to the in-vehicle terminal and includes:

[0088] Step 102, collecting and analyzing driver video data to obtain a first driver state recognition result;

[0089] Step 104, judging the network communication conditions:

[0090] If the bandwidth meets the video transmission requirement, the driver video data is sent to the cloud, and the second driver state recognition result obtained by parsing the driver video data and returned by the cloud is used as the driver state true value;

[0091] If the bandwidth does not meet the video transmission requirement, the first driver state recognition result is used as the driver state true value;

[0092] Step 106, executing a preset automatic driving program according to the true value of the driver state.

[0093] The execution subject of this embodiment is a program running on the vehicle-mounted terminal, and the program can call the vehicle-mounted hardware.

[0094] like Figure 2 As shown, in a preferred embodiment, the vehicle-mounted hardware includes a driver monitoring camera 1, an on-board driver monitoring controller 2, a remote communication module (T-Box) 3, a cloud processor 4, an autonomous driving domain controller 5, a chassis system part (power unit 6, steering device 7, braking device 8), a body control system part (vibrating seat 9, pre-tensioned seat belt 10, ambient light 11) and an on-board entertainment system part (entertainment information controller 12 / IVI12, instrument 13, central control display screen 14).

[0095] In step 102 , the collection of the driver video data is achieved by the program on the vehicle terminal calling the driver monitoring camera 1 ; the analysis of the driver video data is achieved by the program on the vehicle terminal calling the vehicle driver monitoring controller 2 .

[0096] In step 104 , the execution of the judgment is implemented by the program on the vehicle terminal calling the remote communication module (T-Box) 3 .

[0097] In step 106, the execution of the autonomous driving program is achieved by the program on the vehicle terminal calling the autonomous driving domain controller 5, and the autonomous driving domain controller 5 further calling any one or a combination of the chassis system part (power unit 6, steering device 7, braking device 8), the body control system part (vibrating seat 9, pre-tensioned seat belt 10, ambient light 11), and the vehicle entertainment system part (IVI 12, instrument 13, and central control display screen 14).

[0098] The beneficial effects of this embodiment are:

[0099] The driver's video data is monitored and analyzed through the vehicle terminal and the cloud. When the network environment is good, the cloud monitoring with stronger computing power and higher accuracy is used first to obtain the driver's real-time status; when the network environment is poor, the driver monitoring controller is used. The two complement each other and can greatly improve the accuracy and precision of driver status monitoring.

[0100] According to the above embodiment, in this embodiment:

[0101] The step of executing a preset automatic driving program according to the true value of the driver state comprises:

[0102] Determining that the true value of the driver status is a first value, executing a preset automatic driving program to control the vehicle to travel, and controlling the vehicle-mounted audio and video system to perform audio and video prompts;

[0103] The first value refers to the true value of the driver state when the driver is in an abnormal driving state;

[0104] The video and audio prompts include any one or a combination of the following:

[0105] Display prompt information on the instrument panel;

[0106] Displaying prompt information on the display screen;

[0107] Play the reminder tone via the infotainment controller IVI.

[0108] After the step of determining that the true value of the driver status is the first value, executing a preset automatic driving program to control the vehicle to travel, and controlling the vehicle-mounted audio and video system to perform audio and video prompts, the method further includes:

[0109] If it is determined that the driver status true value maintains the first value within a set time period after the in-vehicle audio and video system executes the audio and video prompt, a preset automatic driving program is executed to control the vehicle to decelerate and move to a safe position to stop.

[0110] After the step of determining that the true value of the driver status is the first value, executing a preset automatic driving program to control the vehicle to travel, and controlling the vehicle-mounted audio and video system to perform audio and video prompts, the method further includes:

[0111] Determining that the driver state true value maintains the first value within a set period of time after the vehicle-mounted audio and video system executes the audio and video prompt, then controlling the vehicle to execute the physical prompt through a preset automatic driving program;

[0112] The physical prompts include any one or any combination of the following:

[0113] Control the vibration of the driver's seat;

[0114] Control the seat belt worn by the driver to tighten to the set position;

[0115] Control the ambient light to illuminate the set color.

[0116] After the step of determining that the true value of the driver status is the first value, executing a preset automatic driving program to control the vehicle to travel, and controlling the vehicle-mounted audio and video system to perform audio and video prompts, the method further includes:

[0117] Determining that the driver state true value maintains the first value within a set period of time after the in-vehicle audio and video system executes the audio and video prompt, then controlling the in-vehicle audio and video system to execute the takeover prompt through a preset automatic driving program until the driver takes over the vehicle control;

[0118] The takeover prompt includes any one or any combination of the following:

[0119] Display takeover information on the dashboard;

[0120] Displaying takeover information on the display screen;

[0121] The voice is played back through the entertainment information controller IVI12.

[0122] After the step of determining that the driver state true value maintains the first value within the set period after the in-vehicle audio and video system executes the audio and video prompt, executing a preset automatic driving program to control the vehicle to decelerate and move to a safe position to stop, the method further includes:

[0123] If it is determined that the driver has taken over control of the vehicle, the step of executing the preset automatic driving program to control the vehicle to decelerate and move it to a safe position to stop is terminated.

[0124] The step of executing a preset automatic driving program according to the true value of the driver state comprises:

[0125] If it is determined that the driver state true value is a second value, the preset automatic driving program is executed to control the vehicle driving; the second value refers to the driver state true value when the driver is in a normal driving state.

[0126] After the step of determining that the true value of the driver status is the first value, executing a preset automatic driving program to control the vehicle to travel, and controlling the vehicle-mounted audio and video system to perform audio and video prompts, the method further includes:

[0127] If it is determined that within a set time period after the in-vehicle audio and video system executes the audio and video prompt, the driver status true value is a second value, then the step of controlling the in-vehicle audio and video system to execute the audio and video prompt is terminated, and the preset automatic driving program is executed to control the vehicle driving; the second value refers to the driver status true value when the driver is in a normal driving state.

[0128] This embodiment performs different subsequent actions on the first value and the second value of the driver state true value respectively:

[0129] For the first truth value, that is, the driver is in an abnormal driving state, the subsequent operations of the two stages are performed in sequence to ensure that the driver is out of the abnormal driving state and has the ability to take over the vehicle at any time. Specifically:

[0130] Phase 1: The autonomous driving domain controller 5 will continue to control the vehicle's autonomous driving, and at the same time send reminder instructions to the in-vehicle entertainment system, displaying text and images reminding safe driving on the instrument panel 13 and the central control display screen 14 respectively, and the IVI 12 will emit a safe driving reminder sound; if the driver is received as normal, the autonomous driving will continue without any reminder.

[0131] After the first stage lasts for a set duration (this time can be calibrated according to human factors engineering), if the driver's state returns to normal, the system maintains automatic driving and the reminder information disappears; if the driver's state does not return to normal, the driving state will enter the second stage: the automatic driving controller needs to take emergency measures to control the chassis system's partial power device 6, drive to the safe lane to slow down and stop. At the same time, the body control system controls the vibration seat 9 to vibrate, the pre-tensioned seat belt 10 to warn, the atmosphere light 11 to light up the emergency color, and controls the instrument 13, IVI12, and central control display screen 14 of the in-vehicle entertainment system to display the takeover information. Wait for the driver to take over the vehicle.

[0132] It is worth noting that step 102, step 104 and step 106 are repeatedly executed, that is, the true value of the driver state is updated based on a set interval, the set duration of the first stage corresponds to the set interval, and the set interval is not greater than the set duration.

[0133] This embodiment integrates two sets of driver status recognition methods, one on-board and one on the cloud, and optimizes the vehicle-side algorithm model through data comparison to improve the overall recognition accuracy. When the driver is identified as being in an abnormal driving state, a human-computer interaction reminder is given in the first stage, and emergency measures are taken in the second stage to park the vehicle in a safe lane, and a richer human-computer interaction method is used to remind the driver to take over the vehicle, which effectively ensures the safety of the driver and traffic participants.

[0134] According to any of the above embodiments, in this embodiment:

[0135] If the bandwidth meets the video transmission requirement, the step of sending the driver video data to the cloud, and using the second driver state recognition result obtained by parsing the driver video data and returned by the cloud as the driver state true value includes:

[0136] determining that a second driver state recognition result obtained by parsing the driver video data and returned by the cloud is received, and taking the second driver state recognition result as the driver state true value;

[0137] If it is determined that the second driver state recognition result obtained based on the analysis of the driver video data and returned by the cloud is not received, the first driver state recognition result is used as the true value of the driver state.

[0138] The method of this embodiment also includes:

[0139] It is determined that the vehicle is in a non-operating state, and according to the first driver state recognition result and the second driver state recognition result, the driver video data parsing algorithm deployed on the vehicle terminal is adjusted.

[0140] The step of determining that the vehicle is in a non-operating state and adjusting the driver video data parsing algorithm deployed on the vehicle terminal according to the first driver state recognition result and the second driver state recognition result comprises:

[0141] Uploading the first driver status recognition result to the cloud;

[0142] Determine that the vehicle is in a non-operating state, and receive, via OTA, a driver video data parsing algorithm update file deployed on the vehicle terminal and obtained based on the first driver state recognition result and the second driver state recognition result, which is returned from the cloud;

[0143] It is determined that the vehicle is in a non-operating state, the update file is run, and the driver video data parsing algorithm deployed on the vehicle terminal is updated.

[0144] Based on the above embodiments, this embodiment provides a verification process for returning the second driver status recognition result in the cloud and an update method for the driver video data analysis algorithm deployed on the vehicle terminal, which can further improve the reliability and accuracy of driver status monitoring.

[0145] According to any of the above embodiments, an embodiment will be provided from the perspective of vehicle-mounted hardware.

[0146] The technical problem solved by this embodiment is:

[0147] At present, the driver monitoring system only recognizes the driver's status through the vehicle-side controller, and the recognition accuracy is low. The present invention can simultaneously adopt the driver monitoring controller and the driver video real-time upload cloud monitoring. When the network environment is good, the cloud monitoring with stronger computing power and higher accuracy is used to obtain the driver's real-time status; when the network environment is poor, the driver monitoring controller is used. The two complement each other and can greatly improve the accuracy and precision of driver status monitoring.

[0148] The driver monitoring system software algorithm needs to be updated and iterated through a large amount of data learning, and its state recognition algorithm and model need to be optimized to improve the recognition accuracy. The present invention can quickly update and iterate the driver monitoring controller software through Over-the-Air Technology (OTA) to improve the recognition accuracy of the vehicle-side monitoring system.

[0149] In addition, the existing driver monitoring system only provides auxiliary prompts when it identifies the driver's status, which are easy for the driver to ignore and will not participate in vehicle control. The driver monitoring system designed by the present invention can not only provide a richer human-computer interaction reminder method, but also participate in vehicle control to ensure the safety of the driver and other traffic participants. Autonomous driving vehicles need to fully remind the driver through human interaction (such as hearing, vision, and touch) when the driver is in a state of fatigue, distraction, etc.; if the driver does not return to normal after the reminder, the autonomous driving vehicle needs to take emergency measures to control the vehicle's steering, braking, hazard warning lights, etc., slow down, brake in a safe lane, and turn on the hazard warning lights.

[0150] The advantages of this embodiment over the prior art are:

[0151] 1. Use both in-vehicle and cloud solutions to monitor driver status

[0152] 2. Update the vehicle driver monitoring controller software through OTA

[0153] 3. Design an autonomous driving vehicle driver monitoring system based on the overall vehicle architecture to provide driver reminders and implement emergency control vehicle safety parking strategies after sufficient reminders.

[0154] Therefore, it can further bring:

[0155] 1. Use both in-vehicle and cloud solutions to monitor the driver's status and improve overall recognition rate and accuracy;

[0156] 2. OTA upgrade of the on-board driver monitoring system to more efficiently complete algorithm model optimization and software iteration;

[0157] 3. Reminders and emergency takeovers when the driver is in abnormal condition fully ensure the safety of the driver and traffic participants.

[0158] This embodiment involves the design of the overall functional architecture scheme of the driver monitoring system of an autonomous driving vehicle, which is one of the important contents of the development of the autonomous driving system. This embodiment innovatively designs two driver status monitoring schemes, namely, on-board and cloud-based. When the network condition is good, a high-computing power and high-accuracy cloud-based monitoring method is adopted; when the network condition is poor, an on-board monitoring system is adopted to greatly improve the comprehensive recognition rate. Through OTA technology, the algorithm model of the on-board driver monitoring controller software is optimized. In addition, during the process of vehicle autonomous driving, when the driver's state is abnormal, the driver is fully reminded through human-computer interaction (such as hearing, vision, and touch); if the driver does not return to normal after the reminder, the autonomous driving vehicle needs to take emergency measures to control the vehicle's steering, braking, hazard warning lights, etc., to slow down, brake, and turn on the hazard warning lights in the safe lane. Based on the above content, an autonomous driving driver monitoring system based on the autonomous driving functional architecture level is provided, which mainly includes a driver monitoring camera 1, an on-board driver monitoring controller 2, a remote communication module (T-Box) 3, a cloud processor 4, an autonomous driving domain controller (ADU) 5, a chassis system, a body control system, and an on-board entertainment system.

[0159] The present embodiment provides a driver monitoring system suitable for an autonomous driving vehicle, which can identify the driver's status with higher accuracy, quickly iterate the on-board driver monitoring controller software, and remind the driver when an abnormal status occurs during the autonomous driving process, and take over the vehicle in an emergency.

[0160] In order to achieve the above functions, a driver monitoring system suitable for an autonomous driving vehicle includes: a driver monitoring camera 1, an on-board driver monitoring controller 2, a remote communication module (T-Box) 3, a cloud processor 4, an autonomous driving domain controller (ADU) 5, a chassis system (power unit 6, a steering device 7, a braking device 8), a body control system (including: a vibrating seat 9, a pre-tensioned seat belt 10, an ambient light 11), and an on-board entertainment system (including: IVI 12, an instrument 13, a display screen 14).

[0161] The driver monitoring camera 1 collects the driver's original video data in real time and transmits it to the on-board driver monitoring controller 2. The on-board driver monitoring controller 2 processes the video data and outputs the driver's status. It also sends the video data and the driver's status recognized by the on-board controller (i.e., the on-board driver monitoring controller 2) to T-Box 3 and uploads it to the cloud processor 4. The driver's status recognized by the cloud is returned to the driver monitoring controller 2 through T-Box 3.

[0162] When the network condition is good, both the vehicle and the cloud can obtain two sets of driver status data. At this time, the vehicle driver monitoring controller 2 inputs the driver status recognized by the cloud with higher accuracy to the automatic driving domain controller 5 as the true value for subsequent processing; if the network environment is not good, the vehicle status recognition data is directly used as input.

[0163] The cloud processor 4 obtains two sets of status data, compares the difference between the two sets of data, and uses the original driver video data to determine whether the status recognition is accurate, and adjusts and optimizes the algorithm model of the cloud processor 4 and the on-board driver monitoring controller 2. The optimized and upgraded on-board driver controller software is upgraded through the T-Box3 networking function, using OTA technology to upgrade the on-board driver monitoring controller software, thereby improving the accuracy of the on-board driver monitoring controller 2 status recognition.

[0164] In the system, the autonomous driving domain controller 5 serves as an autonomous driving control and decision-making module. The system has the following functions:

[0165] Function 1: When the vehicle driver monitoring controller 2 inputs to the automatic driving domain controller 5 that the vehicle is in a normal driving state (no distraction, fatigue, smoking, making phone calls, leaving the post, etc.), the automatic driving domain controller 5 controls the vehicle to drive automatically;

[0166] Function 2: When the vehicle driver monitoring controller 2 inputs to the autonomous driving domain controller 5 an abnormal driving state:

[0167] Phase 1: The autonomous driving domain controller 5 will continue to control the vehicle's autonomous driving, and at the same time send a reminder command to the in-vehicle entertainment system, displaying text and images reminding safe driving on the instrument panel 13 and the central control display screen 14, respectively, and the IVI 12 will emit a safe driving reminder sound;

[0168] Phase 2: After fully reminding the driver, if the driver's state does not return to normal driving, the automatic driving controller 5 needs to take emergency measures to control the chassis system, drive to a safe lane to slow down and stop. At the same time, the body system is controlled to vibrate the seat 9, pre-tighten the seat belt 10 to warn, and the atmosphere light 11 lights up the takeover color. The instrument, IVI 12, and central control display screen 14 of the in-vehicle entertainment system are controlled to display the takeover information. Wait for the driver to take over the vehicle.

[0169] One of the innovations of this embodiment is that: when the network condition is good, two sets of driver status data can be obtained, one on the vehicle and the other on the cloud. The driver status obtained by using the high-computing cloud processor 4 is selected as the basis for subsequent processing, which has a higher accuracy rate; even if the network environment is not good, the vehicle-mounted recognition result can be used as the input for subsequent processing.

[0170] The second innovation of this embodiment is that a large amount of actual driver video data can be used as machine learning samples, and the on-board driver monitoring controller software can be updated through OTA technology, which can be quickly iterated and updated to improve its recognition accuracy.

[0171] The third innovation of this embodiment is that during the driving of the autonomous vehicle, when the driver is in an abnormal driving state, the first stage system will remind the driver to drive safely through rich images, texts, sounds, and instrument indications; after the first stage of reminders, if the driver does not return to normal driving state, the system enters the second stage. Control the vehicle to slow down and stop in the safe lane, and at the same time control the body system to vibrate the seat 9, pre-tighten the seat belt 10, and light up the atmosphere light 11 to take over the color, and control the instrument of the in-vehicle entertainment system, IVI12, and the central control display screen 14 to display the takeover information. Wait for the driver to take over the vehicle to ensure the safety of the driver and traffic participants.

[0172] like Figure 2 As shown, the present embodiment provides a driver monitoring system suitable for autonomous driving, including a driver monitoring camera 1, an on-board driver monitoring controller 2, a remote communication module (T-Box) 3, a cloud processor 4, an autonomous driving domain controller 5, a chassis system power unit 6, a steering device 7, a braking device 8, a body control system part vibrating seat 9, a pre-tensioned seat belt 10, an ambient light 11, an on-board entertainment system part entertainment information controller (IVI) 12, an instrument 13, and a central control display screen 14.

[0173] The driver monitoring camera 1 is used to collect the original video image data of the driver in real time, and transmit the original driver data to the on-board driver monitoring controller 2 via an LVDS line or other dedicated video transmission line.

[0174] The on-board driver monitoring controller 2 identifies the real-time driver status through the recognition algorithm, including distraction, fatigue, making phone calls, smoking, leaving the post, etc., and converts the identified status into CAN bus or other commonly used bus signal forms on the vehicle.

[0175] The remote communication module (T-Box) 3 uses 4G / 5G communication to realize communication between the vehicle and the cloud, and is connected to the on-board driver monitoring controller through the on-board Ethernet. The remote communication module (T-Box) 3 needs to determine whether the network environment is normal and whether it is suitable for video data transmission. If the network environment is normal, the on-board driver monitoring controller 2 transmits the driver video data collected by the camera and the identified driver status data to the remote communication module (T-Box) 3, which uploads it to the cloud processor 4.

[0176] The cloud processor 4 receives the video data uploaded by the remote communication module 3, identifies the driver's status and sends it back to the remote communication module 3.

[0177] The autonomous driving domain controller (ADU) 5 is connected to the on-board driver monitoring controller 2 via the CAN bus. The driver status signal sent by the driver monitoring controller is received to the remote communication module 3. In addition, the cloud processor 4 compares the two sets of identification data on the vehicle and the cloud as well as a large amount of external injection data to optimize the algorithm model of the on-board driver monitoring controller 2.

[0178] The autonomous driving domain controller 5 receives the bus data from the onboard driver monitoring controller 2, controls the vehicle human-machine interaction reminder or takes over the vehicle to park in a safe lane until the driver takes over the vehicle

[0179] The power unit 6, steering device 7, brake device 8, vibration seat 9, pre-tensioned seat belt 10, ambient light 11, entertainment information controller 12, instrument 13, and central control display screen 14 are all vehicle-side actuators.

[0180] like Figure 3 As shown, in this embodiment, the specific steps are as follows:

[0181] A. The vehicle enters autonomous driving.

[0182] B. The camera collects driver status video in real time.

[0183] C. When the network condition is good, the on-board driver monitoring controller 2 receives the video data, calculates the result, and uploads the original video data and the vehicle-side recognition result to the cloud through the remote communication module 3.

[0184] D. The cloud processor 4 identifies the driver's status in real time by uploading video data and transmits it back to the on-board driver monitoring controller 2 through the remote communication module 3. At the same time, the vehicle-side and cloud-side data are compared, and the on-board algorithm model is optimized through a large amount of data learning. When the vehicle is idle, the on-board driver monitoring controller 2 software is upgraded through OTA.

[0185] E. The on-board driver monitoring controller 2 inputs the driver status transmitted back from the cloud into the autonomous driving domain controller 5.

[0186] F. If the autonomous driving domain controller 5 receives information that the driver is in an abnormal driving state (distracted, tired, smoking, making phone calls, leaving the post, etc.), the first stage: the autonomous driving domain controller 5 will continue to control the vehicle's autonomous driving, and at the same time send a reminder instruction to the in-vehicle entertainment system, displaying text and images reminding safe driving on the instrument and display screen respectively, and the IVI controller will emit a safe driving reminder sound; if the driver is received as normal, the autonomous driving will continue without any reminder.

[0187] G. After the first stage lasts for a period of time (this time can be calibrated according to human factors engineering), if the driver's state returns to normal, the system maintains automatic driving and the reminder information disappears; if the driver's state does not return to normal, the driving state will enter the second stage: the automatic driving controller needs to take emergency measures, control the chassis system, drive to the safe lane to slow down and stop. At the same time, control the body system to vibrate the seat, pre-tighten the seat belt warning, and light up the emergency color of the atmosphere light, and control the instrument of the in-vehicle entertainment system, IVI controller, and central control display to display the takeover information. Wait for the driver to take over the vehicle.

[0188] This embodiment integrates two sets of driver status recognition methods, one on-board and one on the cloud, and optimizes the vehicle-side algorithm model through data comparison to improve the overall recognition accuracy. When the driver is identified as being in an abnormal driving state, a human-computer interaction reminder is given in the first stage, and emergency measures are taken in the second stage to park the vehicle in a safe lane, and a richer human-computer interaction method is used to remind the driver to take over the vehicle, which effectively ensures the safety of the driver and traffic participants.

[0189] The present embodiment of a driver monitoring system design suitable for autonomous driving is particularly useful in certain aspects in conjunction with a specific type of vehicle, but the vehicle can be any type of vehicle, including but not limited to trucks, cars, buses, buses, ships, airplanes, field vehicles, agricultural equipment, and trams.

[0190] This embodiment proposes a driver monitoring system suitable for autonomous driving. There are two parallel driver status monitoring methods, on-board and in the cloud. When the network condition is good, a high-computing-power, high-accuracy cloud monitoring method is adopted; when the network condition is poor, an on-board monitoring system is adopted. The overall recognition rate is greatly improved. Through OTA technology, the state recognition algorithm model of the on-board driver monitoring controller software is quickly optimized. In addition, during the vehicle's autonomous driving process, when the driver's state is abnormal, the driver is fully reminded through human-computer interaction (such as hearing, vision, and touch); if the driver does not return to normal after the reminder, the autonomous driving vehicle needs to take emergency takeover, control the vehicle's steering, braking, hazard warning lights, etc., slow down, brake and turn on the hazard warning lights in the safe lane. Based on the above design content, an autonomous driving driver monitoring system based on the autonomous driving functional architecture level is provided.

[0191] The driver monitoring device based on the vehicle-mounted terminal and cloud-based analysis provided by the present invention is described below. The driver monitoring device based on the vehicle-mounted terminal and cloud-based analysis described below and the driver monitoring method based on the vehicle-mounted terminal and cloud-based analysis described above can be referenced to each other.

[0192] An embodiment of the present invention provides a driver monitoring system based on an in-vehicle terminal and cloud analysis, which is applied to the in-vehicle terminal and includes:

[0193] The acquisition module 401 is used to acquire and analyze the driver video data to obtain a first driver state recognition result;

[0194] The judgment module 402 is used to perform judgment based on the network communication conditions:

[0195] If the bandwidth meets the video transmission requirement, the driver video data is sent to the cloud, and the second driver state recognition result obtained by parsing the driver video data and returned by the cloud is used as the driver state true value;

[0196] If the bandwidth does not meet the video transmission requirement, the first driver state recognition result is used as the driver state true value;

[0197] The automatic driving module 403 is used to execute a preset automatic driving program according to the true value of the driver state.

[0198] The automatic driving module 403 includes:

[0199] an audio and video prompt submodule, for determining that the true value of the driver state is a first value, executing a preset automatic driving program to control the vehicle to travel, and controlling the vehicle-mounted audio and video system to execute the audio and video prompt;

[0200] The first value refers to the true value of the driver state when the driver is in an abnormal driving state;

[0201] The video and audio prompts include any one or a combination of the following:

[0202] Display prompt information on the instrument panel;

[0203] Displaying prompt information on the display screen;

[0204] Play the reminder tone via the infotainment controller IVI.

[0205] The deceleration and parking submodule is used to determine that within a set period of time after the vehicle-mounted audio and video system executes the audio and video prompt, the driver state true value maintains the first value, and then executes a preset automatic driving program to control the vehicle to decelerate and move to a safe position to stop.

[0206] A physical prompt submodule, for determining that the driver state true value maintains a first value within a set period of time after the vehicle-mounted audio and video system executes the audio and video prompt, and then controlling the vehicle to execute the physical prompt through a preset automatic driving program;

[0207] The physical prompts include any one or any combination of the following:

[0208] Control the vibration of the driver's seat;

[0209] Control the seat belt worn by the driver to tighten to the set position;

[0210] Control the ambient light to illuminate the set color.

[0211] a takeover prompting submodule, for determining that the driver state true value maintains a first value within a set period of time after the vehicle-mounted audio and video system executes the audio and video prompting, and then controlling the vehicle-mounted audio and video system to execute the takeover prompting through a preset automatic driving program until the driver takes over the vehicle control;

[0212] The takeover prompt includes any one or any combination of the following:

[0213] Display takeover information on the dashboard;

[0214] Displaying takeover information on the display screen;

[0215] Playback takes over voice via the infotainment controller IVI.

[0216] The takeover confirmation submodule is used to determine that the driver has taken over the control of the vehicle, and then terminate the step of executing the preset automatic driving program to control the vehicle to slow down and move it to a safe position to stop.

[0217] The first driving submodule is used to determine that the driver state true value is a second value, and then execute a preset automatic driving program to control the vehicle driving; the second value refers to the driver state true value when the driver is in a normal driving state.

[0218] The second driving sub-module is used to determine that within a set time period after the on-board audio and video system executes the audio and video prompt, if the driver's state true value is a second value, then terminate the step of controlling the on-board audio and video system to execute the audio and video prompt, and execute the preset automatic driving program to control the vehicle driving; the second value refers to the driver's state true value when the driver is in a normal driving state.

[0219] The determination module 402 includes:

[0220] a receiving confirmation submodule, configured to determine that a second driver status recognition result returned by the cloud and obtained based on the analysis of the driver video data is received, and then use the second driver status recognition result as a true value of the driver status;

[0221] The non-receiving confirmation submodule is used to determine that the second driver status recognition result obtained by parsing the driver video data and returned by the cloud has not been received, and then use the first driver status recognition result as the true value of the driver status.

[0222] Furthermore, the driver monitoring system based on the vehicle terminal and cloud analysis also includes:

[0223] An algorithm updating module is used to determine that the vehicle is in a non-operating state, and adjust the driver video data parsing algorithm deployed on the vehicle terminal according to the first driver state recognition result and the second driver state recognition result.

[0224] The algorithm updating module comprises:

[0225] An uploading submodule, used for uploading the first driver status recognition result to the cloud;

[0226] a transmission submodule, configured to determine that the vehicle is in a non-operating state, and receive, via OTA, a driver video data parsing algorithm update file deployed on the vehicle terminal and obtained based on the first driver state recognition result and the second driver state recognition result, which is returned from the cloud;

[0227] The update submodule is used to determine that the vehicle is in a non-operating state, run the update file, and update the driver video data parsing algorithm deployed on the vehicle terminal.

[0228] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute a driver monitoring method based on the vehicle terminal and the cloud parsing, the method comprising: collecting and parsing the driver video data to obtain a first driver state recognition result; performing a judgment on the network communication conditions: if the bandwidth meets the video transmission requirements, the driver video data is sent to the cloud, and the second driver state recognition result returned by the cloud and obtained based on the analysis of the driver video data is used as the driver state true value; if the bandwidth does not meet the video transmission requirements, the first driver state recognition result is used as the driver state true value; and a preset automatic driving program is executed according to the driver state true value.

[0229] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0230] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the driver monitoring method based on the vehicle terminal and cloud analysis provided by the above methods, and the method includes: collecting and parsing the driver video data to obtain a first driver state recognition result; performing a judgment based on the network communication conditions: if the bandwidth meets the video transmission requirements, the driver video data is sent to the cloud, and the second driver state recognition result returned by the cloud and obtained based on the analysis of the driver video data is used as the driver state true value; if the bandwidth does not meet the video transmission requirements, the first driver state recognition result is used as the driver state true value; and a preset automatic driving program is executed according to the driver state true value.

[0231] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the driver monitoring method based on vehicle-mounted terminal and cloud analysis provided by the above-mentioned methods, the method comprising: collecting and parsing driver video data to obtain a first driver state recognition result; performing a judgment based on network communication conditions: if the bandwidth meets the video transmission requirements, the driver video data is sent to the cloud, and the second driver state recognition result returned by the cloud and obtained based on the analysis of the driver video data is used as the driver state true value; if the bandwidth does not meet the video transmission requirements, the first driver state recognition result is used as the driver state true value; and a preset automatic driving program is executed according to the driver state true value.

[0232] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0233] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0234] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A driver monitoring method based on an on-board terminal and cloud analysis for an autonomous driving vehicle, characterized in that: The method is implemented by calling the vehicle-mounted hardware, which includes a driver monitoring camera, a vehicle-mounted driver monitoring controller, a remote communication module T-Box, a cloud processor, an automatic driving domain controller, a chassis system, a body control system, and a vehicle-mounted entertainment system. The method is applied to a vehicle-mounted terminal, including: Collecting driver video data, and analyzing the driver video data through an on-vehicle driver monitoring controller to obtain a first driver state recognition result; Execute judgment based on network communication conditions: If the bandwidth meets the video transmission requirement, the driver video data and the first driver state recognition result are sent to the cloud, and the second driver state recognition result obtained by parsing the driver video data and returned by the cloud is used as the true value of the driver state; If the bandwidth does not meet the video transmission requirement, the first driver state recognition result is used as the driver state true value; Executing a preset automatic driving program according to the true value of the driver state; The recognition algorithm model of the on-board driver monitoring controller is obtained by comparing the vehicle-side and cloud-side data based on the cloud server, and is optimized by using a large amount of actual driver video data as machine learning samples. The optimized recognition algorithm model is updated and upgraded in the on-board driver monitoring controller through the networking function of the remote communication module T-Box and the use of the over-the-air download technology OTA; The step of executing a preset automatic driving program according to the true value of the driver state includes: Determining that the driver state true value is a first value, executing a preset automatic driving program to control the vehicle to travel, and controlling the vehicle audio and video system to perform audio and video prompts; the first value refers to the driver state true value when the driver is in an abnormal driving state; After the step of determining that the true value of the driver status is the first value, executing a preset automatic driving program to control the vehicle to travel, and controlling the vehicle-mounted audio and video system to perform audio and video prompts, the method further includes: Determining that the driver state true value maintains the first value within a set period of time after the in-vehicle audio and video system executes the audio and video prompt, executing a preset automatic driving program to control the vehicle to decelerate and move to a safe position to stop; The step of sending the driver video data and the first driver state recognition result to the cloud if the bandwidth meets the video transmission requirement, and using the second driver state recognition result obtained by parsing the driver video data and returned by the cloud as the driver state true value includes: determining that a second driver state recognition result obtained by parsing the driver video data and returned by the cloud is received, and taking the second driver state recognition result as the driver state true value; If it is determined that the second driver state recognition result obtained based on the analysis of the driver video data and returned by the cloud is not received, the first driver state recognition result is used as the true value of the driver state.

2. The driver monitoring method based on vehicle-mounted terminal and cloud analysis according to claim 1 is characterized in that: The video and audio prompts include any one or a combination of the following: Display prompt information on the instrument panel; Displaying prompt information on the display screen; Play the reminder tone via the infotainment controller IVI.

3. The driver monitoring method based on vehicle-mounted terminal and cloud analysis according to claim 1 is characterized in that: After the step of determining that the true value of the driver status is the first value, executing a preset automatic driving program to control the vehicle to travel, and controlling the vehicle-mounted audio and video system to perform audio and video prompts, the method further includes: Determining that the driver state true value maintains the first value within a set period of time after the vehicle-mounted audio and video system executes the audio and video prompt, then controlling the vehicle to execute the physical prompt through a preset automatic driving program; The physical prompts include any one or any combination of the following: Control the vibration of the driver's seat; Control the seat belt worn by the driver to tighten to the set position; Control the ambient light to illuminate the set color.

4. The driver monitoring method based on vehicle-mounted terminal and cloud analysis according to claim 1 is characterized in that: After the step of determining that the true value of the driver status is the first value, executing a preset automatic driving program to control the vehicle to travel, and controlling the vehicle-mounted audio and video system to perform audio and video prompts, the method further includes: Determining that the driver state true value maintains the first value within a set period of time after the in-vehicle audio and video system executes the audio and video prompt, then controlling the in-vehicle audio and video system to execute the takeover prompt through a preset automatic driving program until the driver takes over the vehicle control; The takeover prompt includes any one or any combination of the following: Display takeover information on the dashboard; Displaying takeover information on the display screen; Playback takes over voice via the infotainment controller IVI.

5. The driver monitoring method based on vehicle-mounted terminal and cloud analysis according to claim 1 is characterized in that: After the step of determining that the driver state true value maintains the first value within the set period after the in-vehicle audio and video system executes the audio and video prompt, executing a preset automatic driving program to control the vehicle to decelerate and move to a safe position to stop, the method further includes: If it is determined that the driver has taken over control of the vehicle, the step of executing the preset automatic driving program to control the vehicle to decelerate and move it to a safe position to stop is terminated.

6. The driver monitoring method based on vehicle-mounted terminal and cloud analysis according to claim 1 is characterized in that: The step of executing a preset automatic driving program according to the true value of the driver state comprises: If it is determined that the driver state true value is a second value, the preset automatic driving program is executed to control the vehicle driving; the second value refers to the driver state true value when the driver is in a normal driving state.

7. The driver monitoring method based on vehicle-mounted terminal and cloud analysis according to claim 1 is characterized in that: After the step of determining that the true value of the driver status is the first value, executing a preset automatic driving program to control the vehicle to travel, and controlling the vehicle-mounted audio and video system to perform audio and video prompts, the method further includes: If it is determined that within a set time period after the in-vehicle audio and video system executes the audio and video prompt, the driver status true value is a second value, then the step of controlling the in-vehicle audio and video system to execute the audio and video prompt is terminated, and the preset automatic driving program is executed to control the vehicle driving; the second value refers to the driver status true value when the driver is in a normal driving state.

8. The driver monitoring method based on vehicle-mounted terminal and cloud analysis according to claim 1 is characterized in that: Also includes: It is determined that the vehicle is in a non-operating state, and according to the first driver state recognition result and the second driver state recognition result, the driver video data parsing algorithm deployed on the vehicle terminal is adjusted.

9. The driver monitoring method based on vehicle-mounted terminal and cloud analysis according to claim 8, characterized in that: The step of determining that the vehicle is in a non-operating state and adjusting the driver video data parsing algorithm deployed on the vehicle terminal according to the first driver state recognition result and the second driver state recognition result comprises: Uploading the first driver status recognition result to the cloud; Determine that the vehicle is in a non-operating state, and receive, via OTA, a driver video data parsing algorithm update file deployed on the vehicle terminal and obtained based on the first driver state recognition result and the second driver state recognition result, which is returned from the cloud; It is determined that the vehicle is in a non-operating state, the update file is run, and the driver video data parsing algorithm deployed on the vehicle terminal is updated.

10. A driver monitoring system based on an on-board terminal and cloud analysis for an autonomous driving vehicle, characterized in that: The device is implemented by calling the vehicle-mounted hardware, which includes a driver monitoring camera, a vehicle-mounted driver monitoring controller, a remote communication module T-Box, a cloud processor, an automatic driving domain controller, a chassis system, a body control system, and a vehicle-mounted entertainment system. The device is applied to a vehicle-mounted terminal, including: An acquisition module, used for acquiring driver video data, parsing the driver video data through an on-vehicle driver monitoring controller, and obtaining a first driver state recognition result; The judgment module is used to perform judgments based on network communication conditions: If the bandwidth meets the video transmission requirement, the driver video data and the first driver state recognition result are sent to the cloud, and the second driver state recognition result obtained by parsing the driver video data and returned by the cloud is used as the true value of the driver state; If the bandwidth does not meet the video transmission requirement, the first driver state recognition result is used as the driver state true value; An automatic driving module, configured to execute a preset automatic driving program according to the true value of the driver state; The recognition algorithm model of the on-board driver monitoring controller is obtained by comparing the vehicle-side and cloud data based on the cloud server, and is optimized by using a large amount of actual driver video data as machine learning samples. The optimized recognition algorithm model is updated and upgraded in the on-board driver monitoring controller through the over-the-air download technology OTA; The step of executing a preset automatic driving program according to the true value of the driver state includes: Determining that the driver state true value is a first value, executing a preset automatic driving program to control the vehicle to travel, and controlling the vehicle audio and video system to perform audio and video prompts; the first value refers to the driver state true value when the driver is in an abnormal driving state; After the step of determining that the true value of the driver status is the first value, executing a preset automatic driving program to control the vehicle to travel, and controlling the vehicle-mounted audio and video system to perform audio and video prompts, the method further includes: Determining that the driver state true value maintains the first value within a set period of time after the in-vehicle audio and video system executes the audio and video prompt, executing a preset automatic driving program to control the vehicle to decelerate and move to a safe position to stop; The step of sending the driver video data and the first driver state recognition result to the cloud if the bandwidth meets the video transmission requirement, and using the second driver state recognition result obtained by parsing the driver video data and returned by the cloud as the driver state true value includes: determining that a second driver state recognition result obtained by parsing the driver video data and returned by the cloud is received, and taking the second driver state recognition result as the driver state true value; If it is determined that the second driver state recognition result obtained based on the analysis of the driver video data and returned by the cloud is not received, the first driver state recognition result is used as the true value of the driver state.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program on the vehicle terminal, the steps of the driver monitoring method based on the vehicle terminal and cloud analysis as described in any one of claims 1 to 9 are implemented.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, the steps of the driver monitoring method based on the vehicle terminal and cloud analysis as described in any one of claims 1 to 9 are implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by the processor, the steps of the driver monitoring method based on the vehicle terminal and cloud analysis as described in any one of claims 1 to 9 are implemented.