Driver fatigue driving identification method based on intelligent steering wheel

Abstract

A driver fatigue driving identification method based on an intelligent steering wheel belongs to the field of intelligent driving. In order to solve the problems that real-time video monitoring may be disabled in a specific scene (such as insufficient light), and the real-time video monitoring has large requirements for computing power, power consumption and storage space, a simplified video analysis algorithm is adopted, only the pitching action of the head of a driver is recognized, and the real-time video monitoring efficiency is improved. And the fatigue driving state of the driver is identified and judged more accurately in combination with data acquired by an inertial sensor and a grip sensor which are mounted on the intelligent steering wheel.

Description

Technical Field

[0001] The invention belongs to the field of intelligent driving. Background Art

[0002] With the rapid development of the global automotive industry, the popularity of new energy vehicles has gradually increased, and the technology of intelligent connected vehicles has also made significant progress. In my country, the government has vigorously promoted the transformation and upgrading of the automotive industry and encouraged enterprises to increase their efforts in scientific and technological innovation to achieve sustainable development of the automotive industry. In this context, the function of the car steering wheel, as an important control component of the car, has also shifted from a single control to an intelligent system that integrates information display, interactive control, driving assistance and other functions. In order to meet the development needs of the future automotive industry, the project of developing an intelligent car steering wheel with advanced technology came into being.

[0003] The smart car steering wheel project aims to realize the intelligence and humanization of the car driving assistance system by integrating the latest sensing technology, human-computer interaction technology and intelligent control technology. With the rapid development of emerging technologies such as 5G, Internet of Things, and artificial intelligence, the smart car steering wheel will be able to achieve seamless connection with other vehicle systems, providing drivers with a more convenient, safe, and comfortable driving experience. In addition, the smart car steering wheel can also monitor and feedback the driver's driving behavior in real time through big data analysis, which helps to improve driving safety and reduce the occurrence of traffic accidents.

[0004] In recent years, domestic and foreign automakers have increased their R&D investment in the field of smart cars and launched a number of forward-looking smart steering wheel products, whose functions mainly focus on driver health monitoring and autonomous driving. The technical means adopted for driver health monitoring mainly include adding infrared sensors to the steering wheel to detect whether the driver is holding the steering wheel, adding temperature sensors to detect the driver's body temperature, adding optical heart rate sensors to detect the driver's heart rate, and corresponding buzzer alarms.

[0005] In terms of driver fatigue monitoring, current solutions mainly focus on using video recognition technology to identify the driver's status based on data such as the driver's facial movements and heart rate.

[0006] At present, smart steering wheel technology is in the transition stage from mechatronics to comprehensive intelligence, with wire-controlled steering, interactive innovation and autonomous driving integration as the core driving force. In the next five years, with the improvement of policies and the iteration of technology, its application scenarios will be further expanded, becoming a key carrier for the intelligent transformation of automobiles.

[0007] With the promotion and application of smart steering wheel products and technologies, the problem of effectively monitoring changes in the driver's physical health status has been solved to a certain extent. However, the existing smart steering wheel products on the market still have certain deficiencies in terms of function, performance and user experience, which are mainly reflected in the following aspects:

[0008] 1. Video surveillance has high requirements for vehicle computing power, power consumption, and storage capacity, resulting in high costs for related intelligent surveillance, making it difficult to popularize it on a wider range of mid- and low-end vehicles;

[0009] 2. The expression recognition function of video surveillance is disabled in some scenarios, such as when the driver has small eyes, wears sunglasses, or has insufficient light, which can easily lead to misjudgment and false warnings;

[0010] 3. The monitoring function mainly focuses on the identification of driver fatigue. Due to the individual differences in drivers' physical conditions, such as heart rate, blood pressure, body temperature, etc., there is a certain gap in the standard data range of individuals. It is difficult to use a unified health status assessment algorithm to accurately reflect the driver's real physical condition and to eliminate problems such as false alarms and missed alarms.

[0011] 4. The optical system and video image recognition used in the single-handed or double-handed grip detection technology for steering wheel hand-off also have defects such as high cost, high false alarm rate, and easy obstruction.

[0012] In view of the fact that the aforementioned real-time video monitoring technical solution may be disabled in specific scenarios (small eyes, wearing sunglasses, insufficient light), and the problem that real-time video monitoring requires large computing power, power consumption, and storage space, the present invention adopts a simplified video analysis algorithm to only identify the pitch movement of the driver's head, and combines the data collected by the inertial sensor and grip force sensor installed on the smart steering wheel to more accurately identify and judge the driver's fatigue driving state. Summary of the invention

[0013] The smart steering wheel has a built-in camera, grip force sensor, inertial sensor, central data processing unit, and data storage device.

[0014] The system has two preset signal lights. Signal light 1 and signal light 2 correspond to the grip sensor alarm and video motion recognition alarm events respectively.

[0015] 1. Lightweight recognition of a driver’s single action

[0016] A. Lightweight human detection model (such as MobileNet-SSD) + model compression: Use a lightweight model to identify the driver, and crop the area to focus only on the head area to reduce the amount of input data; after processing, enter the tracking stage, and use a low-computation tracking algorithm (such as KCF or MOSSE) in the tracking stage to reduce the amount of computation for frame-by-frame detection; then enter the action recognition stage.

[0017] B. 2D CNN (two-dimensional neural network) + time series modeling: In the action recognition stage, lightweight 2D CNN (such as MobileNet, ShuffleNet) is used to extract features frame by frame, and then combined with LSTM (LSTM,

[0018] By processing the timing information through Long Short-Term Memory (LSTM) or Temporal Pooling, the computationally intensive 3D convolution calculation can be avoided.

[0019] C. Action classification: Only the tracked driver head area is classified, and only the driver's head pitch motion is captured, which can greatly reduce the processing range of the processor;

[0020] D. Set a preset threshold. Once the video action recognition module detects that the driver's head pitch movement amplitude reaches the threshold, it sends the message to the central data processor.

[0021] 2. Grip force sensor

[0022] The amount of grip force the driver applies to the steering wheel can reflect the degree of control the driver has over the vehicle to a certain extent. The smart steering wheel sends the collected grip force sensing data to the central data processing unit as one of the important bases for determining whether the driver is driving fatigued.

[0023] A. Install a grip force sensor on the smart steering wheel. Considering that drivers have different driving habits, some prefer to hold the steering wheel with both hands, while others prefer to hold it with one hand. If two-point grip force data are collected, the maximum grip force data collected shall prevail.

[0024] B. Preset a grip strength threshold and compare the grip strength data collected above with it. If it is higher than this threshold, it is judged that the driver is in a normal state; if it is lower than this threshold, the information is continuously sent to the central data processing unit at a preset frequency (such as 3 times / second) until the collected grip strength value returns to above the preset threshold.

[0025] 3. Inertial Sensors

[0026] The inertial sensor on the smart steering wheel is used to detect the angular velocity of the steering wheel. When the steering wheel is detected to be rotating, the detected angular velocity data is reported to the central data processor in real time; when the steering wheel is not rotating, there is no need to report data.

[0027] 4. Central Data Processing Unit

[0028] The central data processing unit receives the detection data uploaded by the video motion recognition processor, grip force sensor and inertial sensor, and analyzes the data to determine whether the driver is in a fatigue driving state. After the processing is completed, the relevant data is saved to the data storage unit.

[0029] A. The grip force sensor uploads data and is preset to have the highest priority. After the central data processing unit receives the grip force sensor upload message, it immediately starts to calculate the duration of the message (a timer is set, starting from the receipt of the first message until no such message is received within the preset duration, that is, after the grip force sensor stops uploading data, the timer is reset, and if the signal light 1 is on, it is set to off);

[0030] B. If the abnormal grip sensing duration recorded by the timer exceeds the preset threshold, signal light 1 lights up; C. After receiving information sent by the video action recognition module, the central data processing unit immediately starts to trace back the frequency of receiving such messages in the last time period (the default is 1 minute). If the calculated frequency exceeds the preset threshold, signal light 2 lights up; otherwise, if the frequency does not exceed the preset threshold and signal light 2 is in the lit state, it is set to extinguish;

[0031] D. When signal lights 1 and 2 are both on at the same time, the central data processor reads the angular velocity data uploaded by the inertial sensor last received and stored in the data storage device, calculates the duration from the current moment, and immediately issues a fatigue driving warning message if the duration exceeds a preset threshold.

[0032] The beneficial effects are mainly reflected in two aspects:

[0033] First, by using lightweight video action recognition models, the hardware cost of smart devices can be greatly reduced, which can promote the popularization of smart steering wheels in mid- and low-end models and help expand the application market of smart steering wheels.

[0034] The second is lightweight video motion recognition + grip force sensing + steering wheel rotation sensing. By cross-validating the fatigue driving status through the data of the three, the accuracy of identifying the driver's fatigue driving status can be greatly improved, reducing the occurrence of misjudgment and missed judgment. DETAILED DESCRIPTION

[0035] The smart steering wheel has a built-in camera, grip force sensor, inertial sensor, central data processing unit, and data storage device.

[0036] The system has two preset signal lights. Signal light 1 and signal light 2 correspond to the grip sensor alarm and video motion recognition alarm events respectively.

[0037] 5. Lightweight recognition of a driver’s single action

[0038] A. Lightweight human detection model (such as MobileNet-SSD) + model compression: Use a lightweight model to identify the driver, and crop the area to focus only on the head area to reduce the amount of input data; after processing, enter the tracking stage, and use a low-computation tracking algorithm (such as KCF or MOSSE) in the tracking stage to reduce the amount of computation for frame-by-frame detection; then enter the action recognition stage.

[0039] B. 2D CNN (two-dimensional neural network) + time series modeling: In the action recognition stage, lightweight 2D CNN (such as MobileNet, ShuffleNet) is used to extract features frame by frame, and then combined with LSTM (LSTM,

[0040] By processing the timing information through Long Short-Term Memory (LSTM) or Temporal Pooling, the computationally intensive 3D convolution calculation can be avoided.

[0041] Lightweight 2D CNN (two-dimensional convolutional neural network) is a convolutional neural network designed for low computing resource scenarios. By optimizing the model structure and the number of parameters, it reduces the computational complexity while maintaining high recognition accuracy. Commonly used models include MobileNet, ShuffleNet, SqueezeNet, etc. The present invention uses a 3*3 convolution kernel, the number of input channels is set to 3, and the number of output channels is set to 32.

[0042] The network input is the data type (here is an image) and size, and the output is a category probability vector (such as the N-dimensional vector after Softmax, where N is the number of categories).

[0043] C. Action classification: Only the tracked driver head area is classified, and only the driver's head pitch motion is captured, which can greatly reduce the processing range of the processor;

[0044] A preset threshold is set. Once the video action recognition module detects that the driver's head pitch motion amplitude reaches the threshold, the message is sent to the central data processor. Recognition process: reduce the image pixel value, adjust the size to a fixed size (only the head area), extract features and use global average pooling to replace the fully connected layer to reduce parameters, and use Softmax or Sigmoid (including but not limited to these two) to obtain the category probability.

[0045] The default threshold for head pitch frequency is 5 times per minute.

[0046] 6. Grip force sensor

[0047] The amount of grip force the driver applies to the steering wheel can reflect the degree of control the driver has over the vehicle to a certain extent. The smart steering wheel sends the collected grip force sensing data to the central data processing unit as one of the important bases for determining whether the driver is driving fatigued.

[0048] A. Install a grip force sensor on the smart steering wheel. Considering that drivers have different driving habits, some prefer to hold the steering wheel with both hands, while others prefer to hold it with one hand. If two-point grip force data are collected, the maximum grip force data collected shall prevail.

[0049] B. Preset a grip strength threshold and compare the grip strength data collected above with it. If it is higher than this threshold, it is judged that the driver is in a normal state; if it is lower than this threshold, the information is continuously sent to the central data processing unit at a preset frequency (such as 3 times / second) until the collected grip strength value returns to above the preset threshold.

[0050] The grip strength threshold was taken as the average value of the first 3 minutes when the driver started driving the car.

[0051] 7. Inertial Sensors

[0052] The inertial sensor on the smart steering wheel is used to detect the angular velocity of the steering wheel. When the steering wheel is detected to be rotating, the detected angular velocity data is reported to the central data processor in real time; when the steering wheel is not rotating, there is no need to report data.

[0053] 8. Central Data Processing Unit

[0054] The central data processing unit receives the detection data uploaded by the video motion recognition processor, grip force sensor and inertial sensor, and analyzes the data to determine whether the driver is in a fatigue driving state. After the processing is completed, the relevant data is saved to the data storage unit.

[0055] A. The grip force sensor uploads data and is preset to have the highest priority. After the central data processing unit receives the grip force sensor upload message, it immediately starts to calculate the duration of the message (a timer is set, starting from the receipt of the first message until no such message is received within the preset duration, that is, after the grip force sensor stops uploading data, the timer is reset, and if the signal light 1 is on, it is set to off);

[0056] B. If the abnormal grip sensing duration recorded by the timer exceeds the preset threshold, signal light 1 lights up;

[0057] C. Each time the central data processing unit receives information sent by the video action recognition module, it immediately starts to trace back the frequency of receiving such messages within the last time period (the default is 1 minute). If the calculated frequency exceeds the preset threshold, the signal light 2 is turned on; otherwise, if the frequency does not exceed the preset threshold and the signal light 2 is in the lit state, it is set to be extinguished;

[0058] D. When signal lights 1 and 2 are both on, the central data processor reads the angular velocity data uploaded by the inertial sensor last received and stored in the data storage device, calculates the duration from the current moment, and immediately issues a fatigue driving warning message if the duration exceeds a preset threshold;

[0059] E. The judgment condition for fatigue driving is that signal lights 1 and 2 are on at the same time, and the steering wheel rotation data uploaded by the inertial sensor is not received within a time limit. In other words, the driver's hands are not detected to be tightly gripping the steering wheel for a long time, the head pitch frequency exceeds the standard, and the steering wheel is not turned for a long time. One or a combination of these three conditions is used as the basis for judging fatigue driving. In actual applications, the threshold for holding the steering wheel tightly is set to 6 seconds, the head pitch frequency is set to 5 times / minute, and the threshold for the length of time when the steering wheel does not turn is 10 seconds.

[0060] Effects of the present invention:

[0061] 1. Through lightweight algorithm models, the driver's single action can be effectively recognized with lower hardware configuration and lower energy consumption;

[0062] 2. The smart steering wheel integrates video motion recognition + grip force sensing + angular velocity sensing. By cross-comparing and verifying relevant information between three different dimensions of the driver, compared with traditional video motion recognition technology, it can greatly improve the recognition accuracy of the driver's fatigue driving status.

Claims

1. A method for identifying driver fatigue based on an intelligent steering wheel, characterized in that: The smart steering wheel has a built-in camera, grip force sensor, inertial sensor, central data processing unit, and data storage device. Two signal lights are set, signal light 1 and signal light 2 correspond to the grip force sensor alarm and video motion recognition alarm events respectively. 1) Lightweight recognition of a driver’s single action A. Lightweight human detection model combined with model compression: Use a lightweight model to identify the driver, and crop the area to focus only on the head area to reduce the amount of input data; After the processing is completed, it enters the tracking stage, in which a tracking algorithm is used to reduce the amount of calculation for frame-by-frame detection; then it enters the action recognition stage; B. Two-dimensional neural network 2D CNN joint temporal modeling: In the action recognition stage, a lightweight 2D CNN is used to extract features frame by frame, and then combined with an LSTM long short-term memory network or temporal pooling to process temporal information; C. Action classification: Only the tracked driver head area is classified, and only the driver's head pitch movement is captured; D. Setting a preset threshold value, once the video action recognition module detects that the driver's head pitch motion amplitude reaches the threshold value, the message is sent to the central data processor; The default threshold for head pitch frequency is 5 times per minute; 2) Grip force sensor The smart steering wheel sends the collected grip force sensing data to the central data processing unit as one of the important bases for whether the driver is driving fatigued; A. Install a grip force sensor on the smart steering wheel. Drivers have different driving habits. Some like to hold the steering wheel with both hands, while others like to hold it with one hand. If two-point grip force data is collected, the maximum grip force data collected shall prevail. B. Preset a grip strength threshold and compare the grip strength data collected above with it. If it is higher than this threshold, it is judged that the driver is in a normal state; If the grip strength value is lower than the threshold, the information is sent to the central data processing unit at a preset frequency until the grip strength value collected returns to above the preset threshold; The grip strength threshold was taken as the average value of the first 3 minutes when the driver started driving the car. 3) Inertial Sensors The inertial sensor on the smart steering wheel is used to detect the angular velocity of the steering wheel. When the steering wheel is detected to be rotating, the detected angular velocity data is reported to the central data processor in real time; when the steering wheel is not rotating, there is no need to report data; 4) Central Data Processing Unit The central data processing unit receives the detection data uploaded by the video action recognition processor, grip force sensor and inertial sensor, and performs comprehensive analysis on these data to determine whether the driver is in a fatigue driving state; after the processing is completed, the relevant data is saved to the data storage unit; A. The data uploaded by the grip force sensor is preset to enjoy the highest priority. After the central data processing unit receives the message uploaded by the grip force sensor, it immediately starts to calculate the duration of the message being sent. Specifically, a timer is set to start counting from the time the first message is received until no such message is received within the preset time, that is, after the grip force sensor stops uploading data, Reset the timer to zero and set signal light 1 to off if it is on; B. If the abnormal grip sensing duration recorded by the timer exceeds the preset threshold, signal light 1 lights up; C. Each time the central data processing unit receives information sent by the video action recognition module, it immediately starts to trace back the frequency of receiving such messages in the last time period. If the calculated frequency exceeds the preset threshold, the signal light 2 lights up; otherwise, if the frequency does not exceed the preset threshold, and If signal light 2 is on, set it to off; D. When signal lights 1 and 2 are both on, the central data processor reads the angular velocity data uploaded by the inertial sensor last received and stored in the data storage device, calculates the duration from the current moment, and immediately issues a fatigue driving warning message if the duration exceeds a preset threshold; The threshold for the duration of holding the steering wheel tightly is set to 6 seconds, the frequency of head pitching is set to 5 times / minute, and the threshold for the duration without steering wheel turning is 10 seconds; if the driver's hands are not detected holding the steering wheel tightly, the frequency of head pitching exceeds the standard, or the steering wheel does not turn for more than the time threshold, one or a combination of the above three conditions will be used as the basis for judging fatigue driving.

2. The method for identifying driver fatigue driving based on an intelligent steering wheel according to claim 1, characterized in that: The tracking algorithm is KCF or MOSSE.

3. The method for identifying driver fatigue driving based on an intelligent steering wheel according to claim 1, characterized in that: The lightweight two-dimensional neural network is MobileNet or ShuffleNet.