A method for real-time driver status assessment based on a smart steering wheel

Abstract

This invention relates to a real-time driver status assessment method based on a smart steering wheel, belonging to the field of intelligent driving. Addressing the limitations of existing smart steering wheel driver status monitoring technologies in specific scenarios (such as small eyes, wearing sunglasses, insufficient lighting, open windows, and multiple people chatting), including the inability to collect key core data, incomplete analysis model parameters, and high hardware requirements, this invention supplements the collection of key core data by installing inertial sensors, ECG, and skin conductance sensors on the smart steering wheel. Simultaneously, it wirelessly connects to an external smart terminal, offloading computationally demanding tasks such as video processing to the smart terminal, achieving faster and more accurate identification and judgment of driver status at a lower cost. Furthermore, through personalized baseline data management, a unique status profile is established for each driver, contributing to more accurate assessment results.

Description

Technical Field

[0001] This invention belongs to the field of intelligent driving. Background Technology

[0002] With the rapid development of the global automotive industry, the penetration rate of new energy vehicles is gradually increasing, and intelligent connected vehicle technology has also made significant progress. In my country, the government is vigorously promoting the transformation and upgrading of the automotive industry and encouraging enterprises to increase their technological innovation efforts to achieve sustainable development. Against this backdrop, the steering wheel, as an important control component of a car, has evolved from a simple control mechanism to an intelligent system integrating information display, interactive control, and driver assistance. To meet the future development needs of the automotive industry, the project to develop intelligent car steering wheels with advanced technology has emerged.

[0003] The intelligent car steering wheel project aims to integrate the latest sensing, human-machine interaction, and intelligent control technologies to make automotive driver assistance systems more intelligent and user-friendly. With the rapid development of emerging technologies such as 5G, the Internet of Things, and artificial intelligence, the intelligent car steering wheel will be able to seamlessly connect with other vehicle systems, providing drivers with a more convenient, safe, and comfortable driving experience. Furthermore, the intelligent car steering wheel can use big data analytics to monitor and provide real-time feedback on driver behavior, helping to improve driving safety and reduce traffic accidents.

[0004] In recent years, domestic and international automakers have increased their R&D investment in intelligent vehicles, launching numerous forward-looking intelligent steering wheel products. Their functions primarily focus on driver state monitoring and autonomous driving. The technical means employed for driver state monitoring mainly include acquiring the driver's electrocardiogram (ECG) and electrical skin conductance (ESC) information detected by the intelligent steering wheel's ECG and ESC sensors during actual driving; and the driver's facial expressions and voice information detected by the intelligent steering wheel's camera and voice sensors during actual driving. Then, based on the ECG, ESC, facial expressions, and voice information, the driver's characteristic information is extracted during actual driving. Finally, this characteristic information is input into the driver's corresponding emotion recognition personality model to obtain the driver's emotion recognition results during actual driving. The vehicle is then controlled to provide assisted driving responses based on the emotion recognition results.

[0005] With the widespread application of smart steering wheel products and technologies, the problem of effectively monitoring changes in the driver's physical condition has been solved to some extent. However, existing smart steering wheel products on the market still have shortcomings in terms of functionality, performance, and user experience, mainly in the following aspects:

[0006] 1. Video surveillance places high demands on the vehicle's computing power, power consumption, and storage capacity, resulting in high hardware costs for smart steering wheels and making it difficult to achieve wider adoption in low- and mid-range vehicles.

[0007] 2. Using cameras to obtain driver facial information has significant limitations and may fail in certain scenarios, such as when the driver has small eyes, is wearing sunglasses, or there is insufficient lighting. In addition, the accuracy of voice recognition by voice sensors is difficult to guarantee in scenarios such as when the window is open or when multiple people are chatting in the car.

[0008] 3. The analysis based on electrocardiogram, skin conductance, facial expression and voice is not stable / comprehensive enough. It does not take into account other more valuable and easier-to-obtain information, such as dangerous actions such as sudden steering wheel movements that occur with emotional fluctuations. This results in an incomplete reference system for the assessment model and the assessment accuracy needs to be improved.

[0009] 4. Monitoring functions often employ uniform algorithms and standards. Due to the wide range of individual differences in drivers' physical conditions, such as heart rate, blood pressure, and body temperature, the standard data ranges for individuals vary. Using a uniform state assessment algorithm cannot accurately reflect the true physical condition of individual drivers, making it difficult to avoid problems such as false alarms and missed alarms. Summary of the Invention

[0010] To address the shortcomings of existing intelligent steering wheel driver status monitoring technologies, such as the inability to collect key data in specific scenarios (small eyes, wearing sunglasses, insufficient lighting, open windows, multiple people chatting, etc.), incomplete analysis model parameters, and high hardware requirements, this invention supplements the collection of key data by installing inertial sensors, ECG, and skin conductance sensors on the intelligent steering wheel. Simultaneously, it wirelessly connects to an external intelligent terminal, offloading computationally demanding tasks such as video processing to the intelligent terminal, achieving faster and more accurate identification and judgment of driver status at a lower cost. Furthermore, through personalized baseline data management, a unique status profile is established for each driver, contributing to more accurate assessment results.

[0011] The smart steering wheel incorporates a built-in camera, voice sensor, ECG and skin conductance sensors, inertial sensor, central processing module, wireless transceiver module, and data storage module. It also leverages external computing power from a smart terminal, which comes pre-installed with a video and audio recognition system specifically designed for computationally intensive video and audio analysis tasks.

[0012] Providing computing power support for the smart steering wheel via a smart terminal is optional. When a smart terminal provides assistance, the central processing module on the smart steering wheel can acquire all information parameters and input them into the driver state assessment model for calculation; otherwise, it only provides data collected by the electrocardiogram, skin conductance, and inertial sensors as parameters for calculation.

[0013] 1. Driver Personal Characteristics and Baseline Information Management

[0014] The smart steering wheel has a built-in camera and voice sensor. When the system detects that the vehicle has started, it uses the built-in camera to perform facial recognition on the driver. If the driver is not driving the vehicle for the first time, the system reads the driver's pre-stored personality information from the data storage. Otherwise, the system prompts the driver to complete a specified action for a preset time through voice prompts or text prompts on the in-vehicle screen. The system then acquires the driver's baseline physiological data, such as electrocardiogram, skin conductance, facial features, and voice features, and matches them with a preset emotion recognition model. The information collected is used as a baseline to set the normal range of the driver's various vital signs, and then the data is archived in the data storage module as the driver's personal feature file.

[0015] 2. Data acquisition from ECG sensor, skin conductance sensor, and inertial sensor.

[0016] The smart steering wheel is equipped with an ECG sensor, a skin conductance sensor, and an inertial sensor. When the driver is driving, the system can acquire the driver's ECG and skin conductance information in real time and transmit this information to the central processing module. The ECG and skin conductance information includes heart rate, ECG, heart rate variability, body temperature, skin impedance, sweat rate, and other necessary information.

[0017] The inertial sensors on the smart steering wheel are used to detect the angular velocity and acceleration of the steering wheel rotation, collect data on steering wheel rotation and vibration, and report the detected data to the central processing module in real time.

[0018] 3. Driver emotion recognition based on audio and video

[0019] The computing power of smart terminals is relatively abundant. Here, smart terminals are used to provide additional computing power support for the smart steering wheel.

[0020] The implementation method is as follows: For the first connection, the smart terminal needs to install relevant software and then connect to the smart steering wheel wirelessly (including but not limited to Bluetooth, StarFlash, etc.). The smart steering wheel saves the smart terminal information to the data storage module. Each time the vehicle starts, the smart steering wheel detects surrounding smart terminal signals, retrieves the previously connected smart terminal, and automatically connects. If the previously connected smart terminal is not detected, it retrieves the smart terminal from the previous connection and attempts to connect, and so on, until it successfully connects to a smart terminal stored in the data storage module, which is then used as the computing power provider. If all smart terminals in the data storage fail to connect, the auxiliary computing power is marked as unreachable.

[0021] After the smart steering wheel successfully connects to the smart terminal, it transmits the video captured by the camera and the audio information collected by the sound sensor to the smart terminal in real time via a wireless transceiver module. The smart terminal first performs data cleaning and enhancement, including noise reduction of the original speech (this invention uses a deep learning-based dereverberation algorithm), and image enhancement technology (this invention uses super-resolution reconstruction) to improve the image clarity of the video data. Then, the obtained visual and speech data are aligned with intent using a multimodal large model (this invention uses a CLIP variant, Contrastive Language-Image Pretraining). Finally, the driver's facial expressions and vocal features are extracted from the processed data. After extraction, these feature signals are input into a preset Transformer-based emotion classification model in the smart terminal (the image input uses a standard 224*224 RGB image size, and the final category is determined by the maximum logits value). This model calculates the driver's real-time emotion recognition result during actual driving and then sends the emotion recognition result to the central processing module.

[0022] In scenarios where the driver has small eyes, wears sunglasses, or experiences insufficient light or loud noise, if the smart terminal fails to extract the driver's facial and vocal characteristics, it does not need to send the driver's emotion recognition results to the central processing module.

[0023] 4. The central processing module performs driver status assessment.

[0024] After the vehicle starts, the central processing module retrieves the current driver's personal characteristic profile (mainly baseline information of various vital signs) from the data storage module and keeps it resident in memory;

[0025] The central processing module receives real-time detection data from ECG, ductus skinitis, and inertial sensors. It performs noise reduction on this raw data (ECG signals are suppressed using power frequency noise; ductus skinitis signals are low-pass filtered to eliminate motion artifacts and then decomposed into Phasic transient response and Tonic baseline levels; and inertial sensor data is noise-reduced using Kalman filtering). After baseline correction based on baseline information in the driver's personal profile, normalized signals are generated. These signals are then time-aligned with the real-time driver emotion recognition results submitted from the smart terminal using a sliding window (10-second window, 1-second step). Finally, they are input into a pre-defined driver state assessment model for multimodal composite feature calculation (using a multi-task learning paradigm to associate shared parameters and extracting features from ECG, ductus skinitis, and inertial sensor data). The system calculates driver's heart rate variability (HRV) based on ECG data, driver's respiratory rate based on ECG and skin conductance data, and long-term stability indices based on the mean and variance of steering wheel angular velocity. The resulting calculated values, including time-domain and frequency-domain HRV, skin conductance response (SCR) frequency, skin conductance level (SCL), steering wheel angular velocity variance, and driver emotion recognition type, are concatenated into a multi-dimensional vector. Finally, a state assessment and classification are performed to obtain a three-dimensional real-time driver state evaluation result, including: vital sign parameter assessment (output parameters include two labels: normal and abnormal), current mood state assessment (output parameters include four labels: pleasant mood, mild stress, moderate stress, and anxiety), and fatigue level assessment (output parameters include three labels: energetic, mild fatigue, and extreme fatigue). At least a fatigue level assessment is required.

[0026] The driver condition assessment model determines whether the normalized signal is within the normal range by referring to the current driver's individual characteristic profile. Its personalized baseline information ensures the accuracy of the assessment results.

[0027] The beneficial effects are mainly reflected in three aspects:

[0028] 1. By introducing the computing power of intelligent terminals, the hardware cost of intelligent steering wheels can be greatly reduced, promoting the popularization of intelligent steering wheels in mid-to-low-end models and helping to expand the application market of intelligent steering wheel products;

[0029] 2. By separating video and audio processing from the smart steering wheel, the system architecture becomes more flexible, and the implementation of the smart steering wheel becomes simpler.

[0030] 3. Multi-dimensional composite feature calculation can more accurately assess the driver's condition. Detailed Implementation

[0031] This invention supplements the collection of key core data by installing inertial sensors and ECG / electrodermal sensors on a smart steering wheel. Simultaneously, it wirelessly connects to an external smart terminal, offloading computationally demanding video processing tasks to the smart terminal for faster and more accurate driver status identification and assessment at a lower cost. Furthermore, through personalized baseline data management, a unique status profile is created for each driver, aiding in more accurate evaluation results.

[0032] Analysis shows that traffic accidents caused by driver error are mainly due to three factors: driver fatigue, emotional fluctuations, and abnormal vital signs. This invention uses a smart steering wheel as a platform to focus on large-scale model identification of these three main causes.

[0033] The smart steering wheel incorporates a built-in camera, voice sensor, ECG and skin conductance sensors, inertial sensor, central processing module, wireless transceiver module, and data storage module. It also leverages external computing power from a smart terminal, which comes pre-installed with a video and audio recognition system specifically designed for computationally intensive video and audio analysis tasks.

[0034] Providing computing power support for the smart steering wheel via a smart terminal is optional. When a smart terminal provides assistance, the central processing module on the smart steering wheel can acquire all information parameters and input them into the driver state assessment model for calculation; otherwise, it only provides data collected by the electrocardiogram, skin conductance, and inertial sensors as parameters for calculation.

[0035] 5. Driver Personality Traits and Baseline Information Management

[0036] The smart steering wheel has a built-in camera and voice sensor. When the system detects that the vehicle has started, it uses the built-in camera to perform facial recognition on the driver. If the driver is not driving the vehicle for the first time, the system reads the driver's pre-stored personality information from the data storage. Otherwise, the system prompts the driver to complete relevant actions through voice prompts or text prompts on the in-vehicle screen. After a preset time, the system obtains the driver's baseline physiological data, such as electrocardiogram, skin conductance, facial features, and voice features, and matches them with a preset emotion recognition model. The information collected at this time is used as the baseline to set the normal range of the driver's various vital signs, and then it is archived as the driver's personal feature file in the data storage module.

[0037] 6. Data acquisition from ECG sensors, skin conductance sensors, and inertial sensors.

[0038] The smart steering wheel is equipped with built-in ECG sensors, skin conductance sensors, and inertial sensors. When the driver is driving, the system can acquire the driver's ECG and skin conductance information in real time and transmit this information to the central processing module. The ECG and skin conductance information includes heart rate, ECG, heart rate variability, body temperature, skin impedance, sweat rate, and other necessary information.

[0039] The inertial sensors on the smart steering wheel are used to detect the angular velocity and acceleration of the steering wheel rotation, collect data on steering wheel rotation and vibration, and report the detected data to the central processing module in real time.

[0040] 7. Driver Emotion Recognition Based on Audio and Video

[0041] The computing power of smart terminals is relatively abundant. Here, smart terminals are used to provide additional computing power support for the smart steering wheel.

[0042] The implementation method is as follows: When the smart terminal is connected for the first time, the relevant software needs to be installed first, and then it connects to the smart steering wheel wirelessly (including but not limited to Bluetooth, Star Flash and other technologies). After that, the smart terminal can automatically connect to the smart steering wheel every time.

[0043] Once the smart steering wheel has successfully connected to the smart terminal, it transmits the video captured by the camera and the audio information captured by the sound sensor to the smart terminal in real time. The smart terminal extracts the driver's facial and vocal features from these features and then inputs them into a pre-set driver emotion recognition model in the smart terminal to calculate the driver's real-time emotion recognition results during actual driving. Finally, the emotion recognition results are sent to the central processing module.

[0044] In scenarios where the driver has small eyes, wears sunglasses, or experiences insufficient light or loud noise, if the smart terminal fails to extract the driver's facial and vocal characteristics, it does not need to send the driver's emotion recognition results to the central processing module.

[0045] 8. The central processing module performs driver status assessment.

[0046] After the vehicle starts, the central processing module retrieves the current driver's personal file (mainly the normal range of various vital signs) from the data storage module and keeps it resident in memory;

[0047] The central processing module receives real-time detection data from ECG, skin conductance, and inertial sensors. It performs noise reduction and baseline correction on this raw data to generate a normalized signal. This normalized signal, along with real-time driver emotion recognition results submitted from the smart terminal, is then input into a pre-defined driver state assessment model for composite feature calculation. This yields a three-dimensional real-time driver state assessment, including: vital sign parameter assessment, current mood state assessment, and fatigue level assessment. At least a fatigue level assessment must be included.

[0048] The driver condition assessment model determines whether the normalized signal is within the normal range by referring to the current driver's individual characteristic profile. Its personalized baseline information ensures the accuracy of the assessment results.

[0049] 1. Introduce intelligent terminals to provide auxiliary computing power for intelligent steering wheels;

[0050] 2. By separating video and audio processing tasks from the smart steering wheel, the feature recognition results are provided as optional parameters to the driver state assessment model, making the system architecture more flexible.

[0051] 3. Establish individual characteristic profiles for drivers to provide more reliable baseline information for the evaluation model.

Claims

1. A method for real-time driver status assessment based on a smart steering wheel, characterized in that: The smart steering wheel incorporates a built-in camera, voice sensor, ECG and skin conductance sensors, inertial sensor, central processing module, wireless transceiver module, and data storage module; it also utilizes an intelligent terminal to provide external computing power for video and voice analysis tasks; it provides at least data collected by the ECG, skin conductance, and inertial sensors as parameters for calculation; and the intelligent terminal comes pre-installed with a video and sound recognition system. 1) Driver Personal Characteristics and Baseline Information Management The smart steering wheel has a built-in camera and voice sensor. When the system detects that the vehicle has started, it uses the built-in camera on the smart steering wheel to perform facial recognition on the driver. If the driver is not driving the vehicle for the first time, it reads the driver's pre-stored personality information from the data storage. Otherwise, the driver is required to complete a specified action for a preset time period via voice prompts or text prompts on the in-vehicle screen. Then, the driver's electrocardiogram, skin conductance, facial features, and voice features are obtained and matched with a preset emotion recognition model. The information collected at the moment is used as a baseline to set the normal range of the driver's various vital signs and then archived as the driver's personal feature file in the data storage module. 2) Data acquisition from ECG sensors, skin conductance sensors, and inertial sensors. The smart steering wheel is equipped with an electrocardiogram (ECG) sensor, a skin conductance sensor, and an inertial sensor. When the driver is driving, the ECG and skin conductance information of the driver are acquired in real time and transmitted to the central processing module. Electrocardiogram (ECG) and skin conductance information include heart rate, ECG value, heart rate variability, body temperature, skin impedance, and perspiration rate; The inertial sensors on the smart steering wheel are used to detect the angular velocity and acceleration of the steering wheel rotation, to collect data on the steering wheel rotation and vibration, and to report the detected data to the central processing module in real time. 3) Driver emotion recognition based on audio and video For the first connection of a smart terminal, relevant software needs to be installed. Then, it connects wirelessly to the smart steering wheel. The smart steering wheel saves the smart terminal information to the data storage module. Each time the vehicle starts, the smart steering wheel detects the surrounding smart terminal signals, retrieves the previously connected smart terminal, and automatically connects. If the previously connected smart terminal is not detected, it retrieves the smart terminal that was connected before last and attempts to connect, and so on, until it successfully connects to a smart terminal stored in the data storage module and uses it as a computing power auxiliary provider. If all smart terminals in the data storage fail to connect, the auxiliary computing power is marked as unreachable. After the smart steering wheel discovers that it has successfully connected to the smart terminal, it transmits the video captured by the camera and the audio information captured by the sound sensor to the smart terminal in real time through the wireless transceiver module. The smart terminal first performs data cleaning and enhancement, including noise reduction processing of the original voice and image enhancement technology to improve the image clarity of the video data. Then, the obtained visual data and voice data are aligned with intent through a multimodal large model. Finally, the driver's facial expression and voice feature signals are extracted from the processed data. After extraction, these feature signals are input into the preset Transformer-based emotion classification model in the smart terminal to calculate the driver's real-time emotion recognition results during actual driving. Finally, the emotion recognition results are sent to the central processing module. If the smart terminal fails to extract the driver's facial and vocal features, it does not need to send the driver's emotion recognition results to the central processing module. 4) The central processing module performs driver status assessment. After the vehicle starts, the central processing module retrieves the current driver's personal profile from the data storage module and stores it in memory. The central processing module receives real-time detection data from ECG, skin conductance, and inertial sensors. It reduces noise from these raw data, corrects the baseline based on baseline information in the driver's personal profile, generates normalized signals, and uses a sliding window to time-align these signals with the real-time driver emotion recognition results submitted from the smart terminal. Then, it inputs them together into a preset driver state assessment model for multimodal composite feature calculation. The resulting data is then concatenated into a multidimensional vector, and finally, state judgment and classification are performed to obtain the real-time driver state assessment result.

2. The method according to claim 1, characterized in that: The noise reduction in step 4) specifically involves: ECG signals were suppressed using power frequency noise, skin conductance signals were filtered to eliminate motion artifacts and then decomposed into Phasic transient response and Tonic baseline levels, while inertial sensor data were filtered using Kalman filtering to eliminate noise.

3. The method according to claim 1, characterized in that: In step 4), the sliding window parameters are a 10-second window with a step size of 1 second.