Driver real-time state evaluation method based on intelligent steering wheel

Abstract

The invention discloses a driver real-time state evaluation method based on an intelligent steering wheel and belongs to the field of intelligent driving. Aiming at the problems that the existing intelligent steering wheel driver state monitoring technology cannot collect key core data under specific scenes (small eyes, wearing sunglasses, insufficient light, opening windows, chatting by multiple persons and the like), analysis model parameters are not comprehensive enough, the requirement on hardware configuration is high and the like, an inertial sensor, an electrocardio sensor and a galvanic skin sensor are mounted on an intelligent steering wheel; according to the system, key core data are complementarily collected, and meanwhile, tasks with high computing power requirements such as video processing are unloaded to an intelligent terminal to be completed through wireless connection with the external intelligent terminal, so that the state of a driver can be identified and judged more quickly and accurately at lower cost. Furthermore, through driver personalized baseline data management, a special state file is established for each driver, and a more accurate evaluation effect is helped to be obtained.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent driving. Background Art

[0002] With the rapid development of the global automotive industry, the popularity of new energy vehicles has gradually increased, and significant progress has been made in intelligent connected vehicle technology. In China, the government has vigorously promoted the transformation and upgrading of the automotive industry, encouraging enterprises to increase their efforts in scientific and technological innovation to achieve the sustainable development of the automotive industry. Against this background, as an important control component of the vehicle, the function of the car steering wheel has shifted from a single control to an intelligent system integrating information display, interactive control, driving assistance and other multiple functions. In order to meet the development needs of the future automotive industry, the project of developing an intelligent car steering wheel with advanced technology has emerged as the times require.

[0003] The intelligent car steering wheel project aims to achieve the intelligentization and humanization of the vehicle driving assistance system by integrating the latest sensing technology, human-computer interaction technology and intelligent control technology. With the rapid development of emerging technologies such as 5G, the Internet of Things and artificial intelligence, the intelligent car steering wheel will be able to achieve seamless connection with other vehicle systems, providing a more convenient, safe and comfortable driving experience for drivers. In addition, the intelligent car steering wheel can also conduct real-time monitoring and feedback on the driving behavior of drivers through big data analysis, which helps to improve driving safety and reduce the occurrence of traffic accidents.

[0004] In recent years, automotive manufacturers at home and abroad have increased their R & D investment in the field of intelligent vehicles and launched many forward-looking intelligent steering wheel products, whose functions mainly focus on driver state monitoring and the field of autonomous driving. The technical means adopted for driver state monitoring mainly include obtaining the electrocardiogram information and galvanic skin response information of the driver detected by the electrocardiogram sensor and galvanic skin response sensor of the intelligent steering wheel during the actual driving process, the facial expression information and voice information of the driver detected by the camera and voice sensor of the intelligent steering wheel during the actual driving process, then extracting the characteristic information of the driver during the actual driving process according to the electrocardiogram information, galvanic skin response information, facial expression information and voice information, and finally inputting the characteristic information into the emotion recognition personality model corresponding to the driver to obtain the emotion recognition result information of the driver during the actual driving process, and then controlling the vehicle to perform auxiliary driving response according to the emotion recognition result information.

[0005] With the popularization and application of intelligent steering wheel products and technologies, the problem of effectively monitoring the changes in the physical state of drivers has been solved to a certain extent. However, the existing intelligent steering wheel products on the market still have deficiencies in terms of function, performance and user experience, which are mainly reflected in the following aspects:

[0006] 1. The method of video monitoring has high requirements for on-vehicle computing power, power consumption, and storage capacity, resulting in high hardware costs for intelligent steering wheels and making it difficult to popularize them more widely in mid- to low-end vehicles.

[0007] 2. Using cameras to obtain driver's facial expression information has great limitations, and there are disabling situations in some scenarios, such as small eyes of the driver, wearing sunglasses, insufficient light, etc. In addition, in scenarios such as opening windows and multiple people chatting in the car, it is difficult to ensure the accuracy of voice recognition of the driver through voice sensors.

[0008] 3. The analysis basis based on electrocardiogram, galvanic skin response, facial expressions, and voice is not stable / comprehensive enough, and does not take into account other more valuable and easier-to-obtain information, such as dangerous actions like violently turning the steering wheel that occur along with emotional fluctuations, resulting in an incomplete reference system for the evaluation model and the need to improve the evaluation accuracy.

[0009] 4. Monitoring functions mostly adopt unified algorithms and standards. Due to the wide range of individual differences in the physical conditions of drivers, such as heart rate, blood pressure, body temperature, etc., there are certain differences in the standard data ranges of individuals. It is difficult to truthfully reflect the real physical conditions of individual drivers using a unified state evaluation algorithm, and problems such as false alarms and missed alarms are inevitable. Summary of the Invention

[0010] Aiming at the problems of existing intelligent steering wheel driver status monitoring technologies, such as the inability to collect key core data in specific scenarios (small eyes, wearing sunglasses, insufficient light, opening windows, multiple people chatting, etc.), incomplete analysis model parameters, and high requirements for hardware configuration, the present invention installs inertial sensors, electrocardiogram, and galvanic skin sensors on the intelligent steering wheel to supplement the collection of key core data. At the same time, through wireless connection to an external intelligent terminal, tasks with high computing power requirements such as video processing are offloaded to the intelligent terminal to be completed, realizing faster and more accurate identification and judgment of the driver's status at a lower cost. Further, through driver personalized baseline data management, a proprietary status profile is established for each driver to help obtain a more accurate evaluation effect.

[0011] The intelligent steering wheel is internally equipped with a camera, a voice sensor, an electrocardiogram and galvanic skin sensor, an inertial sensor, a central processing module, a wireless transceiver module, and a data storage module. At the same time, the intelligent terminal is borrowed to provide external computing power support, and a video and voice recognition system is pre-installed on the intelligent terminal, which is specifically used to execute video and voice analysis tasks that consume a large amount of computing power.

[0012] It is an optional item for the intelligent terminal to provide computing power support for the intelligent steering wheel. When there is assistance from the intelligent terminal, the central processing module on the intelligent steering wheel can obtain all information parameters and input them into the driver status evaluation model for calculation; otherwise, only the data collected by the electrocardiogram, galvanic skin, and inertial sensors are provided as parameters for calculation.

[0013] 1. Driver Personal Characteristics and Baseline Information Management

[0014] An in - vehicle camera and a voice sensor are built into the intelligent steering wheel. When the system detects that the vehicle is started, the in - vehicle camera built into the intelligent steering wheel is enabled to perform face recognition on the driver. If the driver is not driving this vehicle for the first time, the pre - stored personal characteristic information of the driver is read from the data storage. Otherwise, through voice prompts or on - vehicle screen text prompts, the driver is required to complete a specified action for a preset duration, and then baseline physiological data such as the driver's electrocardiogram (ECG), galvanic skin response (GSR), facial features, and voice features are obtained and matched with a preset emotion recognition model. The currently collected information is used as a baseline to set the normal data range of the driver's various physical signs, and then it is archived as the driver's personal characteristic file in the data storage module;

[0015] 2. Data Acquisition of Electrocardiogram Sensor, Galvanic Skin Sensor, and Inertial Sensor

[0016] An electrocardiogram sensor, a galvanic skin sensor, and an inertial sensor are built into the intelligent steering wheel. When the driver is driving the vehicle, the driver's electrocardiogram information and galvanic skin information can be obtained in real - time and transmitted to the central processing module; the electrocardiogram and galvanic skin information include heart rate, electrocardiogram, heart rate variability, body temperature, skin impedance, sweat rate, and other necessary information.

[0017] The inertial sensor on the intelligent steering wheel is used to detect the angular velocity and acceleration of the steering wheel rotation, collect data on the steering wheel rotation and jitter, and report the detected data to the central processing module in real - time.

[0018] 3. Driver Emotion Recognition Based on Audio and Video

[0019] The intelligent terminal has relatively abundant computing power. Here, the intelligent terminal is used to provide additional computing power support for the intelligent steering wheel.

[0020] The implementation method is as follows: When the intelligent terminal is first connected, relevant software needs to be installed first, and then it is connected to the intelligent steering wheel wirelessly (including but not limited to technologies such as Bluetooth and XingShan). The intelligent steering wheel saves the intelligent terminal information in the data storage module. Every time the vehicle is started in the future, the intelligent steering wheel detects the surrounding intelligent terminal signals, obtains the previously connected intelligent terminal and automatically connects. If the previously connected intelligent terminal is not detected, the second - previously connected intelligent terminal is obtained and tried to connect, and so on, until a successful connection is made to an intelligent terminal saved in the data storage module, and it is used as the computing power assistance provider. If all the intelligent terminals in the data storage fail to connect, the auxiliary computing power is marked as unreachable.

[0021] After the smart steering wheel finds that it has been successfully connected to the smart terminal, the video collected by the camera and the audio information collected by the sound sensor are transmitted to the smart terminal in real time through the wireless transceiver module. The smart terminal first performs data cleaning and enhancement, including noise reduction processing of the original voice (the present invention adopts a de-reverberation algorithm based on deep learning), and the video data is enhanced by image enhancement technology (the present invention adopts super-resolution reconstruction) to improve the picture clarity, and then the obtained visual data and voice data are aligned by a multimodal large model (the present invention uses a CLIP variant, Contrastive Language-Image Pretraining), and finally the driver's expression and sound feature signals are extracted from the processed data. After the extraction is completed, these feature signals are input into the preset Transformer-based emotion classification model in the smart terminal (the image input adopts a standard RGB image size of 224*224, and the final category is determined by the maximum value of logits), and the real-time emotion recognition results of the driver during the actual driving process are calculated, and finally the emotion recognition results are sent to the central processing module.

[0022] In scenarios such as the driver having small eyes, wearing sunglasses, insufficient light, or loud noise, if the smart terminal fails to extract the driver's facial expression and voice feature information, there is no need to send the driver emotion recognition results to the central processing module.

[0023] 4. The central processing module evaluates the driver's status

[0024] After the vehicle is started, the central processing module retrieves the current driver's personal characteristic file (mainly the baseline information of various physical signs) from the data storage module and stores it in the memory;

[0025] The central processing module receives in real time the detection data uploaded from the electrocardiogram (ECG), galvanic skin response (GSR), and inertial sensors, and denoises these raw data (for the ECG signal, power frequency noise suppression is used; for the GSR signal, low-pass filtering is used to eliminate motion artifacts, and then it is decomposed into Phasic transient response and Tonic baseline level; for the inertial sensor data, Kalman filtering is used to eliminate noise). After baseline correction based on the baseline information in the driver's personal characteristic profile, normalized signals are generated. These signals are time-aligned with the real-time driver emotion recognition results submitted from the smart terminal through a sliding window (10-second window, 1-second step), and then they are input together into a preset driver state evaluation model for multi-modal composite feature calculation (by associating task-sharing parameters through a multi-task learning paradigm, the steering wheel grip state is extracted based on the ECG, GSR, and inertial sensor data, the heart rate variability (HRV) is calculated based on the ECG data, the driver's respiratory rate is calculated based on the ECG and GSR data, and the long-term stability index is calculated based on the mean variance of the steering wheel angular velocity). The obtained composite feature calculation values, such as time domain + frequency domain HRV, skin conductance response (SCR) frequency, skin conductance level (SCL), steering wheel angular velocity variance, driver emotion recognition type, etc., are feature-stitched into a multi-dimensional vector, and finally, state judgment and grading are performed to obtain the real-time driver state evaluation results in three dimensions, including: physical sign parameter evaluation (the output parameters include two labels: normal and abnormal), current mood state evaluation (the output parameters include four labels: happy mood, mild stress, moderate stress, and anxiety), and fatigue degree evaluation (the output parameters include three labels: energetic, mildly fatigued, and extremely fatigued). At least the judgment of the fatigue degree evaluation is included.

[0026] The judgment of whether the normalized signal is within the normal range by the driver state evaluation model is based on the current driver's personal characteristic profile, and its personalized baseline information ensures the accuracy of the evaluation results.

[0027] The beneficial effects are mainly reflected in three aspects:

[0028] 1. By introducing the computing power of the smart terminal, the hardware cost of the smart steering wheel can be greatly reduced, promoting the popularization of the smart steering wheel in mid- to low-end models, and helping to expand the application market of smart steering wheel products;

[0029] 2. Separating the video and sound processing work from the smart steering wheel makes the system architecture more flexible and the implementation of the smart steering wheel more convenient;

[0030] 3. Multi-dimensional composite feature calculation can more accurately evaluate the driver's state. Specific implementation manner

[0031] The present invention supplements the acquisition of key core data by installing inertial sensors, electrocardiogram / galvanic skin sensors on the intelligent steering wheel, and at the same time offloads tasks with high video processing computing power requirements to the external intelligent terminal through wireless connection to be completed, so as to achieve faster and more accurate identification and judgment of the driver's state at a lower cost. Further, through the management of the driver's personalized baseline data, a proprietary state profile is established for each driver to help obtain a more accurate evaluation effect.

[0032] According to analysis, the main causes of traffic accidents caused by the driver's own reasons mainly focus on three aspects: fatigue driving, emotional fluctuations, and abnormal physical sign parameters. The present invention takes the intelligent steering wheel as a carrier and focuses on the large model recognition of the above three main causes.

[0033] The intelligent steering wheel is internally provided with a camera, a voice sensor, an electrocardiogram and galvanic skin sensors, an inertial sensor, a central processing module, a wireless transceiver module, and a data storage module. At the same time, the external computing power support is provided by borrowing the intelligent terminal, and a video and sound recognition system is pre-installed on the intelligent terminal, which is specifically used to execute video and voice analysis tasks with huge computing power consumption.

[0034] It is an optional item for the intelligent terminal to provide computing power support for the intelligent steering wheel. When there is assistance provided by the intelligent terminal, the central processing module on the intelligent steering wheel can obtain the full amount of information parameters and input them into the driver state evaluation model for calculation; otherwise, only the data collected by the electrocardiogram, galvanic skin, and inertial sensors are provided as parameters for calculation.

[0035] 5. Driver Personality Characteristics and Baseline Information Management

[0036] The intelligent steering wheel is internally provided with a camera and a voice sensor. When the system detects that the vehicle starts, the built-in camera on the intelligent steering wheel is enabled to perform face recognition on the driver. If the driver is not driving this vehicle for the first time, the pre-stored personality characteristic information of the driver is read from the data memory; otherwise, through voice prompts or on-vehicle screen text prompts, after the driver is required to complete relevant actions for a preset duration, the baseline physiological data such as the driver's electrocardiogram, galvanic skin, facial features, and voice features are obtained and matched with the preset emotion recognition model. The currently collected information is used as the baseline to set the normal data range of each physical sign of the driver, and then it is archived as the personal characteristic file of the driver into the data storage module;

[0037] 6. Data Acquisition of Electrocardiogram Sensor, Galvanic Skin Sensor, and Inertial Sensor

[0038] The smart steering wheel is equipped with a built-in ECG sensor, skin conduction sensor, and inertial sensor. The driver's ECG and skin conduction information can be obtained in real time while driving the vehicle, and the information is transmitted to the central processing module. The ECG and skin conduction information include heart rate, electrocardiogram, heart rate variability, body temperature, skin impedance, sweat rate and other necessary information.

[0039] The inertial sensor on the smart steering wheel is used to detect the angular velocity and acceleration of the steering wheel, to collect data on the steering wheel rotation and vibration, and to report the detected data to the central processing module in real time.

[0040] 7. Driver emotion recognition based on audio and video

[0041] Smart terminals have sufficient computing power. Here, smart terminals are used to provide additional computing power support for smart steering wheels.

[0042] The implementation method is as follows: When the smart terminal is connected for the first time, you need to install the relevant software first, and then connect to the smart steering wheel via wireless (including but not limited to Bluetooth, Star Flash and other technologies). Every time thereafter, the smart terminal can automatically connect to the smart steering wheel.

[0043] After the smart steering wheel finds that it has been successfully connected to the smart terminal, it transmits the video captured by the camera and the audio information collected by the sound sensor to the smart terminal in real time. The smart terminal extracts the driver's facial expression and sound feature information from it, and then inputs this feature information into the driver emotion recognition model preset in the smart terminal, calculates the real-time emotion recognition results of the driver during actual driving, and finally sends the emotion recognition results to the central processing module.

[0044] In scenarios such as the driver having small eyes, wearing sunglasses, insufficient light, or loud noise, if the smart terminal fails to extract the driver's facial expression and voice feature information, there is no need to send the driver emotion recognition results to the central processing module.

[0045] 8. The central processing module evaluates the driver's status

[0046] After the vehicle is started, the central processing module retrieves the current driver's personal profile (mainly the normal data range of various physical signs) from the data storage module and stores it in the memory;

[0047] The central processing module receives in real time the detection data uploaded from the electrocardiogram, galvanic skin response, and inertial sensors, denoises and corrects the baseline of this raw data to generate a normalized signal, and then inputs it together with the real-time driver emotion recognition result submitted from the intelligent terminal into a preset driver state evaluation model for composite feature calculation, obtaining the real-time driver state evaluation results in three dimensions, including: physical sign parameter evaluation, current mood state evaluation, and fatigue degree evaluation. At least it includes the fatigue degree evaluation. At least it includes the judgment of the fatigue degree evaluation.

[0048] The judgment of whether the normalized signal is within the normal range by the driver state evaluation model is carried out with reference to the current driver's personality profile, and its personalized baseline information ensures the accuracy of the evaluation result.

[0049] 1. Introduce an intelligent terminal to provide auxiliary computing power for the intelligent steering wheel;

[0050] 2. Separate the video and sound processing tasks from the intelligent steering wheel, and use its feature recognition results as optional parameters to provide to the driver state evaluation model, making the system architecture more flexible;

[0051] 3. Establish a personality profile for the driver to provide more reliable baseline information for the evaluation model.

Claims

1. A driver real-time status assessment method based on an intelligent steering wheel, characterized in that: The smart steering wheel has built-in cameras, voice sensors, ECG and skin electrical sensors, inertial sensors, central processing modules, wireless transceiver modules, and data storage modules. It also uses smart terminals to provide external computing power support for video and voice analysis tasks. It at least provides data collected by ECG, skin electrical, and inertial sensors as parameters for calculation. The smart terminal is pre-installed with video and voice recognition systems. 1) Driver personal characteristics and baseline information management The smart steering wheel has a built-in camera and voice sensor. When the system detects that the vehicle is started, the built-in camera on the smart steering wheel is activated to perform facial recognition on the driver. If the driver is not driving the vehicle for the first time, the driver's pre-stored personality feature information is read from the data storage device; Otherwise, the driver is required to complete the specified action for a preset time through voice prompts or text prompts on the vehicle screen, and then the driver's electrocardiogram, skin electricity, facial features, and voice features are obtained and matched with the preset emotion recognition model. The currently collected information is used as a baseline to set the normal data range of the driver's various physical signs, and then archived as the driver's personal feature file in the data storage module; 2) ECG sensor, skin electrical sensor, and inertial sensor data collection The smart steering wheel has built-in ECG sensor, skin electrical sensor and inertial sensor. The driver's ECG and skin electrical information are obtained in real time while driving the vehicle, and the information is transmitted to the central processing module. ECG and skin conductance information includes heart rate, ECG, heart rate variability, body temperature, skin impedance, and sweat rate; The inertial sensor on the smart steering wheel is used to detect the angular velocity and acceleration of the steering wheel, collect data on the steering wheel rotation and vibration, and report the detected data to the central processing module in real time; 3) Driver emotion recognition based on audio and video When a smart terminal is connected for the first time, it is necessary to install the relevant software first, and then connect to the smart steering wheel wirelessly. The smart steering wheel saves the smart terminal information to the data storage module. Each time the vehicle is started, the smart steering wheel detects the surrounding smart terminal signals, obtains the last connected smart terminal and automatically connects; if the last connected smart terminal is not detected, the smart terminal that was connected last is obtained in turn and tries to connect, and so on, until a smart terminal saved in the data storage module is successfully connected and used as a computing power auxiliary provider; if all smart terminals in the data storage device fail to connect, the auxiliary computing power is marked as unreachable; After the smart steering wheel finds that it has been successfully connected to the smart terminal, it transmits the video collected by the camera and the audio information collected by the sound sensor to the smart terminal in real time through the wireless transceiver module; the smart terminal first cleans and enhances the data, including noise reduction of the original voice and image enhancement technology to improve the picture clarity of the video data, and then aligns the obtained visual data with the voice data through a multimodal large model, and finally extracts the driver's facial expression and voice feature signals from the processed data; After the extraction is completed, these feature signals are input into the preset Transformer-based emotion classification model in the intelligent terminal to calculate the real-time emotion recognition results of the driver during the actual driving process, and finally the emotion recognition results are sent to the central processing module; When the intelligent terminal fails to extract the driver's facial expression and voice feature information, there is no need to send the driver's emotion recognition result to the central processing module; 4) The central processing module evaluates the driver's status After the vehicle is started, the central processing module retrieves the current driver's personal feature file from the data storage module and stores it in the memory; The central processing module receives the detection data uploaded by the electrocardiogram, skin conduction and inertial sensors in real time, reduces the noise of these raw data, performs baseline correction according to the baseline information in the driver's personal feature file, generates normalized signals, and uses a sliding window to align the time between the multimodal time series data of these signals and the real-time driver emotion recognition results submitted by the smart terminal. These signals are then input into the preset driver status assessment model for multimodal composite feature calculation, and the obtained data are feature spliced ​​into multidimensional vectors. Finally, status judgment and classification are performed to obtain the real-time driver status assessment results.

2. The method according to claim 1, characterized in that: The noise reduction in step 4) is specifically as follows: The ECG signal uses power frequency noise suppression, the skin electrode signal uses low-pass filtering to eliminate motion artifacts, and then decomposes into Phasic transient response and Tonic base level, and the inertial sensor data uses Kalman filtering to eliminate noise.

3. The method according to claim 1, characterized in that: In step 4), the sliding window parameter is a 10-second window with a step length of 1 second.