Driving state monitoring and feedback method and system based on multi-modal human factors intelligent data analysis, and edge computing terminal device

Abstract

A driving state monitoring and feedback method and system based on multi-modal human factors intelligent data analysis, and an edge computing terminal device. The method comprises: receiving multi-modal human factors data of a tested driver that is collected in real time (S110); pre-processing the multi-modal human factors data, wherein the pre-processing comprises denoising processing and data normalization processing (S120); sending the pre-processed multi-modal human factors data into a pre-trained first state recognition model, so as to obtain a real-time recognized driver state, wherein driver states include a normal state and abnormal states, and the types of the abnormal states include a plurality of states such as a fatigue state, a distracted state and an angry state (S130); and when it is recognized that the driver state is an abnormal state, generating, for different categories of abnormal states, driving state feedback instructions to a driving intervention system, such that the driving intervention system performs state feedback adjustments on the driver on the basis of the received driving state feedback instructions (S140). The method and system can recognize different driving states of a driver in real time and then perform processing on the basis of different driving states, thereby avoiding the occurrence of traffic accidents.

Description

Driving status monitoring and feedback method and system based on multimodal human factors intelligent data analysis, and edge computing terminal device

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the Chinese patent application with application number 202311775914.0 and application date of December 21, 2023, the Chinese patent application with application number 202311782765.0 and application date of December 22, 2023, the Chinese patent application with application number 202311865457.4 and application date of December 29, 2023, and the Chinese patent application with application number 202311659868.8 and application date of December 05, 2023, and claims the priority of the Chinese patent application, and the entire contents of the Chinese patent application are incorporated into this application by introduction. Technical Field

[0003] The present application relates to the field of driving technology, and in particular to a method and system for driving status monitoring and feedback based on multimodal human factors intelligent data analysis, and an edge computing terminal device. Background Art

[0004] With the rapid growth of road traffic demand, traffic safety issues are becoming increasingly prominent. As the primary controllers, regulators, and information decision-makers of the road traffic system, drivers have long been a key research focus for road traffic safety. Driving status monitoring and feedback for vehicle drivers (e.g., identifying issues such as fatigue, distracted driving, and angry driving) is a key research direction in the field of road traffic safety driving intervention. However, how to provide driving status monitoring and feedback for vehicle drivers is a pressing technical challenge.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a method and system for driving status monitoring and feedback based on multimodal human factors intelligent data analysis, and an edge computing terminal device.

[0007] One aspect of the present application provides a method for driving state monitoring and feedback based on multimodal human factors intelligent data analysis, the method comprising the following steps: receiving multimodal human factors data of a test driver collected in real time; preprocessing the multimodal human factors data, the preprocessing including noise reduction and data normalization; sending the preprocessed multimodal human factors data into a pre-trained first state recognition model to obtain a real-time recognized driver state; the driver state includes a normal state and multiple abnormal states, the categories of the abnormal states including one or more of fatigue state, distracted state and angry state; when the driver state is identified as an abnormal state, generating driving state feedback instructions for different categories of abnormal states to a driving intervention system, so that the driving intervention system performs state feedback adjustment on the driver based on the received driving state feedback instructions.

[0008] Another aspect of the present application provides a driving status monitoring and feedback system based on multimodal human factors intelligent data analysis, the system includes a state recognition subsystem and a driving intervention subsystem, the state recognition subsystem is used to receive multimodal human factors data of the test driver collected in real time, preprocess the multimodal human factors data, and send the preprocessed multimodal human factors data into a pre-trained first state recognition model to obtain a real-time recognized driver state; wherein, the human factors data includes multiple types of EEG data, heart rate data, skin conductance data, respiratory data, near-infrared data, blood oxygen data, blood pressure data and skin temperature data, the preprocessing includes noise reduction processing and data normalization processing, the driver state includes a normal state and multiple abnormal states, and the category of the abnormal state includes fatigue The driving intervention subsystem is used to generate driving state feedback instructions to the driving intervention system for different categories of abnormal states when it identifies the driver's state as an abnormal state, so that the driving intervention system can perform state feedback adjustment on the driver based on the received driving state feedback instructions; wherein, the first state recognition model collects the baseline human factors data and demographic data of the test driver, retrieves the driver-related data within a preset similarity range from the baseline human factors data and demographic data of the test driver in the baseline state database, uses the retrieved baseline human factors data of the driver and the driver state data stored in the baseline state database as the training set, and uses the pre-selected normal state and multiple abnormal states as labels, and then iteratively trains to obtain the obtained data.

[0009] Another aspect of the present application provides an edge computing terminal device, including a processor and a memory, wherein the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps of the method described in any one of the above embodiments.

[0010] Another aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method described in any one of the above embodiments when the program is executed by a processor.

[0011] The method and system for driving status monitoring and feedback based on multimodal human factors intelligent data analysis and the edge computing terminal device proposed in this application can use the pre-trained first state recognition model to process the pre-processed multimodal human factors data collected in real time, distinguish the driver's normal state and abnormal state through classification labels, identify the driver's different driving states, and then process different driving states to avoid traffic accidents.

[0012] Additional advantages, purposes, and features of the present application will be described in part in the following description and will become apparent to those skilled in the art upon study of the following or may be learned from practice of the present application. The purposes and other advantages of the present application may be achieved and obtained by the structures specifically pointed out in the specification and drawings.

[0013] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present application are not limited to the above specific description, and the above and other purposes that can be achieved by the present application will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0015] FIG1 is a flow chart of a method for driving status monitoring and feedback based on multimodal human factors intelligence data analysis in one embodiment of the present application;

[0016] FIG2 is a flowchart of EEG data preprocessing in one embodiment of the present application;

[0017] FIG3 is a schematic diagram of the architecture of a driving status detection and warning system according to an embodiment of the present application;

[0018] FIG4 is a flow chart of a driving state detection and warning method according to an embodiment of the present application;

[0019] FIG5 is a flowchart of a method for training a cognitive distraction state recognition model according to an embodiment of the present application;

[0020] FIG6 is a flow chart of another driving state detection and warning method according to an embodiment of the present application;

[0021] FIG7 is a flow chart of a method for detecting driver fatigue in a human-induced intelligent cockpit according to an embodiment of the present application;

[0022] FIG8 is a schematic diagram of driver fatigue status identification and monitoring according to an embodiment of the present application;

[0023] FIG9 is a flowchart of a method for predicting a driver's driving state based on multimodal data according to an embodiment of the present application;

[0024] FIG10 is a schematic diagram of a driving state prediction system according to an embodiment of the present application;

[0025] FIG11 is a schematic diagram of a process for preprocessing EEG data according to an embodiment of the present application;

[0026] FIG12 is a flowchart of obtaining target prediction results according to one embodiment of the present application;

[0027] FIG13 is a flow chart of obtaining a first lateral velocity offset value according to one embodiment of the present application;

[0028] FIG14 is a flow chart of obtaining predicted values ​​of kinematic information according to one embodiment of the present application;

[0029] FIG15 is a flow chart of obtaining a second lateral velocity offset value according to one embodiment of the present application;

[0030] FIG16 is a schematic diagram of the system structure for driving status monitoring and feedback based on multimodal human factors intelligent data analysis in one embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] This application provides a method and system for driving status monitoring and feedback based on multimodal human factors intelligent data analysis, as well as an edge computing terminal device. It identifies the driver's status through multimodal human factors data, and further combines multiple elements such as the driver, vehicle, environment, and road conditions to perform human-vehicle-road loop monitoring and feedback.

[0034] FIG1 is a flow chart of a method for driving status monitoring and feedback based on multimodal human factors intelligence data analysis in one embodiment of the present application. The method includes the following steps:

[0035] Step S110: receiving multimodal human factors data of the test driver collected in real time; the human factors data includes multiple types of EEG data, heart rate data, skin conduction data, respiratory data, near-infrared data, blood oxygen data, blood pressure data and skin temperature data.

[0036] Step S120: preprocessing the multimodal human factor data, including noise reduction and data normalization.

[0037] Step S130: The pre-processed multimodal human factors data is fed into a pre-trained first state recognition model to obtain a real-time recognized driver state; the driver state includes a normal state and multiple abnormal states, and the categories of abnormal states include one or more of a fatigue state, a distracted state, and an angry state.

[0038] In specific implementations, different modalities of human factor data are mapped to driver states with different labels. For example, signals in different regions or bands of EEG data can be used to determine whether the driver is fatigued, distracted, or angry. Heart rate data can be used to analyze whether the driver is angry, breathing data can be used to analyze whether the driver is fatigued, and skin conduction data and near-infrared data can be used to analyze whether the driver is distracted. The above are merely examples, and the present application is not limited thereto.

[0039] Step S140: When the driver's state is identified as an abnormal state, driving state feedback instructions are generated for different types of abnormal states to the driving intervention system, so that the driving intervention system performs state feedback adjustment on the driver based on the received driving state feedback instructions.

[0040] In some embodiments of the present application, the method also includes a pre-training process of the first state recognition model. During the pre-training process of the first state recognition model, the baseline human factors data and demographic data of the test driver are first collected, and the driver-related data within a preset similarity range compared with the baseline human factors data and demographic data of the test driver are retrieved from the baseline state database. The retrieved baseline human factors data of the driver and the driver state data stored in the baseline state database are used as training sets, and the pre-selected normal state and multiple abnormal states are used as labels. The first state recognition model is obtained through iterative training.

[0041] Specifically, the execution subject of the above method can be a vehicle computer. For example, the vehicle computer is connected to various collection devices, and the collection devices are worn on the driver or installed inside the vehicle, so as to collect the driver's multimodal human factors data.

[0042] The method and system for driving status monitoring and feedback based on multimodal human factors intelligent data analysis proposed in this application can use a pre-trained first state recognition model to process the pre-processed multimodal human factors data collected in real time, distinguish the driver's normal state and abnormal state through classification labels, identify the driver's different driving states, and then process different driving states to avoid traffic accidents.

[0043] In some embodiments of the present application, the noise reduction processing step includes: applying one or more of wavelet filtering, Kalman filtering and empirical mode decomposition to the electrocardiogram data; using the value of a time window as the baseline value and the value of a time window as the maximum response value to perform noise reduction on the electrocutaneous data.

[0044] By adopting this application embodiment, purer multimodal human factors data can be obtained.

[0045] In some embodiments of the present application, the preprocessing of EEG data included in the human factor data includes whole-brain average reference, filtering, ICA analysis, and time-frequency analysis to extract the target EEG data band, filtering the EEG data and removing artifacts, identifying and removing blink segments and / or electromyographic segments through ICA analysis, and extracting the target EEG data band through time-frequency analysis; wherein the filtering includes high-pass filtering, low-pass filtering, notch filtering, and band-pass filtering. Among them, one of the analytical methods of time-frequency analysis is power spectral density analysis.

[0046] By adopting the embodiment of the application, purer EEG data can be obtained.

[0047] Specifically, in one embodiment of the present application, the preprocessing of the EEG data included in the human factor data further includes a data cleaning step before whole-brain averaging reference, specifically including: removing artifacts and noise through data cleaning. The preprocessing of the EEG data included in the human factor data further includes: deleting useless channels and removing artifacts based on deep learning methods before ICA analysis.

[0048] By adopting the embodiment of the application, artifacts in the EEG data contained in the human factor data can be removed to obtain purer EEG data, thereby better identifying and providing feedback on the driver's driving status.

[0049] In some embodiments of the present application, the multimodal human factors data further includes image data of the test driver's head. After preprocessing the multimodal human factors data, the method further includes:

[0050] Based on image recognition technology, the image data of the test driver's head is analyzed to obtain the driver's eye movement characteristics, head movement characteristics and facial expression characteristics. The driver's eye movement characteristics and head movement characteristics are analyzed by combining gaze tracking technology, and the facial expression characteristics are analyzed by expression recognition technology to obtain real-time recognition of the driver's status.

[0051] By adopting this embodiment of the application, the angle of real-time identification of the driver's status can be expanded, and the driver's status can be verified from multiple angles to obtain more accurate driver status identification results.

[0052] In some embodiments of the present application, after preprocessing the multimodal human factors data, the method further includes: (1) extracting features from the preprocessed multimodal human factors data to obtain features related to driving behavior; and (2) tagging the features related to driving behavior. When data related to the tag is detected, analysis is performed based on the features related to driving behavior to obtain a real-time identification of the driver's state. The features related to driving behavior include one or more of heart rate, blood pressure, and respiratory rate.

[0053] By adopting this application embodiment, the driver status can be managed in a more detailed manner through label management, thereby supporting more diverse driving status identification and feedback methods.

[0054] In some embodiments of the present application, the first state recognition model is a classification model, and the type of the classification model includes at least any one of a vector machine model, a decision tree model, and a naive Bayes model.

[0055] In some embodiments of the present application, the driver state also includes the driver's cognitive state. After preprocessing the multimodal human factors data, the method also includes: obtaining the driver's operational data including the speed and force of braking during driving, calculating the driver's operating habits based on historical statistics of the driver's historical operational data and setting a driver's cognitive state recognition threshold. When the driver's operational data collected in real time during driving exceeds the driver's cognitive state recognition threshold, it is determined that the driver's cognitive state is abnormal.

[0056] By adopting this application embodiment, the driver's operating habits can be counted by collecting and analyzing the driver's historical operating data, and the driver's driving state abnormality can be identified by identifying abnormal operating conditions during the driving operation process, providing a new way to identify the driver's state from another perspective.

[0057] Furthermore, based on the above driver state identification method, the driver states obtained from different channels are weighted, but if the weighted driver state exceeds a threshold, a driver state identification result, i.e., a driving state, is finally generated.

[0058] In some embodiments of the present application, the driving intervention system includes a voice module and a seat. The steps of the driving intervention system performing state feedback adjustment on the driver based on the received driving state feedback instructions include: (1) using the voice module to broadcast a reminder or play music to soothe the driver's emotions; (2) automatically adjusting the seat angle to remind the driver to adjust the driving state. It should be noted that in today's intelligent driving system, many smart devices on the vehicle (such as smart seats, smart speakers, etc.) are connected to the vehicle computer, and the vehicle computer can issue instructions to them to implement pre-programmed functions. This application does not specifically limit this.

[0059] In some embodiments of the present application, demographic data includes multiple types of the driver's age, gender, height, and weight, and the baseline status database includes corresponding baseline human factor data and driver status data using the driver ID as a tag. The present application is not limited to this, and the above is only an example.

[0060] In some embodiments of the present application, the method is performed in a virtual driving environment simulated using virtual reality technology. The virtual driving environment includes a test driver and a driving vehicle. The virtual driving environment also includes pedestrians and other vehicles, and the pedestrians and other vehicles respond to scene changes in the virtual driving environment.

[0061] In some embodiments of the present application, when the virtual driving environment belongs to a multi-task scenario, the multimodal human factors data includes eye movement data, and the method further includes: (1) first calibrating the baseline of the collected multimodal human factors data; (2) extracting and training the multimodal human factors data during the measurement process; (3) identifying eye movement interest areas based on gaze tracking technology, and extracting relationship features between each eye movement interest area and the driving task scenario based on the SEEV model.

[0062] The virtual driving environment includes a driver's cab and virtual reality road conditions, allowing for the simulation and construction of any desired driving environment, thereby obtaining more accurate physiological information. In one embodiment of the present application, to more accurately obtain physiological information, data is not only collected from the driver's side, but also the reactions of different characters, such as real pedestrians and drivers of other vehicles, are added to the virtual environment. For example, in a scenario where a pedestrian is crossing an intersection and the driver stops, with a few seconds before the pedestrian's side changes direction, the driver's physiological signals, including EEG, EMG, heart rate, blood pressure, and respiratory rate, are collected to obtain their cognitive state corresponding to the current situation. The pedestrian's physiological and motion information is simultaneously detected, such as a gesture to interact with the driver to remind them "I want to cross the road, please wait a moment." The driver's physiological signals, emotional state, and other information are also monitored to determine their driving status and habits. This is a single scenario example and is not specifically limited in this application.

[0063] Furthermore, when the driving task scenario is a multi-task scenario, the relationship features between each area of ​​interest and the driving task scenario are extracted based on the SEEV (Salience S, Effort Value E, Expected Value E, Information Value V) model. The relationship features include: the importance of the driving task scenario (task priority), the relevance of the area of ​​interest and the driving task scenario (relevance), and the amount of information in the area of ​​interest (bandwidth, BW). These are further combined into the visual attention to different areas of interest (AOI) in the rule base under the driving task scenario, and then a rule base is formed to store the (scene, AOI, visual attention) triples. The formula is expressed as:

[0064] By adopting the embodiment of this application, the driver's distraction state can be verified from another perspective based on eye movement data and gaze tracking technology.

[0065] In the above embodiment of the present application, it is also necessary to first calibrate the baseline of the collected multimodal human factors data, and realize the extraction and training of eye movement and physiological characteristics during the measurement process. The specific measurement process is: the human factors data is used to generate characteristic layer physiological indicators through discriminant analysis, the characteristic layer physiological indicators are used to represent the normal or abnormal state of the driver through binary classification data, and the significant changes in the characteristic layer physiological indicators are obtained. The significant changes can be set to the mean ± standard deviation within 4 seconds, or can be customized to other values. This is only an example, and the present application is not limited to this. Among them, the discriminant analysis method can use Bayesian decision theory, linear discriminant function, nonlinear discriminant function and support vector machine to perform discriminant analysis of human factors data.

[0066] Furthermore, when the driver is in a normal state, the eye movement data is used to generate feature-layer eye movement indicators based on the constructed rule base, and the feature-layer eye movement data is matched with the theoretical degree of the eye movement interest area extracted by the rule base to obtain the first evaluation score of the driver in the driving task scenario. For example: correlation coefficient method - eye movement data in the interest area and theoretical attention in the rule base, and another example: σ1(A1,A'1)=0.76, σ2(A2,A'1)=0.96, σ3(A3,A'1)=0.27; among them, A'1 represents the theoretical attention vector of different interest areas, A1, A2, A3 represent the vectors of feature-layer eye movement indicators in different interest areas, σ1, σ2, σ3 represent the first evaluation scores in different interest areas, which are further used to judge the lack of concentration.

[0067] Based on information captured from the pedestrian-vehicle-road loop, a Transformer Encoder is used to encode vehicle data. Two fully connected (FC) layers are added after the Transformer Encoder to ensure that the model output length and dimensions match the input data. The feature extraction model training process is as follows: For the various vehicle inputs, the time dimension is randomly masked by a factor of 0.1. Subsequently, the channel dimension is also randomly masked, masking information from one or two channels. The masked data is fed into the Transformer Encoder and FC layers. The output is compared with the unmasked input data, and the loss is calculated and backpropagated. After training, the model output can perfectly restore the masked portions of the input data. During this process, the Transformer Encoder learns effective features from the data. These features are then combined with other modal data, such as human factors data, emotional data, cognitive data, and environmental data, for classification tasks. Classification tasks include, but are not limited to, driving status and driving habits. Based on the acquired driving habits and status, the model monitors and provides feedback.

[0068] Feedback on driving actions is provided based on the driver's detected emotional state. Emotional state affects the driver's attention and alertness. When a driver is in a positive emotional state, their attention and alertness are enhanced, making it easier for them to detect dangerous situations and emergencies on the road and respond more quickly. Conversely, when a driver is in a negative emotional state, their attention and alertness decrease, making them less sensitive to road conditions, leading to slow reactions or misjudgments. Therefore, when a driver's mood is detected, timely reminders are provided to improve their focus. Alternatively, positive responses can be given to improve the driver's mood.

[0069] Emotional state also influences a driver's decision-making ability: Emotional state has a significant impact on a driver's decision-making ability. When drivers are in a positive emotional state, they are more likely to make wise and rational decisions and have more accurate judgments about driving behavior and road conditions. Conversely, when drivers are in a negative emotional state, they may make impulsive decisions and misjudge driving behavior and road conditions, thereby increasing the risk of traffic accidents.

[0070] Because emotional states influence drivers' driving behavior, emotional states can affect drivers' driving behavior and operations. When drivers are in a positive emotional state, they are more likely to maintain stable driving behavior, comply with traffic rules and instructions, and control their vehicles more accurately. Conversely, when drivers are in a negative emotional state, they may exhibit erratic driving behavior, violate traffic rules and instructions, and may control their vehicles less accurately, increasing the risk of traffic accidents.

[0071] The processing of raw EEG data in this application is relatively complex and includes multiple, more detailed steps. Figure 2 is a flowchart of EEG data preprocessing in one embodiment of this application. First, the obtained raw EEG data is averaged for a whole-brain reference. Then, filtering is performed to remove less necessary frequency bands while retaining frequency bands important for driving state analysis. Independent principal component analysis is then performed, and then power spectral density analysis can be performed to extract the target EEG data bands for driving state recognition.

[0072] In one embodiment of the present application, the raw EEG data is subjected to whole-brain average reference, and band-stop filtering is performed: 0.5-45Hz. ICA principal component analysis is used to identify and remove blink segments and electromyographic segments. Finally, time-frequency analysis is performed to extract power spectral density features including 3-7Hz for theta waves, 8-12Hz for alpha waves, and 13-30Hz for beta waves.

[0073] The basic steps of EEG data preprocessing include the following: (1) Locating channel positions: Based on the electrode arrangement system, each channel is assigned a coordinate position on the scalp to facilitate spatial analysis and visualization. (2) Removing useless channels. (3) Setting a threshold to remove segments with excessive artifacts or outliers to improve data quality.

[0074] When locating the channel, we first use mathematical cleaning to remove artifacts and noise to distinguish clean signals. Choosing a suitable filter based on the frequency range of the artifacts can effectively reduce artifacts in the raw EEG data. For example, a high-pass filter setting of 0.1Hz and a low-pass filter setting of 30Hz can be used. Furthermore, to eliminate mains interference, a notch filter can be selected to remove 50Hz interference. Furthermore, when we are interested in a specific signal frequency range, we can use bandpass filtering. For example, if we are interested in alpha waves, we can use bandpass filtering to retain the 8-13Hz signal frequency band for analysis.

[0075] During the process of removing useless channels, we remove unnecessary channels, such as bilateral mastoid and electrooculogram channels, based on the experimental objectives and design. Combining other multimodal biological data helps us detect and distinguish artifacts in EEG data, such as combining electrooculogram and eye movement measurements. This can help identify blinks and eye movements in EEG signals, identify ECG artifacts in EEG signals by combining them with ECG measurements, and identify head movement artifacts in EEG signals by combining them with acceleration measurements. Multiple electrodes are configured as a single channel, forming a single lead, which can generally be arranged in different channel configurations, such as 16 or 32. If some electrodes within a single lead are not properly placed on the scalp, they cannot accurately collect brain neurophysiological signals, resulting in a bad lead. Causes of a bad lead include channel failure, misplacement, poor contact, series connection, and channel saturation. During data analysis, the first step is to identify and remove bad lead electrodes. This is typically done by comparing them with signals from normal electrodes in other channels to filter out any bad lead signals.

[0076] When setting a threshold to remove segments containing excessive artifacts or outliers, data preprocessing typically involves removing some noise by setting a threshold. This identifies outlier bands and selects appropriate filters based on the artifact frequency range, effectively reducing artifacts in the raw EEG data. For example, a high-pass filter can be set at 0.1Hz and a low-pass filter at 30Hz. Furthermore, to eliminate mains interference, a notch filter can be selected to remove 50Hz interference. Bandpass filtering can also be used when a specific signal frequency range is of interest. For example, if alpha waves are of interest, a bandpass filter can be used to retain the 8-13Hz signal frequency range for analysis. Even after considering all of the above factors, the collected EEG signal may still contain significant noise. This noise can be addressed by using filtering and mathematical algorithms to decompose the signal and noise, improving the signal-to-noise ratio.

[0077] Furthermore, deep learning-based methods can be used to train AI algorithms using existing datasets to automatically detect and reject artifacts in recordings (ideally in real time) to reduce the mean square error between the target signal and the predicted signal.

[0078] The commonly used data preprocessing algorithm is as follows:

[0079] (1) Data capture, for example, capturing a 0.5s time window as the input signal;

[0080] (2) Perform fast Fourier transform on the EEG signal to extract its frequency domain information;

[0081] (3) Using azimuth equidistant projection in the Cartesian coordinate system to project the distribution of EEG electrode points in three-dimensional space onto a two-dimensional image, thus converting the EEG signal into a multispectral image with a spatial topological structure;

[0082] (4) Through the above three steps, the information of the time dimension, frequency domain dimension and spatial dimension of the EEG signal is fully mined.

[0083] In addition, based on the above preprocessing steps, the SNS method, RANSAC method and Cross-Validation method can also be used. These preprocessing methods can improve the signal-to-noise ratio (SNR) of the EEG signal to a certain extent.

[0084] After acquiring EEG signals, further analysis can be performed based on brain topography to determine which frequency bands are most activated within a specific time period, and in which brain regions. For example, when a driver is exposed to a terrifying image, beta waves are more strongly activated, reflecting tension, and are located in the visual cortex in the occipital lobe (the back of the brain). Different EEG signals and their locations determine the individual's current mood, perception, distraction, fatigue, and workload, leading to different responses during actual driving.

[0085] Based on the recognition of driving status based on the state video model recorded in steps S130-S140, the method may also include: driving behavior based on image recognition technology. For example, the driver's eye movement characteristics, head movement characteristics, facial expressions, etc. are used to explore changes in the driver's psychological activities, so as to judge the driver's driving fatigue, driving distraction, etc. This method can determine whether the driver is distracted or driving fatigued by identifying and analyzing eye features. The characteristics of eye movements and rotation angles are extracted through a neural network algorithm. Combined with the angles of the two sides of the rearview mirrors of different related vehicles and the driver's position, it is confirmed whether the driver is normally observing the rearview mirror during driving or is distracted.

[0086] Specifically, the acquired eye movement data and the relationship between the eye rotation angle and the set angle range of the rearview mirror can be combined with the EEG signal to confirm the driving status, thereby performing driver status intervention.

[0087] Alternatively, driving behavior can be assessed based on gaze tracking technology. Specifically, the driver's gaze direction and gaze point can be tracked to determine whether the driver is distracted or fatigued. This method can determine whether the driver is focusing on the road ahead by analyzing the driver's eye movements and gaze direction, thereby determining whether the driver is distracted or fatigued.

[0088] In one embodiment of the present application, driving status can also be identified based on physiological signals. For example, by monitoring the driver's physiological signals, such as heart rate and blood pressure, it is possible to determine whether the driver is distracted or fatigued. This method can determine whether the driver is fatigued or distracted by analyzing the driver's physiological signals. A preset threshold is used to compare the baseline value.

[0089] Furthermore, deep learning algorithm models that capture driving behavior based on physiological signals can also employ feature extraction algorithms to extract features related to driving behavior from the driver's physiological signal data. These features may include heart rate, blood pressure, respiratory rate, and so on. These extracted features are then tagged and managed. When data associated with a tag is detected, these data features (carrying the tag) are analyzed to determine the driver's driving and emotional state. Once the driver's driving and emotional states are captured, they are categorized and managed to form a database with corresponding settings for different driving habits. If the set driving habit level is not met, a reminder is given and improvement plans are suggested.

[0090] The deep learning algorithm model can be a classification algorithm model. Based on the classification algorithm model, the extracted features are classified into different driving behaviors, such as normal driving, distracted driving, and fatigued driving. Common classification algorithms include support vector machines, decision trees, and naive Bayes.

[0091] In a specific embodiment of the present application, since the driver's cognitive state affects driving behavior and decision-making ability, such as the driver's judgment and reaction speed, the driver's driving ability and state can be understood by assessing the person's cognitive state. This requires obtaining data during the driver's operation, including the speed and force of braking, to calculate their operating habits.

[0092] Furthermore, the embodiments of the present application can also evaluate the driving status from three dimensions: driver status, vehicle status, and road environment status, which helps to improve the accuracy of driving status evaluation, issue risk warnings based on the driving status, and improve driving safety.

[0093] The driving status detection and warning method according to an embodiment of the present application will now be described with reference to FIG. 4 to FIG. 6 .

[0094] Figure 3 is an architecture diagram of a driving status detection and warning system according to an embodiment of the present application. As shown in Figure 3, the driving status detection and warning system includes a human-factor intelligent monitoring platform and a real-time monitoring terminal. The human-factor intelligent monitoring platform can monitor one real-time monitoring terminal, or can monitor multiple real-time monitoring terminals simultaneously. When the real-time monitoring terminal is started, the human-factor intelligent monitoring platform and the real-time monitoring terminal communicate with each other through a network protocol, such as HTTP (Hypertext Transfer Protocol) or TCP (Transmission Control Protocol). The real-time monitoring terminal uploads the terminal information to the human-factor intelligent monitoring platform, and the human-factor intelligent monitoring platform matches the uploaded terminal information with the terminal information pre-stored in the platform, thereby determining the monitoring terminal that has been successfully matched. The human-factor intelligent monitoring platform can perform real-time visualization and processing of the warning status of the successfully matched real-time monitoring terminal, and can also perform visualization, data statistics and analysis of the historical information of the warning status of the real-time monitoring terminal.

[0095] Each real-time monitoring terminal includes multiple detection devices, including physiological signal detection devices, video signal detection devices, vehicle signal detection devices, motion capture detection devices, and so on. Physiological signal detection devices are used to detect multimodal physiological data. Multimodal physiological data refers to driver status data collected by multiple sensors. These sensors can be physiological sensors that capture the driver's physiological data while driving, such as eye tracking devices, electrodermal conduction, electrocardiogram (ECG), respiratory, and electroencephalogram (EEG) sensors. Data collected by eye tracking devices includes, but is not limited to, pupil diameter, blink frequency, fixation duration, and number of fixations. Detection devices are used to detect the driver's status, vehicle status, and road environment. The real-time monitoring terminal can simultaneously activate each detection device with a single button, or individually. The sensors for collecting multimodal physiological data can be integrated into a single terminal device or comprised of multiple sensors. A terminal device integrating multiple sensors can be deployed in a local vehicle. Alternatively, the sensors can be installed in the local vehicle, with the terminal device performing analysis and processing located on the server side.

[0096] The real-time monitoring terminal detects the signal and sends it to the cloud server. The cloud server uses a neural network algorithm to determine whether the detected signal is normal—that is, whether it is high-quality and valid. If it is not, the driver is prompted through voice or image to adjust the equipment to obtain a normal detection signal. After the real-time monitoring terminal is connected to the detection equipment, to ensure data quality, the raw data is stored locally on the client. After monitoring is completed, the data is uploaded to the cloud server to form a historical database.

[0097] FIG4 is a flow chart of a driving state detection and warning method according to an embodiment of the present application, which specifically includes the following steps:

[0098] Step S401: Acquire driver status detection data, vehicle status detection data, and road environment status detection data uploaded by the monitoring terminal.

[0099] In this embodiment, multiple detection devices detect and obtain driver status detection data, vehicle status detection data, and road environment status detection data during the driving process. After the monitoring terminal is connected to the detection devices, the monitoring terminal uploads this data to the human factors intelligent monitoring platform. The driver status detection data includes not only multimodal physiological data, but also other behavioral data of the driver during driving, data used to express the driver's emotions or feelings, and data used to determine the driver's perception ability. The perception ability determination data can be obtained by detecting the driver's feedback during driving through different operations. The driver's status during driving is determined based on the above detected data.

[0100] In some embodiments, the driver state includes a fatigue state, a distracted state, and an emotional state.

[0101] Optionally, fatigue states include mild fatigue, moderate fatigue, and severe fatigue. The fatigue state is determined by analyzing data detected by physiological signal detection equipment or video signal detection equipment. For example, EEG (Electroencephalogram) data and PSD (power spectral density) data can be obtained through EEG detection, or changes in the driver's pupil state or blinking state can be detected through in-vehicle camera detection. These data or state changes can be analyzed to determine the driver's fatigue level during driving.

[0102] In some embodiments, distraction states include behavioral distraction and cognitive distraction. Behavioral distraction includes making phone calls, smoking, chatting, and not looking forward for a long time. Cognitive distraction includes low distraction, moderate distraction, and high distraction.

[0103] Optionally, behavioral distraction is obtained by analyzing data detected by a video signal detection device, such as detecting dangerous driving behaviors of the driver due to distraction through a behavioral distraction status detection device (in-vehicle camera, video monitor, etc.); cognitive distraction is obtained by analyzing data detected by a physiological signal detection device, such as detecting HRV (Heart Rate Variability), HR (Heart Rate), IBI (Inter-Beat Interval, i.e., RR interval), SCR (Skin conductance response) data, etc. through a PPG (photoplethysmography) wireless pulse sensor or an EDA (electrodermal activity) wireless electrical skin sensor, and analyzing these data to obtain the degree of cognitive distraction.

[0104] In some embodiments, emotional states include emotional intensity and emotional valence. Emotional intensity includes abnormal and normal emotions; emotional valence includes positive emotions and negative emotions. Positive emotions refer to positive emotions such as happiness and excitement, while negative emotions refer to negative emotions such as anger, irritation, and sadness.

[0105] Optionally, the intensity of emotion is obtained by analyzing data detected by a physiological signal detection device, for example, by using a PPG (photoplethysmography) wireless pulse sensor or an EDA (electrodermal activity) wireless electrodermal sensor to detect HRV (Heart Rate Variability), HR (Heart Rate), IBI (Inter-Beat Interval, i.e., RR interval), SCR (Skin conductance response) data, etc., and analyzing these data. For example, whether the emotion is abnormal is judged based on the change in the SCR (Skin conductance response) value per minute; the valence of emotion is obtained by analyzing data detected by a video signal detection device (such as a facial expression detection device). For example, whether the driver's emotional valence is positive or negative is judged by detecting changes in the driver's facial expression.

[0106] In some embodiments, the vehicle state includes a normal driving state and an abnormal driving state. The vehicle state is determined by analyzing data detected by vehicle signal detection equipment. Specifically, Vehub detects the vehicle's operating speed, steering wheel data, pedal data, GPS (Global Positioning System) data, etc., and analyzes the collected data, including lateral acceleration analysis, center of mass sideslip angle constraint analysis, friction circle constraint analysis, tire sideslip angle analysis, and built-in advanced driving normal driving performance analysis, to determine whether the vehicle is in normal driving state.

[0107] Furthermore, real-time and post-coding can be performed on driving behaviors, which can be analyzed and statistically analyzed. Driving behaviors include driving pedal behavior (brake, accelerator), steering behavior (left turn, right turn), lane change behavior (left lane change, right lane change, reversing, U-turn, driving in the right lane, driving in the left lane), driving speed (high speed, medium speed, low speed, parking or idling), longitudinal acceleration (strong speed, normal acceleration, weak acceleration), and steering wheel behavior (clockwise, counterclockwise), thereby determining whether the vehicle is driving normally. For example, by analyzing driving pedal behavior, it can be determined whether the driver frequently brakes suddenly. If the driver frequently brakes suddenly, it indicates that the current vehicle is driving abnormally.

[0108] In some embodiments, the road environment status includes no vehicle or pedestrian ahead, vehicle collision ahead, lane departure, vehicle too close, frequent lane changes, other vehicles cutting in, vehicle ahead speeding, and pedestrian collision. Vehicle collision ahead, pedestrian collision, vehicle too close, and frequent lane changes include presence and absence; lane departure includes left deviation and right deviation; speeding levels include low, medium, and high; and other vehicles cutting in include left and right deviation.

[0109] Optionally, the road environment state is detected by an ADAS (Advanced Driving Assistance System). Optionally, the road environment state may also include environmental data detected by a third party, such as weather, temperature, humidity, etc.

[0110] Step S402 : Analyze the driver state detection data, the vehicle state detection data, and the road environment state detection data to obtain a driving state evaluation result.

[0111] Specifically, the driver state detection data, vehicle state detection data and road environment state detection data are respectively input into the corresponding state recognition model to identify the driver state, vehicle state and road environment state; and the driving state evaluation result is obtained by combining the driver state, vehicle state and road environment state.

[0112] Optionally, if the driver state is slightly fatigued, the vehicle state is normal driving state, and the road environment state is no vehicle or pedestrian ahead, the driving state assessment result is the first driving risk state;

[0113] If the driver is moderately fatigued, the vehicle is in an abnormal driving state, and the road environment is clear of vehicles or pedestrians, the driving state assessment result is a second driving risk state.

[0114] If the driver is in a severely fatigued state, the vehicle is in an abnormal driving state, and the road environment is in a state where another vehicle is cutting in, the driving state assessment result is the third driving risk state.

[0115] It should be noted that the driving state evaluation of this application may also include other indicators, and the driving state evaluation may also be performed based on the driver state detection data and the vehicle state detection data, or based on the vehicle state detection data and the road environment state detection data, or based on the driver state detection data and the road environment state detection data. For example, if the driver state is slightly fatigued and the vehicle state is an abnormal driving state, the driving evaluation result is the fourth driving risk state, or, if the vehicle state is an abnormal driving state and the road environment state is lane departure, the driving evaluation result is the fifth driving risk state. The above is only an exemplary description, and this application does not limit the division of risk levels and the evaluation indicators of driving states.

[0116] In some embodiments, the acquired multimodal data (including driver status detection data, vehicle status detection data, and road environment status detection data) is input into a convolutional neural network model to map a multimodal matrix. The convolutional neural network model is a 3*3 structured convolutional neural network model. The multimodal matrix is ​​imported into a Transformer model to calculate the correlation weights between different data elements in the multimodal discrete data, that is, to obtain the correlation weights between the multimodal data, and the driving status evaluation results are obtained in combination with the correlation weights.

[0117] Specifically, using this 3x3 convolutional neural network, discrete data items can be mapped into an m-dimensional vector. This vector can be viewed as a feature representation of the discrete data, with each dimension corresponding to a feature of the discrete data item. Assuming n discrete data items, after processing them through the 3x3 convolutional neural network, a 3n × 3m-dimensional multimodal matrix is ​​obtained. In this matrix, each row represents a feature representation of a discrete data item, while each column represents a different feature. The multimodal matrix retains the feature information of the discrete data. Through feature extraction and mapping by the convolutional neural network, the discrete data is converted into a vectorized form. The multimodal matrix is ​​then fed into the Transformer model, where correlation calculation is performed based on the self-attention mechanism. The self-attention mechanism allows the model to analyze each element in the input sequence and adjust weights based on their interrelationships. The resulting correlation weights are then combined to produce a driving state assessment result.

[0118] In some embodiments, the state recognition model includes a driver state recognition model, which includes a cognitive distraction state recognition model. As shown in FIG5 , FIG5 is a flow chart of a method for training a cognitive distraction state recognition model according to an embodiment of the present application. The method for training a cognitive distraction state recognition model includes:

[0119] Step S501 : collecting eye movement and EEG data of one or more drivers when they are performing dual tasks and single tasks multiple times within a preset time interval or at different times.

[0120] In the early stages of model building, a large amount of data from different scenarios is required to train the model. In this application, eye movement and EEG data were collected from a single driver performing both dual-task and single-task, as well as from multiple different drivers performing both dual-task and single-task. Multiple acquisitions and recordings were performed to ensure data integrity.

[0121] Furthermore, data collected for one or more different drivers can be collected multiple times at different time periods or times. For example, data can be collected within preset time intervals or at different times. For example, data can be collected from 8:00 to 11:00, 14:00 to 17:00, or at 6:00, 10:00, 13:00, 15:00, 19:00, midnight, and 2:00 AM. Data under different conditions can be recorded according to different time periods or times. Because each driver's cognitive distraction state varies at different time periods or times, collecting data from different time periods or times during model training can increase the completeness and accuracy of the data, ensure the reliability of the model, and improve the accuracy of driving state assessment, thereby further enhancing driving safety.

[0122] Step S502: Compare the eye movement and EEG data collected multiple times to remove invalid data.

[0123] Specifically, comparing data collected multiple times and deleting invalid data, such as abnormal data or data with large deviations, can improve data accuracy and reduce data processing volume.

[0124] Step S503 , calculating an average value of the eye movement and EEG data collected multiple times after invalid data are removed.

[0125] Specifically, after removing invalid data, the remaining data are averaged and saved.

[0126] Step S504: Perform model training on the cognitive distraction state recognition model using a machine learning algorithm based on the average value.

[0127] However, misjudgment of driver distraction can occur during the driver's distraction detection process. For example, when a driver is driving normally, they may need to turn their head or eyes to check the left or right rearview mirror for safety reasons. Or, due to vehicle bumps, the driver's heart rate or EEG data may change, causing the system to misjudgment that the driver is distracted.

[0128] Furthermore, as shown in FIG6 , FIG6 is a flow chart of another driving state detection and warning method according to an embodiment of the present application, which specifically includes the following steps:

[0129] Step S601 : When a vehicle is traveling along a set route and in a set state, first state detection data of the driver is collected.

[0130] When a vehicle is driving along a set route and in a set state, the set state refers to the driver's normal driving state, for example, the driver is not engaging in distracting behaviors such as making phone calls, smoking, chatting, not looking ahead for extended periods of time, and is not fatigued. At this time, first state detection data of the driver is collected. That is, the driver's first state detection data refers to detection data of the normal driving state. The driver's first state detection data includes at least the driver's physiological signals, first eye movement signals, EEG and brain imaging signals, and behavioral detection data.

[0131] Step S602: performing standardization processing based on the collected first state detection data of the driver.

[0132] Specifically, the collected driver's first-state detection data is converted into a specific unified format to keep the data within a certain range. For example, heart rate data is converted into a data interval format, and heart rate values ​​within this range are considered normal driving conditions. The driver's first-state detection data is standardized to eliminate differences in properties, dimensions, and magnitudes between different detection data, thereby converting it into a dimensionless standardized value.

[0133] Step S603 : Associating the driver status detection data acquired in real time with the first status detection data of the driver after the standardized processing.

[0134] In step S401, multiple detection devices collect in real time the driver status detection data of other drivers to be detected during the driving process, including multimodal physiological data (including EEG data, eye movement data, heart rate, etc.), behavioral data, etc. The driver status detection data collected in step S401 is compared and associated with the driver's first status detection data collected in step S601 to determine which of the driver status detection data belong to the detection data interval of the normal driving state.

[0135] Step S604 : performing noise processing on the driver state detection data based on the driver's first state detection data to obtain the driver's second state detection data.

[0136] In some embodiments, the driver state detection data includes a second eye movement signal, which is obtained by analyzing video content acquired in real time by a video signal detection device. The second eye movement signal is combined with the first eye movement signal in the driver's first state detection data to determine artifact noise in the second eye movement signal, i.e., to determine eye movement signals in the second eye movement signal that are indicative of a normal driving state but are misjudged as a distracted state. The artifact noise is then removed to obtain the second driver state detection data, i.e., the driver state detection data after artifact noise removal.

[0137] Optionally, noise processing may also be performed on other data in the driver status detection data, such as heart rate, EEG signals, behavioral data, etc.

[0138] Step S605 , analyzing the driver's second state detection data, the vehicle state detection data, and the road environment state detection data to obtain a driving state evaluation result.

[0139] The embodiment of the present application obtains the driver's normal driving state data in advance and performs data standardization, compares and correlates the collected driving state detection data with the standardized normal driving state data, and then performs noise processing to find and remove artifact noise, thereby eliminating the impact of misjudgment of distracted state and improving the accuracy of driving state assessment.

[0140] Based on the driving state evaluation result obtained in step S402 or step S605, step S403 is continued to be executed, that is, an early warning is issued based on the driving state evaluation result.

[0141] Specifically, the intelligent human factor monitoring platform uses voice or visual images to remind or warn the driver based on the driving status assessment results. For example, if the driver is fatigued due to driving too long, the intelligent human factor monitoring platform can use voice or visual notifications to remind the driver: "You have been driving for a long time, please stop and rest." Or if the driver is distracted by making a phone call, the current speed is high, and the surrounding traffic conditions are complex, the intelligent human factor monitoring platform can issue an alarm to the vehicle to remind the driver to pay attention to the vehicle ahead.

[0142] Furthermore, based on the driver status detection data, unhealthy driving habits are flagged. These include smoking, talking on the phone, cutting in line, not looking ahead for extended periods, frequent sudden braking, and frequent speeding. Furthermore, the number of occurrences of unhealthy driving habits is counted; when the number of occurrences exceeds a preset number (e.g., three), the driver is reminded or warned. Optionally, for vehicles with autonomous driving systems, the system can pull over by controlling the steering wheel, brakes, and turn signals when it determines that surrounding vehicles are operating safely.

[0143] At the same time, by collecting driver status data, vehicle status data, road environment status data, human-machine interaction data (such as HMI (Human Machine Interface) data and HUD (Head-up Display) data), EEG and eye movement data, the system can assess the driver's basic abilities, such as alertness, attention span, and memory; specific abilities, such as spatial orientation, motor coordination, and peripheral vision; personality traits; and driving suitability. The system can also further construct a data profile of the driver's cognitive abilities, providing data support for driver assessment, training, and selection.

[0144] In an embodiment of the present application, driver status detection data, vehicle status detection data, and road environment status detection data are obtained by simultaneously detecting with multiple different detection devices, and the driver status detection data, vehicle status detection data, and road environment status detection data are comprehensively analyzed to obtain a driving status evaluation result, and then an early warning is issued based on the driving status evaluation result. The present application evaluates the driving status from three dimensions: driver status, vehicle status, and road environment status, which helps to improve the accuracy of driving status evaluation and improve driving safety. At the same time, the driver status, vehicle status, and road environment status are graded. For example, the driving status is divided into fatigue status, distraction status, and emotional status, and the fatigue status, distraction status, and emotional status are further subdivided, thereby further improving the accuracy of the driving status evaluation, issuing risk early warnings based on the driving status, and improving driving safety.

[0145] Furthermore, the embodiments of the present application can also extract at least one independent variable feature of the driver based on the collected physiological data and behavioral data of the driver, and use the at least one independent variable feature to identify the driver's actual fatigue state and control the vehicle to remind the driver of the status, thereby improving the accuracy of the driver's fatigue state identification, increasing the reliability of the identification results, enhancing the driver's experience and customer stickiness, and meeting the different needs of drivers in different scenarios. It solves the problem of detecting the driver's driving status based solely on the driver's facial expression and road information, resulting in a relatively simple fatigue detection method and identification results, and unable to accurately identify the driver's actual driving status, reducing the accuracy and reliability of the detection results, and having a narrow scope of application. It cannot meet the driver's actual detection needs and reduce the driver's usage experience.

[0146] Specifically, FIG7 is a flowchart of a method for detecting driver fatigue in a human-induced intelligent cockpit according to an embodiment of the present application.

[0147] As shown in FIG7 , the method for detecting driver fatigue in a smart cockpit includes the following steps:

[0148] In step S701 , the driver's physiological data and behavioral data are collected.

[0149] It can be understood that physiological data here refers to relevant human physiological signals that can be used to identify the driver's status, such as body temperature, heart rate, blood pressure, etc.; behavioral data here refers to relevant behavioral data that can be used to identify the driver's status, such as some head postures, or some hand movements, etc.

[0150] The embodiment of the present application can collect the driver's physiological data and behavioral data. Because the driver's physiological and behavioral indicators will undergo certain changes during long-term driving, the driver's physiological data and behavioral indicators are collected as the basis for subsequent data processing, and can be further divided to increase the accuracy of detection.

[0151] As an example, collecting the driver's physiological data and behavioral data includes: collecting at least one change data of the driver's heart rate, electroencephalogram signal, and electromyography signal within a preset time period as physiological data; collecting at least one change data of the driver's head posture, blinking frequency, eye opening and closing status, and facial expression within a preset time period as behavioral data.

[0152] Those skilled in the art will understand that physiological data here refers to physiological signals that can be used to measure a person's mental state, including but not limited to: EEG signals, EMG signals, galvanic skin signals, ECG signals, respiratory rate, blood pressure, etc. Behavioral data refers to data records about human behavior and when the behavior occurs, and here refers to the driver's behavior and recorded data during driving; the preset duration is a certain time set in advance, for example, ten minutes.

[0153] The driver's fatigue state can be effectively identified through a variety of physiological and behavioral indicators, such as brain waves, eye movements, head posture, heart rate variability, etc.; during long-term driving, the driver's physiological and behavioral indicators will undergo certain changes, such as changes in eye movement frequency, head posture, heart rate variability, etc. These changes are positively correlated with the fatigue state. By collecting and analyzing the driver's various physiological and behavioral indicators, an effective fatigue state identification model can be constructed.

[0154] For example, heart rate refers to the average number of heartbeats. The heart rates of drivers of different ages, genders, health conditions and living habits will vary. Similarly, the heart rates of drivers in different driving conditions will also vary, for example, the heart rates in normal driving conditions and fatigue driving conditions. Therefore, the driver's heart rate within ten minutes of driving can be collected and used as a physiological data of the driver.

[0155] For example, EEG signals, also known as electroencephalograms (EEG), are electrical signals generated by the activity of brain neurons. The driver's EEG signals during driving will also change with different activities. After collecting EEG signals from different drivers, they are processed using artificial neural networks, mainly to extract typical features of different EEGs in different bands and classify them, and thereby determine whether the driver is fatigued. Therefore, the driver's EEG signals within ten minutes of driving can be collected and used as a physiological data of the driver.

[0156] For example, electromyographic (EMG) signals are the temporal and spatial superposition of motor unit action potentials (MUAPs) in numerous muscle fibers. Surface electromyographic (SEMG) signals are the combined effect of superficial muscle EMG and electrical activity on the skin's surface. They reflect neuromuscular activity to a certain extent and are the electrical signals that accompany muscle contraction. As a driver's body movements change while driving, their muscles contract in different ways, and the EMG signals change accordingly. Therefore, it's possible to collect EMG signals over a ten-minute period while driving and use them as physiological data.

[0157] For example, head posture is generally categorized into three types: raising the head, shaking the head, and turning the head. This refers to the driver's head movements while driving. For example, when a person is tired, their head may unconsciously droop or tilt to the side. Therefore, the driver's head posture can be collected over a ten-minute period while driving and used as behavioral data.

[0158] For example, the blinking frequency is the number of times a driver blinks during a certain period of time while driving. When a person is tired, the number of fast blinks, slow blinks, and the time and number of times are different from those in a normal state. Therefore, the driver's blinking frequency during driving can be collected, for example, the number of times a driver blinks in ten minutes, and this can be used as a behavioral data of the driver.

[0159] For example, the eye opening / closing state refers to the state of the driver's eyes during driving, which can be used to determine whether the driver is in a fatigued state. For example, the time the driver's eyes are closed per unit time (generally 1 minute or 30 seconds) can be calculated. If the eyes are closed for about 80% of the time during this time period, the driver can be determined to be in a fatigued state. Alternatively, the actual pixel values ​​occupied by the driver's eyes in the horizontal and vertical directions can be calculated to calculate the aspect ratio of the eyes. This ratio is relatively fixed for the same person with open or closed eyes, but different people have one thing in common in this value: the value is relatively small (less than 0.3) when the eyes are closed. For example, if the driver's eye aspect ratio is less than 0.3 for 30 seconds within a minute, then the driver is in a fatigued state. Therefore, the driver's eye opening / closing state within one minute of driving can be collected and used as a piece of driver behavioral data.

[0160] For example, facial expressions are those that appear during driving that are quite different from those when the driver is concentrating in normal driving state, such as yawning, daydreaming, or tears flowing from the eyes due to fatigue. At this time, the driver is in a state of fatigue. Therefore, the driver's facial expressions within ten minutes of driving can be collected and used as a behavioral data of the driver.

[0161] The embodiment of the present application can collect different physiological data and behavioral data of the driver over a certain period of time, and use sufficient physiological data and behavioral data to identify the driver's fatigue state as data support for subsequent steps, effectively ensuring the accuracy of detection, increasing the feasibility of identification and the reliability of the results.

[0162] Step S702: extract at least one independent variable feature of the driver based on the physiological data and the behavioral data.

[0163] It can be understood that the independent variable refers to the factor or condition that can be actively manipulated by humans to cause the dependent variable to change. The independent variable can be regarded as the cause of the dependent variable. For example, if the independent variable is that the driver closes his eyes for a long time, then the dependent variable is that the system determines that the driver is driving fatigued. The cause is closing his eyes for a long time, and the result is that the driver is determined to be driving fatigued.

[0164] Exemplarily, the present application can extract at least one independent variable feature of the driver based on at least one change data in physiological data such as heart rate, EEG signals, electromyographic signals, etc. collected within a certain period of time, and at least one change data in behavioral data such as head posture, blinking frequency, eye opening and closing status, facial expressions, etc. within a certain period of time.

[0165] The embodiment of the present application can extract at least one independent variable feature of the driver from the physiological data and behavioral data, and perform the next step of detection based on the independent variable feature, thereby effectively ensuring the accuracy of the detection.

[0166] As an example, extracting at least one independent variable feature of a driver based on physiological data and behavioral data includes: preprocessing the physiological data and behavioral data to obtain processed collected data; and extracting at least one energy ratio index from the collected data as at least one independent variable feature.

[0167] Exemplarily, data preprocessing may include, but is not limited to, removing EOG artifacts - independent component analysis (ICA), noise frequency domain filtering - bandpass filtering, and data downsampling - Nyquist sampling theorem.

[0168] For example, characteristic indicators are screened: The characteristic indicators that best represent driving fatigue are extracted from the preprocessed collected data, with the energy ratio indicator being an effective indicator of the driver's fatigue state. Energy characteristics are extracted by frequency decomposing rhythmic waves using wavelet packet transform. Grey correlation analysis is then used to identify the indicators most closely associated with fatigue status, allowing for optimal features to be selected for studying driving fatigue.

[0169] For example, the significance analysis of characteristic differences: the validity analysis of the driver's fatigue characteristic parameters is to test whether these parameters are significantly different in different fatigue states of the driver. The mean of the parameters under different fatigue states is calculated, and the differences between the means are compared to make a judgment. Analysis of variance is used to test whether these characteristic parameters are different in different states of the driver. According to the theory of variance analysis, the differences in characteristic parameters between different states come from two aspects:

[0170] (1) Parameter error caused by fatigue, that is, the difference caused by the change of the driver's fatigue state, is called the inter-group error. It is expressed by calculating the sum of the squares of the deviations between the mean value of the characteristic parameter in each group and the mean value of the total sample, denoted as E 组间 .

[0171] (2) Random error, that is, the deviation caused by interference during feature extraction, is called intra-group difference. It is expressed by finding the mean value of the feature parameter in each group and the sum of the squares of the difference between the parameters in the group, denoted by E 组内 .

[0172] E 组间 and E 组内 After dividing by the respective degrees of freedom, the mean square values ​​can be obtained, which are: mean square between groups (MS 组间 ) and within-group mean square (MS 组内 ). When these two parameters satisfy the first formula below, different fatigue states will not affect the parameter values, that is, each sample comes from a population. When different fatigue states have a more obvious impact on the extracted feature parameter values, MS 组间 It is caused by the driver's fatigue state and random errors, that is, the samples come from different populations. 组间 and MS 组内 The relationship is shown in the following formula (3). MS 组间 >>MS 组内 (3)

[0173] At this time, MS 组间 and MS 组内The ratio of the two forms the F distribution. By comparing the F value with its critical value, we can determine whether each sample comes from the same population. Assume that there are n driver fatigue samples, and the null hypothesis is set as: the mean value in each fatigue state is the same, that is: u1=u2=u3=u (4)

[0174] Among them, u i represents the mean value of the i-th fatigue state. The original hypothesis is that n samples come from a population (i.e., μ and σ are the same). If the result of the calculation is the same as formula (3), that is, from MS 组内 Much smaller than MS 组间 ; Then, it means that different fatigue states of drivers will cause differences between characteristic parameters and mean values, that is: F>F 0.05 (f 组间 ,f 组内 ) (5)

[0175] Among them, f in the brackets is the degree of freedom. It is generally believed that p<0.05 indicates a statistical difference, p<0.01 indicates a significant statistical difference, and p<0.001 indicates an extremely significant statistical difference.

[0176] Finally, the valid data is counted for use.

[0177] The embodiment of the present application can pre-process the physiological data and behavioral data before extracting at least one independent variable feature from the physiological data and behavioral data to ensure the validity of the detected data and screen out invalid or redundant data. Then, at least one energy ratio index is extracted from the obtained valid data as at least one independent variable feature, thereby effectively improving the efficiency of data detection and ensuring the accuracy and validity of the data.

[0178] Step S703: identifying the actual fatigue state of the driver according to at least one independent variable feature, and controlling the vehicle to provide a status reminder to the driver based on the actual fatigue state.

[0179] Exemplarily, after obtaining at least one independent variable feature from the above example, the actual fatigue state of the driver can be identified. For example, the heart rate change data collected and preprocessed within a certain period of time is used as the independent variable feature. When the driver drives for a long time, the heart rate will decrease, and at this time it can be identified that the driver is in a fatigue state.

[0180] For example, after identifying that the driver is in a fatigue state, the driver's fatigue state can be further divided into levels and defined as an actual fatigue state, and the driver can be reminded according to the driver's actual fatigue state, for example, through a corresponding status voice alarm through a voice alarm system, or through icons or text on the in-car display screen and adjustment of the in-car lights.

[0181] The embodiment of the present application can identify the actual fatigue state of the driver by at least one independent variable feature, and use this identification result to control the vehicle to remind the driver of the status, ensure the accuracy of the identification result, and make relevant reminders to avoid driver fatigue driving and meet the driver's actual usage needs.

[0182] As an example, identifying the actual fatigue state of a driver based on at least one independent variable feature includes: inputting at least one independent variable feature into a pre-constructed fatigue state identification model, and outputting the actual fatigue state, wherein the fatigue state identification model is constructed by KSS scale scores before and after driving and multiple fatigue states.

[0183] Based on other related embodiments, it can be understood that the fatigue state recognition model here is a model that has been trained and established in advance, and can be, but is not limited to, constructed by combining KSS scale scores before and after driving and multiple fatigue states. As shown in Figure 8, Figure 8 is a schematic diagram of driver fatigue state recognition and monitoring in one embodiment of the present application. The method for constructing the fatigue state recognition model can be as follows:

[0184] Exemplarily, the model variables control: independent variables: changes in head posture, changes in blinking frequency, changes in eye opening and closing status, changes in facial expressions, changes in heart rate, changes in EEG signals, changes in EMG signals, etc.; dependent variables: fatigue status (non-fatigue, mild fatigue, moderate fatigue, severe fatigue).

[0185] For example, model subjects and equipment: 9 adults with driving experience were selected as model subjects and the experimental subjects were required to meet the following conditions: aged between 18 and 60, in good health, and without a history of psychological, neurological or vision diseases; using a simulated driving system*1, an electroencephalogram*1, an EEG analysis module*1, an eye tracker*1, a general basic physiological analyzer*1, a wireless high-precision physiological recording system*1, a camera system*1, a head posture detector*1, a wearable surface electromyography measurement system*1, an EMG analysis module*1, a computer system*1, and stopwatches*3.

[0186] For example, model data collection: after explaining the operation and precautions to the model object, the tester sets the test conditions and clarifies the type of data to be collected. After the tester debugs and assembles the data acquisition device, the model object is equipped with the data acquisition device, and after ensuring that the physiological signal acquisition device is well connected and turning on the recording system, the continuous collection of behavioral and physiological data begins.

[0187] The test conditions include: simulated driving environments (e.g., long periods of monotonous high-speed driving, urban roads, and rural roads); driving time (e.g., 30-240 minutes); fatigue levels (e.g., alert, mild fatigue, moderate fatigue, and severe fatigue); and model subject characteristics (e.g., age, gender, driving experience, seat angle adjustment, etc.). Data types collected include: behavioral data (e.g., changes in head posture, blink rate, eye opening and closing, and facial expression); and physiological data (e.g., changes in heart rate, EEG signals, and EMG signals).

[0188] Exemplarily, the fatigue state labeling method is: a method combining the model object's subjective evaluation Karolinska Sleepiness Scale (KSS) and a state recording button is used to record the model object's fatigue state and occurrence time during driving, so as to determine different fatigue state labels corresponding to the collected data samples.

[0189] The five ratings of 1 to 5 represent a normal driver state, while 6 to 10 represents fatigue, with 6 and 7 representing mild fatigue, 8 representing moderate fatigue, and 9 and 10 representing severe fatigue. The model subject uses the KSS scale to determine changes in its own state. When the state reaches mild, moderate, and severe fatigue, it signals the recorder, who records the time points to facilitate segmentation of EEG and ECG signals. When the model subject reaches severe fatigue, data collection continues for 10 minutes, which is then stored and exported, completing data collection.

[0190] It's important to note that the subjective scale is used as a supplementary tool, primarily employing the state recording and annotation method. The model subject undergoes a subjective KSS evaluation before and after the simulated driving. By recording the time points at which the model subject's state changes, combined with the scale's display before and after the driving session, the corresponding time periods for the four different states can be determined. This data is then annotated to ultimately create a fatigue state recognition model.

[0191] The KSS scale divides fatigue levels into 10 levels, with higher levels indicating deeper fatigue. Table 1 shows the KSS scale scores and various fatigue states, as shown below:

[0192] Table 1

[0193] Among them, 1 means extremely alert, 2 means very alert, 3 means alert, 4 means somewhat alert, 5 means neither alert nor sleepy, 6 means some signs of sleepiness, 7 means sleepy but still able to stay awake, 8 means sleepy and needs effort to stay awake, 9 means very sleepy and needs great effort to stay awake and try hard to overcome it, and 10 means extreme sleepiness and cannot stay awake; further divided according to the state: 1-5 means non-fatigue state, 6-7 means mild fatigue, 8 means moderate fatigue and 9-10 means severe fatigue.

[0194] The embodiment of the present application can input at least one independent variable feature into a pre-trained fatigue state recognition model and output the actual fatigue state. The fatigue state recognition model is pre-constructed through professional KSS scale scoring and multiple fatigue states, and is divided into different fatigue levels. The convergence and reliability of the model are guaranteed through offline training, thereby improving the feasibility and reliability of the present application.

[0195] As an example, identifying the actual fatigue state of the driver based on at least one independent variable feature includes: obtaining the current operating condition and / or current environment of the vehicle; generating a weight for each independent variable feature in at least one independent variable feature based on the current operating condition and / or current environment; and determining the actual fatigue state based on the at least one independent variable feature and the corresponding weight.

[0196] It can be understood that the current operating condition of the vehicle refers to the condition of the vehicle driving on the current road, for example, driving on a monotonous highway, driving on a city road, or driving on a rural road.

[0197] For example, the vehicle's current operating conditions and surroundings can both affect the driver's driving state. For example, when the vehicle is in relatively good operating conditions, such as during monotonous highway driving, the driver is relaxed and the EMG frequency shows a downward trend, but this does not indicate fatigue or increased fatigue. However, when the vehicle is exposed to strong light, the driver's eyes may unconsciously squint for extended periods of time, and their eyes may close more deeply, but they do not experience drowsiness, meaning they are not experiencing moderate fatigue.

[0198] For example, since different working conditions and different environments will affect the driver's physiological data and behavioral data, the weight of each independent variable feature in at least one independent variable feature can be generated according to the current working condition and / or the current environment. For example, when the current working condition is better, the weight of the electromyographic signal needs to be reduced, and under strong light, the weight of the eye opening and closing state needs to be reduced. The actual fatigue state is then determined based on at least one independent variable feature and the corresponding weight.

[0199] The embodiment of the present application can determine the driver's actual fatigue state based on the collected physiological data and behavioral data of the driver, further combining the vehicle's current operating conditions and / or current environment, effectively improving the accuracy of detection and meeting the different needs of the driver in actual use.

[0200] As an example, providing a status reminder to the driver based on the actual fatigue status includes: determining whether the actual fatigue status meets the preset reminder conditions; if the actual fatigue status meets the preset reminder conditions, matching the target reminder type and reminder signal according to the actual fatigue status, and controlling the vehicle to provide a status reminder based on the reminder signal according to the target reminder type.

[0201] It can be understood that the preset reminder conditions can be understood as certain conditions that are pre-set and can control the vehicle to remind the driver, for example, the driver is in a moderately fatigued driving state.

[0202] For example, the preset reminder condition is that the driver's fatigue level is moderate fatigue and above. Then, when the driver's actual fatigue state reaches moderate fatigue, it can be determined that certain reminder conditions are met. At this time, the target reminder type and reminder signal can be matched according to the driver's actual fatigue state, and the vehicle can be controlled to provide status reminders based on the reminder signal.

[0203] For example, if the driver's actual fatigue state is mild fatigue, the matching target reminder type is the voice alarm type, and the reminder signal is the broadcast reminder. That is, the vehicle can be controlled to broadcast through the in-car loudspeaker: "Hello, driver, it is detected that you have been driving continuously for more than 3 hours. Continuing to drive may cause fatigue driving. Please take a rest!"; for another example, if the driver is in a moderate fatigue state, the matching target reminder type is the action type, and the reminder signal is seat adjustment. The seat angle that the driver usually adjusts to is adjusted to a more comfortable angle, such as slightly raising the backrest by 15 degrees to remind the driver.

[0204] The embodiment of the present application can pre-set certain reminder conditions, so that when the actual fatigue state meets certain conditions, the driver can be reminded of the status through the generated reminder type and reminder signal, reminding the driver at the critical moment to avoid the possibility of safety accidents, improve the driver's experience, and meet the driver's usage needs in actual scenarios.

[0205] As an example, the preset reminder condition is that the fatigue level of the actual fatigue state is greater than the preset level.

[0206] It is understandable that the preset level can be understood as a certain fatigue state level that is pre-set and requires a reminder.

[0207] For example, the preset level is mild fatigue. Then, when the driver is in a fatigued state and the fatigue level is moderate fatigue or severe fatigue, it is already greater than the preset level, that is, certain reminder conditions have been met, and the driver can be reminded of certain status.

[0208] The embodiment of the present application can pre-set certain reminder conditions and fatigue levels to determine that certain reminder conditions are met when the actual fatigue level is greater than a certain fatigue level, thereby reminding the driver, ensuring driving safety, improving the driver's experience, and ensuring customer stickiness.

[0209] Research shows that driver distraction significantly reduces a driver's ability to perceive danger, impairing their ability to react to it. This can lead to untimely and inaccurate judgment of the road environment, improper driving behavior, and accidents. Prompting drivers to react when distracted can effectively prevent some accidents. Statistics show that if a driver's reaction time can be improved by 0.5 seconds, the likelihood of an accident is reduced by approximately 60%, and even if an accident does occur, its severity can be mitigated.

[0210] In one embodiment, the driver's status can be predicted using traditional machine learning and deep learning methods. Traditional machine learning methods require less computation and lower hardware platform requirements, making them suitable for embedded platforms. However, they require a complex feature extraction process. Deep learning methods can automatically extract effective features, simplify the classification process, and improve generalization. With sufficient data, they offer excellent classification performance, but they also require more computation and higher hardware platform requirements.

[0211] Deep learning is an end-to-end algorithm, a type of representation learning. It requires only data input and produces a corresponding target output, eliminating the need for complex feature engineering. It can self-learn and extract useful features. Due to its advantages, such as self-learning features and excellent performance, deep learning methods can be applied to the problem of distracted driving.

[0212] For example, a spatiotemporal convolutional neural network based on EEG signals can be used to detect driver distraction. By combining the time and frequency domain features of EEG signals, high accuracy is achieved. Compared to traditional algorithms, this framework offers advantages in computational efficiency and parameter update speed, making it suitable for use in online distraction monitoring systems. For example, a long short-term memory (LSTM) model can be built to detect driver fatigue. This model leverages the temporal characteristics of EEG signals and optimizes the model through training process visualization, significantly improving accuracy compared to traditional machine learning algorithms. For example, a channel-based convolutional neural network can be used to predict driver fatigue. This method leverages the channel correlation characteristics of EEG signals and achieves good results. For example, by constructing simulated driving experiments with different driving scenarios, the driver's braking reaction time is used to evaluate attention. A deep convolutional network is then used to process EEG signals, enabling detection of braking intentions under different attention states.

[0213] In one example, driving behavior experiments are mainly divided into simulated driving experiments and real-car experiments. For example, in a real-car environment, a camera is used to collect image data of the driver's driving process, including images of normal driving status and images of distracted status (making a phone call with both hands, sending text messages with both hands, picking up items behind the driver's seat, and operating the car radio), and recognition is performed through image technology. For example, a simulator with visual feedback, auditory feedback, and vehicle sensors can be used to conduct distraction research. For example, a simulator can be used to conduct vehicle turning experiments and collect driving parameters such as steering wheel angle, vehicle speed, brake pedal signal, lateral acceleration, environmental road information, and driving time during driving. The steering mode under low attention is explored, and the driver's distraction can be monitored based on the car's steering performance. On this basis, real-car experiments are conducted to eliminate the impact of extreme working conditions on the steering sensor and enhance robustness.

[0214] In one example, with the continuous development of computer hardware and the increased availability of large amounts of distracted driving data, the application of machine learning algorithms to the study of distracted driving has become possible. For example, one-dimensional discrete wavelets can be used to analyze EEG signals, obtaining various wavelet frequency bands. The statistics of the wavelet coefficients in the delta, theta, alpha, and beta wave bands can be extracted as features, input into a neural network, and combined with a fuzzy model to classify the driver's attention. For example, the EEG power spectrum and driving scene can be combined as input features, combined with a support vector machine (SVM), and the particle swarm algorithm can be used to optimize the system parameters to obtain a driver distraction detection model for different driving scenarios. For example, the frequency domain features of the EEG data can be extracted, and the feature dimension can be reduced based on principal component analysis (PCA). The reduced features can then be input into the maximum likelihood estimation (MLE) and K-nearest neighbor (KNN) model for classification. For example, EEG data of a driver in a distracted state can be collected, and its power spectrum can be extracted as a feature, which is then input into a radial basis function (RBF)-based SVM to build a driver attention focus tracking system.

[0215] As you can see, a driver distraction monitoring system monitors and predicts a driver's driving state, correcting distractions before they become dangerous. This reduces the driver's misperception and misjudgment of the environment, and shortens reaction time to dangerous situations. This has important theoretical and practical value for reducing accident rates and improving traffic safety.

[0216] In view of this, the present application proposes a method for predicting the driving state of a driver based on multimodal data. The method predicts the driver's state by combining multimodal data, where the multimodal data includes the driver's physiological state data and vehicle driving data, and feeds the monitoring information back to the Advanced Driver Assistance System (ADAS) to warn or remind distracted drivers. This function, combined with the existing ADAS, can reduce the driver's distracted behavior, thereby improving driving safety.

[0217] FIG9 is a flowchart of a method for predicting a driver's driving state based on multimodal data according to an embodiment of the present application.

[0218] As shown in FIG9 , the method for predicting a driver's driving state based on multimodal data includes S901 - S902 .

[0219] S901: Obtain an initial prediction result for the driver's driving state based on the driver's physiological state data.

[0220] S902: Obtain a target prediction result for the vehicle's driving state based on the initial prediction result and the vehicle driving data.

[0221] Specifically, accidents are often caused by driver distraction. To improve driving safety, it is necessary to predict the driver's driving state. For example, sensors can be used to collect physiological information about the driver's driving state. This collected physiological information can, to a certain extent, reflect the driver's attention. Based on the driver's physiological information, an initial prediction of the driving state is obtained. The initial prediction of the driving state can be either a distracted driving state or a non-distracted driving state. Based on the driver's physiological data, an initial prediction of the vehicle's driving state can be obtained using a deep learning network or machine learning method. Based on the initial prediction of the driver and vehicle driving data, a target prediction of the driving state is obtained. The target prediction may include a prediction of dangerous driving. Combining the initial prediction of the driving state with the vehicle driving data can predict whether the vehicle is driving dangerously, thereby improving the accuracy and efficiency of the prediction. Furthermore, when dangerous driving is predicted, a warning or alert is provided to the distracted driver, thereby improving driving safety.

[0222] FIG10 is a schematic diagram of a driving state prediction system according to an embodiment of the present application.

[0223] As shown in FIG10 , the driving state prediction system of the present application can be divided into two major modules. The first module includes a model training module, which is used to train a deep learning model. The deep learning model can be a cognitive distraction classification model. The first module can be used to obtain physiological state data under different distraction states, and use it as training set data for model training to obtain a cognitive distraction classification model. In one example, the physiological state data includes at least one of eye movement data and EEG data. Of course, the physiological state data can also include other types of data, such as motion data of specific body parts (hands, feet, etc.). The motion data of specific body parts can reflect whether the driver is distracted to a certain extent. For example, when a driver uses his hands to operate a mobile phone while driving, the hand motion data reflects the driver's distraction.

[0224] The second module includes an online cognitive distraction detection module. The second module is composed of at least three systems: a data acquisition system, a data processing system, and a model prediction system. The data acquisition system is used to collect EEG data, eye movement data, and vehicle data (i.e., vehicle driving data) in real time. The data processing system is used to process the collected data in real time, which is divided into processing EEG data, eye movement data, and vehicle driving data. The processing results of the EEG data and eye movement data are then sent to the cognitive distraction classification model trained in the first module for prediction, and the cognitive distraction classification result (i.e., the initial prediction result) is obtained. The collected vehicle driving data is processed to obtain the vehicle position prediction error rate. Combined with the vehicle position prediction error rate, the driving state is predicted according to certain rules in the model prediction system.

[0225] When making predictions based on EEG data and eye movement data, the EEG data and eye movement data may be processed first, and the data processing includes preprocessing.

[0226] As an example, based on the driver's physiological state data, an initial prediction result for the driving state is obtained, including: inputting at least one of the eye movement data and the EEG data into a trained deep learning model for prediction to obtain an initial prediction result.

[0227] Specifically, physiological state information of the driver is collected. The physiological state information is used to determine whether the driver is distracted, so the collected physiological state information includes at least one of eye movement data and EEG data. At least one of the eye movement data and EEG data is input into a trained deep learning model. The deep learning model extracts features from the input eye movement data and EEG data, predicts the driving state, and obtains an initial prediction result. The initial prediction result is used to indicate whether the driver is distracted.

[0228] As an example, the trained deep learning model includes a trained cognitive distraction classification model. When training the model, a training data set is first obtained, and the eye movement data and EEG data of one or more drivers performing dual-task driving experiments and single-task driving can be collected respectively. In the early stage of establishing each classification model, a large amount of accumulated data in different scenarios is required. In this application, the eye movement data and EEG data information of a driver performing dual tasks and single tasks can be collected respectively, as well as the eye movement data and EEG data information of multiple different drivers in dual-task and single-task states. Among them, when collecting data from a driver, it can be collected multiple times and recorded. Furthermore, the data collected multiple times can be compared to remove data with large errors, and the others can be averaged and saved. Dual-task means that the driver performs distraction tasks while performing the driving task, and single-task means that the driver only performs the driving task.

[0229] Collecting data from multiple different drivers also requires multiple collections at different times, such as at set time intervals or at different time periods. For example, in the morning, this could be at 6:00 AM, 10:00 AM, 1:00 PM, 3:00 PM, 7:00 PM, 12:00 AM, and 2:00 AM. Eye movement and EEG data are recorded at different times and under different conditions. Because each person's distraction state varies at different times, comprehensive considerations are required when collecting model training data to increase data completeness and accuracy. This will improve the accuracy of the dataset and the accuracy of driving state predictions, thereby enhancing driving safety.

[0230] The prediction results from training the deep learning model can include labeling results. Data in the dual-task state is labeled 1, indicating cognitive distraction, and data in the single-task state is labeled 0, indicating no cognitive distraction. After training, a trained cognitive distraction classification model is obtained. During online prediction, at least one of the eye movement data and the EEG data is input into the trained cognitive distraction classification model for prediction, resulting in an initial prediction result.

[0231] As an example, the initial prediction result may also include a label result. In the dual-task state, the data label is 1, indicating cognitive distraction; in the single-task driving state, the data label is 0, indicating no cognitive distraction.

[0232] As an example, preprocessing the eye movement data includes at least one of the following: removing data in the eye movement data indicating abnormal pupil size changes, pupil occlusion, and artifacts at the pupil edge; removing data in the eye movement data indicating gaze deviation; removing data in the eye movement data indicating that the line of sight is outside the area of ​​interest; and removing data in the eye movement data indicating that the scanning angular velocity is greater than a preset angular velocity.

[0233] Specifically, collected eye movement data must be preprocessed before being input into a trained deep learning model to improve the accuracy of model predictions. Eye movement data can include raw pupil size time series and user gaze position information captured by an eye tracker and live cameras. Abnormal pupil size changes in eye movement data include non-positive pupil size, missing eye targets, eyelid occlusion, or blinking.

[0234] As an example, to remove data in which the gaze is outside the area of ​​interest in the eye movement data, a velocity threshold identification fixation filter (I-VT) can be used to extract the features of gaze and saccade. The threshold can be set to 30 degrees. That is, when the angular velocity of the saccade is greater than the threshold of 30 degrees, it indicates interference caused by turning the head. This part of the interference needs to be removed to identify gaze and saccade eye movement behaviors.

[0235] As an example, data in the eye movement data indicating that the scanning angle velocity is greater than the preset angular velocity can be removed. The viewing area can be determined as the area of ​​interest by taking the distance between the center point of the road in front of the vehicle and the eyes as the diameter or radius and the eyes as the center. That is, the line between the driver's eyes as the starting point and the center point of the road in front is taken as the angle bisector, and the area within the 16° viewing angle is the area of ​​interest. The data of the non-interest area is deleted, and only the gaze points in the area of ​​interest are recorded.

[0236] As an example, preprocessing of EEG data includes at least one of the following: averaging EEG data of multiple channels to obtain a mean value, retaining the difference between the EEG data of each channel and the mean value; filtering the EEG data to retain data in a preset band; removing interference data in the EEG data caused by blinking or body movement; and extracting features from the EEG data to obtain power spectral density feature data for a specific band.

[0237] Specifically, the collected EEG data must also be pre-processed before being input into the trained deep learning model. EEG data can be collected using a head-mounted EEG device.

[0238] As an example, in the process of preprocessing EEG data as shown in Figure 11, the raw EEG data is averaged for a whole-brain reference, that is, the data collected by multiple channels are averaged to obtain an average value, and the difference between the data of each channel and the average value can be used as the whole-brain average reference data. Band-stop filtering is then performed, and the filter frequency can be set to 0.5-45Hz, that is, signals outside this band are removed. Principal component analysis (Independent Component Correlation Algorithm, ICA) is then used to identify and remove interference data caused by blinking or body movement. Blinking can interfere with EEG signals, and signals caused by body movement interfere with EEG signals, both of which need to be removed. Finally, feature extraction is performed through power spectral density analysis to obtain power spectral density feature data for specific bands. Power spectral density features can be extracted, including 3-7Hz for theta waves, 8-12Hz for alpha waves, and 13-30Hz for beta waves.

[0239] As an example, the initial prediction result represents whether the driver is distracted during driving, and the target prediction result represents whether the vehicle driving deviation is caused by the driver's distraction during driving.

[0240] Specifically, at least one of the preprocessed eye movement data and EEG data is input into a trained deep learning model to obtain an initial prediction result. The initial prediction result can indicate whether the driver was distracted during driving. For example, if the driver was distracted, the output label is 1, and if the driver was not distracted, the output label is 0. To further improve the accuracy of the prediction, the vehicle driving data can be combined for judgment. By processing the vehicle driving data, it can be determined whether the vehicle has deviated. For example, by combining the initial prediction result with the vehicle driving data, a target prediction result for the driving state is obtained. The target prediction result can indicate whether the vehicle's driving deviation was caused by driver distraction during driving.

[0241] As an example, the vehicle driving data includes lateral speed offset information; based on the initial prediction result and the vehicle driving data, a target prediction result for the driving state is obtained, including: when the initial prediction result and the lateral speed offset information both indicate that the driving state is risky driving, determining that the driver's driving state is risky driving.

[0242] Specifically, during normal vehicle driving, the vehicle's lateral velocity offset is relatively stable or small. For example, when the vehicle is traveling in a straight line, the theoretical value of the vehicle's lateral velocity offset is zero. However, in the event of an emergency, the driver instinctively turns the steering wheel, resulting in a larger lateral velocity offset. Therefore, the vehicle's lateral velocity offset information can be analyzed to determine whether the driving state is risky. The initial prediction result characterizes whether the driver is distracted during driving. If the driver is distracted, it is risky driving; otherwise, it is non-risky driving. Combining the initial prediction result with the vehicle's driving data, a target prediction result for the driving state is obtained. If the initial prediction result indicates risky driving, and the result of the lateral velocity offset information also indicates risky driving, the target prediction result for the driving state is risky driving. This risky driving is caused by the driver's distraction, resulting in vehicle driving deviation.

[0243] In another example, the vehicle driving data includes lateral velocity values ​​and lateral velocity offset information. Compared to determining risky driving conditions based on the initial prediction results and lateral velocity offset information, the lateral velocity value can be further considered to improve accuracy. As shown in Figure 12, based on the initial prediction results and the vehicle driving data, a target prediction result for the vehicle's driving state is obtained, including S1201-S1202.

[0244] S1201: When the lateral speed value is less than or equal to a preset speed threshold, determine the driving state of the driver based on the initial prediction result and the lateral speed offset information.

[0245] S1202: When both the initial prediction result and the lateral speed offset information indicate that the driving state is risky driving, determine that the driver's driving state is risky driving.

[0246] Specifically, driving behaviors such as U-turns and lane changes also affect lateral speed offset information. Therefore, if a normal U-turn or lane change results in a large lateral speed offset, it can be determined that there is no driving risk. Specifically, when the lateral speed value is less than or equal to a preset speed threshold (i.e., the lateral speed value is less than or equal to the lateral speed value at the time of the U-turn or lane change), and if both the initial prediction result and the lateral speed offset information indicate a risky driving state, the driver's driving state is determined to be risky.

[0247] As an example, the lateral speed value can be directly measured by a speed detector on the vehicle, the preset speed threshold can be set to 2m / s, and the lateral speed can be recorded as The lateral velocity value is the lateral velocity The absolute value of the lateral velocity is recorded as When the lateral speed value is less than or equal to 2m / s, it indicates that the vehicle is not in a U-turn or lane change situation. At this time, if the initial prediction result and the lateral speed offset information both indicate that the driving state is risky driving, the driving state is determined to be risky driving.

[0248] As an example, when the lateral speed value is greater than a preset speed threshold and the initial prediction result is risky driving, the driving state is not risky driving. For example, when the lateral speed value is greater than 2m / s, it is not monitored as cognitive distraction.

[0249] As an example, as shown in FIG13 , the vehicle driving data includes the true value of the kinematic information, and the lateral velocity offset information includes the first lateral velocity offset value; the method for predicting the driving state of the driver based on multimodal data also includes S1301 - S1303 .

[0250] S1301: Based on the true value of the kinematic information, a predicted value of the kinematic information is predicted.

[0251] S1302: Obtain a prediction error of the kinematic information based on the true value of the kinematic information and the predicted value of the kinematic information.

[0252] S1303: Obtain a first lateral velocity offset value based on the prediction error.

[0253] Specifically, the vehicle driving data includes the true value of the kinematic information, and the true value of the kinematic information can be directly obtained through collection. The kinematic information may include information such as the position, speed, acceleration, etc. of the vehicle. Based on the true value of the kinematic information, the predicted value of the kinematic information is predicted. The predicted value of the kinematic information may correspond one-to-one with the true value of the kinematic information. For example, the position, speed, acceleration, etc. of the vehicle at the next moment are predicted one by one to obtain the corresponding predicted value. Based on the true value of the kinematic information and the predicted value of the kinematic information, the prediction error of the kinematic information is obtained (the prediction error includes the vehicle position prediction error rate mentioned above). The predicted value at the next moment is compared with the true value obtained through collection to obtain the prediction error of the kinematic information. Based on the prediction error, the first lateral velocity offset value is obtained.

[0254] As an example, all data of vehicle driving data need to be collected synchronously, that is, all data have the same starting point in time. The real value of the kinematic information can be sampled at a certain frequency band, such as 10HZ. t Represents the true value of the vehicle kinematic information collected at time t. Each Z t A vector representing the q-dimensional (e.g., position, velocity, acceleration, etc.) vehicle driving data measured at time t. t-1 Represents all q-dimensional vehicle kinematic information up to time t-1. It can be obtained through the state space model according to Z t-1 (The true value of all q-dimensional vehicle kinematic information as of time t-1) predicts the predicted value of the vehicle kinematic information at time t, and the predicted value of the kinematic information at time t is recorded as ( Represents the q-dimensional prediction result of position, velocity, acceleration, etc.). Based on the true value of the kinematic information and the predicted value of the kinematic information, the prediction error of the kinematic information is obtained. The lane position prediction error can be used as the evaluation scale of the prediction error between the true value and the predicted value, or the average lane position prediction error can be used as the evaluation scale of the prediction error between the true value and the predicted value. The average lane position prediction error represents the average value of the lane position prediction error at multiple moments in the time window. If the lane position prediction error is small, the driving behavior has not deviated significantly; if the lane position prediction error is large, the driving behavior has deviated significantly, and dangerous driving may occur. Based on the lane position prediction error, the first lateral velocity offset value is calculated.

[0255] As an example, as shown in FIG14 , based on the true value of the kinematic information, the predicted value of the kinematic information is predicted, including S1401 - S1402 .

[0256] S1401 , using multiple spatial state models to perform prediction based on the true value of kinematic information, to obtain multiple predicted values ​​corresponding one-to-one to the multiple spatial state models.

[0257] S1402 : Perform weighted calculation on multiple prediction values ​​based on weights corresponding to multiple spatial state models to obtain a prediction value of kinematic information.

[0258] Specifically, the true value of kinematic information can be predicted using a spatial state model. The spatial state model has multiple driving motion patterns c∈{1,...,C}, where C is the number of spatial state models, for example, C=6. The six spatial state models include constant velocity (CV), constant acceleration (CA), constant turn rate and velocity (CTRV), constant turn rate and acceleration (CTRA), constant steering angle and velocity (CSAV), and constant curvature and acceleration (CCA). Multiple spatial state models are fused to obtain multiple predicted values ​​that correspond to each other. For example, a weighted calculation is performed on the multiple predicted values ​​to obtain the predicted value of the kinematic information.

[0259] As an example, the autonomous multiple model algorithm (AMM) can be used to fuse multiple spatial state models. The fused prediction value is obtained by formula (6):

[0260] Formula (6) indicates that each of the six spatial state models outputs a prediction value, obtaining six prediction values, and performing weighted averaging on the six prediction values ​​to obtain the fused prediction value.

[0261] in, is the normalized weight of each state space model at time t, The weight of each state-space model can be set as needed. For example, different weights may be assigned to different state-space models under different road conditions. The predicted values ​​from each state-space model are weighted together to produce the predicted kinematic information. It should be noted that the predicted kinematic information is multi-dimensional (q-dimensions, such as position, velocity, and acceleration).

[0262] The lane position prediction error can be used as a prediction error evaluation scale between the true value and the predicted value, or the average lane position prediction error can be used as a prediction error evaluation scale between the true value and the predicted value.

[0263] As an example, let the lane position prediction error be ε t , which is the predicted value of kinematic information and the true value Z of the kinematic information t The difference between t It is the lane position prediction error in multiple dimensions (position, velocity, acceleration, etc. q dimensions) at time t.

[0264] As an example, let the average lane position prediction error be In the i-th time window, a time window includes multiple moments. If a time window includes n moments, the average lane position prediction error

[0265] As an example, based on the lane position prediction error, the first lateral velocity offset value is calculated and the first lateral velocity offset value is calculated by formula (7):

[0266] Among them, σ f is the lane position prediction error ε in the fusion state t The standard deviation of , εt in the fusion state is the weighted average of the lane position deviation in different states, the weight of each state is the weight corresponding to the C state space model above, and the weighted average of the lane position deviation in C different states is obtained in formula (2) is the first lateral velocity offset value of the i-th time window in the positive direction, is the first lateral velocity offset value of the 0th time window in the positive direction, which is zero. is the first lateral velocity offset value of the i-th time window in the negative direction, is the first lateral velocity offset value of the 0th time window in the negative direction, which is zero. k is a preset value, which can be set to 0.8.

[0267] Multiple spatial state models are used to predict the true value of the collected kinematic information to obtain the predicted value of the kinematic information. The kinematic prediction error is obtained based on the difference between the true value of the kinematic information and the predicted value of the kinematic information. Based on the prediction error, the first lateral velocity offset is calculated to determine whether the driving state is risky driving.

[0268] As an example, the lateral speed offset information indicates that the driver's driving state is risky driving, including: when the first lateral speed offset value is greater than a preset speed offset threshold, determining that the lateral speed offset information indicates that the driver's driving state is risky driving.

[0269] Specifically, the first lateral speed offset value includes lateral speed offset values ​​in two directions, namely, the first lateral speed offset value in the positive direction and the first lateral velocity offset in the negative direction When the first lateral speed offset value is greater than the preset speed offset threshold, the first lateral speed offset value in the positive direction is and the first lateral velocity offset in the negative direction The preset speed offset threshold can be set to 8. When the first lateral speed offset value in the positive direction is greater than the preset speed offset threshold, and the first lateral velocity offset in the negative direction When at least one of the values ​​is greater than 8, it is determined that the lateral speed offset information indicates that the driving state is risky driving.

[0270] In addition to obtaining the lateral velocity offset information through the prediction error, the lateral velocity offset information can also be obtained through the lateral velocity.

[0271] As an example, as shown in FIG15 , the vehicle driving data includes a lateral velocity value, and the lateral velocity offset information includes a second lateral velocity offset value; the method for predicting the driving state of a driver based on multimodal data also includes S1501 - S1502 .

[0272] S1501: Determine an initial lateral velocity offset value based on the lateral velocity value.

[0273] S1502: Determine an average of the initial lateral velocity offset values ​​based on the initial lateral velocity offset values ​​and the smoothing coefficient as a second lateral velocity offset value.

[0274] Specifically, the lateral velocity can be directly measured. The lateral velocity value is the absolute value of the lateral velocity. The initial lateral velocity offset value is obtained based on the lateral velocity value. The initial lateral velocity offset value is obtained from the lateral velocity values ​​in multiple windows. For example, the initial time window can be 1s. The initial lateral velocity offset value in the time window i∈{1,...,N} is shown in formula (8):

[0275] Next, the mean of the initial lateral velocity offset value is determined based on the initial lateral velocity offset value and the smoothing coefficient, as shown in formula (9):

[0276] in, is the mean of the initial lateral velocity offset values, is the initial lateral velocity offset value in the i-th time window, is the initial lateral velocity offset value of the i-1th time window, λ is a smoothing parameter, and the mean of the initial lateral velocity offset values ​​can be used as the second lateral velocity offset value. The second lateral velocity offset value is used to determine whether the driving state is risky.

[0277] As an example, the lateral speed offset information indicates that the driver's driving state is risky driving, including: when the second lateral speed offset value is in the risk confidence interval, determining that the lateral speed offset information indicates that the driver's driving state is risky driving.

[0278] Specifically, a risk confidence interval can be set. If the second lateral speed offset value is 0 or relatively small, it means that the degree of lateral offset is small, that is, there is no risky driving. If the second lateral speed offset value falls within the risk confidence interval, it means that the degree of lateral offset is large, that is, there is risky driving.

[0279] As an example, setting the risk confidence interval 1-ɑ can be expressed by formula (10):

[0280] Where L is The 1-ɑ / 2 percentile of the standard normal distribution, σ0 is the standard deviation. ɑ can be set to 0.0027, L = 3. UCL i Represents the upper limit of the risk confidence interval, LCL i Indicates the lower limit of the confidence interval. By judging the second lateral velocity offset value Whether the driving process is risky driving is determined by whether it is within the risk confidence interval. Outside the risk confidence interval, that is, there is no risk driving, if the second lateral speed offset value Falling within the risk confidence interval indicates a large degree of lateral deviation, which means risky driving.

[0281] By inputting the driver's physiological state data into the trained cognitive distraction classification model, an initial prediction result is obtained. The initial prediction result characterizes whether the driver is distracted. If the driver is distracted, it is dangerous driving. If the driver is not distracted, it is not dangerous driving. This result is a preliminary prediction result and needs to be further analyzed in combination with the vehicle driving data. The vehicle driving data includes lateral speed offset information. The lateral speed offset is calculated to determine whether the vehicle's driving state is risky driving. This application proposes two examples of methods for calculating the lateral speed offset, which will not be repeated here. Combining the initial prediction result and the vehicle driving data, a target prediction result is obtained. The target prediction result characterizes whether the vehicle driving deviation is caused by the driver's distraction during the driving process.

[0282] As an example, when the initial prediction result and the lateral speed offset information both indicate that the driving state is risky driving of the driver, determining the driver's driving state is risky driving includes: when the initial prediction result indicates that the driver's driving state within a target number of time periods is risky driving, and the lateral speed offset information indicates that the driver's driving state is risky driving, determining the risky driving level based on the target number, wherein the target number is positively correlated with the level of risky driving, and the risk degree of a larger level is greater than the risk degree of a smaller level.

[0283] Specifically, after the pre-processed eye movement data and EEG data are input in real time, they are passed through the cognitive distraction classification model to obtain a classification result label. For example, a classification result label of 1 indicates that the driving state is risky driving, and a classification label of 0 indicates that the driving state is not risky driving. The lateral speed offset information is in units of time windows. For example, the eye movement data and EEG data of each time window are input into the distraction classification model to obtain the driving state within the corresponding time window period. The risky driving is graded according to the number of time periods (i.e., time windows) in which the driving state is risky driving. Obviously, the more time periods in which the driving state is risky driving, the higher the level of risky driving. That is, the number of targets is positively correlated with the number of risky driving levels, and the risk level of a large level is greater than the risk level of a small level.

[0284] As an example, the risk level can be divided into three levels according to the number of targets, and the target number is denoted as Di. If Di=1: it means that the prediction label result corresponding to the i-th time window is 1 (the physiological state data in the i-th time window indicates that the driver is distracted), and the lateral speed offset information indicates that the driving state is risky driving, the ADAS system warns of risky driving level one; Di=2: it means that the prediction label result corresponding to the i-th time window and the i-1-th time window is 1 (the physiological state data in the i-th time window and the i-1-th time window indicate that the driver is distracted), and the lateral speed offset information indicates that the driving state is risky driving, the ADAS system warns of risky driving level two; Di=3: it means that the prediction label result corresponding to the i-th time window and the i-1 and i-2 time windows is 1 (the physiological state data in the i-th time window and the i-1 and i-2 time windows indicate that the driver is distracted), and the lateral speed offset information indicates that the driving state is risky driving, the ADAS system warns of risky driving level three. Level three is greater than level two, and level two is greater than level one. The higher the level, the greater the risk. It is understandable that more levels of risk levels can be set according to actual conditions.

[0285] As an example, the method for predicting a driver's driving state based on multimodal numbers further includes: outputting risk warning information in a corresponding risk warning manner according to the risk driving level.

[0286] Specifically, different risk warning methods can be output according to the risk driving level. For example, the greater the risk driving level, the stronger the output warning information, such as the louder the warning sound.

[0287] It should be noted that in order to avoid continuous warnings for the same distracting behavior, when the highest level 3 is reached, the first lateral speed offset value and the second lateral speed offset value are reset to 0.

[0288] It should also be noted that when the data is unstable or lost, or when the data transmitted by sensors such as EEG and eye movement is null, Di can be set to 0.

[0289] This application obtains an initial prediction result for the driving state based on the driver's physiological state data; and obtains a target prediction result for the driving state based on the initial prediction result and vehicle driving data. The multimodal data-based driver driving state prediction method of this application can combine the driver's physiological state data and vehicle driving data to predict the driver's state, thereby warning or reminding distracted drivers, thereby improving driving safety.

[0290] In another example, the vehicle's driving mode may be switched based on the target prediction result.

[0291] For example, after obtaining the target prediction result, the vehicle's driving mode can be switched based on the target prediction result. For example, the vehicle's driving mode can initially be either regular driving mode or assisted driving mode. Regular driving mode is controlled by the driver, while assisted driving mode provides assistance to the driver in addition to the automatic driving mode. This shows that switching the vehicle's driving mode based on the target prediction result can improve driving safety.

[0292] In one example, if the target prediction result indicates that the vehicle is in a risky driving state, the vehicle's driving mode is switched to the autonomous driving mode. For example, if the target prediction result indicates that the vehicle is in a risky driving state, it further indicates that the driver's driving condition is poor, such as distraction or fatigue, when driving in the conventional driving mode or the assisted driving mode. In this case, switching to the autonomous driving mode is necessary to ensure driving safety.

[0293] In one example, after a vehicle's driving mode is switched to autonomous driving mode, the driver's comfort level in autonomous driving mode can be monitored in real time while the vehicle is operating in autonomous driving mode. This comfort level indicates whether the driver trusts the autonomous driving mode, is able to relax, and maintain emotional stability.

[0294] When the comfort information indicates that the driver's comfort level is lower than the preset level, it means that the driver does not trust the automatic driving mode enough. In the automatic driving mode, the driver is in a state of tension, fatigue, and emotional instability. At this time, the vehicle's driving mode can be switched from the automatic driving mode to the assisted driving mode or the conventional driving mode to improve the driver's comfort.

[0295] In one example, when a vehicle is traveling in an automatic driving mode, at least one of the driver's physiological information and vehicle driving information can be collected, and based on at least one of the physiological information and vehicle driving information, the driver's comfort information can be detected. Physiological information includes, for example, the driver's EEG data, emotional data, fatigue data, skin conductance data, etc. The physiological information can be processed to obtain the driver's comfort, for example, whether the driver is nervous or tired. Vehicle driving information includes, for example, the vehicle's steering angle, speed, acceleration, whether the lane is changed during driving, etc. The vehicle's driving information will affect the driver's comfort to a certain extent. For example, the driver may be nervous when the vehicle is turning sharply, driving fast, or changing lanes frequently. Therefore, the physiological information and vehicle driving information can be combined when detecting the driver's comfort information.

[0296] Another aspect of the present application provides a driving state monitoring and feedback system based on multimodal human factor intelligent data analysis, also known as a driving state monitoring and feedback system based on multimodal human factor data analysis, and a corresponding driving state monitoring and feedback method based on multimodal human factor data analysis. The system includes a state recognition subsystem and a driving intervention subsystem. The state recognition subsystem is used to receive multimodal human factor data of the test driver collected in real time, pre-process the multimodal human factor data, and send the pre-processed multimodal human factor data into a pre-trained first state recognition model to obtain a real-time recognition of the driver's state; wherein the human factor data includes multiple types of EEG data, heart rate data, skin conductance data, respiratory data, near-infrared data, blood oxygen data, blood pressure data and skin temperature data, the pre-processing includes noise reduction processing and data normalization processing, the driver's state includes a normal state and multiple abnormal states, and the categories of abnormal states include multiple types of fatigue state, distraction state and anger state. The driving intervention subsystem is used to generate driving status feedback instructions to the driving intervention system for different categories of abnormal states when it identifies the driver's state as an abnormal state, so that the driving intervention system can adjust the driver's state feedback based on the received driving status feedback instructions.

[0297] Among them, the first state recognition model collects the baseline human factors data and demographic data of the test driver, retrieves the driver-related data that is within a preset similarity range compared with the baseline human factors data and demographic data of the test driver in the baseline state database, uses the retrieved baseline human factors data of the driver and the driver state data stored in the baseline state database as the training set, and uses the pre-selected normal state and multiple abnormal states as labels, and then iterative training is obtained.

[0298] Specifically, in one embodiment of the present application, driver demographic data, such as age, gender, height, and weight, is recorded, and baseline human factor data is collected for 5 minutes. The baseline human factor data includes heart rate, electrodermal conduction, respiration, EEG, near-infrared, blood oxygenation, blood pressure, and skin temperature data. During the training process of the first state recognition model, based on the driver demographic data and baseline human factor data, a relevant algorithm is used to find similar driver data in the system's existing baseline state database (driver demographic data and baseline human factor data can be filtered according to a preset deviation range). This data is then trained and classified into fatigue, distraction, anger, and normal states. During the testing process after model training, the driver's human factor physiological data collection device is used to collect and monitor heart rate, electrodermal conduction, respiration, EEG, near-infrared, blood oxygenation, blood pressure, and skin temperature data in real time after signal correction. When using the first state recognition model to identify driving states, the preprocessed data is fed into the trained model to obtain the driving state recognition result.

[0299] Preprocessing of ECG data involves applying denoising algorithms, such as wavelet filtering, Kalman filtering, and empirical mode decomposition, to the raw data to remove myoelectric interference, power frequency interference, and baseline drift correction. The data is then normalized. Preprocessing of electrodermal data involves applying denoising algorithms to the raw data, using the values ​​of a time window as the baseline and the values ​​of a time window as the maximum response. The data is then normalized. Noise reduction and data normalization are performed on data such as respiration, skin temperature, blood pressure, and blood oxygen. Data normalization involves converting data into data that follows a normal distribution. The normal distribution is a common probability distribution characterized by symmetry, uniformity, and predictability. Data normalization can improve the accuracy and efficiency of data analysis and is widely used in machine learning, statistics, data mining, and other fields.

[0300] Figure 16 is a schematic diagram of the system structure for driving state monitoring and feedback based on multimodal human factors intelligent data analysis in one embodiment of the present application. Figure 16 specifically illustrates the contents of the driving state recognition system and the driving intervention system.

[0301] The driving state recognition system monitors real-time heart rate, electrodermal conductivity, respiration, EEG, near-infrared, blood oxygen, blood pressure, and skin temperature data, and feeds this data into a driving state recognition model, which then detects fatigue, distraction, anger, or normal states in the form of labels. The driving state model is trained based on a baseline state database. Normal state labels require no processing, but abnormal state labels (fatigue, distraction, or anger) are addressed through commands from the driving intervention system, which control the fragrance module to spray scents, the voice module to provide voice or music prompts, and seat vibration, cooling air, and backrest angle adjustment to provide reminders.

[0302] Generally speaking, the system is installed on the vehicle system, and the vehicle system establishes a command connection with the fragrance module, voice module and smart seat in the vehicle.

[0303] This system is divided into two parts: the first part is the driving state recognition system, and the other part is the driving intervention system. In the driving state detection system, the driver's physiological monitoring data, including heart rate, skin conduction, respiration, EEG, near infrared, blood oxygen, blood pressure and skin temperature data, can be collected and multiple states can be identified through the model trained by the deep learning algorithm. For example, it can include: (1) fatigue state, (2) distraction state, (3) anger state, and (4) normal state. After the state is identified, the fragrance, voice module and seat in the driving intervention system are used to issue warnings and intervene.

[0304] Once the driving state is identified, the driving intervention system takes action. Specifically, if a negative emotional state is detected, the system can provide feedback such as soothing music or positive stories to adjust the driver's state. It can also instruct the vehicle to enter alert mode and take emergency action if the driver engages in excessive driving. If driver fatigue is detected, the system further assesses the driver's mood. Because drivers need to avoid excessive fatigue, which can lead to negative emotions, further assessment is required to assess the fatigue state, obtain physiological information, and further assess the emotional state. If the emotional state is negative, proactive feedback and adjustment are required. For example, the in-vehicle entertainment system can be activated. It can provide music, radio, videos, and other features to alleviate driver fatigue and boredom. These features can help drivers relax and improve driving comfort and safety. Alternatively, the intelligent voice interaction system can be activated, using its voice recognition and natural language processing technologies to interact with the driver, providing services such as navigation, traffic information, and phone calls. These services can help drivers better understand traffic conditions and use various vehicle functions, improving driving convenience and safety. Alternatively, a simultaneous broadcast can be enabled, simultaneously sending the vehicle's information as a tag to vehicles within the currently set area. The vehicle's communication system can also exchange information with the driver and other vehicles, notifying them of the vehicle's location and current driver status. This status is broadcast within the permitted range with the driver's prior permission, thus preventing accidents. It can also provide feedback on the vehicle's assisted autonomous driving. Intelligent driving assistance systems monitor the driver's status and vehicle driving conditions, providing warnings and assistive actions such as automatic braking, lane keeping, blind spot monitoring, and adaptive cruise control.

[0305] Corresponding to the above method, the present application also provides an edge computing terminal device, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps of the method described above.

[0306] In addition, the edge computing terminal device may also include smart sensors and programmable logic controllers (PLCs) for collecting data of the steps of the method described in the above embodiments.

[0307] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the art.

[0308] The method and system for driving status monitoring and feedback based on multimodal human factors intelligent data analysis proposed in this application can use at least a pre-trained first state recognition model to process the pre-processed multimodal human factors data collected in real time, distinguish the driver's normal state and abnormal state through classification labels, identify the driver's different driving states, and then process different driving states to avoid traffic accidents.

Claims

1. A driving status monitoring and feedback method based on multimodal human factors intelligence data analysis, characterized in that: The method comprises: Receive multimodal human factor data of the test driver collected in real time; Preprocessing the multimodal human factor data, wherein the preprocessing includes noise reduction processing and data normalization processing; The pre-processed multimodal human factor data is fed into a pre-trained first state recognition model to obtain a real-time recognized driver state; the driver state includes a normal state and a plurality of abnormal states; the category of the abnormal state includes one or more of a fatigue state, a distracted state and an angry state; When it is identified that the driver's state is an abnormal state, driving state feedback instructions are generated for different categories of abnormal states to the driving intervention system, so that the driving intervention system performs state feedback adjustment on the driver based on the received driving state feedback instructions.

2. The method according to claim 1, characterized in that: During the pre-training process of the first state recognition model, the baseline human factor data and demographic data of the test driver are first collected, and the driver-related data that is within a preset similarity range compared to the baseline human factor data and demographic data of the test driver is retrieved from the baseline state database. The retrieved driver's baseline human factor data and the driver's state data stored in the baseline state database are used as training sets, and the pre-selected normal state and multiple abnormal states are used as labels. The first state recognition model is obtained through iterative training.

3. The method according to claim 1, characterized in that The multimodal human factors data includes one or more of electroencephalogram data, heart rate data, electromyography data, eye movement data, skin conductance data, respiratory data, near infrared data, blood oxygen data, blood pressure data and skin temperature data.

4. The method according to claim 1, characterized in that: The noise reduction processing step comprises: Applying one or more of wavelet filtering, Kalman filtering and empirical mode decomposition to the ECG data; The value of a time window is used as the baseline value, and the value of a time window is used as the maximum response value, and the skin electrical data is subjected to noise reduction processing.

5. The method according to claim 1, characterized in that The preprocessing of the EEG data included in the multimodal human factors data includes whole-brain average reference, filtering processing, ICA analysis and time-frequency analysis to extract the target band of the EEG data, the EEG data is screened and artifacts are removed by the filtering processing, the blink segments and / or electromyography segments are identified and removed by the ICA analysis, and the target band of the EEG data is extracted by the time-frequency analysis; wherein the filtering processing includes high-pass filtering processing, low-pass filtering processing, notch filtering processing and band-pass filtering processing.

6. The method according to claim 5, characterized in that The preprocessing of the EEG data included in the multimodal human factors data also includes a data cleaning step before the whole brain average reference, and artifacts and noise are removed by the data cleaning method; The preprocessing of the EEG data included in the multimodal human factors data also includes a step of deleting useless channels and a step of removing artifacts based on a deep learning method before ICA analysis.

7. The method according to claim 1, characterized in that The multimodal human factor data also includes image data of the test driver's head. After preprocessing the multimodal human factor data, the method further includes: Based on image recognition technology, the image data of the test driver's head is analyzed to obtain the driver's eye movement characteristics, head movement characteristics and facial expression characteristics. The eye movement characteristics and head movement characteristics of the driver are analyzed in combination with gaze tracking technology. The facial expression characteristics are analyzed through expression recognition technology to obtain real-time recognition of the driver's status.

8. The method according to claim 1, characterized in that After preprocessing the multimodal human factor data, the method further includes: Extract features from the preprocessed multimodal human factor data to obtain features related to driving behavior; Tagging the features related to the driving behavior, and when data related to the tags are detected, analyzing the features related to the driving behavior to obtain a real-time identification of the driver's state; The characteristics related to driving behavior include one or more of heart rate, blood pressure and respiratory rate.

9. The method according to claim 1, characterized in that: The first state recognition model is a classification model, and the type of the classification model includes any one of a vector machine model, a decision tree model and a naive Bayes model.

10. The method according to claim 1, characterized in that The driver state also includes the driver's cognitive state. After preprocessing the multimodal human factors data, the method also includes: obtaining the driver's operating data including the speed and force of braking during driving, counting the driver's operating habits based on the driver's historical operating data collected from historical statistics and setting a driver's cognitive state recognition threshold; when the driver's operating data collected in real time during driving exceeds the driver's cognitive state recognition threshold, it is determined that the driver's cognitive state is abnormal.

11. The method according to claim 1, characterized in that: The driving intervention system comprises a voice module and a seat, and the step of the driving intervention system performing state feedback adjustment on the driver based on the received driving state feedback instruction comprises: Use the voice module to give voice reminders or play music to soothe the driver; Automatically adjust the seat angle to remind the driver to adjust the driving state.

12. The method according to claim 11, characterized in that When the virtual driving environment belongs to a multi-task scenario, the multimodal human factors data includes eye movement data, and the method further includes: First, calibrate the baseline of the collected multimodal human factors data; Extract and train multimodal human factors data during the measurement process; The eye movement interest areas are identified based on gaze tracking technology, and the relationship characteristics between each eye movement interest area and driving task scene are extracted based on the SEEV model.

13. The method according to claim 1, wherein generating driving state feedback instructions for different types of abnormal states to a driving intervention system so that the driving intervention system performs state feedback adjustment on the driver based on the received driving state feedback instructions comprises: Acquire driver status detection data, vehicle status detection data, and road environment status detection data uploaded by a monitoring terminal, wherein the monitoring terminal includes a plurality of detection devices, and the detection devices include at least a physiological signal detection device, a video signal detection device, and a vehicle signal detection device; Analyzing the driver state detection data, the vehicle state detection data, and the road environment state detection data to obtain a driving state evaluation result; A warning is issued based on the driving status evaluation result.

14. The method according to claim 13, characterized in that Also includes: When the vehicle is traveling in a set state along a set route, collecting first state detection data of the driver; Performing standardization processing according to the collected first state detection data of the driver; The driver state detection data acquired in real time is associated with the first state detection data of the driver after standardization.

15. The method according to claim 14, characterized in that After associating the driver state detection data acquired in real time with the first state detection data of the driver after standardization, the method further includes: The driver state detection data is subjected to noise processing based on the first state detection data of the driver to obtain the second state detection data of the driver, wherein the first state detection data of the driver at least includes the physiological signal, the first eye movement signal, the electroencephalogram and brain imaging signal, and the behavior detection data of the driver; The second state detection data of the driver, the vehicle state detection data and the road environment state detection data are analyzed to obtain the driving state evaluation result.

16. The method according to claim 15, characterized in that The driver state detection data is subjected to noise processing based on the first state detection data of the driver, comprising: The driver state detection data includes a second eye movement signal, and the second eye movement signal is obtained by parsing the video content acquired in real time by the video signal detection device; In combination with the first eye movement signal, artifact noise of the second eye movement signal is determined, and the artifact noise is removed.

17. The method according to claim 13, characterized in that: The analyzing the driver state detection data, the vehicle state detection data and the road environment state detection data comprises: Inputting the driver state detection data, the vehicle state detection data and the road environment state detection data into corresponding state recognition models respectively, and identifying the driver state, the vehicle state and the road environment state; The driving state evaluation result is obtained by combining the driver state, the vehicle state and the road environment state.

18. The method according to claim 17, characterized in that The combining the driver state, the vehicle state and the road environment state to obtain the driving state evaluation result comprises: If the driver state is mild fatigue, the vehicle state is normal driving state, and the road environment state is no vehicle or pedestrian ahead, then the driving state assessment result is a first driving risk state; If the driver state is moderate fatigue, the vehicle state is an abnormal driving state, and the road environment state is no vehicle or pedestrian ahead, the driving state assessment result is a second driving risk state; If the driver state is severe fatigue, the vehicle state is an abnormal driving state, and the road environment state is another vehicle cutting in, then the driving state assessment result is a third driving risk state.

19. The method according to claim 13, characterized in that Also includes: Marking bad driving habits based on the driver status detection data, wherein the bad driving habits at least include smoking, making phone calls and cutting in; Counting the number of occurrences of the bad driving habits; When the number of occurrences of the bad driving habit behavior is greater than a preset number, the driver is reminded or warned.

20. The method according to claim 13, characterized in that Also includes: Determine whether the signal detected by the detection device is a normal signal by using a neural network algorithm, If the signal is an abnormal signal, the driver is prompted to debug the equipment through voice or image.

21. The method according to claim 1, characterized in that After receiving the multimodal human factors data of the test driver collected in real time, it also includes: Collecting behavior data of the driver; extracting at least one independent variable feature of the driver according to the multimodal human factor data and the behavior data; The actual fatigue state of the driver is identified according to the at least one independent variable feature, and the vehicle is controlled to provide a status reminder to the driver based on the actual fatigue state.

22. The method according to claim 21, characterized in that The collecting of driver's behavior data includes: At least one change data of the driver's head posture, blinking frequency, eye opening and closing state, and facial expression within a preset time period is collected as the behavior data.

23. The method according to claim 21, characterized in that The extracting at least one independent variable feature of the driver according to the multimodal human factor data and the behavior data comprises: Preprocessing the multimodal human factor data and the behavioral data to obtain processed collected data; At least one energy ratio index is extracted from the collected data as the at least one independent variable feature.

24. The method according to claim 21, characterized in that The identifying the actual fatigue state of the driver according to the at least one independent variable feature comprises: The at least one independent variable feature is input into a pre-constructed fatigue state recognition model, and the actual fatigue state is output, wherein the fatigue state recognition model is constructed by the KSS scale scores before and after driving and multiple fatigue states.

25. The method according to claim 21, characterized in that The identifying the actual fatigue state of the driver according to the at least one independent variable feature comprises: Acquiring the current operating condition and / or current environment of the vehicle; Generating a weight of each independent variable feature in the at least one independent variable feature according to the current working condition and / or the current environment; The actual fatigue state is determined according to the at least one independent variable feature and the corresponding weight.

26. The method according to claim 21, characterized in that The providing a status reminder to the driver based on the actual fatigue status includes: Determining whether the actual fatigue state meets a preset reminder condition; If the actual fatigue state meets the preset reminder condition, a target reminder type and a reminder signal are matched according to the actual fatigue state, and the vehicle is controlled to perform a status reminder based on the reminder signal according to the target reminder type.

27. The method according to claim 26, characterized in that The preset reminder condition is that the fatigue level of the actual fatigue state is greater than a preset level.

28. The method according to claim 3, characterized in that After receiving the multimodal human factors data of the test driver collected in real time, it also includes: Inputting the multimodal human factor data of the driver into a trained deep learning model for prediction, and obtaining an initial prediction result for the driving state of the driver, wherein the initial prediction result represents whether the driver is distracted during driving; Based on the initial prediction result and the vehicle driving data, a target prediction result for the driving state of the vehicle is obtained, wherein the target prediction result represents whether the vehicle driving deviation is caused by the driver's distraction during the driving process.

29. The method according to claim 28, characterized in that The vehicle driving data includes a lateral speed value and lateral speed offset information; the target prediction result for the driving state of the vehicle is obtained based on the initial prediction result and the vehicle driving data, including: When both the initial prediction result and the lateral speed deviation information indicate that the driving state is risky driving, determining that the driving state of the driver is risky driving; When the lateral speed value is less than or equal to a preset speed threshold, determining the driving state of the driver based on the initial prediction result and the lateral speed offset information; When both the initial prediction result and the lateral speed deviation information indicate that the driving state is risky driving, it is determined that the driving state of the driver is risky driving.

30. The method according to claim 29, characterized in that The vehicle driving data includes a true value of kinematic information, and the lateral velocity offset information includes a first lateral velocity offset value; the method further includes: Using multiple spatial state models to make predictions based on the real values ​​of the kinematic information, a plurality of predicted values ​​corresponding to the multiple spatial state models are obtained, and based on the weights corresponding to the multiple spatial state models, the plurality of predicted values ​​are weightedly calculated to predict the predicted values ​​of the kinematic information; Obtaining a prediction error of the kinematic information based on a true value of the kinematic information and a predicted value of the kinematic information; Based on the prediction error, the first lateral speed offset value is obtained, so as to determine that the lateral speed offset information indicates that the driving state of the driver is risky driving when the first lateral speed offset value is greater than a preset speed offset threshold.

31. The method according to claim 28 or 29, characterized in that The vehicle driving data further includes a lateral speed value, and the lateral speed offset information includes a second lateral speed offset value; the method further includes: determining an initial lateral velocity offset value based on the lateral velocity value; Based on the initial lateral velocity offset value and the smoothing coefficient, a mean value of the initial lateral velocity offset value is determined as the second lateral velocity offset value, and when the second lateral velocity offset value is in a risk confidence interval, it is determined that the lateral velocity offset information indicates that the driving state of the driver is risky driving.

32. The method according to claim 29, characterized in that In a case where both the initial prediction result and the lateral speed offset information indicate that the driving state is risky driving of the driver, determining that the driving state of the driver is risky driving includes: When the initial prediction result indicates that the driving state of the driver within a target number of time periods is risky driving, and the lateral speed offset information indicates that the driving state of the driver is risky driving, determining a risky driving level based on the target number, The target quantity is positively correlated with the level of risky driving, and the risk level of a larger level is greater than the risk level of a smaller level.

33. The method according to claim 28, characterized in that: The step of inputting the multimodal human factor data of the driver into a trained deep learning model for prediction to obtain an initial prediction result for the driving state of the driver includes preprocessing the eye movement data, including: A. removing data with abnormal pupil size changes, pupil occlusion, and artifacts at the pupil edge from the eye movement data; B. removing data representing gaze deviation from the eye movement data; C. removing the data in the eye movement data where the sight line is outside the area of ​​interest; D. Remove data in the eye movement data indicating that the scanning angular velocity is greater than a preset angular velocity.

34. The method according to claim 28, characterized in that The method of inputting the multimodal human factor data of the driver into a trained deep learning model for prediction to obtain an initial prediction result for the driving state of the driver also includes preprocessing the EEG data, including: A. Averaging the EEG data of multiple channels to obtain the mean value, and retaining the difference between the EEG data of each channel and the mean value; B. filtering the EEG data and retaining data in a preset band; C. removing interference data caused by blinking or body movement in the EEG data; D. Extract features from the EEG data to obtain power spectrum density feature data for a specific band.

35. The method according to claim 28 or 29, characterized in that The method further comprises: When the target prediction result indicates that the vehicle is in a risky driving state, switching the driving mode of the vehicle to an automatic driving mode; When the vehicle is traveling in the automatic driving mode, collecting at least one of the driver's physiological information and the vehicle's driving information to detect the driver's comfort information; When the comfort information indicates that the driver's comfort level is less than a preset level, the driving mode of the vehicle is switched from the automatic driving mode to the assisted driving mode or the conventional driving mode.

36. A driving status monitoring and feedback system based on multimodal human factors intelligent data analysis, characterized in that: The system comprises a state recognition subsystem and a driving intervention subsystem, wherein: The state recognition subsystem is used to receive multimodal human factor data of the test driver collected in real time, preprocess the multimodal human factor data, and send the preprocessed multimodal human factor data to a pre-trained first state recognition model to obtain a real-time recognized driver state; wherein the human factor data includes multiple types of EEG data, heart rate data, skin conductance data, breathing data, near infrared data, blood oxygen data, blood pressure data and skin temperature data, the preprocessing includes noise reduction processing and data normalization processing, the driver state includes a normal state and multiple abnormal states, and the categories of the abnormal state include multiple types of fatigue state, distraction state and anger state; The driving intervention subsystem is used to generate driving state feedback instructions to the driving intervention system for different types of abnormal states when recognizing that the driver's state is an abnormal state, so that the driving intervention system performs state feedback adjustment on the driver based on the received driving state feedback instructions; Among them, the first state recognition model collects baseline human factor data and demographic data of the test driver, retrieves driver-related data within a preset similarity range compared to the baseline human factor data and demographic data of the test driver in the baseline state database, uses the retrieved driver's baseline human factor data and the driver's state data stored in the baseline state database as a training set, and uses pre-selected normal states and multiple abnormal states as labels, and then iteratively trained to obtain it.

37. An edge computing terminal device, comprising a processor and a memory, characterized in that: The memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps of the method as claimed in any one of claims 1 to 35.

38. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 35 are implemented.

Images