A driving situation understanding method based on human-machine augmented perception

Abstract

The application discloses a driving situation understanding method based on human-computer enhanced perception, and the method comprises the following steps: first, integrating multi-source human-computer perception data; second, analyzing driving attributes of a driver; third, constructing a human-computer enhanced perception model; and fourth, constructing a driving situation understanding model; the beneficial effects are as follows: a personalized driving habit calculation model is constructed, so that the automatic driving perception system is more humanized and personalized; autonomous perception is realized according to the driving habit type, the driver's perception mode and the perception logic at the current moment, human-computer fusion perception semantic inference of the current driving situation is realized, accurate and comprehensive perception information is provided for intelligent vehicle decision-making, and the driving situation complexity and the danger degree can be calculated in real time, and the traffic situation can be evaluated.

Description

Technical Field

[0001] This invention relates to a driving situation understanding method, and more particularly to a driving situation understanding method based on human-machine augmented perception. Background Technology

[0002] Currently, autonomous driving perception systems are a crucial component in realizing autonomous driving technology. These systems acquire data through high-precision, multi-sensor systems mounted on vehicles. After data processing and fusion, they enable precise perception of the vehicle's surroundings, providing accurate and comprehensive information for the autonomous vehicle's decision-making system. Compared to human perception, machine perception offers advantages in terms of high precision and continuity. Machine perception has a broad dimension, acquiring massive amounts of data during the perception process. However, some data has no substantial impact on the autonomous driving system's understanding of the surrounding environment, and processing and fusing this data can lead to resource waste. Furthermore, machine perception cannot cover complex traffic situations and extreme scenarios, and has not yet reached the level of human intelligence. When performing perception tasks, drivers can quickly and effectively capture key elements in a scene and, under certain conditions, can understand the current driving situation well and make reasonable decisions. However, drivers are easily affected by physiological and psychological factors, making it difficult to maintain a good driving state for extended periods, thus posing safety hazards to driving tasks.

[0003] Due to the immaturity of autonomous driving technology and legal constraints, human-machine co-driving technology, where humans are in the loop, is currently a hot research topic. At present, compared to the control layer of human-machine co-driving technology, research on the perception layer is relatively limited. How to leverage the respective strengths of both driver and machine perception to achieve a driver-centric autonomous driving perception system is crucial for a comprehensive and accurate understanding of driving situations. Research on driving situation understanding methods based on enhanced human-machine perception has significant guiding implications for autonomous driving behavior decision-making. Summary of the Invention

[0004] The purpose of this invention is to leverage the respective strengths of driver and machine perception to realize a driver-centric autonomous driving perception system, thereby achieving a comprehensive and accurate understanding of the driving situation. This invention provides a driving situation understanding method based on human-machine enhanced perception.

[0005] The driving situation understanding method based on human-machine enhanced perception provided by this invention includes the following steps:

[0006] The first step is to integrate multi-source human-machine perception data. The specific process is as follows:

[0007] Step 1: Integration of driver physiological data, which includes electrical signals from electroencephalography (EEG), electromyography (EMG), electrocardiography (ECG), electrooculography (EOG), and electrodermal osmosis (EDS).

[0008] Step 2: Integration of driver external characteristic data. When performing driving tasks, the driver's external characteristics are manifested in head movement, eye movement, and limb operation to control the steering wheel, accelerator and brake pedals.

[0009] Step 3: Integration of traffic environment and vehicle status data. In addition to integrating the data from Step 1 and Step 2, the driver's perception data from Step 1 and Step 2 and the machine perception data from Step 3 are processed to achieve time and space synchronization. For different driving behaviors, parameter variables describing different driving behaviors are extracted to obtain a driving behavior feature matrix and analyze the relationship with the driver's driving behavior.

[0010] The second step is to analyze the driver's driving attributes. The specific process is as follows:

[0011] Step 1: Online driving intention calculation model – Analyze the driver's driving attributes and establish an online driving intention model;

[0012] Step 2: Personalized driving habit calculation model. In order to improve the driving safety and comfort of intelligent vehicles and make the autonomous driving system more humanized and personalized, the driver's driving habits are judged. The driver's driving habits are divided into robust, general and aggressive types. By constructing a personalized driving habit calculation model, the driver's driving attributes are analyzed.

[0013] Step 3: Driver Perception Pattern Analysis. In order to study the driving attributes of drivers, the perception patterns of drivers with different habits are analyzed, including the following perception patterns, lane changing perception patterns, fatigue perception patterns, and emergency condition perception patterns of aggressive, conservative, and general drivers.

[0014] The third step is to construct a human-machine enhanced perception model, and the specific steps are as follows:

[0015] Step 1: Establish a driver perception mechanism model;

[0016] Step 2: Establish a human-centered perception model for the autonomous driving system. The specific method is as follows:

[0017] By inputting the driver's lateral and longitudinal perception mechanism models into the human-centered autonomous driving system perception model, on the one hand, the autonomous driving perception system is driver-centric. It autonomously perceives based on the driver's current habit type, perception mode, and perception logic, and determines the priority of traffic participants based on the region of interest. If the human-centered autonomous driving perception system obtains the lane-changing intention of the current aggressive driver, the autonomous driving system assigns higher weights to traffic participants of interest in the current lane and the left lane, and lower weights to traffic participants perceived in the right lane. The information is incorporated into the autonomous driving system perception information step by step according to the weight. On the other hand, the autonomous driving system needs to evaluate the rationality of the driver's perception information, whether the current driver's perception information is accurate and whether there are any omissions, and complete the integration of longitudinal and lateral perception information of the human-centered autonomous driving perception system.

[0018] Step 3: Human-Machine Perception Consistency Comparison Model:

[0019] The driver perception mechanism model from step one and the human-centered autonomous driving system perception mechanism model from step two are input into the human-machine perception consistency comparison model. The driver's perception results and the human-centered autonomous driving system perception results are compared. If the driver's perception results and the autonomous driving system perception results are consistent, the perception results are directly output. If the driver's perception results and the autonomous driving system perception results are inconsistent, the autonomous driving system determines the driver's current state, verifies the accuracy and comprehensiveness of the perception information caused by the driver's state, and compensates the information perceived by the autonomous driving system into the human-machine perception consistency comparison model with a set weight, and finally outputs the human-machine perception results.

[0020] Step 4: Human-Machine Perception Data Fusion Model. After comparing the perception results of the human-machine perception consistency model, the driver's perception information and the autonomous driving system's perception information are determined, and valuable information is input into the human-machine perception data fusion model;

[0021] Step 4: Construct a driving situation understanding model. The specific steps are as follows:

[0022] Step 1: Human-machine fusion perception and semantic inference;

[0023] Step 2: Driving Situation Analysis, the specific methods are as follows:

[0024] The driving situation evolves continuously during driving. How the driver's area of ​​interest changes is analyzed and predicted to obtain open driving situation results. If the vehicle changes lanes, how the trajectory of the vehicle and surrounding traffic vehicles evolves to analyze the driving situation situation.

[0025] Step 3: Assessment of driving scenario complexity, the specific methods are as follows:

[0026] Driving situation complexity is represented by road dynamic traffic environment complexity and road topology complexity. Several drivers with rich driving experience evaluate the dynamic traffic environment complexity and road topology complexity. The evaluation indicators for dynamic traffic environment complexity include, but are not limited to, the number, density, and type of traffic participants; the evaluation indicators for road topology complexity include, but are not limited to, road slope, road curvature, adhesion coefficient, water accumulation, and flatness. Using the analytic hierarchy process (AHP), the complexity weights of different evaluation indicators for different dynamic traffic environments and different road topologies are obtained by comparing the importance of the above evaluation indicators. The road dynamic traffic environment complexity and road topology complexity are calculated according to the evaluation indicators and their weights, and mapped to the interval [0,1].

[0027] Step 4: Risk assessment of the driving situation, the specific methods are as follows:

[0028] The factors influencing the hazard level of a driving situation are the collision time between the vehicle and surrounding vehicles and the driver's fatigue state. The driver's fatigue state is defined as a dimensionless number between 0 and 1, where "0" represents a good driver condition and "1" represents an extremely fatigued driver. The higher the value, the greater the hazard level of the driving situation. The shorter the collision time, the lower the hazard level of the driving situation. When the driver is extremely fatigued, the collision time is shorter, making it easier for a traffic accident to occur, and the resulting driving situation hazard level is also higher. The hazard level of the driving situation is assessed based on the above two factors.

[0029] The first step, step one, of integrating driver physiological data includes the following steps:

[0030] Step 1: Acquisition of driver's bioelectrical signals, the specific method is as follows:

[0031] By collecting bioelectrical signals from drivers wearing sensors, it is possible to study the longitudinal and lateral driving intentions of drivers. The sensors also collect electromyographic signals from the driver's lower leg and forearm surface, as well as electrocardiogram, electrooculogram, and electrodermal signals.

[0032] Step 2: Noise reduction processing of driver's bioelectrical signals, the specific methods are as follows:

[0033] Electroencephalogram (EEG) signals are extremely weak physiological electrical signals that are easily affected by external environmental interference. Therefore, preprocessing of EEG data is necessary. The frequency range of EEG signals is selected from 1-30Hz, and filters are used to remove high and low frequency noise mixed into the EEG signals, thus completing the preprocessing of EEG signals. Based on the frequency of EEG waves, they are divided into delta waves, theta waves, alpha waves, and beta waves. At the same time, the electrical signals of electromyography (EMG), electrocardiography (ECG), electrooculography (EOG), and electrodermal conductance (EDC) of drivers are also preprocessed.

[0034] Step 3: Synchronization processing of driver's bioelectrical signals, the specific methods are as follows:

[0035] There is a time difference in the data collected by different sensors, so the above signals are preprocessed for time synchronization.

[0036] Step 4: Extraction of driver's bioelectrical signal features, the specific methods are as follows:

[0037] By extracting feature parameters from bioelectrical signals to form feature vectors, which serve as important components of the driver's intention recognition model, the driver's EEG features are obtained through various EEG data feature extraction methods. The driver's electromyography (EMG) signals are then analyzed using time-frequency domain feature analysis to extract the main features of the EMG signals that reflect the driver's actions. Similarly, the driver's electrocardiogram (ECG), electrooculogram (EOG), and electrodermal signal features are extracted. Finally, the driver's EEG, EMG, ECG, EOG, and EEG signals are integrated to achieve a comprehensive representation of the driver's state by combining the driver's bioelectrical signals.

[0038] The integration of driver external characteristic data in Step 1 and Step 2 includes the following steps:

[0039] Step 1: Acquisition of driver's limb control, head movement, and eye movement signals. The specific methods are as follows:

[0040] The camera extracts video of the driver's head movement, and the eye tracker obtains the driver's gaze. It records the relationship between the gaze area and gaze time when the driver performs different driving actions such as following, braking and changing lanes. If the driver is preparing to change lanes, the time and number of times the driver looks at the rearview mirror are collected. Different steering wheel speeds, acceleration and brake pedal speeds can reflect the driver's driving intentions. If it is normal braking and emergency braking, the sensors obtain steering wheel, acceleration and brake pedal signals.

[0041] Step 2: Signal synchronization processing, the specific methods are as follows:

[0042] The signals collected by the different sensors mentioned above have a time difference, so it is also necessary to perform time synchronization preprocessing on the collected data, match the time series of head movement, eye movement behavior and driving behavior data, and complete the integration of driver external characteristic data to achieve time synchronization;

[0043] Step 3: Extraction of driver's external characteristics. The specific methods are as follows:

[0044] Features representing the driver's external characteristics are extracted, including saccade time, saccade amplitude, standard deviation of head rotation angle, rearview mirror attention allocation rate, steering wheel speed, accelerator pedal, brake pedal position and speed.

[0045] The first step, step three, integrates traffic environment and vehicle status data, including the following steps.

[0046] Step 1: Information collection on this vehicle and surrounding vehicles and pedestrians. The specific methods are as follows:

[0047] The vehicle acquires dynamic traffic participant information, including the number, location, speed, and acceleration of traffic participants, as well as the collision time between the vehicle and other vehicles, through onboard sensors and vehicle-road cooperative equipment.

[0048] Step Two: Collection of information on lane lines, signs, and traffic lights. The specific methods are as follows:

[0049] Collect traffic environment information, including lane lines, traffic signs, traffic lights, and static obstacles.

[0050] Step 1 of the second step includes the following steps:

[0051] Step 1: Feature selection of input data. The specific method is as follows:

[0052] The online driving intention calculation model uses a neural network model, with the vehicle speed V as the input feature parameter. ego Steering wheel angle θ steer Steering wheel speed w steer Accelerator pedal displacement S throttle Accelerator pedal speed V throttle Brake pedal displacement S braking Brake pedal speed V braking The output is A = (a1,a2,a3,a4,a5), where (1,0,0,0,0), (0,1,0,0,0), (0,0,1,0,0), (0,0,0,1,0), and (0,0,0,0,1) represent driving intentions of changing lanes to the left, changing lanes to the right, accelerating, braking, and holding, respectively.

[0053] Step 2: Determining the neural network model structure, the specific method is as follows:

[0054] In the first step, the data integrated in step two is divided into training and test sets. First, the number of hidden layer nodes is determined. The data of the training set is input into the input layer. Left lane change, right lane change, acceleration, braking and hold are selected as the output layer of the network. The connection weights between the input layer and the hidden layer, and between the hidden layer and the output layer are determined through the forward and backward propagation algorithm.

[0055] Step 3: Output results of the neural network, the specific method is as follows.

[0056] By establishing an online driving intention calculation model, the data collected during driving is input into the online driving intention calculation model to obtain the driver's intention at each sampling time.

[0057] The personalized driving habit calculation model in step two of the second step includes the following components:

[0058] Step 1: Calculation method for personalized driving habits. The specific method is as follows:

[0059] The multi-source human-machine perception data integrated in the first step, after feature extraction, is input into the personalized driving habit calculation model. This includes the signal features of EEG, EMG, ECG, SCEA, and EOG extracted from the driver's physiological data in Step 1; the driver's head movement and eye movement features, as well as the lateral and longitudinal control features, extracted from the driver's external characteristics integrated in Step 2; and the features extracted from the traffic environment and vehicle state data integrated in Step 3, including the relative position, speed, and relative acceleration of the driver and the target vehicle. Due to the high dimensionality of the extracted feature parameters, principal component analysis is used to reduce the dimensionality of the data, retaining the information of the high-dimensional feature parameters. Based on the above feature parameters, the feature data is represented by a sample matrix Z, denoted as:

[0060]

[0061] For an m-dimensional random variable, n experimental observations are performed, z1, z2, ..., z n Indicates the observed sample;

[0062] z j =(z 1j ,z 2j ,…,z mj ) T This represents the j-th observation sample;

[0063] z ij Let i represent the i-th variable of the j-th observation sample, where j = 1, 2, ..., n;

[0064] Standardize the feature parameters, obtain the sample covariance matrix, and calculate the cumulative variance contribution rate of the principal components;

[0065]

[0066] The principal components with a cumulative variance contribution rate of over 90% are selected as the components after dimensionality reduction to obtain a new dataset. The new dataset is then divided into training and test sets.

[0067] Step Two: Calculation Results of Personalized Driving Habits. The specific method is as follows:

[0068] Through cluster analysis, driver styles are divided into three types: aggressive, average, and conservative. Driver-related characteristic data are input into a personalized driving habit calculation model to determine the current driver's driving habits.

[0069] The steps in step three of the second step are as follows:

[0070] Step 1: Analysis of the patterns of car-following perception. The specific methods are as follows:

[0071] Drivers’ visual attention changes when performing driving tasks. The driver’s field of vision is divided into 5 areas, including the area in front 1, the left rearview mirror area 2, the right rearview mirror area 3, the instrument panel area 4, and the interior rearview mirror area 5. The driver’s visual fixation and gaze shift characteristics are studied when performing car-following tasks.

[0072] First, input the driver's head movement and eye movement signals collected in Step 1 and Step 2, and then statistically analyze the gaze duration of each region based on information entropy of the visual gaze characteristics.

[0073] The discrete variable entropy information is E, and the fixation entropy rate value is E. e Calculation formula:

[0074]

[0075] E max =log2 N

[0076]

[0077] In the formula, N represents the number of fixation regions, N = 5. It is the probability of focusing on a certain area; The average gaze time of a driver in a certain area, where n represents the area number;

[0078] Driver gaze behavior mainly includes the number of times and duration of gaze at the target object, which characterizes the distribution and degree of interest of the driver in the car-following task. Driver saccadic behavior reflects the shifting characteristics and visual search patterns of the driver in the region of interest.

[0079] A driver gaze shift probability matrix was constructed using Markov chains to analyze the gaze shift characteristics of drivers with different driving habits in car-following tasks.

[0080] M is the one-step transition probability P ij The state transition matrix is ​​represented as follows:

[0081]

[0082] P ij This represents the element in the i-th row and j-th column of the transition matrix probability, where the sum of the transition probabilities in each row is 1;

[0083] Using Markov chain theory, we explore the gaze shift patterns and stationary distribution characteristics of the gaze point between different gaze regions; we analyze the differences in gaze distribution during the car-following phase for drivers with different habits, and obtain the perceptual patterns of drivers with different habits in the car-following task.

[0084] Section Two: Analysis of Lane Change Perception Patterns, with the following specific methods.

[0085] Similar to the driver's perception pattern analysis method during the following phase, lane change perception pattern analysis is performed to analyze the difference in the driver's gaze distribution during the lane change intention phase and the lane change execution phase, and to obtain the driver's gaze shift pattern between the area directly in front of the current lane, the rearview mirror area, and other target areas during the lane change phase.

[0086] Section 3: Analysis of fatigue perception patterns, the specific methods are as follows.

[0087] Fatigue can easily lead to driving risks. By characterizing the driver's perception patterns under fatigue conditions through eye movement, head movement information, and driving behavior characteristics, we can identify the patterns of driver fatigue perception. Eye movement and head movement characteristics include pupil closure time, blinking frequency, head drooping angle, number of nodding, and number of yawning. Driving behavior includes steering wheel correction frequency. By determining the fatigue detection parameters of eye movement, head movement information, and driving behavior, we can characterize the patterns of driver fatigue perception.

[0088] Step 4: Analysis of the patterns of emergency condition perception. The specific methods are as follows.

[0089] This study analyzes drivers' reaction time in emergency situations. Based on the length of drivers' reaction time to emergency situations, an analysis of variance is used to test the results. The length of reaction time significantly affects the probability of saccades of interest and horizontal eye activity. The study analyzes the familiarity and sensitivity to emergency situations with different reaction times to obtain the emergency situation perception patterns of drivers with different habits.

[0090] Step 3, Step 1, includes the following steps:

[0091] Step 1: Driver's longitudinal perception mechanism model, the specific method is as follows:

[0092] The input to the driver's longitudinal perception mechanism model is the current driver's driving intention, driving habits and perception patterns in the second step. If the online calculation model of the aggressive driver's intention outputs acceleration intention at the current moment, based on the longitudinal perception patterns of the aggressive driver, mainly including head movement and eye movement patterns, the current driver's gaze duration patterns and scanning area shift patterns for the dashboard area and the area in front, it is inferred that the current driver needs to perform the accelerator pedal action at the current moment.

[0093] Section Two: Driver Lateral Perception Mechanism, the specific methods are as follows:

[0094] The driver's current driving intention, driving habits, and perception patterns are also input into the driver's lateral perception mechanism model to infer the driver's perception. For example, if the driver intends to change lanes to the left at the current moment, based on the driver's current lane-changing perception patterns, the driver's focus is on the left rearview mirror area and the area in front. Based on the traffic information of the vehicle's current lane and the left lane, it is inferred that the driver intends to change lanes to the left at the current moment. The driver's longitudinal and lateral perception mechanism model outputs the comprehensive information of the driver's perception at the current moment.

[0095] Step 3, Step 4 includes the following steps:

[0096] Step 1: Representation of the driver's perceived information set, the specific method is as follows.

[0097] The driver's perceived information set D = {Target_number, Target_type, Target_position, Target_speed, Lane_information}, where Target_number represents the number of traffic participants currently perceived by the driver, Target_type represents the type of target object, which can be divided into pedestrians, bicycles, passenger cars, trucks, and animals, Target_position represents the position of traffic participants, which can be divided into directly in front, left front, right front, left side, right side, directly behind, left rear, and right rear, and Target_speed represents the speed of movement of traffic participants;

[0098] Part Two: Representation of Autonomous Driving Perception Information Sets, with the following specific methods:

[0099] The information set perceived by the autonomous driving system is A = {target_ID, target_type, target_size, target_distance, target_speed, target_acceleration, lane_ID, lane_style}, where target_ID represents the ID of the target object obtained by the sensor; target_type represents the type of the target object; target_size represents the size of the target object; target_distance represents the distance of the target object; target_speed represents the speed of the target object; target_acceleration represents the acceleration of the target object; lane_ID represents the ID of the lane; "0" represents the current lane; "-1" represents the left lane; "1" represents the right lane; and lane_style represents the lane type. By mapping the information set elements perceived by the driver and the autonomous driving system, the format of the human-machine longitudinal and lateral fusion perception data message is determined, and a human-machine perception data fusion model is established.

[0100] The specific steps in step one of the fourth step are as follows:

[0101] Step 1: Input of human-machine fusion perception results, the specific method is as follows:

[0102] Input the human-machine fusion perception results data to perform human-machine fusion perception semantic inference and extract key semantic information describing the current driving situation;

[0103] Step 2: Human-machine fusion perception and semantic information representation, the specific methods are as follows:

[0104] If the information set perceived by the autonomous driving system is A = {(Target_number = 2), (lane_ID = 0, target_distance = 50, target_speed = 40, target_acceleration = 0), (lane_ID = -1, target_distance = -60, target_speed = 70, target_acceleration = 0)}, and the semantic information represents that there are two target vehicles in the current scenario, one of which is traveling at 40 km / h 50m ahead of the current lane, and the other is traveling at 70 km / h 60m behind the left lane, then similarly, the human-machine fusion perception semantic inference for the current driving situation can be completed.

[0105] The beneficial effects of this invention are:

[0106] 1) The driving situation understanding method based on human-machine enhanced perception described in this invention provides a multi-source perception dataset that integrates the driver and the autonomous driving system.

[0107] 2) The driving situation understanding method based on human-machine enhanced perception described in this invention improves the driving safety and comfort of intelligent vehicles, constructs a personalized driving habit calculation model, and makes the autonomous driving perception system more humanized and personalized.

[0108] 3) The driving situation understanding method based on human-machine enhanced perception described in this invention obtains the perception patterns of drivers with different driving habits, including lane change perception patterns, following perception patterns, fatigue perception patterns, and emergency condition perception patterns.

[0109] 4) The driving situation understanding method based on human-machine enhanced perception described in this invention is a human-centered autonomous driving perception system that enables autonomous perception based on the driver's habit type, driver's perception mode and perception logic at the current moment.

[0110] 5) The driving situation understanding method based on human-machine enhanced perception described in this invention constructs a human-machine fusion perception model, semantically represents the human-machine fusion perception information, and realizes semantic inference of the human-machine fusion perception of the current driving situation. This provides accurate and comprehensive perception information for intelligent vehicle decision-making.

[0111] 6) The driving situation understanding method based on human-machine enhanced perception described in this invention provides a driving situation understanding method that can calculate the complexity and danger of driving situations in real time and assess traffic conditions. Attached Figure Description

[0112] Figure 1 This is a schematic diagram illustrating the overall steps of the driving situation understanding method described in this invention.

[0113] Figure 2 This is a schematic diagram of the overall architecture of the driving situation understanding method described in this invention.

[0114] Figure 3 This is a schematic diagram of the overall architecture of the first step described in this invention.

[0115] Figure 4 This is a schematic diagram of the overall architecture of the second step described in this invention.

[0116] Figure 5 This is a schematic diagram of the overall architecture of the third step described in this invention.

[0117] Figure 6 This is a schematic diagram of the overall architecture of the fourth step described in this invention.

[0118] Figure 7 This is a flowchart of the algorithm for step two in the second step of the present invention.

[0119] Figure 8 This is a flowchart of the algorithm for step three in the second step of the present invention.

[0120] Figure 9 This is a flowchart of the algorithm for step two in the third step of the present invention.

[0121] Figure 10 This is an example diagram of the calculation results in step three of the fourth step of the present invention.

[0122] Figure 11 This is an example diagram of the calculation results in step four of the fourth step of the present invention. Detailed Implementation

[0123] Please see Figures 1 to 11 As shown:

[0124] The driving situation understanding method based on human-machine enhanced perception provided by this invention includes the following steps:

[0125] The first step is to integrate multi-source human-machine perception data;

[0126] The second step is to analyze the driver's driving attributes.

[0127] The third step is to construct a human-machine enhanced perception model.

[0128] Step 4: Construct a driving situation understanding model.

[0129] The process in the first step is as follows:

[0130] Step 1: Driver Physiological Data Integration. Driver physiological data includes electrical signals such as electroencephalogram (EEG), electromyography (EMG), electrocardiogram (ECG), electrooculogram (EOG), and ductal nerve conduction (TEF). Driver physiological data integration includes the following steps.

[0131] Step 1: Acquisition of driver's bioelectrical signals, the specific method is as follows:

[0132] The driver's bioelectrical signals were collected by wearing sensors. When performing driving tasks, the driver's brain needs to coordinate to complete driving operations, and electroencephalogram (EEG) signals can reflect the driver's perceptual activities. To study the driver's longitudinal and lateral driving intentions, electromyographic signals from the driver's lower leg and forearm were collected. Similarly, electrocardiogram (ECG), electrooculogram (EOG), and electrodermal signaling (EDS) signals were also collected.

[0133] Step 2: Noise reduction processing of driver's bioelectrical signals, the specific methods are as follows:

[0134] Electroencephalogram (EEG) signals are extremely weak physiological electrical signals, easily affected by external environmental interference. Therefore, preprocessing of EEG data is necessary. The EEG signal frequency range of 1-30Hz is selected, and filters are used to remove high and low frequency noise aliased into the EEG signal, completing the preprocessing. EEG waves are categorized into delta waves, theta waves, alpha waves, and beta waves based on their frequency. Simultaneously, preprocessing is also performed on the driver's electromyography (EMG), electrocardiography (ECG), electrooculography (EOG), and electrodermal activity (EDA) signals.

[0135] Step 3: Synchronization processing of driver's bioelectrical signals, the specific methods are as follows:

[0136] There is a time difference in the data collected by different sensors, so the above signals are preprocessed for time synchronization.

[0137] Step 4: Extraction of driver's bioelectrical signal features, the specific methods are as follows:

[0138] Feature vectors are constructed by extracting characteristic parameters from bioelectrical signals, serving as crucial components in a driver intention recognition model. Driver EEG features are obtained using various EEG data feature extraction methods. Time-frequency domain feature analysis is employed to analyze driver electromyography (EMG) signals, extracting key features reflecting driver actions. Similarly, feature extraction is performed on driver electrocardiogram (ECG), electrooculogram (EOG), and electrodermal signaling (EDS). Finally, these EEG, EMS, ECG, EOG, and EDS signals are integrated to achieve a comprehensive representation of the driver's state through bioelectrical signal combination.

[0139] Step Two: Driver External Characteristic Data Integration. When performing driving tasks, a driver's external characteristics manifest as head movements, eye movements, and limb operations controlling the steering wheel, accelerator, and brake pedals. Driver external characteristic data integration includes the following steps.

[0140] Step 1: Acquisition of driver's limb control, head movement, and eye movement signals. The specific methods are as follows:

[0141] Video of the driver's head movements is extracted using a camera, and an eye tracker obtains the driver's gaze patterns, recording data on the relationship between the gaze area and gaze duration as the driver performs different driving actions such as following, braking, and lane changing. For example, when the driver is preparing to change lanes, the time and number of times the driver looks at the rearview mirror are collected. Different steering wheel rotation speeds, accelerator pedal speeds, and brake pedal speeds can reflect the driver's driving intentions, such as normal braking and emergency braking. Sensors are used to obtain signals from the steering wheel, accelerator, and brake pedals.

[0142] Step 2: Signal synchronization processing, the specific methods are as follows:

[0143] The signals collected by the different sensors mentioned above have time differences, so it is also necessary to perform time synchronization preprocessing on the collected data, match the time series of head movement, eye movement behavior and driving behavior data, and complete the integration of driver external characteristic data to achieve time synchronization.

[0144] Step 3: Extraction of driver's external characteristics. The specific methods are as follows:

[0145] Extract features characterizing the driver's external traits, including saccade time, saccade amplitude, standard deviation of head rotation angle, and rearview mirror attention allocation rate. Also extract steering wheel speed, accelerator pedal and brake pedal position and speed.

[0146] Step 3: Integration of Traffic Environment and Vehicle Status Data. In addition to integrating the data from Steps 1 and 2, it is also necessary to integrate traffic environment and vehicle status data, including the following steps.

[0147] Step 1: Information collection on this vehicle and surrounding vehicles and pedestrians. The specific methods are as follows:

[0148] Information on dynamic traffic participants, such as vehicles, pedestrians, and animals, is acquired through vehicle-mounted sensors and vehicle-to-infrastructure (V2I) devices. This includes the number, location, speed, and acceleration of traffic participants, as well as the collision time between this vehicle and other vehicles.

[0149] Step Two: Collection of information on lane lines, signs, and traffic lights. The specific methods are as follows:

[0150] Collect traffic environment information, including lane lines, traffic signs, traffic lights, and static obstacles.

[0151] By processing the driver's perception data in steps one and two and the machine perception data in step three, time and space synchronization is achieved. For different driving behaviors, parameter variables describing different driving behaviors are extracted to obtain a driving behavior feature matrix and analyze the relationship with the driver's driving behavior.

[0152] The process in the second step is as follows:

[0153] Step 1: Online Driving Intent Calculation Model. This involves analyzing the driver's driving attributes and establishing an online driving intent model, including the following steps.

[0154] Step 1: Feature selection of input data. The specific method is as follows.

[0155] The online driving intention calculation model uses a neural network model, with the vehicle speed V as the input feature parameter. ego Steering wheel angle θ steer Steering wheel speed w steer Accelerator pedal displacement S throttle Accelerator pedal speed V throttle Brake pedal displacement S braking Brake pedal speed V braking The output is A = (a1,a2,a3,a4,a5), where (1,0,0,0,0), (0,1,0,0,0), (0,0,1,0,0), (0,0,0,1,0), and (0,0,0,0,1) represent driving intentions of changing lanes to the left, changing lanes to the right, accelerating, braking, and holding, respectively.

[0156] Step 2: Determining the neural network model structure, the specific methods are as follows.

[0157] In the first step, step two integrates data into training and testing sets. First, the number of hidden layer nodes is determined. The training data is then input into the input layer, and left lane change, right lane change, acceleration, braking, and hold are selected as the network's output layers. The connection weights between the input and hidden layers, and between the hidden and output layers, are determined using forward and backward propagation algorithms.

[0158] Step 3: Output results of the neural network, the specific method is as follows.

[0159] By establishing an online driving intention calculation model, the collected data information during driving is input into the online driving intention calculation model to obtain the driver's intention at each sampling time.

[0160] Step Two: Personalized Driving Habit Calculation Model. To improve the driving safety and comfort of intelligent vehicles and make autonomous driving systems more human-centered and personalized, it is necessary to determine the driver's driving habits, which can be categorized as conservative, average, and aggressive. A personalized driving habit calculation model is constructed to analyze the driver's driving attributes. The personalized driving habit calculation model includes the following steps.

[0161] Section 1: Calculation method for personalized driving habits. The specific method is as follows.

[0162] The multi-source human-machine perception data integrated in the first step is input into the personalized driving habit calculation model after feature extraction. This includes signal features extracted from the driver's physiological data in Step 1, such as EEG, EMG, ECG, SCEA, and EOG; data integrated from the driver's external characteristics in Step 2, including extracted head movement and eye movement features, as well as lateral and longitudinal control features; and features extracted from the traffic environment and vehicle state data in Step 3, including the relative position, speed, and relative acceleration of the driver and target vehicles. Due to the high dimensionality of the extracted feature parameters, principal component analysis is used to reduce the dimensionality of the data, preserving information from high-dimensional feature parameters. Based on the above feature parameters, the feature data is represented by a sample matrix Z, denoted as Z.

[0163]

[0164] For an m-dimensional random variable, n experimental observations are performed, z1, z2, ..., z n Indicates the observed sample;

[0165] z j =(z 1j ,z 2j ,…,z mj ) T This represents the j-th observation sample;

[0166] z ij Let i represent the i-th variable of the j-th observation sample, where j = 1, 2, ..., n.

[0167] Standardize the feature parameters, obtain the sample covariance matrix, and calculate the cumulative variance contribution rate of the principal components.

[0168]

[0169] Principal components with a cumulative variance contribution rate exceeding 90% are selected as components after dimensionality reduction to obtain a new dataset. The new dataset is then divided into training and test sets.

[0170] Step 2: Calculation results of personalized driving habits. The specific method is as follows.

[0171] Through cluster analysis, driver styles are categorized into three types: aggressive, average, and conservative. Driver-related characteristic data are then input into a personalized driving habit calculation model to determine the current driver's driving habits.

[0172] Step 3: Driver Perception Pattern Analysis. To study driver attributes, the perception patterns of drivers with different driving habits are analyzed, including the following perception patterns, lane-changing perception patterns, fatigue perception patterns, and emergency condition perception patterns of aggressive, conservative, and average drivers.

[0173] Step 1: Analysis of the patterns of car-following perception. The specific methods are as follows.

[0174] Drivers' visual attention changes when performing driving tasks. The driver's field of vision is divided into five areas: area 1 (directly in front), area 2 (left rearview mirror), area 3 (right rearview mirror), area 4 (instrument panel), and area 5 (interior rearview mirror). The analysis of following tasks examines the driver's visual fixation and gaze shift characteristics.

[0175] First, input the driver's head movement and eye movement signals collected in Step 1 and Step 2. Visual fixation features are then statistically analyzed based on information entropy to determine the fixation duration in each region.

[0176] The discrete variable entropy information is E, and the fixation entropy rate value is E. e Calculation formula

[0177]

[0178] E max =log2 N

[0179]

[0180] In the formula, N represents the number of fixation regions, N = 5. It is the probability of focusing on a certain area; Let n be the average gaze time of a driver in a certain area, where n represents the area number.

[0181] Driver gaze behavior mainly includes the number of times and duration of gaze at a target object, characterizing the distribution and degree of interest in the driver's region of interest during car-following tasks. Driver saccadic behavior reflects the shifting characteristics of the driver's region of interest and visual search patterns.

[0182] A driver gaze shift probability matrix is ​​constructed using Markov chains to analyze the gaze shift characteristics of drivers with different driving habits in car-following tasks.

[0183] M is the one-step transition probability P ij The state transition matrix is ​​represented as follows:

[0184]

[0185] P ij This represents the element in the i-th row and j-th column of the transition matrix probability, where the sum of the transition probabilities in each row is 1.

[0186] Using Markov chain theory, this study explores the gaze shift patterns and stationary distribution characteristics between different gaze regions; it also analyzes the differences in gaze distribution during the car-following phase among drivers with different habits. The perceptual patterns of drivers with different habits in the car-following task are thus derived.

[0187] Section Two: Analysis of Lane Change Perception Patterns, with the following specific methods.

[0188] Similar to the method used to analyze the perception patterns of drivers during the following phase, a lane-change perception pattern analysis was conducted. This analysis examined the differences in the driver's gaze distribution between the lane-change intention phase and the lane-change execution phase. The analysis revealed the pattern of gaze shifts between the area directly in front of the current lane, the rearview mirror area, and other target areas during the lane-change phase.

[0189] Section 3: Analysis of fatigue perception patterns, the specific methods are as follows.

[0190] Fatigue-prone drivers pose a significant risk to road safety. This study characterizes driver perception patterns under fatigue by analyzing eye movement, head movement, and driving behavior. Eye movement and head movement characteristics include pupil closure time, blinking frequency, head drooping angle, number of nods, and number of yawns. Driving behavior includes steering wheel correction frequency. By determining fatigue detection parameters based on eye movement, head movement, and driving behavior, the study aims to characterize driver perception patterns under fatigue.

[0191] Step 4: Analysis of the patterns of emergency condition perception. The specific methods are as follows.

[0192] This study analyzes drivers' reaction times in emergency situations. Based on the length of drivers' emergency situation perception reaction time, analysis of variance is used to test the results. The results show that reaction time significantly affects the probability of saccades of interest and horizontal eye activity. The study further analyzes the familiarity and sensitivity to emergency situations with different reaction times, revealing the patterns of emergency situation perception among drivers with different habits.

[0193] The process in the third step is as follows:

[0194] Step 1: Driver Perception Mechanism Model. This includes the following components.

[0195] Section 1: Driver's longitudinal perception mechanism model, the specific methods are as follows.

[0196] The input to the driver's longitudinal perception mechanism model is the current driver's driving intention, driving habits, and perception patterns from the second step. For example, the online calculation model of an aggressive driver's intention at the current moment outputs the acceleration intention. Based on the longitudinal perception patterns of aggressive drivers, mainly including head movement and eye movement patterns, the current driver's gaze duration patterns and scan area shift patterns for the dashboard and the area in front, it is inferred that the current driver needs to perform an accelerator pedal action at the current moment.

[0197] Section Two: Driver Lateral Perception Mechanism, the specific methods are as follows.

[0198] The driver's current driving intentions, driving habits, and perception patterns are also input into the driver's lateral perception mechanism model. The model infers from the driver's perceptions, such as the driver's current intention to change lanes to the left. Based on the driver's current lane-changing perception patterns, the driver's focus is on the left rearview mirror area and the area in front. Based on the traffic information of the vehicle's current lane and the left lane, the model infers that the driver intends to change lanes to the left at that moment. The driver's longitudinal and lateral perception mechanism model outputs comprehensive information about the driver's current perceptions.

[0199] Step 2: Human-centered perception model for autonomous driving systems. The specific methods are as follows.

[0200] The driver's lateral and longitudinal perception mechanism models are input into the human-centered autonomous driving system's perception model. On one hand, the autonomous driving perception system is driver-centric, autonomously perceiving based on the driver's current behavior type, perception pattern, and perception logic. Traffic participants are prioritized according to their regions of interest. For example, if the human-centered autonomous driving perception system detects an aggressive driver's lane-changing intention, the system assigns higher weights to traffic participants of interest in the current lane and the left lane, and lower weights to those in the right lane, incorporating them into the system's perception information progressively based on weight. On the other hand, the autonomous driving system needs to evaluate the rationality of the driver's perception information, assessing its accuracy and identifying any omissions. This completes the integration of longitudinal and lateral perception information from the human-centered autonomous driving perception system.

[0201] Step 3: Human-Machine Perception Consistency Comparison Model.

[0202] The driver perception mechanism model from Step 1 and the human-centered autonomous driving system perception mechanism model from Step 2 are input into the human-machine perception consistency comparison model. The driver's perception results are compared with those of the human-centered autonomous driving system. If the driver's perception results are consistent with those of the autonomous driving system, the perception result is directly output. If the driver's perception results are inconsistent with those of the autonomous driving system, the autonomous driving system determines the driver's current state, verifies the accuracy and comprehensiveness of the perception information caused by the driver's state, and compensates for the information perceived by the autonomous driving system with a certain weight in the human-machine perception consistency comparison model, finally outputting the human-machine perception result.

[0203] Step 4: Human-Machine Perception Data Fusion Model. After comparing the perception results through the human-machine perception consistency comparison model, the driver's perception information and the autonomous driving system's perception information are determined, and valuable information is input into the human-machine perception data fusion model. This includes the following steps.

[0204] Step 1: Representation of the driver's perceived information set, the specific method is as follows.

[0205] The driver's perceived information set D = {Target_number, Target_type, Target_position, Target_speed, Lane_information}. Here, Target_number represents the number of traffic participants currently perceived by the driver. Target_type represents the type of target, which can be categorized as pedestrians, bicycles, passenger cars, trucks, animals, etc. Target_position represents the position of the traffic participant, which can be categorized as directly in front, left front, right front, left side, right side, directly behind, left rear, and right rear. Target_speed represents the speed of movement of the traffic participant.

[0206] Section 2: Representation of the set of perception information for autonomous driving, the specific method is as follows.

[0207] The information set perceived by the autonomous driving system is A = {target_ID, target_type, target_size, target_distance, target_speed, target_acceleration, lane_ID, lane_style}, where target_ID represents the ID of the target object obtained by the sensor, target_type represents the target object type, target_size represents the target object size, target_distance represents the distance to the target object, target_speed represents the target object speed, and target_acceleration represents the target object acceleration. lane_ID represents the lane ID, with "0" representing the current lane, "-1" representing the left lane, and "1" representing the right lane. lane_style represents the lane type. The information sets perceived by the driver and the autonomous driving system are mapped to each other, the format of the human-machine longitudinal and lateral fusion perception data message is determined, and a human-machine perception data fusion model is established.

[0208] The process in step four is as follows:

[0209] Step 1: Human-machine fusion perception and semantic inference.

[0210] Step 1: Input of human-machine fusion perception results, the specific method is as follows.

[0211] The human-machine fusion perception result data is input into step one to perform human-machine fusion perception semantic inference and extract some key semantic information describing the current driving situation.

[0212] Step 2: Human-machine fusion perception and semantic information representation, the specific methods are as follows.

[0213] For example, {(Target_number=2),(lane_ID=0,target_distance=50,target_speed=40,target_acceleration=0),(lane_ID=-1,target_distance=-60,target_speed=70,target_acceleration=0)}. The semantic information represents that there are two target vehicles in the current scene, one traveling at 40 km / h 50m ahead in the current lane, and the other traveling at 70 km / h 60m behind in the left lane. Similarly, this completes the human-machine fusion perception semantic inference for the current driving situation.

[0214] Step 2: Driving Situation Analysis, the specific methods are as follows.

[0215] The driving situation constantly evolves during driving. Analyzing how the driver's area of ​​interest changes yields results in the analysis and prediction of open driving scenarios. For example, after a lane change, how do the trajectories of the vehicle and surrounding traffic evolve? This allows for the analysis of the driving situation.

[0216] Step 3: Assessment of driving scenario complexity, the specific methods are as follows.

[0217] Driving situation complexity is represented by road dynamic traffic environment complexity and road topology complexity. Multiple experienced drivers evaluated the dynamic traffic environment complexity and road topology complexity. Evaluation indicators for dynamic traffic environment complexity include, but are not limited to, the number, density, and type of traffic participants. Evaluation indicators for road topology complexity include, but are not limited to, road slope, road curvature, adhesion coefficient, water accumulation, and flatness. Using the analytic hierarchy process (AHP), the complexity weights of different evaluation indicators for different dynamic traffic environments and different road topologies are obtained by comparing the importance of the above evaluation indicators. The road dynamic traffic environment complexity and road topology complexity are calculated according to the evaluation indicators and their weights, mapped to the interval [0,1]. Driving situation complexity is as follows: Figure 10 As shown.

[0218] Step 4: Assessment of the risk level of the driving situation, the specific methods are as follows.

[0219] The factors influencing the hazard level of a driving situation are the collision time between the vehicle and surrounding vehicles, and the driver's fatigue level. Driver fatigue level is defined as a value between 0 and 1, where "0" represents a good driver condition and "1" represents extreme fatigue. A higher value indicates a greater hazard in the driving situation. A shorter collision time results in a lower hazard level. Extreme driver fatigue and a shorter collision time increase the likelihood of a traffic accident, thus increasing the hazard level of the driving situation. Based on these two factors, the hazard level of a driving situation can be assessed as follows: Figure 11 As shown.

Claims

1. A driving situation understanding method based on human-machine augmented perception, characterized in that: The method includes the following steps: The first step is to integrate multi-source human-machine perception data. The specific process is as follows: Step 1: Integration of driver physiological data, which includes electrical signals from electroencephalography (EEG), electromyography (EMG), electrocardiography (ECG), electrooculography (EOG), and electrodermal osmosis (EDS). Step 2: Integration of driver external characteristic data. When performing driving tasks, the driver's external characteristics are manifested in head movement, eye movement, and limb operation to control the steering wheel, accelerator and brake pedals. Step 3: Integration of traffic environment and vehicle status data. In addition to integrating the data from Step 1 and Step 2, the driver's perception data from Step 1 and Step 2 and the machine perception data from Step 3 are processed to achieve time and space synchronization. For different driving behaviors, parameter variables describing different driving behaviors are extracted to obtain a driving behavior feature matrix and analyze the relationship with the driver's driving behavior. The second step is to analyze the driver's driving attributes. The specific process is as follows: Step 1: Online driving intention calculation model – Analyze the driver's driving attributes and establish an online driving intention model; Step 2: Personalized driving habit calculation model. In order to improve the driving safety and comfort of intelligent vehicles and make the autonomous driving system more humanized and personalized, the driver's driving habits are judged. The driver's driving habits are divided into robust, general and aggressive types. By constructing a personalized driving habit calculation model, the driver's driving attributes are analyzed. Step 3: Driver Perception Pattern Analysis. To study driver attributes, the perception patterns of drivers with different driving habits are analyzed, including the following perception patterns of aggressive, conservative, and average drivers. Lane change perception patterns, fatigue perception patterns, and emergency condition perception patterns; The third step is to construct a human-machine enhanced perception model, and the specific steps are as follows: Step 1: Establish a driver perception mechanism model; Step 2: Establish a human-centered perception model for the autonomous driving system. The specific method is as follows: By inputting the driver's lateral and longitudinal perception mechanism models into the human-centered autonomous driving system perception model, on the one hand, the autonomous driving perception system is driver-centric. It autonomously perceives based on the driver's current habit type, perception mode, and perception logic, and determines the priority of traffic participants based on the region of interest. If the human-centered autonomous driving perception system obtains the lane-changing intention of the current aggressive driver, the autonomous driving system assigns higher weights to traffic participants of interest in the current lane and the left lane, and lower weights to traffic participants perceived in the right lane. The information is incorporated into the autonomous driving system perception information step by step according to the weight. On the other hand, the autonomous driving system needs to evaluate the rationality of the driver's perception information, whether the current driver's perception information is accurate and whether there are any omissions, and complete the integration of longitudinal and lateral perception information of the human-centered autonomous driving perception system. Step 3: Human-Machine Perception Consistency Comparison Model: Input the driver perception mechanism model in step one and the human-centered autonomous driving system perception mechanism model in step two into the human-machine perception consistency comparison model, compare the driver's perception results with the human-centered autonomous driving system perception results, and output the perception results directly when the driver's perception results are consistent with the autonomous driving system perception results. If the driver's perception results are inconsistent with those of the autonomous driving system, the autonomous driving system determines the driver's current state, verifies the accuracy and comprehensiveness of the perception information caused by the driver's state, compensates the information perceived by the autonomous driving system with a set weight into the human-machine perception consistency comparison model, and finally outputs the human-machine perception result. Step 4: Human-machine perception data fusion model. After comparing the perception results of the human-machine perception consistency model, the driver's perception information and the autonomous driving system's perception information are determined, and valuable information is input into the human-machine perception data fusion model. Step 4: Construct a driving situation understanding model. The specific steps are as follows: Step 1: Human-machine fusion perception and semantic inference; Step 2: Driving Situation Analysis, the specific methods are as follows: The driving situation evolves continuously during driving. How the driver's area of ​​interest changes is analyzed and predicted to obtain open driving situation results. If the vehicle changes lanes, how the trajectory of the vehicle and surrounding traffic vehicles evolves to analyze the driving situation situation. Step 3: Assessment of driving scenario complexity, the specific methods are as follows: Driving situation complexity is represented by road dynamic traffic environment complexity and road topology complexity. Several drivers with rich driving experience evaluate the dynamic traffic environment complexity and road topology complexity. The evaluation indicators for dynamic traffic environment complexity include, but are not limited to, the number, density, and type of traffic participants; the evaluation indicators for road topology complexity include, but are not limited to, road slope, road curvature, adhesion coefficient, water accumulation, and flatness. Using the analytic hierarchy process (AHP), the complexity weights of different evaluation indicators for different dynamic traffic environments and different road topologies are obtained by comparing the importance of the above evaluation indicators. The road dynamic traffic environment complexity and road topology complexity are calculated according to the evaluation indicators and their weights, and mapped to the interval [0,1]. Step 4: Risk assessment of the driving situation, the specific methods are as follows: The factors influencing the hazard level of a driving situation are the collision time between the vehicle and surrounding vehicles and the driver's fatigue state. The driver's fatigue state is defined as a dimensionless number between 0 and 1, where "0" represents a good driver condition and "1" represents an extremely fatigued driver. The higher the value, the greater the hazard level of the driving situation. The shorter the collision time, the lower the hazard level of the driving situation. When the driver is extremely fatigued, the shorter the collision time, the more likely a traffic accident will occur, and the higher the resulting hazard level of the driving situation. The hazard level of the driving situation is assessed based on the collision time between the vehicle and surrounding vehicles and the driver's fatigue state.

2. The driving situation understanding method based on human-machine augmented perception according to claim 1, characterized in that: The integration of driver physiological data in the first step (step one) includes the following steps: Step 1: Acquisition of driver's bioelectrical signals, the specific method is as follows: By collecting bioelectrical signals from drivers wearing sensors, it is possible to study the longitudinal and lateral driving intentions of drivers. The sensors also collect electromyographic signals from the driver's lower leg and forearm surface, as well as electrocardiogram, electrooculogram, and electrodermal signals. Step 2: Noise reduction processing of driver's bioelectrical signals, the specific methods are as follows: Electroencephalogram (EEG) signals are extremely weak physiological electrical signals, easily affected by external environmental interference. Therefore, preprocessing of the EEG data is necessary, and the appropriate frequency band for the EEG signal should be selected. Filters are used to remove high and low frequency noise aliased into the EEG signal, completing the preprocessing of the EEG signal. The EEG signal is then categorized according to its frequency. Wave, Wave, Wave, The wave also preprocesses the electrical signals of the driver's electromyography, electrocardiography, electrooculography, and electrodermal conductance. Step 3: Synchronization processing of driver's bioelectrical signals, the specific methods are as follows: There is a time difference in the data collected by different sensors, so the above signals are preprocessed for time synchronization. Step 4: Extraction of driver's bioelectrical signal features, the specific methods are as follows: By extracting feature parameters from bioelectrical signals to form feature vectors, which serve as important components of the driver's intention recognition model, the driver's EEG features are obtained through various EEG data feature extraction methods. The driver's electromyography (EMG) signals are then analyzed using time-frequency domain feature analysis to extract the main features of the EMG signals that reflect the driver's actions. Similarly, the driver's electrocardiogram (ECG), electrooculogram (EOG), and electrodermal signal features are extracted. Finally, the driver's EEG, EMG, ECG, EOG, and EEG signals are integrated to achieve a comprehensive representation of the driver's state by combining the driver's bioelectrical signals.

3. The driving situation understanding method based on human-machine augmented perception according to claim 1, characterized in that: The integration of driver external characteristic data in the first step and step two includes the following steps: Step 1: Acquisition of driver's limb control, head movement, and eye movement signals. The specific methods are as follows: The camera extracts video of the driver's head movement, and the eye tracker obtains the driver's gaze. It records the relationship between the gaze area and gaze time when the driver performs different driving actions such as following, braking and changing lanes. If the driver is preparing to change lanes, the time and number of times the driver looks at the rearview mirror are collected. Different steering wheel speeds, acceleration and brake pedal speeds can reflect the driver's driving intentions. If it is normal braking and emergency braking, the sensors obtain steering wheel, acceleration and brake pedal signals. Step 2: Signal synchronization processing, the specific methods are as follows: The signals collected by the different sensors mentioned above have a time difference, so it is also necessary to perform time synchronization preprocessing on the collected data, match the time series of head movement, eye movement behavior and driving behavior data, and complete the integration of driver external characteristic data to achieve time synchronization; Step 3: Extraction of driver's external characteristics. The specific methods are as follows: Features representing the driver's external characteristics are extracted, including saccade time, saccade amplitude, standard deviation of head rotation angle, rearview mirror attention allocation rate, steering wheel speed, accelerator pedal, brake pedal position and speed.

4. The driving situation understanding method based on human-machine augmented perception according to claim 1, characterized in that: The integration of traffic environment and vehicle status data in the first step, step three, includes the following steps: Step 1: Information collection of this vehicle and surrounding vehicles and pedestrians. The specific methods are as follows: The vehicle acquires dynamic traffic participant information, including the number, location, speed, and acceleration of traffic participants, as well as the collision time between the vehicle and other vehicles, through onboard sensors and vehicle-road cooperative equipment. Step Two: Collection of information on lane lines, signs, and traffic lights. The specific methods are as follows: Collect traffic environment information, including lane lines, traffic signs, traffic lights, and static obstacles.

5. The driving situation understanding method based on human-machine augmented perception according to claim 1, characterized in that: The second step, step one, includes the following steps: Step 1: Feature selection of input data. The specific method is as follows: The online driving intent calculation model uses a neural network model, with the vehicle speed as the input feature parameter. , Steering wheel angle Steering wheel speed Accelerator pedal displacement Accelerator pedal speed Brake pedal displacement Brake pedal speed The output is ,in , , , , These respectively indicate driving intentions: left lane change, right lane change, acceleration, braking, and holding. Step 2: Determining the neural network model structure, the specific method is as follows: In the first step, the data integrated in step two is divided into training and test sets. First, the number of hidden layer nodes is determined. The data of the training set is input into the input layer. Left lane change, right lane change, acceleration, braking and hold are selected as the output layer of the network. The connection weights between the input layer and the hidden layer, and between the hidden layer and the output layer are determined through the forward and backward propagation algorithm. Step 3: Outputting the neural network results, the specific method is as follows: By establishing an online driving intention calculation model, the data collected during driving is input into the online driving intention calculation model to obtain the driver's intention at each sampling time.

6. The driving situation understanding method based on human-machine augmented perception according to claim 1, characterized in that: The personalized driving habit calculation model in step two of the second step includes the following components: Step 1: Calculation method for personalized driving habits. The specific method is as follows: The multi-source human-machine perception data integrated in the first step, after feature extraction, is input into the personalized driving habit calculation model. This includes the signal features of EEG, EMG, ECG, SCEA, and EOG extracted from the driver's physiological data in Step 1; the driver's head movement and eye movement features, as well as the lateral and longitudinal control features, extracted from the driver's external characteristics integrated in Step 2; and the features extracted from the traffic environment and vehicle state data integrated in Step 3, including the relative position, speed, and relative acceleration of the driver and the target vehicle. Due to the high dimensionality of the extracted feature parameters, principal component analysis is used to reduce the dimensionality of the data, retaining the information of the high-dimensional feature parameters. Based on the above feature parameters, the feature data is represented by a sample matrix Z, denoted as: ； right 3D random variables This experiment observation, Indicates the observed sample; Indicates the first One observation sample; Indicates the first The first observation sample One variable, ; Standardize the feature parameters, obtain the sample covariance matrix, and calculate the cumulative variance contribution rate of the principal components; ； The principal components with a cumulative variance contribution rate of over 90% are selected as the components after dimensionality reduction to obtain a new dataset. The new dataset is then divided into training and test sets. Step Two: Calculation Results of Personalized Driving Habits. The specific method is as follows: Through cluster analysis, driver styles are divided into three types: aggressive, average, and conservative. Driver-related characteristic data are input into a personalized driving habit calculation model to determine the current driver's driving habits.

7. The driving situation understanding method based on human-machine augmented perception according to claim 1, characterized in that: The steps in step three of the second step are as follows: Step 1: Analysis of the patterns of car-following perception. The specific methods are as follows: Drivers’ visual attention changes when performing driving tasks. The driver’s field of vision is divided into 5 areas, including the area in front 1, the left rearview mirror area 2, the right rearview mirror area 3, the instrument panel area 4, and the interior rearview mirror area 5. The driver’s visual fixation and gaze shift characteristics are studied when performing car-following tasks. First, input the driver's head movement and eye movement signals collected in Step 1 and Step 2, and then statistically analyze the gaze duration of each region based on information entropy of the visual gaze characteristics. The entropy information of discrete variables is gaze entropy rate value Calculation formula: ； ； ； In the formula, Indicates the number of fixation areas. , It is the probability of focusing on a certain area; The average gaze time a driver spends in a certain area. Indicates the region number; Driver gaze behavior mainly includes the number of times and duration of gaze at the target object, which characterizes the distribution and degree of interest of the driver in the car-following task. Driver saccadic behavior reflects the shifting characteristics and visual search patterns of the driver in the region of interest. A driver gaze shift probability matrix was constructed using Markov chains to analyze the gaze shift characteristics of drivers with different driving habits in car-following tasks. M is the probability of a one-step transition. The state transition matrix is ​​represented as follows: ； The probability of the transition matrix is ​​represented by the first... Line number The elements in the column have a total transition probability of 1 for each row; Using Markov chain theory, we explore the gaze shift patterns and stationary distribution characteristics of the gaze point between different gaze regions; we analyze the differences in gaze distribution during the car-following phase for drivers with different habits, and obtain the perceptual patterns of drivers with different habits in the car-following task. Section Two: Analysis of Lane Change Perception Patterns, with the following specific methods: Similar to the driver's perception pattern analysis method during the following phase, lane change perception pattern analysis is performed to analyze the difference in the driver's gaze distribution during the lane change intention phase and the lane change execution phase, and to obtain the driver's gaze shift pattern between the area directly in front of the current lane, the rearview mirror area, and other target areas during the lane change phase. Step 3: Analysis of fatigue perception patterns, the specific methods are as follows: Fatigue can easily lead to driving risks. By analyzing the driver's eye movement, head movement information and driving behavior characteristics, we can characterize the driver's perception patterns when fatigued. Eye movement and head movement characteristics include pupil closure time, blinking frequency, head drooping angle, number of nodding and number of yawning. Driving behavior includes steering wheel correction frequency. By determining eye movement, head movement information, and fatigue detection parameters of driving behavior, the driver's fatigue perception pattern can be characterized. Step 4: Analysis of Emergency Condition Perception Patterns. The specific methods are as follows: This study analyzes drivers' reaction time in emergency situations. Based on the length of drivers' reaction time to emergency situations, an analysis of variance is used to test the results. The length of reaction time significantly affects the probability of saccades of interest and horizontal eye activity. The study analyzes the familiarity and sensitivity to emergency situations with different reaction times to obtain the emergency situation perception patterns of drivers with different habits.

8. The driving situation understanding method based on human-machine augmented perception according to claim 1, characterized in that: The third step, step one, includes the following steps: Step 1: Driver's longitudinal perception mechanism model, the specific method is as follows: The input to the driver's longitudinal perception mechanism model is the current driver's driving intention, driving habits and perception patterns in the second step. If the online calculation model of the aggressive driver's intention outputs acceleration intention at the current moment, based on the longitudinal perception patterns of the aggressive driver, mainly including head movement and eye movement patterns, the current driver's gaze duration patterns and scanning area shift patterns for the dashboard area and the area in front, it is inferred that the current driver needs to perform the accelerator pedal action at the current moment. Section Two: Driver Lateral Perception Mechanism, the specific methods are as follows: The driver's current driving intention, driving habits, and perception patterns are also input into the driver's lateral perception mechanism model to infer the driver's perception.

9. The driving situation understanding method based on human-machine augmented perception according to claim 1, characterized in that: The third step, step four, includes the following steps: Step 1: Representation of the driver's perceived information set, using the following method: Driver-perceived information set D = {Target_number, Target_type, Target_position, Target_...} The function is defined as follows: Target_number represents the number of traffic participants perceived by the driver, Target_type represents the type of target, which can be divided into pedestrians, bicycles, passenger cars, trucks and animals, Target_position represents the position of traffic participants, which can be divided into front, left front, right front, left side, right side, rear, left rear and right rear, and Target_speed represents the speed of movement of traffic participants. Step 2: Representation of the perception information set for autonomous driving, the specific method is as follows: The set of information perceived by the autonomous driving system is A={target_ID,target_type,target_size,target_di- The model is defined as follows: `stance, target_speed, target_acceleration, lane_ID, lane_style`. `target_ID` represents the ID of the target object obtained by the sensor; `target_type` represents the type of the target object; `target_size` represents the size of the target object; `target_distance` represents the distance to the target object; `target_speed` represents the speed of the target object; `target_acceleration` represents the acceleration of the target object; `lane_ID` represents the ID of the lane; "0" represents the current lane; "-1" represents the left lane; "1" represents the right lane; `lane_style` represents the lane type. The model maps the information sets perceived by the driver and the autonomous driving system to determine the format of the human-machine longitudinal and lateral fusion perception data message and establishes a human-machine perception data fusion model.

10. The driving situation understanding method based on human-machine augmented perception according to claim 1, characterized in that: The specific steps in step one of the fourth step are as follows: Step 1: Input of human-machine fusion perception results, the specific method is as follows: Input the human-machine fusion perception results data to perform human-machine fusion perception semantic inference and extract key semantic information describing the current driving situation; Step 2: Human-machine fusion perception and semantic information representation, the specific methods are as follows: If the information set perceived by the autonomous driving system is A={(Target_number=2),(lane_ID=0, target_distance=50,target_speed=40,target_acceleration=0),(lane_ID=-1,target_distance=-60,target_speed=70,target_acceleration=0)}; and the semantic information represents that there are two target vehicles in the current scene, one of which is traveling at 40km / h 50m ahead of the current lane and the other is traveling at 70km / h 60m behind the left lane, then similarly, the human-machine fusion perception semantic inference for the current driving situation can be completed.

Images