An autonomous takeover request method that synthesizes driver and traffic environment data

Abstract

The application discloses an automatic driving takeover request method integrating driver and traffic environment data, and belongs to the technical field of automatic driving, and comprises the following steps: 1. constructing a database of driver visual / auditory / tactile / olfactory sensory characteristic information and traffic environment information; 2. real-time monitoring of typical state key features of the driver, such as fatigue, distraction, drinking and the like; 3. real-time monitoring of traffic environment, including key features of vehicles, pedestrians, road signs and signal lamps and the like; 4. providing different decision suggestions according to a double-channel model combined with a rule engine algorithm. The system can make safe and efficient driving decisions according to the perceived driver and traffic environment; the automatic driving system will encounter various uncertainties in actual driving, and through the double-channel model combined with the rule engine algorithm, the system can better handle the uncertainties in a timely manner, thereby improving the robustness and safety of the system.

Description

TECHNICAL FIELD

[0001] The application discloses an automatic driving takeover request method integrating driver and traffic environment data, and belongs to the technical field of automatic driving and artificial intelligence vehicle networking information perception. BACKGROUND

[0002] According to the classification standard formulated by the American Society of Automotive Engineers (SAE), the automatic driving vehicle is currently at the conditional automation (L3) level, and the driver still needs to take over the vehicle in time in a partially or conditionally automatic driving vehicle, and how to make the driver respond to the takeover prompt of the automatic driving system and realize the smooth transition of the control right in a relatively short time (within 10-15s) has been a key problem to be solved for the automatic driving vehicle. However, in the long automatic driving process of the L3-level automatic driving vehicle, the driver is prone to fatigue, and in this case, if an emergency occurs in the automatic driving process and the driver needs to take over, the driver is difficult to recover to a state capable of taking over the vehicle in time due to the fatigue, and at this time, if the vehicle is not taken over or is taken over in the fatigue state, traffic danger will be caused. The takeover request and the control right switching are key core technologies of the automatic driving vehicle, and at present, the research on the control right switching has not well improved the takeover quality, and the takeover control of the vehicle still has a large development space. SUMMARY

[0003] In order to overcome the deficiencies in the prior art, the application provides a personalized visual / auditory / tactile / olfactory takeover request prompting method, which improves the cognitive condition of the driver in a short time so as to safely and smoothly take over the vehicle, and the technical scheme of the application is as follows:

[0004] An automatic driving takeover request method integrating driver and traffic environment data. The method comprises the following steps:

[0005] Step 1, constructing a driver visual / auditory / tactile / olfactory sensory characteristic information and traffic environment information database;

[0006] Step 2, monitoring and analyzing and identifying the typical state key features of the driver fatigue, distraction and drinking in real time;

[0007] Step 3, monitoring and analyzing and identifying the key features of the traffic environment including the vehicle, the pedestrian, the road sign and the signal lamp in real time;

[0008] Step 4, providing different decision suggestions according to the double-channel model combined with the rule engine algorithm.

[0009] Further, the specific process of step 1 is as follows:

[0010] First, the sensors arranged in the vehicle are as follows:

[0011] A camera is installed in the car to capture the driver's facial expressions and eye movements; a microphone is installed to capture the driver's voice and vocal characteristics; a touch sensor is configured to capture the driver's hand movements and touch operations; a gas sensor is integrated to capture the odor in the car cabin and the driver's odor feedback; real-time data from various sensors, including video streams, audio, touch operations, and gas signals, are captured;

[0012] The collected data is transmitted to the on-board computing unit or cloud server; real-time processing and analysis of perception data on the cloud server; using computer vision and image processing techniques to extract visual features of facial expressions and eye movements; using speech processing and acoustic analysis techniques to identify the driver's voice characteristics and emotional state; analyzing data from touch sensors and gas sensors to identify the driver's hand movements and odor perception;

[0013] According to the collected data, feature extraction and sensory feature detection algorithms are developed; by analyzing facial expressions and eye movements to determine the driver's emotional state and attention level; through speech analysis to determine the driver's speech rate, tone and content, to determine whether they are tired or distracted; analyze hand movements and touch events to determine the driver's operating behavior and reaction speed; analyze gas sensor data to determine the driver's olfactory feedback to the environment;

[0014] Establish a driver sensory information database: store the obtained driver sensory feature data in the database; the database contains personal features and data of the driver in different behavioral states, such as normal state, fatigue, distraction, drinking, etc.; the database is scalable and can be operated in real time to adapt to the data storage and management needs of different drivers and driving situations.

[0015] Further, the specific process of step 2 is:

[0016] Introduce deep learning technology to more accurately monitor and identify the driver's fatigue, distraction and drinking, etc. state; by introducing convolutional neural network (CNN) and cross-modal attention mechanism into the system, the driver's behavior and physiological characteristics are analyzed and identified, which includes detecting fatigue, distraction, eye closure frequency, odor, etc. key features; by monitoring these features, the system can determine in real time whether the driver's state is suitable for driving or whether a warning or takeover request is needed;

[0017] The convolutional neural network (CNN) architecture is as follows:

[0018] 1. Image processing branch

[0019] Input layer: 32x32 grayscale image

[0020] Convolution layer 1: 32 3x3 convolution kernels, ReLU activation

[0021] Pooling layer 1: 2x2 max pooling

[0022] Convolution layer 2: 64 3x3 convolution kernels, ReLU activation

[0023] Pooling layer 2: 2x2 max pooling

[0024] Flatten layer: flatten the feature map into a one-dimensional vector X img

[0025] Output: image feature vector

[0026] 2. Sound processing branch

[0027] Input layer: sound data containing 20 spectral features

[0028] Fully connected layer 1: 128 neurons, ReLU activation

[0029] Output layer: sound feature vector X audio

[0030] 3. Physiological signal processing branch

[0031] Input layer: physiological signal data containing 10 features

[0032] Fully connected layer 1: 64 neurons, ReLU activation

[0033] Output layer: sound feature vector X physio

[0034] The input image is processed through convolution and pooling layers to extract features; the input sound spectrogram is processed through convolution and pooling layers to extract features; the tactile and olfactory physiological signals are processed through fully connected layers to extract features;

[0035] To effectively fuse the features of different modal data and focus on the importance of each modality at different stages, we introduce a cross-modal attention mechanism;

[0036] We introduce an attention weight calculation mechanism to calculate the importance of different modal data in the feature fusion process;

[0037] We use a multi-layer perceptron (MLP) to learn the attention weights of different modal data to adjust the contribution of different modal data in feature fusion, and the calculation formula of input attention weight is as follows:

[0038] α img = σ(W img X img +b img )

[0039] α audio= σ(W audio X audio + b audio )

[0040] α physio = σ(W physio X physio + b physio )

[0041] where σ is the Sigmoid activation function, which limits the attention weights between 0 and 1, representing the relative importance of the weights; α img , α audio , and α physio are the attention weights for image, sound, and physiological modalities, respectively, whose values are between 0 and 1, representing the importance of each modality data in feature fusion; W img is the weight matrix for calculating the attention weights of image features; b img is the bias for calculating the attention weights of image features; W audio is the weight matrix for calculating the attention weights of sound features; b audio is the bias for calculating the attention weights of sound features; W physio is the weight matrix for calculating the attention weights of physiological features; b physio is the bias for calculating the attention weights of physiological features.

[0042] The features of different modality data are weighted and fused using the attention weights to obtain the final driver state recognition result; the final fused features are represented as:

[0043] X combined = α img × X img + α audio × X audio + α physio × X physio

[0044] where X combined is the final fused feature vector, in which the contribution of each modality data is weighted by the corresponding attention weight;

[0045] Finally, the fused feature vector X combined is input to the input layer; the fully connected layer 2 contains 128 neurons, which accepts X combined and performs further feature extraction and classification; through the Softmax layer, the classification probability distribution of the driver state is output.

[0046] Further, the specific process of step 3 is as follows:

[0047] Introducing deep learning techniques to identify potential dangerous situations or abnormal driving behaviors in the road traffic environment, including vehicles, pedestrians, road signs, and traffic lights; By introducing Convolutional Neural Networks (CNN) and cross-modal attention mechanisms into the system, the system can analyze and identify features of the surrounding traffic environment, including the number of vehicles, speed, distance, number of pedestrians, location, moving trajectory, road signs, and traffic lights. By monitoring these features, the system can determine potential dangerous situations or abnormal driving behaviors in real time and issue warnings or take over requests to the driver;

[0048] 1. Image processing branch

[0049] Input layer: 32x32 color image (RGB)

[0050] Convolutional layer 1: 32 3x3 convolutional kernels, ReLU activation

[0051] Pooling layer 1: 2x2 max pooling

[0052] Convolutional layer 2: 64 3x3 convolutional kernels, ReLU activation

[0053] Pooling layer 2: 2x2 max pooling

[0054] Fully connected layer: 128 neurons, ReLU activation

[0055] Flattening layer: flatten feature maps into one-dimensional vector X img

[0056] Output: image feature vector

[0057] 2. Sound processing branch:

[0058] Input layer: spectral data containing 40 spectral features (sound spectrogram)

[0059] Fully connected layer: 64 neurons, ReLU activation

[0060] Output layer: sound feature vector X audio

[0061] 3. Text processing branch:

[0062] Input layer: text data containing 20 features (word vectors or other representations)

[0063] Fully connected layer: 32 neurons, ReLU activation

[0064] Output layer: sound feature vector X text

[0065] The input image is extracted by convolution and pooling layer; the input sound spectrum is extracted by convolution and pooling layer; the text is extracted by full connection layer;

[0066] In order to effectively fuse the features of different modal data, and pay attention to the importance of each modality in different stages, we introduce a cross-modal attention mechanism;

[0067] An attention weight calculation mechanism is introduced to calculate the importance of different modal data in the feature fusion process;

[0068] A multi-layer perception (MLP) is used to learn the attention weight of different modal data to adjust the contribution of different modal data in feature fusion, and the calculation formula of input attention weight is as follows:

[0069] α img =σ(W img X img +b img )

[0070] α audio =σ(W audio X audio +b audio )

[0071] α text =σ(W text X text +b text )

[0072] In the formula, σ is a Sigmoid activation function, which is used to limit the attention weight between 0 and 1, representing the relative importance of the weight; α img , α audio , α text are the attention weights of image, sound and text modalities respectively, their values are between 0 and 1, representing the importance of each modality data in feature fusion; W img is the weight matrix for calculating the attention weight of image features; b img is the bias for calculating the attention weight of image features; W audio is the weight matrix for calculating the attention weight of sound features; b audio is the bias for calculating the attention weight of sound features; W text is the weight matrix for calculating the attention weight of physiological features; b text is the bias for calculating the attention weight of physiological features;

[0073] The features of different modal data are weighted and fused using the attention weight to obtain the final driver state recognition result; the final fused feature is represented as:

[0074] X combined = a img X img + a audio X audio + a text X text

[0075] where X combined is the final fused feature vector, in which the contribution of each modality data is weighted by the corresponding attention weight;

[0076] Finally, the fused feature vector X commbined is input to the input layer; the fully connected layer contains 3 neurons, which accepts X combined and performs further feature extraction and classification; through the Softmax layer, the classification probability distribution of the traffic environment state is output.

[0077] Further, the specific process of step 4 is as follows:

[0078] The driver state recognition priority is usually higher than the traffic environment perception and prediction, because ensuring the state and safety of the driver is the primary task of the intelligent driving system; the dual-channel takes the output state information in steps 2 and 3 as the input of the decision engine; the system automatically makes decisions, such as triggering visual, auditory, tactile or olfactory takeover requests or warnings, according to the current driver state and environmental conditions through learning and optimization;

[0079] According to different traffic environments, the rule engine algorithm is used for decision-making, and the specific priority and decision rules are as follows:

[0080] 1. Driver state: fatigue detection

[0081] Condition: Driver's eye closure frequency exceeds threshold (eye closure frequency exceeds 10 times per minute);

[0082] Decision: Trigger visual warning, suggest driver to rest or park for rest;

[0083] 2. Driver state: distraction detection

[0084] Condition: Driver's head frequently turns, not paying attention to the road conditions ahead;

[0085] Decision: Trigger sound warning, remind the driver to pay attention to the road conditions ahead;

[0086] 3. Driver state: alcohol detection

[0087] Condition: Alcohol concentration exceeds legal limit (exceeds 20mg / 100ml);

[0088] Decision: Trigger visual and audio alerts, suggest parking or seeking alternative route;

[0089] 4. Traffic environment: Distance to vehicle ahead

[0090] Condition: Vehicle ahead is too close (at speeds above 100 km / h, a distance of 100 meters should be maintained from the vehicle ahead in the same lane, at speeds below 100 km / h, the distance can be reduced, but the minimum distance should not be less than 50 meters);

[0091] Decision: Trigger haptic alerts, reminding the driver to maintain a safe distance or slow down;

[0092] 5. Traffic environment: Intersection

[0093] Condition: Distance to intersection is less than 20 meters and the traffic light is red;

[0094] Decision: Trigger visual and audio alerts, reminding the driver to stop and wait for the signal;

[0095] 6. Driver state and traffic environment combination: Fatigue + Intersection

[0096] Condition: Driver is in a state of fatigue and the distance to the intersection is less than 20 meters;

[0097] Decision: Trigger visual and audio alerts, forcing the driver to stop and wait for the signal;

[0098] 7. Driver state: Adverse weather

[0099] Condition: Adverse weather (such as rain, snow, etc.) is detected;

[0100] Decision: Trigger haptic alerts, reminding the driver to slow down and maintain safe driving; 8. Driver state: Hands off the steering wheel

[0101] Condition: Driver has not held the steering wheel for 5 seconds;

[0102] Decision: Trigger audio alerts, reminding the driver to keep their hands on the steering wheel; 9. Traffic environment: Pedestrians crossing the road

[0103] Condition: Signs of pedestrians crossing the road are detected;

[0104] Decision: Trigger visual and audio alerts, reminding the driver to slow down and drive carefully; 10. Traffic environment: Road signs

[0105] Condition: Speed limit signs or other traffic signs are detected;

[0106] Decision: Trigger haptic and audio alerts to remind the driver to slow down;

[0107] Decision: Trigger haptic and audio alerts to remind the driver to slow down;

[0108] 12. Driver state: Abnormal heart rate

[0109] Condition: Driver's heart rate is abnormally elevated;

[0110] Decision: Trigger visual and audio alerts, suggesting the driver to park or seek medical attention;

[0111] 13. Traffic environment: Road congestion

[0112] Condition: Road congestion is detected;

[0113] Decision: Trigger audio alerts to remind the driver to choose an alternative route or prepare for delays;

[0114] Condition: The vehicle is approaching a sharp turn section;

[0115] Decision: Trigger audio alerts to remind the driver to slow down and drive cautiously;

[0116] Condition: The driver uses a mobile phone or other devices for more than 10 seconds;

[0117] Decision: Trigger visual and audio alerts to remind the driver to focus on driving;

[0118] Condition: Driver's emotional state is detected to be abnormal;

[0119] Decision: Trigger visual and audio alerts to remind the driver to keep calm while driving;

[0120] 17. Traffic environment: Road potholes

[0121] Condition: The vehicle is approaching a road pothole area;

[0122] Decision: Trigger haptic alerts to remind the driver to pay attention to avoid bumpy driving;

[0123] 18. Traffic environment: Height-limited bridge / tunnel

[0124] Condition: The vehicle is about to enter a height-limited bridge or tunnel;

[0125] Decision: Trigger audio alerts to remind the driver to pay attention to the vehicle height and slow down;

[0126] 19. Driver state and traffic environment combined: Fatigue + road signs

[0127] Condition: Driver fatigue and limited speed sign ahead;

[0128] Decision: Trigger visual and audio alerts to remind the driver of the speed limit sign and slow down.

[0129] Beneficial effects: In terms of vision, the icon is combined through the HUD and the central interface; in terms of hearing, through the way of voice prompt; in terms of touch, through the form of vibrating seat back; in terms of smell, it is divided into two categories: pleasant smell and aversive smell, and released for 10s through the mixing of two or more gases. In addition, the method combining two CNNs and cross-modal attention mechanism solves several key technical problems in intelligent driving systems: multi-modal data fusion, driver state recognition, complex environment perception and prediction, and intelligent decision engine optimization. Intelligent driving systems need to process multiple types of data from different sensors, such as images, sounds, and physiological signals, to fully understand the driving environment and driver state. Multiple CNN branches are used to process different modal data, and then cross-modal attention mechanism is used to fuse features of different modalities. This method effectively realizes the fusion and information interaction of multi-modal data, and improves the system's ability to perceive complex environments. Intelligent driving systems need to monitor the driver's state in real time, such as fatigue, distraction, drunk driving, and dangerous situations in the traffic environment. Through CNN, driver behavior and physiological characteristics are identified and analyzed, including multi-modal data such as images, sounds, and physiological signals. Cross-modal attention mechanism can dynamically adjust the weight of different modal data in feature fusion, improving the accurate perception of driver state changes. Intelligent driving systems need to perceive and predict the surrounding traffic environment in real time, including vehicle, pedestrian, road sign, traffic signal and other information. CNN is used to process and analyze image and sound data in the traffic environment to extract key features. Through cross-modal attention mechanism, the features of different modal data are fused to achieve accurate perception and prediction of complex traffic environment. This method effectively improves the understanding and response ability of intelligent driving systems to traffic scenarios. Intelligent driving systems need to make intelligent decisions based on driver state and traffic environment data to trigger appropriate takeover requests or warnings to ensure driving safety. Comprehensive use of multi-modal data fusion and cross-modal attention mechanism results in the design of intelligent decision engine rules. These rules are based on deep learning models for comprehensive analysis of driver state and traffic environment, which can automatically make decisions and take appropriate measures to ensure driving safety and optimize driving experience.

[0130] The innovation of this invention lies in the fact that the dual-channel system uses the state information output from steps 2 and 3 as input to the decision engine; it utilizes different learning algorithms to specifically fuse and process complex sensing processes of vision, hearing, touch, or smell; and based on the current driver state and environmental conditions, the system automatically makes decisions through learning and optimization, triggering visual, auditory, tactile, or olfactory takeover requests or warnings. Attached Figure Description

[0131] Figure 1 This is a flowchart of the present invention;

[0132] Figure 2 This is a diagram illustrating the priority of different traffic environments. Detailed Implementation

[0133] like Figure 1 As shown, an autonomous driving takeover request method that integrates driver and traffic environment data includes the following steps:

[0134] Step 1: Construct a database of driver's visual / auditory / tactile / olfactory sensory characteristics and traffic environment information;

[0135] Step 2: Real-time monitoring of key characteristics of typical driver states such as fatigue, distraction, and alcohol consumption;

[0136] Step 3: Monitor the traffic environment in real time, including key features such as vehicles, pedestrians, road signs, and traffic lights;

[0137] Step 4: Based on the dual-channel model combined with the rule engine algorithm, different decision suggestions are provided.

[0138] The specific process of step 1 above is as follows:

[0139] The sensors deployed inside the vehicle are as follows:

[0140] An in-vehicle camera is installed to capture the driver's facial expressions and eye movements; a microphone is installed to capture the driver's voice and speech characteristics; a touch sensor is configured to detect the driver's hand movements and touch operations; an olfactory sensor is integrated to detect the smell of the air inside the vehicle and the driver's olfactory feedback; data from various sensors are collected in real time, including video streams, audio, touch events, and olfactory signals.

[0141] The collected data is transmitted to the vehicle computing unit or cloud server; on the cloud server, the perceived data is processed and analyzed in real time; using computer vision and image processing technology, visual features of facial expressions and eye movements are extracted; using speech processing and acoustic analysis technology, the driver's voice features and emotional state are identified; and data from touch sensors and olfactory sensors are analyzed to identify the driver's hand movements and olfactory perception.

[0142] Based on the collected data, develop algorithms for feature extraction and recognition of sensory characteristics; analyze facial expressions and eye movements to identify the emotional state and attention level of the driver; through voice analysis, identify the driver's speech speed, tone, and content to determine if they are fatigued or distracted; analyze hand movements and touch events to determine the driver's operating behavior and reaction speed; analyze olfactory sensor data to identify the driver's olfactory feedback to the environment;

[0143] Establish a database of driver sensory information: store the identified driver sensory characteristic data in the database; the database includes individual characteristics of the driver and data under different behavior states, such as normal state, fatigue state, distraction state, and drinking state; the database is expandable and real-time to meet the data storage and management needs of different drivers and driving situations.

[0144] The specific process of step 2 is as follows:

[0145] Introduce deep learning technology to more accurately monitor and identify the driver's fatigue, distraction, and drinking status; by introducing convolutional neural networks (CNN) and cross-modal attention mechanisms into the system, analyze and identify the driver's behavior and physiological characteristics, including detecting fatigue, distraction, eye closure frequency, and odor; by monitoring these features, the system can determine in real-time whether the driver is in a suitable state for driving or whether a warning or takeover request is needed;

[0146] The convolutional neural network (CNN) architecture is as follows:

[0147] 1. Image processing branch

[0148] Input layer: 32x32 grayscale image

[0149] Convolution layer 1: 32 3x3 convolution kernels, ReLU activation

[0150] Pooling layer 1: 2x2 max pooling

[0151] Convolution layer 2: 64 3x3 convolution kernels, ReLU activation

[0152] Pooling layer 2: 2x2 max pooling

[0153] Flattening layer: flatten the feature map into a one-dimensional vector X img

[0154] Output: image feature vector

[0155] 2. Sound processing branch

[0156] Input layer: sound data containing 20 spectral features

[0157] Fully connected layer 1: 128 neurons, ReLU activation

[0158] Output layer: sound feature vector X audio

[0159] 3. Physiological signal processing branch

[0160] Input layer: physiological signal data containing 10 features

[0161] Fully connected layer 1: 64 neurons, ReLU activation

[0162] Output layer: sound feature vector X physio

[0163] The input image is extracted features through convolution and pooling layers; the input sound spectrogram is extracted features through convolution and pooling layers; the physiological signals of touch and smell are extracted features through fully connected layers;

[0164] In order to effectively fuse the features of different modal data, and pay attention to the importance of each modality at different stages, we introduce a cross-modal attention mechanism;

[0165] An attention weight calculation mechanism is introduced to calculate the importance of different modal data in the feature fusion process;

[0166] A multi-layer perceptron (MLP) is used to learn the attention weights of different modal data to adjust the contribution of different modal data in feature fusion, and the calculation formula of input attention weight is as follows:

[0167] α img = σ(W img X img + b img )

[0168] α audio = σ(W audio X audio + b audio )

[0169] α physio = σ(W physio X physio + b physio )

[0170] In the formula, σ is the Sigmoid activation function, which limits the attention weight to between 0 and 1, representing the relative importance of the weight; α img , α audio , α physio are the attention weights of image, sound, and physiological modalities, respectively, whose values are between 0 and 1, representing the importance of each modality data in feature fusion; W imgis a weight matrix for calculating attention weights of image features; b img is a bias for calculating attention weights of image features; W audio is a weight matrix for calculating attention weights of sound features; b audio is a bias for calculating attention weights of sound features; W physio is a weight matrix for calculating attention weights of physiological features; b physio is a bias for calculating attention weights of physiological features;

[0171] The features of different modal data are weighted and fused using the attention weights to obtain the final driver state recognition result; the final fused feature is represented as:

[0172] X combined = a img X img + a audio X audio + a physio X physio

[0173] where X combined is the final fused feature vector, and the contribution of each modal data is weighted by the corresponding attention weight;

[0174] Finally, the fused feature vector X commbined is input to the input layer; the fully connected layer 2 contains 128 neurons, accepts X combined and performs further feature extraction and classification; through the Softmax layer, the classification probability distribution of the driver state is output.

[0175] The specific process of the above step 3 is:

[0176] Deep learning technology is introduced to analyze the road traffic environment, including vehicle, pedestrian, road sign and signal light information, and identify potential dangerous situations or abnormal driving behaviors; by introducing convolutional neural network (CNN) and cross-modal attention mechanism into the system, the surrounding traffic environment features are analyzed and identified, which includes detecting the number of vehicles, vehicle speed, vehicle distance, number of pedestrians, position, moving track, road signs and signal lights, etc. By monitoring these features, the system can judge potential dangerous situations or abnormal driving behaviors in real time and issue warnings or take over requests to the driver;

[0177] 1. Image processing branch

[0178] Input layer: 32x32 color image (RGB)

[0179] Convolutional layer 1: 32 3x3 convolution kernels, ReLU activation

[0180] Pooling layer 1: 2x2 max pooling

[0181] Convolution layer 2: 64 3x3 convolutional kernels, ReLU activation

[0182] Pooling layer 2: 2x2 max pooling

[0183] Fully connected layer: 128 neurons, ReLU activation

[0184] Flattening layer: flatten the feature maps into a one-dimensional vector X img

[0185] Output: image feature vector

[0186] 2. Sound processing branch:

[0187] Input layer: spectral data containing 40 spectral features (sound spectrogram)

[0188] Fully connected layer: 64 neurons, ReLU activation

[0189] Output layer: sound feature vector X audio

[0190] 3. Text processing branch:

[0191] Input layer: text data containing 20 features (word vectors or other representations)

[0192] Fully connected layer: 32 neurons, ReLU activation

[0193] Output layer: sound feature vector X text

[0194] The input image is processed through convolution and pooling layers to extract features; the input sound spectrogram is processed through convolution and pooling layers to extract features; the text is processed through a fully connected layer to extract features;

[0195] To effectively fuse the features of different modal data and focus on the importance of each modality at different stages, we introduce a cross-modal attention mechanism;

[0196] We introduce an attention weight calculation mechanism to calculate the importance of different modal data in the feature fusion process;

[0197] We use a multi-layer perceptron (MLP) to learn the attention weights of different modal data to adjust the contribution of different modal data in feature fusion. The calculation formula of input attention weight is as follows:

[0198] α img = σ(W img X img +b img )

[0199] α audio = σ(W audio X audio + b audio )

[0200] α text = σ(W text X text + b text )

[0201] where σ is the Sigmoid activation function, which limits the attention weight between 0 and 1, representing the relative importance of the weight; α img , α audio , α text are the attention weights of image, sound, and text modalities, respectively, whose values are between 0 and 1, representing the importance of each modality data in feature fusion; W img is the weight matrix for calculating the attention weight of image features; b img is the bias for calculating the attention weight of image features; W audio is the weight matrix for calculating the attention weight of sound features; b audio is the bias for calculating the attention weight of sound features; W text is the weight matrix for calculating the attention weight of physiological features; b text is the bias for calculating the attention weight of physiological features.

[0202] The features of different modality data are weighted and fused using the attention weights to obtain the final driver state recognition result; the final fused features are represented as:

[0203] X combined = α img X img + α audio X audio + α text X text

[0204] where X combined is the final fused feature vector, and the contribution of each modality data is weighted by the corresponding attention weight;

[0205] Finally, the fused feature vector X commbined is input to the input layer; the fully connected layer contains 3 neurons, which accept X combined and perform further feature extraction and classification; through the Softmax layer, the classification probability distribution of the traffic environment state is output.

[0206] The specific process of step 4 is as follows:

[0207] Driver state recognition priority is usually higher than traffic environment perception and prediction, because ensuring the state and safety of the driver is the primary task of the intelligent driving system; the dual-channel takes the state information output in steps 2 and 3 as the input of the decision engine; the system automatically makes decisions, such as triggering a takeover request or warning of vision, hearing, touch, or smell, according to the current driver state and environmental conditions through learning and optimization;

[0208] Figure 2 Including making decisions by using rule engine algorithms according to different traffic environments, the specific priority and decision rules are as follows:

[0209] 1. Driver state: fatigue detection

[0210] Condition: The frequency of the driver closing his eyes exceeds the threshold (the number of closing eyes exceeds 10 times per minute);

[0211] Decision: Trigger a visual warning and suggest that the driver rest or park for a rest;

[0212] 2. Driver state: distraction detection

[0213] Condition: The driver's head frequently turns and does not pay attention to the road conditions ahead;

[0214] Decision: Trigger an audible warning to remind the driver to pay attention to the road conditions ahead;

[0215] 3. Driver state: alcohol detection

[0216] Condition: Alcohol concentration exceeds the legal limit (more than 20 mg / 100 ml);

[0217] Decision: Trigger a visual and audible warning and park or seek alternative driving;

[0218] 4. Traffic environment: distance in front of the vehicle

[0219] Condition: The distance between the front vehicle and the vehicle is too close (when the vehicle speed is more than 100 km / h, the distance should be more than 100 meters from the front vehicle in the same lane, and when the vehicle speed is less than 100 km / h, the distance can be appropriately shortened, but the minimum distance should not be less than 50 meters);

[0220] Decision: Trigger a tactile warning to remind the driver to maintain a safe distance or slow down;

[0221] 5. Traffic environment: intersection

[0222] Condition: The distance to the intersection is less than 20 meters, and the signal light is red;

[0223] Decision: Trigger a visual and audible warning to remind the driver to stop and wait for the signal;

[0224] 6. Driver state and traffic environment synthesis: fatigue + intersection

[0225] Condition: Driver is in a state of fatigue and is less than 20 meters from an intersection;

[0226] Decision: Trigger visual and audio alerts, forcing the driver to stop and wait for the signal;

[0227] 7. Driver state: adverse weather

[0228] Condition: Adverse weather (e.g. rain, snow, etc.) is detected;

[0229] Decision: Trigger haptic alerts, reminding the driver to slow down and maintain safe driving;

[0230] 8. Driver state: hands off the wheel

[0231] Condition: Driver has not held the steering wheel for 5 seconds;

[0232] Decision: Trigger audio alerts, reminding the driver to keep their hands on the wheel;

[0233] 9. Traffic environment: pedestrians crossing

[0234] Condition: Signs of pedestrians crossing the road are detected;

[0235] Decision: Trigger visual and audio alerts, reminding the driver to slow down and drive carefully;

[0236] 10. Traffic environment: road signs

[0237] Condition: Speed limit signs or other traffic signs are detected;

[0238] Decision: Trigger audio alerts, reminding the driver to obey traffic rules; 11. Driver state: excessive acceleration Condition: Driver is accelerating excessively, with a vehicle instantaneous acceleration greater than 0.5g;

[0239] Decision: Trigger haptic and audio alerts, reminding the driver to slow down;

[0240] 12. Driver state: abnormal heart rate

[0241] Condition: Driver's heart rate is abnormally elevated;

[0242] Decision: Trigger visual and audio alerts, advising the driver to stop or seek medical attention;

[0243] 13. Traffic environment: road congestion

[0244] Condition: Road congestion is detected;

[0245] Decision: Trigger a sound alert to remind the driver to choose an alternative route or prepare for a delay; 14. Traffic environment: sharp turn

[0246] Condition: The vehicle is approaching a sharp turn section;

[0247] Decision: Trigger a sound alert to remind the driver to slow down and drive carefully; 15. Driver state: distraction

[0248] Condition: The driver uses a mobile phone or other device for more than 10 seconds;

[0249] Decision: Trigger a visual and sound alert to remind the driver to focus on driving; 16. Driver state: emotional abnormality

[0250] Condition: Emotional abnormality of the driver is detected;

[0251] Decision: Trigger a visual and sound alert to remind the driver to drive calmly; 17. Traffic environment: road potholes

[0252] Condition: The vehicle is about to pass through a road pothole area;

[0253] Decision: Trigger a tactile alert to remind the driver to pay attention to avoid bumpy driving;

[0254] 18. Traffic environment: height-limited bridge / tunnel

[0255] Condition: The vehicle is about to enter a height-limited bridge or tunnel;

[0256] Decision: Trigger a sound alert to remind the driver to pay attention to the height of the vehicle and slow down; 19. Combination of driver state and traffic environment: fatigue + road sign

[0257] Condition: The driver is in a state of fatigue and there is a speed limit sign ahead;

[0258] Decision: Trigger a visual and sound alert to remind the driver to pay attention to the speed limit sign and slow down.

[0259] Finally, it should be noted that the above only describes the preferred embodiments of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent replacements to some technical features. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for requesting automatic driving takeover based on integrated driver and traffic environment data, characterized in that, The method comprises the following steps: Step 1, constructing a database of driver visual, auditory, tactile and olfactory sensory characteristics information and traffic environment information; Step 2, real-time monitoring of driver fatigue, distraction, drinking typical state key features and analysis and identification; Step 3, real-time monitoring of traffic environment, including vehicle, pedestrian, road sign and signal light key features and analysis and identification; Step 4, providing different decision suggestions according to the double-channel model combined with the rule engine algorithm; The specific process of step 2 is as follows: Introduce convolutional neural network and cross-modal attention mechanism to analyze and identify the behavior and physiological characteristics of the driver, including the detection of fatigue, distraction, eye closure frequency and odor key features; by monitoring these features, the system can determine in real time whether the driver is in a suitable state for driving, and whether a warning or request for vehicle takeover is needed; The convolutional neural network architecture is as follows: (1) Image processing branch Input layer: 32x32 gray image, Convolutional layer 1: 32 3x3 convolutional kernels, ReLU activation, Pooling layer 1: 2x2 max pooling, Convolutional layer 2: 64 3x3 convolutional kernels, ReLU activation, Pooling layer 2: 2x2 max pooling, Flattening layer: flattens the feature map into a one-dimensional vector X img , Output: image feature vector; (2) Sound processing branch Input layer: sound data containing 20 spectral features, Fully connected layer 1: 128 neurons, ReLU activation, Output layer: sound feature vector X andio , (3) Physiological signal processing branch Input layer: physiological signal data containing 10 features, Fully connected layer 1: 64 neurons, ReLU activation, Output layer: sound feature vector X physio ; The input image extracts features through convolutional and pooling layers; the input sound spectrum image extracts features through convolutional and pooling layers; The tactile and olfactory physiological signals extract features through fully connected layers; In order to effectively fuse the features of different modal data and pay attention to the importance of each modality at different stages, a cross-modal attention mechanism is introduced; An attention weighting algorithm is introduced to calculate the importance of different modal data in the feature fusion process; A multi-layer perceptron is used to learn the attention weights of different modal data to adjust the contribution of different modal data in feature fusion, and the calculation formula of input attention weight is as follows: a img = σ(W img X img + b img ) a audio = σ(W audio X audio + b audio ) a physio = σ(W physio X physio + b physio ) where σ is a Sigmoid activation function to set the attention weights between 0 and 1, representing the relative importance of the weights; a img , a audio , a physio are the attention weights for image, sound, and physiological modalities, respectively, whose values are between 0 and 1, representing the importance of each modality data in feature fusion; W img is the weight matrix for computing the attention weights of image features; b img is the bias for computing the attention weights of image features; W audio is the weight matrix for computing the attention weights of sound features; b audio is the bias for computing the attention weights of sound features; W physio is the weight matrix for computing the attention weights of physiological features; b physio is the bias for computing the attention weights of physiological features; The features of different modal data are weighted and fused using attention weights to obtain the final driver state recognition result; the final fused feature result is represented as: X combined = a img x X img + a audio x X audio + a physio x X physio where X combined is the final fused feature vector, where the contribution of each modality data is weighted by the corresponding attention weight; Finally, the fused feature vector X combined is input to the input layer; the fully connected layer 2 contains 128 neurons that accept X combined and performs further feature extraction and classification; through the Softmax layer, the classification probability distribution of the driver state is output.

2. The method of claim 1, wherein the method further comprises: The specific process of step 1 is as follows: First, deploy sensors inside the vehicle as follows: Install a camera inside the vehicle to capture the driver's facial expressions and eye movements; install a microphone to capture the driver's voice and vocal characteristics; configure a touch sensor to capture the driver's hand movements and touch operations; Integrate a gas sensor to capture the odor in the vehicle cabin and the driver's odor feedback; Real-time capture of data from various sensors, including video stream, audio, touch operation and gas signal; Transfer the collected data to the vehicle-mounted computing unit or cloud server; process and analyze the perception data in real time on the cloud server; extract visual features of facial expressions and eye movements; Identify the driver's voice features and emotional state; Analyze data from touch sensors and gas sensors to identify driver hand movements and odor perception; According to the collected data, determine the emotional state and attention level of the driver by analyzing facial expressions and eye movements; Determine the driver's speech rate, tone and content through voice analysis to determine whether they are tired or distracted; analyze hand movements and touch events to determine the driver's operating behavior and reaction speed; analyze gas sensor data to determine the driver's olfactory feedback to the environment; Establish a driver sensory information database: store the obtained driver sensory feature data in the database; the database contains personal features and data of the driver in different behavior states, including normal state, fatigue, distraction, alcohol consumption; the database is scalable and can be operated in real time to meet the data storage and management needs of different drivers and driving situations. 3.The method of claim 1, wherein, Step 3 specific process: Introduce deep learning technology to analyze the road traffic environment, including vehicles, pedestrians, road signs and signal light information, and identify potential dangerous situations or abnormal driving behaviors; by introducing convolutional neural networks and cross-modal attention mechanisms into the system, analyze and identify the characteristics of the surrounding traffic environment, which includes detecting the number of vehicles, speed, distance, number of pedestrians, location, moving track, road signs and signal light information, by monitoring these features, the system can judge potential dangerous situations or abnormal driving behaviors in real time, and issue warnings or take over requests to the driver; (1) Image processing branch Input layer: 32x32 color image, Convolution layer 1: 32 3x3 convolution kernels, ReLU activation, Pooling layer 1: 2x2 max pooling, Convolution layer 2: 64 3x3 convolution kernels, ReLU activation, Pooling layer 2: 2x2 max pooling, Fully connected layer: 128 neurons, ReLU activation, Flattening layer: flattens the feature map into a one-dimensional vector X img , Output: image feature vector; (2) Sound processing branch: Input layer: spectral data containing 40 spectral features, Fully connected layer: 64 neurons, ReLU activation, Output layer: sound feature vector X andio ; (3) Text processing branch: Input layer: text data containing 20 features, Fully connected layer: 32 neurons, ReLU activation, Output layer: sound feature vector X text ; The input image is extracted by convolution and pooling layers; the input sound spectrum is extracted by convolution and pooling layers; the text is extracted by fully connected layers; In order to effectively fuse the features of different modal data and pay attention to the importance of each modality at different stages, a cross-modal attention mechanism is introduced; Introduce an attention weight calculation mechanism to calculate the importance of different modal data in the feature fusion process; Use a multi-layer perceptron to learn the attention weights of different modal data to adjust the contribution of different modal data in feature fusion, the calculation formula of input attention weight is as follows: a img = σ(W img X img + b img ) a audio = σ(W audio X audio + b audio ) a text = σ(W text X text + b text ) where σ is a Sigmoid activation function to limit the attention weights between 0 and 1, representing the relative importance of the weights; a img , a audio , a text are the attention weights of image, sound, text modalities respectively, whose values are between 0 and 1, representing the importance of each modality data in feature fusion; W img is the weight matrix for calculating the attention weight of image features; b img is the bias for calculating the attention weight of image features; W audio is the weight matrix for calculating the attention weight of sound features; b audio is the bias for calculating the attention weight of sound features; W text is the weight matrix for calculating the attention weight of physiological features; b text is the bias for calculating the attention weight of physiological features; Use attention weights to weight and fuse the features of different modal data to get the final driver state recognition result; the final fused feature is represented as: X combined = a img × X img + a audio × X audio + a text × X text where X combined is the final fused feature vector, where the contribution of each modality data is weighted by the corresponding attention weight; Finally, the fused feature vector X combined is input to the input layer; the fully connected layer contains 3 neurons, which accepts X combined and performs further feature extraction and classification; through the Softmax layer, the classification probability distribution of the traffic environment state is output. 4.The method of claim 1, wherein, Step 4 specific process: Driver state recognition priority is higher than traffic environment perception and prediction, the dual-channel takes the state information of the output in step 2 and step 3 as the input of the decision engine; the system automatically makes decisions according to the current driver state and environmental conditions through learning and optimization; According to different traffic environment, use rule engine algorithm to make decisions, the specific priority and decision rules are as follows: (1) Driver state: fatigue detection Condition: Driver's eye closure frequency exceeds the threshold, eye closure frequency exceeds 10 times per minute; Decision: Trigger visual warning, suggest driver to rest or park for rest; (2) Driver state: distraction detection Condition: Driver's head frequently turns, does not pay attention to the road conditions in front; Decision: Trigger sound warning, remind the driver to pay attention to the road conditions in front; (3) Driver state: alcohol detection Condition: Blood alcohol concentration exceeds the provisions of the road traffic safety law, exceeds 20mg / 100ml; Decision: Trigger visual and sound warning, park or seek alternative driving; (4) Traffic environment: vehicle distance Condition: The distance between the front vehicle and the vehicle is too close, when the speed is more than 100 kilometers per hour, the distance should be more than 100 meters from the front vehicle in the same lane, when the speed is less than 100 kilometers per hour, the distance can be appropriately shortened, but the minimum distance shall not be less than 50 meters; Decision: Trigger tactile warning, remind the driver to maintain a safe distance or slow down; (5) Traffic environment: intersection Condition: The distance to the intersection is less than 20 meters, and the signal light is red; Decision: Trigger visual and sound warning, remind the driver to stop and wait for the signal; (6) Driver state and traffic environment: fatigue + intersection Condition: Driver fatigue state and distance to intersection less than 20 meters; Decision: Trigger visual and sound warning, force the driver to stop and wait for the signal; (7) Driver state: bad weather Condition: Detect bad weather; Decision: Trigger tactile warning, remind the driver to slow down and keep safe driving; (8) Driver state: hand off the steering wheel Condition: Driver does not hold the steering wheel for 5 seconds; Decision: Trigger sound warning, remind the driver to keep his hand on the steering wheel; (9) Traffic environment: pedestrian crossing Condition: Detect signs of pedestrians crossing the road; Decision: Trigger visual and sound warning, remind the driver to slow down and drive carefully; (10) Traffic environment: road signs Condition: Detect speed limit signs or other traffic signs; Decision: Trigger sound warning, remind the driver to obey traffic rules; (11) Driver state: excessive acceleration Condition: Driver steps on the accelerator too much, vehicle instantaneous acceleration greater than 0.5g; Decision: Trigger tactile and sound warning, remind the driver to slow down; (12) Driver state: abnormal heart rate Condition: Driver's heart rate is abnormally high; Decision: Trigger visual and sound warning, suggest the driver to park or seek medical treatment; (13) Traffic environment: road congestion Condition: Detect road congestion; Decision: Trigger sound warning, remind the driver to choose other routes or prepare for delay; (14) Traffic environment: sharp turn Condition: The vehicle is about to pass through a sharp turn section; Decision: Trigger sound warning, remind the driver to slow down and drive carefully; (15) Driver state: Distraction Condition: Driver uses phone or other device for more than 10 seconds; Decision: Trigger visual and audio alerts to remind driver to focus on driving; (16) Driver state: Emotional abnormality Condition: Driver emotional abnormality is detected; Decision: Trigger visual and audio alerts to remind driver to keep calm driving; (17) Traffic environment: Road potholes Condition: Vehicle is about to pass through road pothole area; Decision: Trigger tactile alerts to remind driver to pay attention to avoid bumpy driving; (18) Traffic environment: Height-limited bridge or tunnel Condition: Vehicle is about to enter height-limited bridge or tunnel; Decision: Trigger audio alerts to remind driver to pay attention to vehicle height and slow down; (19) Driver state and traffic environment combination: Fatigue + Road sign Condition: Driver fatigue state and there is speed limit sign ahead; Decision: Trigger visual and audio alerts to remind driver to pay attention to speed limit sign and slow down.

Images