Attention reminding algorithm based on multi-mode intelligent driving

Abstract

The invention discloses an attention reminding algorithm for intelligent driving based on multiple modes, and relates to the technical field of intelligent driving and active safety, and the algorithm comprises the steps: firstly, collecting data in real time through a multi-mode sensor group, carrying out the alignment, and then calculating the quantization features, such as the fixation deviation degree, the steering wheel disturbance entropy and the heart rate variability; then, mapping scores by adopting an S-type function, analyzing a historical sequence by utilizing a long short-term memory network to generate a dynamic weight, and performing adjustment and weighted calculation on a single-mode score in combination with an environment complexity weight to obtain a comprehensive attention score; and finally, according to the continuous driving duration, calculating a linear decreasing dynamic reminding threshold value, executing dual logic judgment in combination with the environmental risk level, and triggering graded feedback through sound and light or vibration. According to the method, the problems of poor adaptability and high false alarm rate of traditional single-mode monitoring in a dynamic environment are effectively solved, the robustness of the system is improved, and adaptive evaluation and accurate early warning of driver distraction behaviors in a complex scene are realized.

Description

Technical Field

[0001] This application relates to the fields of intelligent driving and active safety technology, and in particular to an attention reminder algorithm for intelligent driving based on multimodal driving. Background Technology

[0002] With the popularization of intelligent driving technology, driver monitoring systems have become an important feature to ensure driving safety. Existing monitoring solutions usually rely on a single modality or simple multimodal overlay, such as monitoring eyelid closure and gaze direction through cameras, or analyzing operational characteristics through steering angle sensors to determine whether the driver is distracted or fatigued.

[0003] However, existing attention detection technologies still have limitations in practical applications. Firstly, a major technical flaw lies in the disconnect between driver state assessment and external environmental risks. Most existing solutions focus solely on extracting features from the driver's facial or body movements, failing to consider the road environment in which the vehicle is located. In low-risk and high-risk road sections (such as close following or sharp curves), the safety hazards caused by the same duration of gaze deviation are drastically different. Ignoring environmental dimensions in the assessment can lead to delayed warnings at critical moments or frequent disruptions during safe times.

[0004] Secondly, in the fusion processing of multi-sensor data, existing technologies mostly use preset fixed weights or simple logical rules for discrimination. This processing method is difficult to adapt to dynamically changing driving scenarios. For example, the confidence level of visual sensor data will drop significantly in strong backlight conditions, or the steering wheel operation features will naturally decrease when driving on a long straight highway. If the system cannot dynamically adjust the weight contribution of each modality according to the temporal context, it is very easy to generate false alarms or missed alarms due to the poor environmental adaptability of a single sensor, affecting the overall robustness of the system.

[0005] Furthermore, existing alarm decision-making logic is typically based on fixed static thresholds. This setting ignores the cumulative effect of driving tasks over time and the decline in human physiological functions. As continuous driving time increases, the driver's reaction speed and cognitive ability gradually decrease. If the judgment criteria used in the early stages of driving are still applied, it will be impossible to identify subtle inattentional behaviors in time during the fatigue accumulation phase, thus failing to provide effective safety redundancy. Summary of the Invention

[0006] The purpose of this invention is to provide an attention reminder algorithm for intelligent driving based on multimodal thinking, which at least solves the problems of insufficient monitoring accuracy and timely warning in complex driving scenarios caused by existing technologies ignoring the dynamic changes in attention requirements of the external environment, the inability of fixed weight fusion mechanisms to cope with sensor confidence fluctuations, and the inability of static thresholds to adapt to the cumulative laws of human fatigue.

[0007] This invention provides an attention reminder algorithm for intelligent driving based on multimodal sensors. The algorithm collects data through a multimodal sensor array and uses feature extraction, multimodal fusion, and dynamic decision-making mechanisms to evaluate and intervene in the driver's state.

[0008] The algorithm first performs data acquisition and preprocessing steps. A multimodal sensor array is used to acquire real-time data on driver behavior, physiological characteristics, and vehicle driving environment; all data is labeled with a unified timestamp. The feature extraction and preprocessing module performs time alignment on the heterogeneous data and calculates quantized feature vectors for visual, posture, operational, physiological, and environmental modalities within a set time window.

[0009] During feature calculation, for the visual modality, the algorithm calculates gaze deviation and eyelid closure rate features. The gaze deviation feature is obtained by calculating the Euclidean distance between the gaze point coordinates and the safe gaze center in each frame within the window, and taking the arithmetic mean of this distance within the window. The eyelid closure rate feature is obtained by statistically analyzing the percentage of frames within the window where the eye aspect ratio is lower than a preset closure threshold. For the posture modality, the algorithm reads the head yaw and pitch angle data sequences, and takes the maximum absolute value of the two as the head deviation angle feature.

[0010] For the operational mode, the algorithm calculates the steering wheel disturbance entropy feature and the hand-off-wheel duration feature. The calculation process for the steering wheel disturbance entropy feature includes discretizing the steering wheel angular velocity value into several intervals, statistically calculating the probability of the angular velocity falling within each interval, and calculating the information entropy. The calculation logic for the hand-off-wheel duration feature involves real-time monitoring of the hand grip status. When the status indicates that the hand is gripping the steering wheel, the feature value is reset to zero. When the status indicates that the hand is off the steering wheel, the difference between the current time and the timestamp of the last detected hand gripping the steering wheel is calculated. For the physiological mode, the algorithm uses adjacent heartbeat interval data within a window to calculate the root mean square of the difference between adjacent heartbeat intervals as a time-domain indicator of heart rate variability. Power spectral density analysis is performed on the heart rate variability signal, and the ratio of low-frequency power values ​​to high-frequency power values ​​is calculated as a frequency-domain stress index.

[0011] For each environmental modality, the algorithm calculates the environmental complexity weights. This process receives road curvature, distance to the vehicle in front, and normalized illumination intensity, and uses a weighted summation method to integrate these environmental factors. During the calculation, the reciprocal of the distance to the vehicle in front is taken to reflect the logic that the closer the distance, the higher the environmental complexity weight.

[0012] Subsequently, the algorithm performs a multimodal fusion evaluation step. The multimodal fusion module first maps the features of each modality to a numerical range of a unified dimension to obtain a single-modal attention score. This mapping process uses a sigmoid activation function with a decay coefficient as the mapping kernel. The decay coefficient controls the sensitivity of a specific modality to attention decline, ensuring that the function's output tends towards zero when feature values ​​deviate abnormally. Next, a temporal neural network model is used to analyze historical feature sequences and generate dynamic weights for each modality. This process utilizes a long short-term memory network to capture the temporal context features of driving behavior, infers the importance of each modality at the current moment based on the hidden layer state vector of the previous moment, and processes it through a normalization function to ensure that the sum of the dynamic weights of all modalities equals one.

[0013] The final comprehensive attention score is obtained by multiplying the individual attention scores of each modality with their corresponding dynamic weights, multiplying by the environmental complexity weight as a modulation factor, and finally summing the results of all monitored modalities. This fusion method ensures that the final score depends not only on the driver's state but also on the direct modulation of external environmental risks.

[0014] Finally, the algorithm performs dynamic decision-making and tiered alert steps. The dynamic decision-making module obtains the vehicle's continuous driving time and calculates the real-time dynamic alert threshold using a linear correction model. This calculation process sets a base alert threshold and a maximum reference driving time, calculates the cumulative fatigue correction based on the ratio of the current continuous driving time to the maximum reference driving time, and subtracts this correction from the base alert threshold, thus making the real-time dynamic alert threshold decrease linearly with the increase of driving time.

[0015] The logic for determining the alert level is as follows: When the overall attention score is less than the severe distraction threshold, or when the overall attention score is less than the real-time dynamic alert threshold and the environmental complexity weight is greater than the high-risk environment threshold, it is determined to be a Level 2 dangerous state, and a Level 2 alert instruction is generated; when the overall attention score is less than the real-time dynamic alert threshold but greater than or equal to the severe distraction threshold, and the environmental complexity weight does not exceed the high-risk environment threshold, it is determined to be a Level 1 distraction state, and a Level 1 alert instruction is generated; otherwise, it is determined to be a normal driving state. The human-machine interaction module outputs corresponding physical feedback signals based on the determination result.

[0016] The above solution achieves the following beneficial technical effects:

[0017] This application improves the safety of attention assessment in complex traffic scenarios by introducing environmental complexity weights as modulation factors during multimodal fusion. Existing technologies typically focus only on the driver's physiological or behavioral characteristics, neglecting the dynamic changes in the external environment's attention requirements. This application quantifies road curvature, distance to the vehicle ahead, and light intensity as environmental weights and directly incorporates them into the calculation logic of the comprehensive attention score. This means that in high-risk scenarios such as following too closely or navigating sharp curves, the system will forcibly lower the attention score, thereby triggering warnings earlier when the driver has not yet experienced physiological fatigue but the environmental risk has increased sharply, achieving a coupled assessment of environmental risk and driving status.

[0018] This application utilizes a temporal neural network to construct a dynamic allocation mechanism for the weights of each modality, effectively solving the problem of misjudgment caused by the failure of a single sensor or fluctuations in confidence. Driving scenarios are highly dynamic; for example, the reliability of visual features decreases under strong backlighting, or steering wheel operation features are not obvious when maintaining a straight path. By analyzing the contextual relationships of historical feature sequences, this application can adaptively reduce the weight of disturbed modalities and increase the weight of high-confidence modalities (such as physiological indicators or posture features) at the current moment. This mechanism fully utilizes the complementarity of multi-source data, ensuring that the system can still output stable attention evaluation results even when some sensor data is missing or disturbed.

[0019] This application establishes a dynamic threshold model that decreases linearly with continuous driving time, conforming to the physiological law of human fatigue accumulation. Traditional fixed threshold schemes struggle to balance false alarm control in the early stages of driving with false alarm prevention in later stages. This application automatically adjusts the severity of distraction detection by monitoring the vehicle's running time after startup. As driving time increases, the alert threshold gradually decreases, making the system more sensitive to even minor distractions. This design ensures that the system can provide more timely intervention when the driver is in a period of fatigue accumulation, avoiding the risk of delayed reaction due to declining physiological function. Attached Figure Description

[0020] Figure 1 This is a system overall structure block diagram of an attention reminder system for intelligent driving based on multimodal driving according to the present invention.

[0021] Figure 2 This is a flowchart of the overall algorithm of an attention reminder algorithm for intelligent driving based on multimodal driving according to the present invention.

[0022] Among them, 100 is a multimodal sensor group; 101 is a visual acquisition unit; 102 is a posture perception unit; 103 is an operation signal acquisition unit; 104 is a physiological monitoring unit; 105 is an environmental perception unit; 200 is a feature extraction and preprocessing module; 300 is a multimodal fusion module; 400 is a dynamic decision-making module; and 500 is a human-computer interaction module. Detailed Implementation

[0023] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] See attached document Figure 1 This invention provides an attention alert system for intelligent driving based on multimodal sensors. The system includes: a multimodal sensor group 100, a feature extraction and preprocessing module 200, a multimodal fusion module 300, a dynamic decision-making module 400, and a human-computer interaction module 500. These modules interact with each other via an onboard bus or an on-chip bus to achieve real-time data transmission and processing.

[0025] The multimodal sensor array 100 is configured to collect real-time driver behavior data, physiological data, and vehicle driving environment data. Specifically, the multimodal sensor array 100 includes a vision acquisition unit 101, an attitude perception unit 102, an operation signal acquisition unit 103, a physiological monitoring unit 104, and an environmental perception unit 105. The vision acquisition unit 101 is mounted above the dashboard or on the A-pillar in the driver's cab and is configured to acquire image data including the driver's eye and facial features. The attitude perception unit 102 includes an inertial measurement unit and a cockpit camera and is configured to acquire three-dimensional attitude data of the driver's head.

[0026] The operation signal acquisition unit 103 is connected to the vehicle chassis controller local area network and is configured to read steering wheel angular velocity, steering torque, and hand-off-wheel status signals. The physiological monitoring unit 104 is integrated into the steering wheel grip or configured as a wearable device interface and is configured to collect the driver's electrocardiogram or pulse wave signals. The environmental perception unit 105 includes a forward-facing camera, a positioning device, and a vehicle network communication terminal, and is configured to acquire data on road curvature, distance to the vehicle in front, and ambient light intensity.

[0027] The feature extraction and preprocessing module 200 is communicatively connected to the multimodal sensor group 100. The feature extraction and preprocessing module 200 is configured to receive raw sensor data and perform time synchronization and standardization processing on the multi-source heterogeneous data. The feature extraction and preprocessing module 200 internally has a sliding time window for calculating visual modality features, posture modality features, operational modality features, physiological modality features, and environmental modality features within this window. The feature extraction and preprocessing module 200 transmits the calculated quantized feature vectors to the multimodal fusion module 300.

[0028] The multimodal fusion module 300 is connected to the feature extraction and preprocessing module 200. The multimodal fusion module 300 is configured to execute an attention evaluation algorithm. The multimodal fusion module 300 maps the features of each modality to a numerical range of a unified dimension, obtaining a single-modal attention score. The multimodal fusion module 300 uses a pre-built temporal neural network model to process historical feature sequences, generating dynamic weights for each modality. Based on the dynamic weights and environmental complexity weights, the multimodal fusion module 300 performs a weighted summation of the single-modal attention scores to generate a comprehensive attention score reflecting the driver's current state.

[0029] The dynamic decision-making module 400 receives a comprehensive attention score from the multimodal fusion module 300. The dynamic decision-making module 400 internally stores basic threshold parameters and a fatigue accumulation correction model. The dynamic decision-making module 400 is configured to monitor the vehicle's continuous driving time and calculate the current dynamic alert threshold based on the driving time. The dynamic decision-making module 400 compares the comprehensive attention score with the dynamic alert threshold and, in conjunction with the environmental risk level provided by the environmental perception unit 105, generates a tiered control command.

[0030] The human-machine interface module 500 responds to control commands output by the dynamic decision-making module 400. The human-machine interface module 500 includes an in-vehicle speaker, a seat vibration actuator, and a central control display screen. The human-machine interface module 500 is configured to execute primary or secondary alerts based on control commands to prompt the driver to concentrate.

[0031] See attached document Figure 2 This invention provides an attention reminder algorithm for intelligent driving based on multimodal operation, which is executed based on the above-mentioned system architecture.

[0032] In step S100, the multimodal sensor group 100 collects data. During vehicle operation, the vision acquisition unit 101 captures the driver's facial image, the posture perception unit 102 measures the head yaw angle, the operation signal acquisition unit 103 records steering wheel operation data, the physiological monitoring unit 104 detects heart rate variability indicators, and the environmental perception unit 105 simultaneously acquires road geometry information and traffic flow information. All data are labeled with a unified timestamp.

[0033] In step S200, the feature extraction and preprocessing module 200 performs feature calculations. The feature extraction and preprocessing module 200 resamples and aligns the data from each sensor based on a preset sampling frequency. The feature extraction and preprocessing module 200 calculates the gaze deviation, eyelid closure rate, head deviation angle, steering wheel perturbation entropy, root mean square error of heart rate variability, and environmental complexity weights within the current time window.

[0034] In step S300, the multimodal fusion module 300 performs a fusion evaluation. First, the multimodal fusion module 300 converts the features of each physical dimension into normalized attention scores. Then, by analyzing the feature states of historical time steps, the multimodal fusion module 300 outputs the confidence weight of each modality at the current moment. Finally, the multimodal fusion module 300 combines the environmental complexity weight and the confidence weight to calculate the final comprehensive attention score.

[0035] In step S400, the dynamic decision-making module 400 determines the alert level. The dynamic decision-making module 400 obtains the current single continuous driving duration and calculates the real-time dynamic alert threshold using a linearly decreasing function. The dynamic decision-making module 400 then determines whether the overall attention score is lower than the dynamic alert threshold. If it is lower than the threshold, it further determines whether the current environmental complexity weight exceeds the high-risk threshold or whether the score is lower than the severe distraction threshold.

[0036] In step S500, the human-computer interaction module 500 outputs feedback. If step S400 determines that the distraction is mild, the human-computer interaction module 500 triggers an audio prompt; if it determines that the distraction is severe or in a high-risk scenario, the human-computer interaction module 500 simultaneously triggers an audio alarm and seat vibration; if the determination result shows that the attention is normal, the system continues to execute step S100 for cyclical monitoring.

[0037] In this embodiment, each subunit of the multimodal sensor group 100 achieves synchronous data acquisition through a specific hardware layout and communication protocol.

[0038] The visual acquisition unit 101 includes a near-infrared camera sensor and an infrared auxiliary light. The near-infrared camera sensor is mounted above the steering column cover or in the dashboard visor area within the driver's cab, with the lens optical axis pointing towards the driver's face. The infrared auxiliary light is configured to provide active illumination with a wavelength of 850nm or 940nm in low-light environments, ensuring all-weather image clarity. The visual acquisition unit 101 is configured to output a grayscale image stream at a frame rate of 30Hz or higher. The image processing chip performs real-time analysis of the grayscale image stream, extracting the coordinates of key facial points, specifically including the coordinates of the inner corner of the eye, the outer corner of the eye, the pupil center, and the upper and lower eyelid edges. Based on these key point coordinates, the visual acquisition unit 101 outputs normalized gaze vector data and eyelid vertical distance data.

[0039] The attitude perception unit 102 includes a six-axis inertial measurement sensor and a cockpit wide-angle camera. The six-axis inertial measurement sensor is fixed to the vehicle chassis or under the driver's seat to collect the vehicle's own acceleration and angular velocity data, eliminating interference from vehicle motion on head attitude determination. The cockpit wide-angle camera is configured to capture upper-body images including the driver's head and shoulders. The attitude perception unit 102 combines the facial normal vector with the vehicle coordinate system to calculate the Euler angles of the driver's head in three-dimensional space, specifically outputting the head yaw and pitch angles.

[0040] The operation signal acquisition unit 103 is physically connected to the vehicle's chassis high-speed CAN bus or FlexRay bus. The operation signal acquisition unit 103 is configured to read data frames from the vehicle chassis domain controller at a sampling period of 10ms. The read data frames include the absolute steering angle value output by the steering wheel angle sensor, the real-time torque value output by the steering wheel torque sensor, and the hand grip status flag bit output by the capacitive steering wheel sensor. The operation signal acquisition unit 103 performs low-pass filtering on the read raw signals to remove high-frequency noise interference and outputs smoothed steering wheel angular velocity data and hand-off-wheel duration data.

[0041] The physiological monitoring unit 104 includes dry electrode sensors or photoplethysmography (PPG) sensors embedded in the grip areas on both sides of the steering wheel rim. When the driver holds the steering wheel with both hands, the physiological monitoring unit 104 collects electrocardiogram (ECG) signals or pulse wave signals through skin contact. The physiological monitoring unit 104 integrates analog front-end circuitry configured to amplify and convert weak bioelectrical signals from analog to digital. The signal processing circuit detects the peak position of the R wave in the heartbeat waveform, calculates the time interval sequence between adjacent R waves, and outputs successive heartbeat interval (RR interval) data.

[0042] The environmental perception unit 105 includes a forward-looking multi-function camera, a millimeter-wave radar, and an onboard GPS receiver. The forward-looking multi-function camera is mounted on the upper edge of the windshield and is configured to identify lane line directions and traffic signs. The millimeter-wave radar is configured to detect the relative distance and relative speed of vehicles ahead. The environmental perception unit 105 calculates the radius of curvature of the current lane centerline based on the quadratic coefficient of the lane line equation and outputs road curvature data. Simultaneously, the environmental perception unit 105 uses the light-sensing module built into the forward-looking multi-function camera to measure ambient illuminance and outputs a normalized illuminance value. The onboard GPS receiver, combined with high-precision map data, assists in outputting the road type label for the current road segment.

[0043] The present invention provides a feature extraction and preprocessing method, which is executed by a feature extraction and preprocessing module 200. The feature extraction and preprocessing module 200 is configured to clean, align and calculate features of the raw data stream output by the multimodal sensor group 100.

[0044] The feature extraction and preprocessing module 200 is set with a fixed-length time window. and a unified system sampling frequency The feature extraction and preprocessing module 200 uses the current time... As the cutoff point, extract the past. The data sequence within a given time period. For heterogeneous data with inconsistent sampling frequencies, the feature extraction and preprocessing module 200 uses linear interpolation or nearest neighbor matching algorithms to align the timestamps of each sensor's data to the time axis of the system's sampling frequency. In this embodiment, the time window... Set to 2 seconds, the number of frames contained within the window. It is 60 frames per second.

[0045] For visual modal data, the feature extraction and preprocessing module 200 calculates gaze deviation features. The feature extraction and preprocessing module 200 defines the central area of ​​the windshield as the safe gaze center. The feature extraction and preprocessing module 200 calculates the gaze coordinates of each frame within the window. With the center The Euclidean distance between them is calculated, and the arithmetic mean of this distance within the window is calculated using the following formula:

[0046] ;

[0047] in, For the first The gaze coordinate vector of the frame.

[0048] Simultaneously, the feature extraction and preprocessing module 200 calculates the eyelid closure rate feature. The feature extraction and preprocessing module 200 obtains the eye aspect ratio for each frame. and compare it with the preset closure threshold. A comparison is then made. The feature extraction and preprocessing module calculates the percentage of frames within a statistical window that satisfy the closure condition, using the following formula:

[0049] ;

[0050] in, This is an indicator function that takes a value of 1 when the condition is met, and 0 otherwise. In this embodiment, The value is 0.25.

[0051] For the posture modal data, the feature extraction and preprocessing module 200 calculates the head deviation angle feature. The feature extraction and preprocessing module 200 reads the head yaw angle. and pitch angle Given a data sequence, the maximum absolute value of the two values ​​is taken as the current head deviation metric, and the calculation formula is expressed as:

[0052] ;

[0053] This feature is used to characterize whether the driver's head rotates or lowers significantly.

[0054] For the operational modal data, the feature extraction and preprocessing module 200 calculates the hand-off-disk duration feature. Features of steering wheel perturbation entropy The feature extraction and preprocessing module 200 records the timestamp of the last time a hand was detected gripping the steering wheel. Calculate the current time and The difference is taken as the duration of hand-off from the steering wheel. For the steering wheel perturbation entropy, the feature extraction and preprocessing module 200 discretizes the steering wheel angular velocity value into several intervals. Statistically calculate the probability of the angular velocity falling within each interval. And calculate the information entropy, the calculation formula is expressed as:

[0055] ;

[0056] This entropy value reflects the degree of disorder in driving operations and is used to identify fatigued driving behavior with excessive fine-tuning.

[0057] For operational modal data, the feature extraction and preprocessing module 200 uses piecewise function logic to calculate the hand-off-disk duration feature. The feature extraction and preprocessing module 200 monitors the hand grip status bit uploaded by the operation signal acquisition unit 103 in real time. When the status bit shows that the hands are gripping the steering wheel, the module will... Reset to 0. When the status indicator shows the hand is off the steering wheel, the module records the timestamp of the status transition as... (i.e., the last time you held the steering wheel), and set the current time. and The difference is taken as the duration. The specific formula for calculation is as follows:

[0058] ;

[0059] This formula can accurately quantify the duration of time a driver is out of physical control of the vehicle, and is used to assess operational-level inattention deficit.

[0060] For physiological modal data, the feature extraction and preprocessing module 200 calculates the time-domain index of heart rate variability. With frequency domain pressure index The feature extraction and preprocessing module 200 utilizes adjacent heartbeat interval data within a window. The root mean square of the difference between adjacent RR intervals is calculated using the following formula:

[0061] ;

[0062] In addition, the feature extraction and preprocessing module 200 performs power spectral density analysis on the heart rate variability signal and extracts the low-frequency band. High frequency band The power value is used to calculate the ratio of the two values ​​as the pressure index. The calculation formula is expressed as:

[0063] ;

[0064] For environmental modal data, the feature extraction and preprocessing module 200 calculates the environmental complexity weights. The feature extraction and preprocessing module 200 receives the road curvature. Distance to the vehicle in front and normalized light intensity The feature extraction and preprocessing module 200 uses a weighted summation method to integrate the above environmental factors, and the calculation formula is expressed as follows:

[0065] ;

[0066] in, , , The environmental impact weighting coefficients are preset. The distance to the vehicle in front is used in this formula. Taking the reciprocal indicates that the closer the distance, the higher the weight of the environment complexity. In this embodiment, it is set... , , The feature extraction and preprocessing module 200 packages all the feature vectors calculated above and transmits them to the multimodal fusion module 300.

[0067] This invention provides a multimodal attention fusion method, which is executed by a multimodal fusion module 300. The multimodal fusion module 300 is configured to receive the feature vectors of each modality output by the feature extraction and preprocessing module 200, and generate the final driver attention score through a unified mapping space and temporal dynamic weighting mechanism.

[0068] The multimodal fusion module 300 first performs the single-modal attention scoring mapping step. The multimodal fusion module 300 then processes the received features from each modality... (in (Representing visual, posture, operational, or physiological modalities) mapped to intervals Internally, to eliminate differences in physical dimensions. The multimodal fusion module 300 uses a Sigmoid function with a decay coefficient as the mapping kernel, and the calculation formula is expressed as:

[0069] ;

[0070] in, These are the feature values ​​after Z-score standardization, with a mean of 0 and a standard deviation of 1. For modality The decay coefficient is a pre-trained learnable parameter used to control the sensitivity of a specific modality to attention decay. The Sigmoid activation function is defined as follows: The physical meaning of this formula is: when the eigenvalue... When the value is above the mean (i.e., there is an abnormal deviation), the input item The value becomes negative, causing the output score to change. A decrease and tendency toward 0 indicates a degradation of attention in that modal dimension.

[0071] The multimodal fusion module 300 then performs a dynamic weight allocation step. Considering the time-varying characteristics of the dominant modality during driving, the multimodal fusion module 300 introduces a Long Short-Term Memory (LSTM) network to capture the temporal context features of driving behavior. The multimodal fusion module 300 utilizes the hidden state from the previous time step. To infer the importance weights of each modality at the current time, the calculation formula is expressed as:

[0072] ;

[0073] in, Indicates that the LSTM network is in The hidden layer state vector at each time step is used to represent the historical driving behavior pattern. and These are the trainable weight matrices for the attention layer and the hidden layer, respectively. This is the bias vector. The function is used to normalize the output vector, ensuring the weights of all modes are normalized. The sum equals 1. This mechanism can adaptively boost the weights of key modes and suppress the weights of noisy modes based on the current driving context (e.g., lane changing, following other vehicles, or congestion).

[0074] The multimodal fusion module 300 finally executes the fusion attention score generation step. The multimodal fusion module 300 then combines the attention scores from each single modality... Its corresponding dynamic weight Perform a weighted combination and introduce environmental complexity weights. Modulation is performed, and the final comprehensive attention score is calculated. The calculation formula is expressed as follows:

[0075] ;

[0076] Among them, the summation symbol Traverse all monitored modalities. Perform a comprehensive attention score. The range of values ​​is This fusion formula introduces environmental complexity weights. As a modulation factor, the final score depends not only on the driver's physiological and behavioral state but also on the constraints of external environmental risks, thereby enabling more stringent identification of driver distraction in high-risk scenarios. The multimodal fusion module 300 will calculate... Transmitted to dynamic decision module 400.

[0077] This invention provides a dynamic threshold decision-making and hierarchical reminder method, which is mainly executed collaboratively by a dynamic decision-making module 400 and a human-computer interaction module 500.

[0078] The dynamic decision module 400 is configured to receive a comprehensive attention score from the multimodal fusion module 300. and the environmental complexity weights from the feature extraction and preprocessing module 200. Meanwhile, the dynamic decision-making module 400's internal timer records the continuous driving time after the vehicle starts in real time. The dynamic decision-making module 400 uses a linear correction model to calculate the real-time alert threshold for the current moment. This is to accommodate the cumulative physiological fatigue that drivers experience over time. The calculation formula is expressed as:

[0079] ;

[0080] in, The system's preset basic reminder threshold, For maximum reference driving time, This is the fatigue accumulation correction factor. In this embodiment, Set to 0.6. Set to 4 hours. The value is set to 0.1. This formula establishes a functional relationship between the alert threshold and driving duration, ensuring that the evaluation criteria can dynamically follow changes in the driving cycle.

[0081] The hierarchical reminder logic executed by the dynamic decision-making module 400 specifically follows a preset segmented discrimination function. The dynamic decision-making module 400 will calculate the comprehensive attention score. Environmental complexity weight and dynamic reminder thresholds Substitute into the reminder decision function This function defines the output state under different risk levels, and the specific calculation formula is as follows:

[0082] ;

[0083] in, A sound prompt is triggered in response to the first-level reminder command; The corresponding level 2 alert command triggers a combined sound and vibration alarm. This corresponds to the silent state.

[0084] This logical formula ensures that the system has a dual safety boundary:

[0085] The first boundary is the absolute bottom line, that is, when At any time, regardless of the environment or driving duration, a level two alarm will be triggered directly;

[0086] The second boundary is the environmental sensitivity boundary, which occurs when the attention score is below the dynamic threshold. However, if the absolute bottom line has not been reached, and the complexity of the environment... The system is forcibly upgraded to a Level 2 alert, thereby enabling stricter security intervention in high-risk environments.

[0087] The dynamic decision-making module 400 is based on a comprehensive attention score. Real-time alert threshold and environmental complexity weights The hierarchical judgment logic is executed. The dynamic decision module 400 first judges the comprehensive attention score. Is it below the real-time alert threshold? .like The dynamic decision module 400 determines that the driver is in a normal driving state and does not generate a reminder instruction.

[0088] like The dynamic decision-making module 400 further combines a severe distraction threshold (set to 0.4) and a high-risk environment threshold (set to 0.7) for secondary judgment. The specific logic is as follows:

[0089] If attention score is taken into account Less than the severe distractibility threshold of 0.4, or in Under the premise of environmental complexity weight If the value is greater than 0.7, the dynamic decision-making module 400 determines that the current situation is a level two danger state. At this time, regardless of the driving time, the system believes that the driver's attention is severely lacking or the environmental risk is extremely high, and the dynamic decision-making module 400 generates a level two warning instruction.

[0090] If attention score is taken into account Located at the severe distraction threshold of 0.4 and the real-time alert threshold Between, and environmental complexity weight If the value is less than or equal to 0.7, the dynamic decision module 400 determines that the driver is currently in a level one distraction state. At this time, the system believes that the driver has a slight inattention and the environmental risk is controllable, and the dynamic decision module 400 generates a level one reminder instruction.

[0091] The human-machine interface module 500 is electrically connected to the dynamic decision-making module 400 and is configured to respond to the aforementioned commands and execute physical feedback. Upon receiving a Level 1 alert command, the human-machine interface module 500 controls the in-vehicle audio system to output a specific frequency alert tone or a pre-recorded voice prompt. Upon receiving a Level 2 alert command, the human-machine interface module 500 executes a multimodal strong interference warning, specifically including controlling the in-vehicle audio system to output a high-decibel alarm tone, and simultaneously controlling a haptic feedback actuator integrated into the driver's seat or steering wheel to generate pulse vibrations. This tiered mechanism reduces unnecessary interference to the driver in low-risk scenarios while ensuring safety warnings.

[0092] For ease of description, the above system is described by dividing it into various units and modules based on their functions. Of course, in implementing this application, the functions of each unit and module can be implemented in one or more software and / or hardware.

[0093] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art and should not be interpreted in an idealized or overly formal sense unless specifically defined.

[0094] For the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0095] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-modal based attention reminding algorithm for intelligent driving, characterized in that, The method comprises the following steps: S100, real-time collection of behavior feature data, physiological feature data and vehicle driving environment data of a driver by using a multi-modal sensor group, all data being provided with a unified time stamp label; S200, resampling alignment of the collected data by a feature extraction and preprocessing module, and calculation of visual modal features, posture modal features, operation modal features, physiological modal features and environment modal features in a set time window to obtain a quantized feature vector; S300, fusion evaluation by a multi-modal fusion module, mapping of the modal features to a unified dimension value range to obtain a single-modal attention score, analysis of a historical feature sequence by a time sequence neural network model to generate dynamic weights of the visual modal features, the posture modal features, the operation modal features, the physiological modal features and the environment modal features, and calculation of a comprehensive attention score in combination with an environment complexity weight; S400, acquisition of a continuous driving duration of a vehicle by a dynamic decision module, calculation of a real-time dynamic reminding threshold by a linear correction model, comparison of the comprehensive attention score with the real-time dynamic reminding threshold, and determination of a reminding level in combination with the environment complexity weight; S500, output of a corresponding feedback signal according to the determined reminding level by a man-machine interaction module. 2.The attention reminding algorithm for intelligent driving based on multi-modal according to claim 1, wherein, The calculation of the visual modal features specifically comprises: calculation of Euclidean distances between gaze point coordinates of each frame in the calculation window and a safe gaze center, and taking an arithmetic mean value of the Euclidean distances in the window as a gaze deviation degree feature, and statistics of a frame number ratio of eye length-width ratios lower than a preset closing threshold in the statistics window as an eyelid closing rate feature; the calculation of the posture modal features specifically comprises: reading of data sequences of a head yaw angle and a pitch angle, and taking a maximum value of absolute values of the two as a head deviation angle feature. 3.The attention reminding algorithm for intelligent driving based on multi-modal according to claim 1, wherein, The calculation of the operation modal features specifically comprises: discretization of a steering wheel rotation angle speed value into several intervals, statistics of probabilities of the rotation angle speed falling in each interval and calculation of information entropy as a steering wheel disturbance entropy feature, and real-time monitoring of a hand holding state bit, resetting of a hand-off-wheel duration feature to zero when the state bit shows that the hand is holding the steering wheel, and calculation of a difference value between a current time and a time stamp of the last time when the hand holding the steering wheel is detected as the hand-off-wheel duration feature.

4. The attention reminding algorithm for intelligent driving based on multi-modal according to claim 1, characterized in that, The calculation of the physiological modal features specifically comprises: calculation of a root mean square of adjacent heartbeat interval difference values by using adjacent heartbeat interval data in the window as a heart rate variability time domain index, and power spectral density analysis of the heart rate variability signal to extract power values of low and high frequency bands, and calculation of a ratio of the low frequency band power value to the high frequency band power value as a frequency domain stress index.

5. The attention reminding algorithm for intelligent driving based on multi-modal according to claim 1, characterized in that, The calculation of the environment modal features specifically comprises: reception of road curvature, front vehicle distance and normalized illumination intensity, and calculation of the environment complexity weight by using a weighted summation method to integrate the above environment factors; wherein, the front vehicle distance is taken as an inverse number to represent that the closer the distance is, the higher the weight is.

6. The attention reminding algorithm for intelligent driving based on multi-modal according to claim 1, characterized in that, The obtaining of the single-modal attention score specifically comprises: The visual modality feature, the posture modality feature, the operation modality feature, the physiological modality feature, and the environmental modality feature are standardized, and a Sigmoid activation function with a decay coefficient is used as a mapping kernel; The decay coefficient is used to control the sensitivity of a specific modality to attention decline, and when the feature value deviates abnormally, the output of the Sigmoid activation function tends to zero.

7. The attention reminding algorithm for intelligent driving based on multi-modal according to claim 6, characterized in that, The dynamic weights of the generated visual modality feature, posture modality feature, operation modality feature, physiological modality feature, and environmental modality feature specifically include: The long short-term memory network is used to capture the time sequence context feature of the driving behavior, the importance of each modality at the current time is inferred according to the hidden layer state vector at the previous time, and the normalization function is processed to ensure that the sum of the dynamic weights of all modalities is equal to one. 8.The attention reminding algorithm for intelligent driving based on multi-modal according to claim 7, wherein, The calculation of the comprehensive attention score specifically includes: The single-modality attention score of the visual modality feature, the posture modality feature, the operation modality feature, the physiological modality feature, and the environmental modality feature is multiplied by the dynamic weight corresponding to the visual modality feature, the posture modality feature, the operation modality feature, the physiological modality feature, and the environmental modality feature, and then multiplied by the environmental complexity weight as a modulation factor, and finally the results of all monitoring modalities are summed to obtain the comprehensive attention score. 9.The attention reminding algorithm for intelligent driving based on multi-modal according to claim 1, wherein, The calculation of the real-time dynamic reminding threshold specifically includes: The basic reminding threshold and the maximum reference driving time are set, the fatigue accumulation correction amount is calculated according to the proportion of the current continuous driving time to the maximum reference driving time, and the fatigue accumulation correction amount is subtracted from the basic reminding threshold, so that the real-time dynamic reminding threshold decreases linearly with the increase of the driving time.

10. The attention reminding algorithm for intelligent driving based on multi-modal according to claim 9, characterized in that, The determination of the reminding level specifically follows the following logic: When the comprehensive attention score is less than the serious distraction threshold or when the comprehensive attention score is less than the real-time dynamic reminding threshold and the environmental complexity weight is greater than the high-risk environment threshold at the same time, it is determined as a secondary dangerous state, and a secondary reminding instruction is generated; When the comprehensive attention score is less than the real-time dynamic reminding threshold, but greater than or equal to the serious distraction threshold, and the environmental complexity weight does not exceed the high-risk environment threshold, it is determined as a primary distraction state, and a primary reminding instruction is generated; Otherwise, it is determined as a normal driving state, and no reminding instruction is generated.

Images