AR-hud evaluation adjustment method and system fusing multi-source physiological data and driving performance

Abstract

The present application relates to a method and system for evaluating and adjusting AR-HUD based on fusion of multi-source physiological data and driving performance. The method comprises the following steps: collecting and preprocessing multi-source physiological and driving performance data of drivers, and constructing multi-modal time sequence input samples; then inputting the preprocessed samples into a driver state evaluation model based on MMA-Net, and synchronously outputting the probabilities of fatigue, visual distraction and high cognitive load states; then determining the three state probabilities based on the preset probability threshold and state priority rules, and outputting the comprehensive state grade; then generating and executing adaptive adjustment instructions for AR-HUD interface display parameters based on the comprehensive state grade; finally, collecting the driver state change data after adjustment execution as feedback signals for dynamic updating. The present application collects multi-source data to construct an evaluation model, realizes deep fusion and collaborative analysis of physiological signals and operation behaviors, synchronously outputs the probabilities of three states, comprehensively and objectively evaluates the comprehensive influence of AR-HUD interface on drivers, and overcomes the deficiency of single-dimensional evaluation.

Description

Technical Field

[0001] This invention relates to the field of intelligent cockpit function evaluation and in-vehicle human-machine interaction technology, and in particular to an AR-HUD evaluation and adjustment method and system that integrates multi-source physiological data and driving performance. Background Technology

[0002] Augmented Reality Head-Up Display (AR-HUD) is one of the core technologies in the field of human-machine interaction in smart cockpits. AR-HUD can overlay navigation guidance, driving assistance, and risk warning information as virtual images onto the real road scene in front of the driver. This allows drivers to obtain key driving information without shifting their gaze from the road, reducing the frequent switching between the road and the in-vehicle display during driving, thus improving driving safety and optimizing the intuitiveness of information interaction. Compared to traditional combined head-up displays, AR-HUD typically has a greater virtual image distance and a wider field of view, enabling spatial registration and fusion display of virtual information with the real traffic environment. With the development of automotive intelligence and connectivity, and the continuous advancement of cockpit interaction technology, AR-HUD has gradually become an important carrier for information presentation and human-machine interaction in smart vehicles.

[0003] In existing technologies, the evaluation of AR-HUD interfaces mainly revolves around dimensions such as display optical performance, information layout rationality, visual readability, and user subjective experience. Some evaluation methods focus on subjective questionnaires, expert ratings, or hardware optical parameters to conduct static testing and comprehensive scoring of the interface; others combine physiological signals such as eye movements, electroencephalograms (EEGs), and electrocardiograms (ECGs) to analyze the driver's attention level, fatigue level, or cognitive state, and attempt to adaptively adjust display parameters based on this. In addition, some studies have used methods such as the fusion of subjective and objective indicators or weighted analysis to quantitatively evaluate the comprehensive performance of AR-HUD interfaces, thereby improving the systematicity and comparability of evaluation results. However, the aforementioned existing technologies still have certain limitations.

[0004] (i) Existing evaluation methods often focus on single-dimensional information, analyzing the AR-HUD interface only from one aspect, such as subjective perception, optical parameters, or physiological signals, lacking a comprehensive portrayal of the driver's multidimensional driving state. Especially in actual driving, the driver's state is not only reflected in changes in physiological signals such as eye movements, EEG, and ECG, but also directly reflected in driving performance indicators such as lane keeping ability, speed control stability, and reaction time. However, existing technologies usually do not conduct a synergistic analysis of the driver's physiological state and driving performance, making it difficult to comprehensively and objectively assess the combined impact of the AR-HUD interface on the driver's cognitive load, operational behavior, and driving safety.

[0005] (ii) Existing technologies generally lack closed-loop linkage capabilities in interface adjustment mechanisms. Although some existing solutions can classify, compare, or optimize AR-HUD interfaces based on evaluation results, these results are mostly used only for offline design improvements and have not yet established effective linkage with the real-time operation of the display system. In other words, when the system identifies that the driver is in a state of high cognitive load, distracted attention, or visual fatigue, existing AR-HUD systems usually cannot dynamically and adaptively adjust the displayed content, information density, layout, presentation format, or warning intensity according to the driver's real-time state and driving task requirements. Therefore, it is difficult to reduce interface interference and alleviate visual burden in a timely manner, thereby improving driving safety. Summary of the Invention

[0006] In view of the shortcomings of the prior art, the purpose of this invention is to provide an AR-HUD evaluation and adjustment method and system that integrates multi-source physiological data and driving performance, so as to solve one or more problems in the prior art.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows:

[0008] An AR-HUD evaluation modulation method integrating multi-source physiological data and driving performance includes the following steps:

[0009] S1. Data Acquisition and Preprocessing: Collect multi-source physiological data and driving performance data of drivers and perform preprocessing to construct multimodal time-series input samples;

[0010] S2. Driver state evaluation model construction: By inputting the preprocessed multimodal temporal input samples into the constructed driver state evaluation model based on the multimodal multi-task dual attention network MMA-Net, the probability of driver fatigue state, visual distraction state and high cognitive load state are output simultaneously.

[0011] S3. Driver status classification determination: Based on preset probability thresholds and status priority rules determined by objective indicators, the probability of fatigue state is assessed. Probability of visual distraction and the probability of a high cognitive load state Perform a fusion judgment and output the driver's overall status level;

[0012] S4. Adaptive AR-HUD adjustment: Based on the driver's comprehensive state level, generate and execute adaptive adjustment instructions for AR-HUD interface display parameters, wherein the display parameters include at least one of information content, display position, visual attributes, and presentation mode;

[0013] S5. Closed-loop feedback optimization: Dynamically update the system by collecting driver status change data after the AR-HUD interface adjustment is executed as a feedback signal.

[0014] Furthermore, in step S1:

[0015] The multi-source physiological data includes at least two of the following: eye-tracking data, electrocardiogram data, and electroencephalogram data.

[0016] The driving performance data includes at least one of lateral control data, longitudinal control data, and reaction time data.

[0017] Furthermore, the data preprocessing in step S1 includes the following steps:

[0018] Denoising was performed, with moving average filtering applied to eye-tracking data and bandpass filtering and independent component analysis applied to remove artifacts from ECG and EEG data.

[0019] Time synchronization alignment unifies all data streams to a preset sampling rate through linear interpolation;

[0020] Feature normalization is performed by applying z-score normalization to each sample, as shown in the following expression:

[0021]

[0022] In the formula: This represents the raw measurement value of a certain physiological indicator or driving performance indicator. This indicates the average value of the indicator when the driver is at rest. The standard deviation of the same indicator at rest.

[0023] The sample segmentation process based on sliding time windows generates samples with a preset window length and sliding step size, where each sample contains a video frame sequence and a physiological time series matrix.

[0024] Furthermore, the driver state evaluation model in step 2 includes:

[0025] Multimodal encoders, including video encoders and temporal encoders, obtain corresponding visual feature vectors based on 3D convolutional neural networks and 1D convolutional neural networks, respectively. and physiological feature vectors ;

[0026] Bidirectional cross-modal attention module BCMA is used for each task Calculate video features respectively With time series characteristics The cross-modal fusion is expressed as follows:

[0027]

[0028] In the formula: Indicates the first In each task, video modalities are used as a query matrix; Indicates the first A matrix in which temporal modes are used as keys in each task; Indicates the first A matrix of temporal modes as values ​​in each task; Indicates the first The temporal modalities are used as the query matrix in each task; Indicates the first A matrix in which video modalities serve as keys in each task; Indicates the first A matrix of video modalities as values ​​in each task; The key vector dimension for each attention head;

[0029] The Multi-Task Exchange Block (MTEB) is used for feature exchange, sharing, and enhancement among multiple tasks, including fatigue recognition, visual distraction recognition, and high cognitive load recognition. Its expression is shown below:

[0030]

[0031]

[0032] In the formula: Indicates splicing; The current task The query matrix, It is a task output by BCMA. Features; Other tasks The key matrix; Other tasks The value matrix; It is a task Learnable linear transformation matrix;

[0033] Task-specific decoder, by each task A two-layer fully connected network is then used, and the enhanced fused features are mapped to fatigue state probabilities based on the Sigmoid function. Probability of visual distraction and the probability of a high cognitive load state .

[0034] Furthermore, the driver state evaluation model is based on multi-task joint optimization, and the total loss is set as the average of the binary cross-entropy losses of the three tasks, as shown in the following expression:

[0035]

[0036] In the formula: This represents the total loss value used for backpropagation; The binary cross-entropy loss function; , , These represent the probabilities of fatigue state, distraction state, and high load state predicted by the model, with values ​​ranging from 0 to 1. , , This is the actual label for the corresponding task, and its value can be 0 or 1.

[0037] Furthermore, in step S3, the driver's overall state level includes normal state, fatigue state, visual distraction state, and high cognitive load state; wherein the fatigue state further includes fatigue warning level and fatigue high risk level.

[0038] Furthermore, the state priority rule in step S3 is as follows: visual distraction state has a higher priority than fatigue state, fatigue state has a higher priority than high cognitive load state, and high cognitive load state has a higher priority than normal state.

[0039] Furthermore, the specific determination process for the state priority rule is as follows:

[0040] If the probability of visual distraction When this occurs, it is determined to be a state of visual distraction;

[0041] If the conditions for visual distraction are not met and the probability of fatigue is high When this occurs, it is determined to be a state of fatigue, where when At the time, it was at the fatigue warning level. The current situation is at a high level of fatigue risk.

[0042] If the conditions for visual distraction and fatigue are not met, and the probability of a high cognitive load state is high. When this occurs, it is determined to be a state of high cognitive load;

[0043] If none of the above conditions are met, the condition is considered normal.

[0044] Furthermore, the adaptive AR-HUD adjustment strategy in step S4 includes:

[0045] When the condition is determined to be normal, maintain the standard display mode;

[0046] When a state of high cognitive load is identified, the information content is simplified and integrated, and the information layout is adjusted to reduce the visual search range.

[0047] When visual distraction is detected, non-emergency information is paused or weakened, the visual prominence of warning information is enhanced, and AR spatial fusion is used to align the warning information with the real hazard source.

[0048] When fatigue is detected, a progressive adjustment scheme is implemented based on the fatigue warning level or high-risk fatigue level. This includes turning off unnecessary information, moving key information to the center of the field of vision, reducing the overall brightness of the interface, increasing the size of warning icons, and adding voice reminders at the high-risk sub-level.

[0049] An evaluation and adjustment system, the system being applied to the aforementioned AR-HUD evaluation and adjustment method that integrates multi-source physiological data and driving performance, the system comprising:

[0050] The data acquisition unit is used to collect multi-source physiological data and driving performance data of the driver in real time;

[0051] The data preprocessing unit is used to perform noise reduction, alignment, normalization, and window segmentation on the collected data;

[0052] The driver state evaluation unit is equipped with a driver state evaluation model based on MMA-Net, which is used to output the probability of each driver state.

[0053] The driver state fusion determination unit is used to determine the final comprehensive state level based on probability thresholds and priority rules;

[0054] The adaptive adjustment unit is used to generate AR-HUD control commands based on the overall status level and adjust the display parameters.

[0055] An AR-HUD display unit is used to receive and execute the control commands, presenting optimized augmented reality information to the driver.

[0056] Compared with the prior art, the beneficial technical effects of the present invention are as follows:

[0057] (i) This invention collects multi-source physiological data such as eye movement, electrocardiogram, and electroencephalogram, as well as driving performance data such as lateral control, longitudinal control, and reaction time, and constructs a driver state evaluation model based on a multimodal, multi-task, dual-attention network. This achieves deep fusion and collaborative analysis of physiological signals and operational behavior, and can simultaneously output the probability of fatigue state, visual distraction state, and high cognitive load state. This allows for a comprehensive and objective assessment of the combined impact of the AR-HUD interface on driver cognitive load, operational behavior, and driving safety, overcoming the shortcomings of single-dimensional evaluation.

[0058] (II) This invention refines the driver's overall state level based on objective thresholds and priority rules, and generates differentiated adaptive adjustment instructions for different state levels such as normal, fatigued, visually distracted, and high cognitive load, including information content, display position, visual attributes, and presentation mode. At the same time, by collecting changes in the adjusted driver's state as feedback signals, the model and strategy are continuously iterated and optimized using reinforcement learning or incremental learning methods, forming a complete "perception-judgment-adjustment-feedback" closed-loop operation mechanism. This enables the AR-HUD system to dynamically respond to and adaptively adjust the driver's real-time state, effectively reducing interface interference, alleviating visual burden, and improving driving safety. Attached Figure Description

[0059] Figure 1 This illustration shows a flowchart of the AR-HUD evaluation and adjustment method and system that integrates multi-source physiological data and driving performance according to an embodiment of the present invention.

[0060] Figure 2 The MMA-Net architecture diagram of the AR-HUD evaluation and modulation method and system that integrates multi-source physiological data and driving performance according to an embodiment of the present invention is shown.

[0061] Figure 3 The flowchart of the driver state determination method and system of AR-HUD evaluation adjustment method and system that integrates multi-source physiological data and driving performance according to an embodiment of the present invention is shown. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this invention clearer, the AR-HUD evaluation and adjustment method and system integrating multi-source physiological data and driving performance proposed in this invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The advantages and features of this invention will become clearer from the following description. It should be noted that the accompanying drawings are in a very simplified form and use non-precise proportions, only for the purpose of conveniently and clearly illustrating the embodiments of this invention. Please refer to the accompanying drawings to make the objectives, features, and advantages of this invention more apparent and understandable. It should be understood that the structures, proportions, sizes, etc., depicted in the accompanying drawings are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed in the specification, and are not intended to limit the implementation conditions of this invention. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportional relationships, or adjustments to the size, without affecting the effects and objectives achieved by this invention, should still fall within the scope of the technical content disclosed in this invention.

[0063] Please refer to the following: Figure 1 and Figure 2 The AR-HUD evaluation modulation method, which integrates multi-source physiological data and driving performance, includes the following steps:

[0064] S1. Data Acquisition and Preprocessing: Collect multi-source physiological data and driving performance data of drivers and perform preprocessing to construct multimodal time-series input samples.

[0065] The multi-source physiological data includes at least two of the following: eye-tracking data, electrocardiogram (ECG) data, and electroencephalogram (EEG) data. The eye-tracking data includes, but is not limited to, fixation point coordinates, pupil diameter, blink frequency, and eyelid closure. The ECG data includes, but is not limited to, heart rate variability indices and low-frequency / high-frequency power ratios. The EEG data includes, but is not limited to, frontal lobe Theta wave power and other frequency domain characteristics reflecting fatigue or cognitive load.

[0066] The driving performance data includes at least one of lateral control data, longitudinal control data, and reaction time data. The lateral control data includes, but is not limited to, the standard deviation of lateral displacement and the standard deviation of steering wheel angle. The longitudinal control data includes, but is not limited to, the standard deviation of speed, the standard deviation of acceleration, and the headway. The reaction time data includes, but is not limited to, braking reaction time.

[0067] The collection of the above data can simultaneously obtain complete information on the driver's internal physiological fluctuations and external operational performance, providing a comprehensive data foundation for subsequent comprehensive condition assessment.

[0068] Furthermore, the preprocessing of the collected data includes, in sequence:

[0069] The noise reduction process involves moving average filtering for eye-tracking data and bandpass filtering and independent component analysis for ECG and EEG data within a preset range to remove artifacts, thereby eliminating interference from irrelevant signals such as electromyography and electrooculography and ensuring the purity of the input data.

[0070] Time synchronization alignment unifies all data streams to a preset sampling rate through linear interpolation, ensuring that data from different sources remain matched in the time dimension and avoiding temporal misalignment from affecting subsequent model training and inference.

[0071] Feature normalization is performed on each sample using z-score normalization to eliminate dimensional differences between different physiological indicators. Simultaneously, baseline statistics of individual resting states are introduced to adapt to the physiological differences among different drivers, improving the model's generalization ability to different individuals. The expression is shown below:

[0072]

[0073] In the formula: This represents the raw measurement value of a certain physiological indicator or driving performance indicator, such as eyelid closure degree, pupil diameter, heart rate variability, or vehicle speed standard deviation. This represents the mean value of the indicator when the driver is in a resting state (i.e., a baseline state of normal driving, without significant fatigue or distraction). It represents the standard deviation of the same indicator at rest.

[0074] The sample segmentation process based on sliding time windows generates samples with a preset window length and sliding step size. Each sample contains a video frame sequence and a physiological time series matrix, thereby constructing input samples that can reflect the temporal changes in the driver's state and meet the model's requirements for temporal context information.

[0075] S2. Driver state evaluation model construction: By inputting the preprocessed multimodal temporal input samples into the constructed driver state evaluation model based on the multimodal multi-task dual attention network MMA-Net, the model simultaneously outputs the probabilities of driver fatigue, visual distraction, and high cognitive load. Specifically, the driver state evaluation model in step 2 includes:

[0076] A multimodal encoder is used to extract features from video modal data and physiological temporal modal data respectively, obtaining representational features under different modalities. This includes video encoders and temporal encoders. The video encoder employs a 3D convolutional neural network (I3D skeleton), taking a sequence of video frames as input and outputting visual feature vectors. The temporal encoder employs a 1D convolutional neural network, containing 5 convolutional layers (with the number of filters increasing layer by layer: 64, 128, 256, 256, 512). Each layer is followed by batch normalization and ReLU activation, and finally global average pooling is applied to output a physiological feature vector. .

[0077] The Bidirectional Cross-Modal Attention Module (BCMA) is used to uncover the correlations between features from different modalities, enabling intermodal information exchange and fusion. For each task... Calculate video features respectively With time series characteristics The cross-modal fusion is expressed as follows:

[0078]

[0079] In the formula: Indicates the first In each task, the video modalities are used as a query matrix, consisting of... Calculations show that The weight matrix is ​​a learnable matrix. The feature vector output by the video encoder; Indicates the first The matrix of temporal modes as keys in each task is composed of... Calculations show that The weight matrix is ​​a learnable matrix. This is the feature vector output by the timing encoder. Indicates the first The matrix of temporal modes as values ​​in each task is composed of... Calculations show that This is a learnable weight matrix. Wherein, Represents the modal source, For video modality, This is a sequential mode. Similarly, Indicates the first The temporal modalities are used as the query matrix in each task; Indicates the first A matrix in which video modalities serve as keys in each task; Indicates the first A matrix of video modalities as values ​​in each task; The key vector dimension for each attention head.

[0080] In the bidirectional cross-modal attention module, a multi-head attention mechanism is employed to achieve bidirectional fusion of video modality and temporal modality. In this embodiment, the number of attention heads is set to [number missing]. The key vector dimension of each head is The visual feature vector output by the video encoder The timing feature vector output by the timing encoder is For each task, the multi-head attention output in the video-to-temporal direction and the multi-head attention output in the temporal-to-video direction are calculated separately, and the two are added together to obtain the cross-modal fusion feature for that task. The calculation formula is shown below:

[0081]

[0082] In the formula: This indicates a multi-head attention output that uses video modal features as queries and temporal modal features as keys and values. This represents the multi-head attention output using temporal modality features as the query and video modality features as the key and value. Each multi-head attention output is calculated according to the standard multi-head attention formula, as shown below:

[0083]

[0084] Each attention head in the formula , , , These are the query, key, and value matrices obtained through linear projection from the corresponding headers. This is the learnable output projection matrix.

[0085] The Multi-Task Exchange Block (MTEB) is used to exchange, share, and enhance features among multiple tasks, including fatigue recognition, visual distraction recognition, and high cognitive load recognition, to improve the performance of multi-task joint recognition. Its expression is shown below:

[0086]

[0087]

[0088] In the formula: Indicates splicing. The current task The query matrix is ​​composed of Calculations show that This is a learnable weight matrix. It is a task output by BCMA. Its characteristics. Other tasks The key matrix, by Calculations show that This is a learnable weight matrix. Other tasks The value matrix, by Calculations show that This is a learnable weight matrix. It is a task The learnable linear transformation matrix is ​​designed to map the concatenated high-dimensional features back to the original 512 feature dimensions.

[0089] Task-specific decoder, by each task A two-layer fully connected network is then used, and the enhanced fused features are mapped to fatigue state probabilities based on the Sigmoid function. Probability of visual distraction and the probability of a high cognitive load state .

[0090] Furthermore, the driver state evaluation model is based on multi-task joint optimization, and the total loss is set as the average of the binary cross-entropy losses of the three tasks, as shown in the following expression:

[0091]

[0092] In the formula: This represents the total loss value used for backpropagation. This is the binary cross-entropy loss function, suitable for the probability output of binary classification problems. , , These represent the probabilities of fatigue state, distraction state, and high load state predicted by the model, respectively, with values ​​ranging from 0 to 1. , , This is the actual label for the corresponding task, and its value is 0 or 1 (0 indicates that the state does not exist, and 1 indicates that the state exists).

[0093] Preferably, the Adam optimizer is used during model training. By presetting the initial learning rate, batch size, number of training rounds, and early stopping strategy, the model is prevented from overfitting. Training is terminated early when the validation set loss does not decrease for several consecutive rounds, thus ensuring the model's generalization ability.

[0094] Through the above model structure, deep fusion of multimodal features and joint optimization of multiple tasks are achieved, thereby improving the accuracy, robustness and real-time performance of driver state recognition.

[0095] S3. Driver status classification determination: Based on preset probability thresholds and status priority rules determined by objective indicators, the probability of fatigue state is assessed. Probability of visual distraction and the probability of a high cognitive load state Perform a fusion judgment and output the driver's overall status level.

[0096] Based on the state probability results output by the driver state evaluation model and combined with threshold rules formed by objective index calibration, the driver's comprehensive state level is determined. The threshold setting establishes a mapping relationship between the model output probability and physical indicators based on objective criteria. Fatigue state is calibrated according to the relevant provisions in the "Rules for Determining Fatigue Driving by Motor Vehicle Drivers" (GA / T2372-2026) regarding EEG characteristic judgment values ​​and physiological fatigue eye closure (both eyelids completely closed for more than 2 seconds). Visual distraction state is calibrated based on objective indicators such as the duration and frequency of gaze deviation from the road area. High cognitive load state is calibrated based on the objective change range of physiological indicators such as pupil diameter change rate and the ratio of Theta wave power spectrum in EEG. The probability threshold results set in this embodiment are shown in the table below:

[0097]

[0098] Furthermore, the driver's overall state level includes four categories: normal state, fatigued state, visually distracted state, and high cognitive load state. Preferably, the fatigued state can be further divided into a fatigue warning level and a high-risk fatigue level to support differentiated control of AR-HUD adjustment intensity under different risk levels. During the overall state determination process, considering the varying degrees of impact of different states on driving safety and the priority relationships when multiple abnormal states occur simultaneously, this embodiment sets the priority of visually distracted state higher than fatigued state, the priority of fatigued state higher than high cognitive load state, and high cognitive load state priority higher than normal state. Based on the above priority rules, a unique overall state level can be output under multiple concurrent states, serving as the control basis for subsequent AR-HUD interface adjustments. Please refer to [reference needed]. Figure 3 The specific determination rules for state priority are as follows:

[0099] First, determine whether the visual distraction state condition is met. When the probability of the visual distraction state is... When this threshold is reached, the driver is determined to be visually distracted. This threshold corresponds to the visual distraction judgment standard of a single gaze deviating from the road area for more than 2 seconds or a cumulative deviation exceeding 4 seconds within 10 consecutive seconds.

[0100] Secondly, if the visual distraction condition is not met (i.e.) If the fatigue state probability is met, then it is further determined whether the fatigue state condition is satisfied. At that time, it is determined that the driver is currently fatigued. Specifically, when... When this occurs, it is classified as a fatigue warning sub-level. At that time, it is determined to be a high-risk fatigue sub-level. The above thresholds correspond to the EEG characteristics used to judge the degree of fatigue and related objective indicators such as physiological fatigue and eye closure.

[0101] Secondly, if neither the conditions for visual distraction nor fatigue are met (i.e.) and If the probability of a high cognitive load state is met, then determine whether the conditions for a high cognitive load state are satisfied. At this point, the driver is determined to be in a state of high cognitive load. This threshold corresponds to a physiological indicator change of more than 25% in frontal lobe Theta wave power from baseline or more than 7% in pupil diameter dilation.

[0102] Finally, if none of the above three driver status conditions are met (i.e.) , and If the driver is in a normal state, then the driver is considered to be in a normal state.

[0103] Through the above-mentioned judgment rules, the present invention can prioritize the identification of the state with the highest safety risk when multiple states are concurrent, and perform fine classification of fatigue state, so as to provide accurate and reliable state input basis for subsequent adaptive adjustment of AR-HUD.

[0104] S4. Adaptive AR-HUD Adjustment: Based on the driver's overall state level, this component generates and executes adaptive adjustment instructions for the AR-HUD interface display parameters. These display parameters include at least one of information content, display position, visual attributes, and presentation mode. Information content adjustment includes retaining key information and weakening or hiding secondary information. Display position adjustment includes optimizing the projection area of ​​information in the driver's forward field of vision. Visual attribute adjustment includes adaptive adjustments to parameters such as brightness, contrast, color, size, and flicker frequency. Presentation mode adjustment includes switching or combining AR spatial fusion, hierarchical presentation, and interface aggregation methods.

[0105] Furthermore, different AR-HUD adjustment strategies correspond to different driver state levels. Specifically, for different sub-levels of fatigue, differentiated adjustment schemes can be adopted to achieve progressive control from mild intervention to enhanced warnings, thereby reducing visual interference and cognitive burden, and improving interface adaptability and driving safety. Specifically, the following adjustment strategies are adopted for different driver state levels:

[0106] Normal state: Maintains standard display mode, displays all information completely, and automatically adjusts brightness according to ambient light.

[0107] Fatigue Status: Based on the fatigue warning level or high-risk fatigue level, a progressive adjustment scheme is implemented, including turning off unnecessary information, moving key information to the center of the field of vision, reducing the overall brightness of the interface, increasing the size of warning icons, and adding voice prompts at high-risk sub-levels. Specifically, this includes: turning off entertainment information, detailed navigation traffic conditions, and non-critical vehicle status, retaining only vehicle speed, speed limits, and key warnings. Key information is moved to the center of the field of vision (lower center edge of the windshield). Warning icons are enlarged by 20%–30%. Brightness is reduced by 30%–50%, using a dark background with bright warnings (red / yellow). Voice prompts are added at high-risk sub-levels.

[0108] Visual Distraction Status: Non-emergency information is paused or weakened, while the visual salience of warning information is enhanced. AR spatial fusion is used to align warning information with the actual hazard source. Specifically, this includes pausing navigation prompts, entertainment information, call reminders, and other non-emergency information, retaining only warnings and necessary vehicle speed displays (reduced to 60%). Warning icons are enlarged 1.5–2 times. A rapid flashing frequency of 5–8Hz is used. AR spatial fusion display uses AR arrows / highlighted boxes to align warnings with the actual hazard source.

[0109] High cognitive load state: Simplify and integrate information content, adjust information layout to reduce the visual search range, specifically including: simplifying navigation 3D models to arrow guidance, and simplifying lengthy text prompts to icon sequences. Display long commands step by step. Integrate information such as vehicle speed, speed limit, navigation, and driving assistance into the same area, with spacing controlled within a 2° to 5° field of view. Use soft, low-saturation colors.

[0110] S5. Closed-loop feedback optimization: Dynamically update the system by collecting driver status change data after the AR-HUD interface adjustment is executed as a feedback signal.

[0111] The system continuously collects multi-source driver state data and dynamically updates the driver state evaluation model and interface adjustment strategy by using changes in driver state and improvements in driving performance after AR-HUD interface adjustments as feedback signals. Preferably, reinforcement learning, online learning, or incremental learning methods can be used to continuously iterate and optimize model parameters and adjustment strategies. This allows the system to achieve personalized adaptive adjustments based on individual differences among drivers, historical interaction behaviors, and long-term usage characteristics, forming a closed-loop operation mechanism of "perception-judgment-adjustment-feedback," further enhancing the intelligence level, interactive comfort, and safety assurance capabilities of the AR-HUD system.

[0112] An evaluation and adjustment system, the system being applied to the aforementioned AR-HUD evaluation and adjustment method that integrates multi-source physiological data and driving performance, the system comprising:

[0113] The data acquisition unit is used to collect multi-source physiological data and driving performance data of the driver in real time. It consists of an eye-tracking camera, steering wheel-integrated ECG electrodes, head-mounted dry electrode EEG acquisition device, vehicle CAN bus interface, and forward radar / camera.

[0114] The data preprocessing unit is used to perform noise reduction, alignment, normalization, and window segmentation on the collected data.

[0115] The driver state evaluation unit is equipped with an MMA-Net-based driver state evaluation model, which outputs the probabilities of various driver states, including the fatigue probability. Probability of visual distraction High cognitive load probability .

[0116] The driver state fusion determination unit is used to determine the final comprehensive state level based on probability thresholds and priority rules.

[0117] The adaptive adjustment unit is used to generate AR-HUD control commands based on the driver's state level and adjust the display parameters.

[0118] An AR-HUD display unit is used to receive and execute the control commands, presenting optimized augmented reality information to the driver.

[0119] Through the collaborative work of the data acquisition unit, preprocessing unit, status evaluation unit, fusion judgment unit, adaptive adjustment unit and AR-HUD display unit, the entire process from driver status perception to adaptive adjustment of the AR-HUD interface is automated. It can dynamically adjust the information presentation method according to the driver's real-time status, effectively reducing the driving safety risks in abnormal driving conditions.

[0120] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0121] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. An AR-HUD evaluation and modulation method integrating multi-source physiological data and driving performance, characterized by: Includes the following steps: S1. Data Acquisition and Preprocessing: Collect multi-source physiological data and driving performance data of drivers and perform preprocessing to construct multimodal time-series input samples; S2. Driver state evaluation model construction: By inputting the preprocessed multimodal temporal input samples into the constructed driver state evaluation model based on the multimodal multi-task dual attention network MMA-Net, the probability of driver fatigue state, visual distraction state and high cognitive load state are output simultaneously. S3. Driver status classification determination: Based on preset probability thresholds and status priority rules determined by objective indicators, the probability of fatigue state is assessed. Probability of visual distraction and the probability of a high cognitive load state Perform a fusion judgment and output the driver's overall status level; S4. Adaptive AR-HUD adjustment: Based on the driver's comprehensive state level, generate and execute adaptive adjustment instructions for AR-HUD interface display parameters, wherein the display parameters include at least one of information content, display position, visual attributes, and presentation mode; S5. Closed-loop feedback optimization: Dynamically update the system by collecting driver status change data after the AR-HUD interface adjustment is executed as a feedback signal.

2. The AR-HUD evaluation and adjustment method integrating multi-source physiological data and driving performance as described in claim 1, characterized in that, In step S1: The multi-source physiological data includes at least two of the following: eye-tracking data, electrocardiogram data, and electroencephalogram data. The driving performance data includes at least one of lateral control data, longitudinal control data, and reaction time data.

3. The AR-HUD evaluation and adjustment method integrating multi-source physiological data and driving performance as described in claim 1, characterized in that, The data preprocessing in step S1 includes the following steps: Denoising was performed, with moving average filtering applied to eye-tracking data and bandpass filtering and independent component analysis applied to remove artifacts from ECG and EEG data. Time synchronization alignment unifies all data streams to a preset sampling rate through linear interpolation; Feature normalization is performed by applying z-score normalization to each sample, as shown in the following expression: In the formula: This represents the raw measurement value of a certain physiological indicator or driving performance indicator. This indicates the average value of the indicator when the driver is at rest. The standard deviation of the same indicator at rest. The sample segmentation process based on sliding time windows generates samples with a preset window length and sliding step size, where each sample contains a video frame sequence and a physiological time series matrix.

4. The AR-HUD evaluation and adjustment method integrating multi-source physiological data and driving performance as described in claim 1, characterized in that, The driver state evaluation model in step 2 includes: Multimodal encoders, including video encoders and temporal encoders, obtain corresponding visual feature vectors based on 3D convolutional neural networks and 1D convolutional neural networks, respectively. and physiological feature vectors ; Bidirectional cross-modal attention module BCMA is used for each task Calculate video features respectively With time series characteristics The cross-modal fusion is expressed as follows: In the formula: Indicates the first In each task, video modalities are used as a query matrix; Indicates the first A matrix in which temporal modes are used as keys in each task; Indicates the first A matrix of temporal modes as values ​​in each task; Indicates the first The temporal modalities are used as the query matrix in each task; Indicates the first A matrix in which video modalities serve as keys in each task; Indicates the first A matrix of video modalities as values ​​in each task; The key vector dimension for each attention head; The Multi-Task Exchange Block (MTEB) is used for feature exchange, sharing, and enhancement among multiple tasks, including fatigue recognition, visual distraction recognition, and high cognitive load recognition. Its expression is shown below: In the formula: Indicates splicing; The current task The query matrix, It is a task output by BCMA. Features; Other tasks The key matrix; Other tasks The value matrix; It is a task Learnable linear transformation matrix; Task-specific decoder, by each task A two-layer fully connected network is then used, and the enhanced fused features are mapped to fatigue state probabilities based on the Sigmoid function. Probability of visual distraction and the probability of a high cognitive load state .

5. The AR-HUD evaluation and adjustment method integrating multi-source physiological data and driving performance as described in claim 4, characterized in that: The driver state evaluation model is based on multi-task joint optimization, and the total loss is set as the average of the binary cross-entropy losses of the three tasks, as shown in the following expression: In the formula: This represents the total loss value used for backpropagation; The binary cross-entropy loss function; , , These represent the probabilities of fatigue state, distraction state, and high load state predicted by the model, with values ​​ranging from 0 to 1. , , This is the actual label for the corresponding task, and its value can be 0 or 1.

6. The AR-HUD evaluation and adjustment method integrating multi-source physiological data and driving performance as described in claim 1, characterized in that: In step S3, the driver's overall condition level includes normal condition, fatigue condition, visual distraction condition, and high cognitive load condition; wherein the fatigue condition further includes fatigue warning level and fatigue high risk level.

7. The AR-HUD evaluation and adjustment method integrating multi-source physiological data and driving performance as described in claim 6, characterized in that, In step S3, the priority rules for states are as follows: visual distraction state has a higher priority than fatigue state, fatigue state has a higher priority than high cognitive load state, and high cognitive load state has a higher priority than normal state.

8. The AR-HUD evaluation and adjustment method integrating multi-source physiological data and driving performance as described in claim 1, characterized in that: The specific determination process for the state priority rule is as follows: If the probability of visual distraction When this occurs, it is determined to be a state of visual distraction; If the conditions for visual distraction are not met and the probability of fatigue is high When this occurs, it is determined to be a state of fatigue, where when At the time, it was at the fatigue warning level. The current situation is at a high level of fatigue risk. If the conditions for visual distraction and fatigue are not met, and the probability of a high cognitive load state is high. When this occurs, it is determined to be a state of high cognitive load; If none of the above conditions are met, the condition is considered normal.

9. The AR-HUD evaluation and adjustment method integrating multi-source physiological data and driving performance as described in claim 1, characterized in that, The adaptive AR-HUD adjustment strategy in step S4 includes: When the condition is determined to be normal, maintain the standard display mode; When a state of high cognitive load is identified, the information content is simplified and integrated, and the information layout is adjusted to reduce the visual search range. When visual distraction is detected, non-emergency information is paused or weakened, the visual prominence of warning information is enhanced, and AR spatial fusion is used to align the warning information with the real hazard source. When fatigue is detected, a progressive adjustment scheme is implemented based on the fatigue warning level or high-risk fatigue level. This includes turning off unnecessary information, moving key information to the center of the field of vision, reducing the overall brightness of the interface, increasing the size of warning icons, and adding voice reminders at the high-risk sub-level.

10. An evaluation and adjustment system, said system being applied to the AR-HUD evaluation and adjustment method as described in any one of claims 1 to 9, characterized in that, The system includes: The data acquisition unit is used to collect multi-source physiological data and driving performance data of the driver in real time; The data preprocessing unit is used to perform noise reduction, alignment, normalization, and window segmentation on the collected data; The driver state evaluation unit is equipped with a driver state evaluation model based on MMA-Net, which is used to output the probability of each driver state. The driver state fusion determination unit is used to determine the final comprehensive state level based on probability thresholds and priority rules; The adaptive adjustment unit is used to generate AR-HUD control commands based on the overall status level and adjust the display parameters. An AR-HUD display unit is used to receive and execute the control commands, presenting optimized augmented reality information to the driver.

Images