Multi-modal data quality monitoring and repair method for human-machine co-driving experience evaluation

Abstract

The application discloses a multi-modal data quality monitoring and repairing method for human-machine co-driving experience evaluation, relates to the technical field of human-machine co-driving systems, collects multi-modal data in the human-machine co-driving process, performs real-time segmentation processing on electroencephalogram data in the multi-modal data, calculates quality evaluation indexes corresponding to contact impedance, signal amplitude range, signal-to-noise ratio and electromyographic artifact power for each data segment; the quality evaluation indexes are fused and calculated to generate dynamic quality labels corresponding to the electroencephalogram data segments, the dynamic quality labels include normal labels, suspicious labels and abnormal labels; for the electroencephalogram data segment corresponding to the suspicious label, adaptive filtering parameters corresponding to the artifact type represented by the dynamic quality label are selected. Through real-time quality evaluation, dynamic label management and generative data repairing technology, the application realizes online monitoring, abnormal processing and continuous recovery of multi-modal data, and improves the reliability and intelligent level of human-machine co-driving experience evaluation.

Description

Technical Field

[0001] This application relates to the field of human-machine co-driving system technology, specifically to a method for multimodal data quality monitoring and repair for human-machine co-driving experience evaluation. Background Technology

[0002] In human-machine co-driving systems, in order to accurately evaluate the interaction experience, psychological state, and cognitive load between the driver and the intelligent driving system, it is usually necessary to collect and analyze multimodal physiological perception data. Among them, electroencephalogram (EEG) signals are widely used in scenarios such as driving experience evaluation, psychological load analysis, and human-machine interaction effect assessment because they can reflect the driver's real-time brain activity state.

[0003] Existing human-machine co-driving experience evaluation systems typically acquire multi-source data during the driving process using EEG, eye-tracking, ECG, and other physiological sensors, and then combine this data with signal processing algorithms and pattern recognition models to decode the driver's state. However, in real-world driving environments, the acquired multimodal data, especially EEG signals, is easily affected by factors such as driver head movements, facial muscle activity, sweating, changes in electrode contact status, and electromagnetic interference in the vehicle environment. This can lead to problems such as signal drift, noise pollution, artifact interference, and data loss.

[0004] Existing technologies typically employ offline signal processing methods to optimize the quality of acquired data. For example, algorithms such as Independent Component Analysis (ICA) are used to separate and remove artifact components such as electrooculography (EOG) and electromyography (EMG), or filtering and noise reduction are used to improve signal quality in order to ensure the accuracy of subsequent psychological state recognition and experience evaluation processes.

[0005] However, existing technologies still have certain shortcomings in practical applications. First, existing signal processing methods based on offline analysis typically require processing after data acquisition, making it impossible to monitor and adjust data quality changes in real time during driving. This results in a significant lag in experience evaluation results, making it difficult to meet the real-time closed-loop optimization requirements of intelligent driving scenarios. Second, existing methods mainly passively process existing noise or artifacts, usually only reducing some interference. When sensor contact abnormalities, equipment detachment, or prolonged signal loss occur, abnormal data segments are often discarded directly, leading to discontinuous valid data and affecting the integrity of driver psychological state decoding. Furthermore, traditional signal processing algorithms typically rely on fixed parameters and human experience for adjustment. They struggle to achieve adaptive optimization when facing changes in the driving environment, individual differences, and complex dynamic noise interference, resulting in limited data quality control capabilities.

[0006] Therefore, existing technologies cannot achieve real-time, online, and intelligent quality monitoring and missing data repair for multimodal data that affects the quality of human-machine co-driving experience evaluation. This can easily lead to biased evaluation results and reduce the overall reliability and application value of the human-machine co-driving experience evaluation system.

[0007] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0008] The purpose of this application is to provide a multimodal data quality monitoring and repair method for human-machine co-driving experience evaluation, so as to solve the problems in the background art mentioned above.

[0009] To achieve the above objectives, this application provides the following technical solution: a method for multimodal data quality monitoring and repair for human-machine co-driving experience evaluation, comprising the following steps: Multimodal data was collected during the human-machine co-driving process. The EEG data in the multimodal data was segmented in real time, and the quality evaluation indicators corresponding to contact impedance, signal amplitude range, signal-to-noise ratio and electromyography artifact power were calculated for each data segment. The quality evaluation indicators are fused and calculated to generate dynamic quality labels for the corresponding EEG data segments. The dynamic quality labels include normal labels, suspicious labels, and abnormal labels. For the EEG data segments corresponding to suspicious tags, the corresponding adaptive filtering parameters are selected according to the artifact type represented by the dynamic quality tag, and the EEG data segments are filtered in real time. For the EEG data segment corresponding to the abnormal label, the historical EEG data of the target channel and the current EEG data of the adjacent normal channel are obtained. The historical EEG data and the current EEG data of the adjacent normal channel are input into the prediction model to perform generative prediction and repair of the missing EEG data. The repaired EEG data segments are marked with repair status, and the data credibility in the subsequent human-machine co-driving experience evaluation process is adjusted according to the repair status marking to maintain the continuity of the multimodal data evaluation process.

[0010] The EEG data in the multimodal data is continuously divided according to a preset time window, and each time window corresponds to an independent data quality evaluation unit. For each EEG data segment, electrode contact status information, signal amplitude change information, spectral energy distribution information, and artifact interference information are collected. The reliability of the current EEG data segment is described by multiple dimensions of quality evaluation indicators to avoid the problem that a single evaluation parameter cannot accurately reflect the EEG acquisition status in complex driving environments.

[0011] Dynamic quality labels are generated based on the fusion calculation results of quality evaluation indicators. Dynamic quality labels are used to distinguish the usability status of EEG data segments. Among them, the normal label corresponds to the data segment that can directly participate in the evaluation of human-machine co-driving experience, the suspicious label corresponds to the data segment that contains mild artifacts but still retains effective EEG characteristics, and the abnormal label corresponds to the data segment that has signal loss or abnormal acquisition. The dynamic quality labels are used as the basis for judging the selection of subsequent data processing strategies.

[0012] The quality evaluation indicators include contact impedance evaluation indicators, signal amplitude range evaluation indicators, signal-to-noise ratio evaluation indicators, and electromyography artifact power evaluation indicators. Among them, the contact impedance evaluation indicator is used to determine the connection status between the EEG acquisition electrodes and the human body; the signal amplitude range evaluation indicator is used to identify abnormal signal attenuation or sensor detachment; the signal-to-noise ratio evaluation indicator is used to analyze the proportion of effective EEG components; and the electromyography artifact power evaluation indicator is used to determine the interference caused by the driver's actions.

[0013] The adaptive filtering parameters are adjusted based on the artifact feature information in the dynamic quality label. When the suspicious label corresponding to the EEG data segment contains EMG artifact features, the bandpass filter range is adjusted to reduce the impact of EMG interference. When the suspicious label contains power frequency interference features, the corresponding notch filter processing method is selected according to the interference frequency to improve the effectiveness of EEG data in dynamic driving environment.

[0014] The prediction model employs either an autoregressive model or a lightweight Seq2Seq neural network model. The prediction model receives historical normal data from the target EEG channel and current EEG data from adjacent normal channels. By analyzing the temporal variation patterns of EEG signals and the spatial correlations between different brain regions, it predicts the EEG waveforms of data-missing regions corresponding to abnormal labels.

[0015] The historical EEG data of the target channel includes normal EEG waveforms continuously acquired before the abnormality occurred. The current EEG data of adjacent normal channels includes data from multiple EEG acquisition channels that are spatially related to the target channel. By jointly inputting the prediction model, the generated missing EEG data maintains the brain activity change pattern corresponding to the current driving state of the target driver.

[0016] During the generative predictive repair process, missing data is generated based on the frequency characteristics of the EEG signal and the correlation characteristics between brain regions in the current state of the driver. The generated EEG waveform maintains the alpha band oscillation mode and the theta band oscillation mode in the target driver's driving state, and avoids the problem of insufficient restoration of the temporal continuity of EEG data caused by linear interpolation.

[0017] The repaired state markers are used to distinguish between the original collected EEG data and the predicted generated EEG data. The reliability of the data in the subsequent psychological state decoding process is adjusted according to the repaired state markers. The EEG data after generative prediction repair carries the corresponding uncertainty information to reduce the impact of prediction data errors on the human-machine co-driving experience evaluation results.

[0018] A correspondence is established between dynamic quality labels and subsequent data processing strategies. When an EEG data segment is identified as a normal label, the original collected data is retained for experience evaluation. When an EEG data segment is identified as a suspicious label, an adaptive filtering processing mechanism is invoked. When an EEG data segment is identified as an abnormal label, a generative prediction and repair mechanism is invoked, thereby realizing an automated quality control process under different data states.

[0019] The technical effects and advantages provided by this application in the above technical solution are as follows: This application performs real-time quality assessment of multimodal data during the human-machine co-driving experience evaluation process and dynamically adjusts processing strategies based on data quality status. It manages data acquisition, anomaly identification, and data optimization processes online, enabling timely detection of abnormal data affecting evaluation accuracy during driving and reducing analytical bias caused by noise, artifacts, and signal anomalies. Dynamic quality labels differentiate between different data states, allowing subsequent processing to select appropriate filtering or repair methods based on data reliability. This provides stable data quality assurance for the human-machine co-driving experience evaluation process, improves the reliability of the evaluation optimization closed-loop process, and provides fundamental support for the system's application in real-world driving environments.

[0020] This application utilizes a generative predictive repair approach to recover abnormally missing EEG data. It jointly analyzes historical data from the target channel with current data from adjacent channels to generate continuous EEG data that conforms to the driver's current brain activity patterns, avoiding data gaps caused by sensor anomalies or signal interruptions. Simultaneously, by labeling the repaired data with its state and adjusting its reliability, the subsequent psychological state decoding process can effectively utilize the repaired data, ensuring evaluation continuity while reducing the impact of data uncertainty. This method enables automated operation of data quality monitoring, anomaly handling, and missing data repair, making it suitable for long-term, continuous human-machine co-driving experience evaluation tasks in in-vehicle environments. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0022] Figure 1 This is a flowchart illustrating the overall method of this application. Detailed Implementation

[0023] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the description of this disclosure will be more complete and fully convey the concept of the exemplary embodiments to those skilled in the art.

[0024] like Figure 1 As shown, this application provides a multimodal data quality monitoring and repair method for human-machine co-driving experience evaluation, including the following steps: Multimodal data was collected during the human-machine co-driving process. The EEG data in the multimodal data was segmented in real time, and the quality evaluation indicators corresponding to contact impedance, signal amplitude range, signal-to-noise ratio and electromyography artifact power were calculated for each data segment. The quality evaluation indicators are fused and calculated to generate dynamic quality labels for the corresponding EEG data segments. The dynamic quality labels include normal labels, suspicious labels, and abnormal labels. For the EEG data segments corresponding to suspicious tags, the corresponding adaptive filtering parameters are selected according to the artifact type represented by the dynamic quality tag, and the EEG data segments are filtered in real time. For the EEG data segment corresponding to the abnormal label, the historical EEG data of the target channel and the current EEG data of the adjacent normal channel are obtained. The historical EEG data and the current EEG data of the adjacent normal channel are input into the prediction model to perform generative prediction and repair of the missing EEG data. The repaired EEG data segments are marked with repair status, and the data credibility in the subsequent human-machine co-driving experience evaluation process is adjusted according to the repair status marking to maintain the continuity of the multimodal data evaluation process.

[0025] Specific implementation method one: Real-time quality monitoring and dynamic quality label generation during multimodal data acquisition.

[0026] In this embodiment, a multimodal data quality monitoring and repair method for evaluating human-machine co-driving experience is applied to the driver state acquisition process in an intelligent driving environment. This method continuously collects, analyzes in real-time, and assesses the data quality of driver physiological state-related data during vehicle operation, human-machine interaction, and driving task execution. Specifically, during vehicle operation, driver state data is acquired through multimodal sensing devices deployed on the driver's side. These devices may include electroencephalogram (EEG) acquisition devices, electrocardiogram (ECG) acquisition devices, eye-tracking acquisition devices, electrodermal conductance (EDA) acquisition devices, electromyography (EMG) acquisition devices, and other sensing devices capable of reflecting the driver's physiological or behavioral state. Among these, EEG data, as an important physiological signal reflecting the driver's cognitive state, psychological load, emotional changes, and human-machine interaction experience, is continuously collected by the EEG acquisition devices and serves as the primary object of subsequent quality monitoring and repair processing.

[0027] During EEG data acquisition, the driving environment differs from the laboratory environment. Drivers may exhibit changes such as head rotation, body posture, facial muscle movements, and driving maneuvers. Furthermore, vibrations during vehicle operation, changes in the electromagnetic environment, and variations in electrode wearing status can all affect the quality of EEG signal acquisition. Therefore, this implementation first performs real-time segmentation processing on the continuously acquired EEG signals, dividing the continuous EEG data into multiple consecutive data segments according to a preset time window. Each data segment corresponds to an independent quality evaluation unit. For example, the EEG signal can be divided into time windows of 0.5 seconds, allowing the system to continuously assess the validity of the current EEG data at short intervals. EEG data within each time window enters the quality evaluation process independently, thereby avoiding the influence of local abnormal signals on the entire long-term data sequence and improving the real-time performance and accuracy of data quality monitoring.

[0028] For each segmented EEG data segment, the system calculates multiple quality evaluation indicators to comprehensively reflect the effectiveness of the current EEG signal. Specifically, it obtains electrode connection status evaluation parameters for the current EEG channel by detecting the contact impedance between the EEG acquisition electrodes and the human skin. Contact impedance directly reflects the contact stability between the EEG electrodes and the human body. When the electrodes maintain good contact with the skin, the contact impedance is in a low range, and the acquired EEG signal has good stability. When the electrodes become loose, fall off, or the contact state changes due to human movement, the contact impedance will increase significantly, which may lead to a decrease in EEG signal quality. Therefore, by monitoring changes in contact impedance in real time, data quality problems caused by abnormal electrode contact can be identified in advance.

[0029] Simultaneously, the signal amplitude range in the EEG data segments is analyzed. By detecting the trend of EEG signal amplitude changes, it is determined whether there is abnormal signal attenuation, channel failure, or sensor detachment. Under normal acquisition conditions, EEG signals typically fluctuate within a certain range. When a particular acquisition channel is affected by equipment malfunction, the signal amplitude may suddenly drop to near zero, indicating a risk of data loss for that channel. For example, if an EEG channel drops from a normal fluctuation state to near 0 microvolts within a short period, even if the contact impedance parameter is temporarily within the normal range, it may indicate an acquisition anomaly in that channel, requiring a further reduction in the quality evaluation level of that data segment.

[0030] Furthermore, by calculating the ratio between the power of the target frequency band and the power of the entire frequency band of the EEG signal, a signal-to-noise ratio (SNR) evaluation index for the current EEG data segment is obtained. This index reflects the proportion of effective EEG activity components in the currently acquired signal. When the EEG activity energy in the target frequency band is high and the noise energy in the non-target frequency band is low, it indicates that the current data has a good proportion of effective information. When external interference, motion artifacts, or equipment noise cause an increase in the energy of the non-target frequency band, the SNR decreases, indicating a decline in the reliability of the current EEG data. This index can further distinguish between normal EEG changes and abnormal fluctuations caused by external noise.

[0031] Furthermore, by detecting energy changes in the frequency bands corresponding to electromyographic artifacts, an evaluation index for the degree of electromyographic interference can be obtained. During actual driving, drivers may generate muscle activity due to head turning, facial expressions, chewing, body adjustments, or steering wheel operations. The electrical signals generated by these muscle activities can mix into the EEG signals, forming electromyographic artifacts. By analyzing the power changes in the relevant frequency bands of electromyographic artifacts, it is possible to determine whether the current EEG data is significantly affected by electromyographic interference, and to provide a basis for selecting subsequent abnormality handling strategies.

[0032] After evaluating contact impedance, signal amplitude range, signal-to-noise ratio (SNR), and EMG artifact power, the quality evaluation indicators from multiple dimensions are fused to obtain a comprehensive quality evaluation result for the corresponding EEG data segment. The fusion calculation process can assign corresponding weights based on the importance of different evaluation indicators, comprehensively judging the overall reliability of the current data segment and avoiding misjudgments caused by abnormalities in a single indicator. For example, if an EEG data segment has slight EMG interference but normal contact impedance and a high SNR, it can be judged that the data segment still has some validity; conversely, if the contact impedance is abnormally high, the signal amplitude is close to zero, and the SNR is significantly reduced, it can be determined that the data segment has a serious data quality problem.

[0033] Based on the comprehensive quality evaluation results, a corresponding dynamic quality label is generated for each EEG data segment. The dynamic quality label characterizes the reliability of the current EEG data segment and serves as an important basis for selecting subsequent data processing strategies. The dynamic quality label can include normal, questionable, and abnormal labels. A normal label indicates that the current EEG data segment meets the experience evaluation requirements and can be directly input into the subsequent mental state decoding model; a questionable label indicates that the current EEG data segment contains a certain degree of noise or artifacts and requires further adaptive filtering; an abnormal label indicates that the current EEG data segment may have serious signal loss, sensor malfunctions, or be unusable directly, requiring subsequent generative repair processes.

[0034] For example, during EEG acquisition, the contact impedance of a certain EEG channel is detected at 15kΩ. Since this value is below the preset normal threshold of 20kΩ, it can be determined that the EEG electrode is currently maintaining good contact with the body, and the initial acquisition conditions are normal. However, in a subsequent time period, the amplitude of the EEG signal acquired by this channel rapidly drops to near 0 microvolts within 0.2 seconds, indicating that the current channel may have experienced electrode detachment, abnormal signal transmission, or a malfunction in the acquisition equipment. Based on the abnormal amplitude change and combined with other quality evaluation indicators, the system makes a comprehensive judgment and adjusts the dynamic quality label of the corresponding 0.5-second EEG data segment to an abnormal label.

[0035] Through the aforementioned real-time quality monitoring process, the multimodal data quality status of the driver can be continuously acquired during the human-machine co-driving experience evaluation. This allows for the timely detection of transient artifacts, signal anomalies, and potential missing data in EEG data. Furthermore, dynamic quality labels record the reliability of different data segments, providing accurate data quality data for subsequent adaptive filtering, generative data repair, and psychological state decoding processes. This implementation avoids the problem of traditional methods that only perform offline processing after data acquisition, achieving real-time perception of multimodal data quality status in the driving environment and improving the data continuity, reliability, and intelligence level during the human-machine co-driving experience evaluation process.

[0036] Specific implementation method 2: Adaptive filtering of mild artifacts based on dynamic quality labels.

[0037] In this embodiment, a multimodal data quality monitoring and repair method for human-machine co-driving experience evaluation is applied to the real-time data optimization process under mild EEG signal abnormalities. In Specific Implementation One, the system performs multi-dimensional quality evaluation on the EEG data collected during human-machine co-driving and generates corresponding data quality labels based on indicators such as contact impedance, signal amplitude range, signal-to-noise ratio, and EMG artifact power. This embodiment further utilizes dynamic quality labels as a basis for selecting data processing strategies, performing real-time adaptive filtering on data segments with mild quality abnormalities but still retaining valid EEG information, thereby reducing the impact of changes in the driving environment, driver actions, and external interference factors on the quality of EEG signal acquisition.

[0038] In actual driving, the types of interference to EEG signals exhibit significant dynamic changes. For example, when a driver changes lanes, observes the road environment, operates vehicle control equipment, or performs human-machine interaction, head movements, facial muscle activities, and changes in body posture may occur. These actions can cause electromyography (EMG) signals to mix with the EEG acquisition signals, resulting in varying degrees of EMG artifacts in the EEG data. Simultaneously, the operation of electrical equipment, power supply changes, and the operation of surrounding electronic devices during vehicle operation may also introduce fixed-frequency power frequency interference. Because different types of interference correspond to different frequency distributions and degrees of impact, using fixed filtering parameters is insufficient to adapt to complex dynamic driving environments; therefore, it is necessary to dynamically adjust the filtering processing strategy based on real-time monitoring results.

[0039] In practice, when a segment of EEG data is marked as suspicious after quality evaluation, the system further reads the corresponding quality evaluation index information to analyze and determine the cause of the anomaly. The dynamic quality label not only indicates the current EEG data quality level but also contains characteristic information related to the anomaly type, such as changes in EMG artifact power, power frequency interference intensity, and signal spectrum characteristics. Based on the anomaly characteristics in the dynamic quality label, the system automatically matches the corresponding filtering method and adjusts the filtering parameters in real time according to the current data quality status, enabling the filtering process to adapt to changes in EEG signals under different driving environments.

[0040] For example, when the system detects a significant increase in the power of electromyography (EMG) artifacts in a certain EEG data segment, it indicates that the current EEG signal may be affected by high-frequency interference from the driver's head movements, facial muscle activity, or body movements. To address this type of anomaly, the system automatically adjusts the bandpass filter parameters, reducing the likelihood of high-frequency EMG signals entering the effective EEG frequency band by lowering the upper limit frequency of the bandpass filter. During the adjustment process, the system does not directly delete abnormal data, but rather reduces interference components while preserving the main characteristics of the EEG signal, ensuring that the processed EEG data still reflects the driver's true brain activity state. For instance, when the driver quickly turns their head to observe the vehicle's surroundings, the activity of the frontal and temporal muscles increases, and EMG signals may be superimposed on the EEG acquisition results. By dynamically lowering the upper limit frequency of the filter, this type of high-frequency interference can be effectively weakened, increasing the proportion of effective components in the EEG signal.

[0041] Furthermore, when the system identifies significant power frequency interference in the current EEG data segment through dynamic quality tags, it automatically enhances notch filtering based on the interference frequency characteristics. Since internal vehicle electronics, power systems, and the external electromagnetic environment may generate periodic interference around 50Hz or 60Hz, this type of interference can affect the EEG signal spectrum analysis results. To address this, the system automatically selects the corresponding 50Hz or 60Hz notch filtering strategy based on the detected interference frequency and adjusts the notch processing strength to reduce the impact of power frequency signals on the EEG data while preserving as many effective frequency components as possible in the EEG signal.

[0042] In this embodiment, the adaptive filtering process automatically completes anomaly identification, filtering mode selection, and filtering parameter adjustment based on real-time quality monitoring results, without requiring manual parameter configuration. Compared to traditional fixed-parameter filtering methods, this embodiment can change the processing strategy in real time according to dynamic environmental changes during driving. For example, when the driver maintains a stable driving state, the EEG signal is less affected by interference, and the system can use a wider range of filtering parameters to preserve the detailed features of the EEG signal to the greatest extent; when the driver makes significant movements or external interference increases, the system automatically adjusts the filtering parameters according to dynamic quality labels to improve the signal's anti-interference capability.

[0043] The EEG data, after adaptive filtering, continues into the human-machine co-driving experience evaluation process and is input into the subsequent psychological state analysis model for further analysis of the driver's psychological load, emotional state, cognitive state, and human-machine interaction experience. Because the filtering process can dynamically adjust according to changes in data quality, it effectively reduces data fluctuations caused by transient noise, minor artifacts, and environmental interference, avoiding deviations in psychological state decoding results due to local abnormal signals.

[0044] Furthermore, the adaptive filtering process in this embodiment can also form a continuous data quality control flow with the dynamic quality label generation process in Specific Implementation Method 1. When an EEG data segment is identified as a suspicious label, it first enters the adaptive filtering process; when the data quality recovers to a state that meets the evaluation requirements after filtering, it continues to be used for subsequent experience evaluation analysis; when the data quality requirements are still not met after filtering, it can further enter the subsequent generative data repair process according to the quality label status. Through the above methods, a continuous processing flow from data quality detection, anomaly type identification, adaptive filtering to subsequent repair judgment is realized, improving the data processing capability of the human-machine co-driving experience evaluation system in complex driving environments.

[0045] This implementation method enables automatic identification and dynamic suppression of mild EEG artifacts without affecting the real-time performance of human-machine co-driving experience evaluation. It reduces processing delays caused by traditional manual parameter tuning, improves the stability and reliability of EEG data in complex driving environments, and provides continuous, high-quality data input for subsequent driver psychological state decoding and human-machine co-driving experience evaluation.

[0046] Specific implementation method 3: Online repair and credibility labeling of missing EEG data based on generative prediction model.

[0047] In this embodiment, the multimodal data quality monitoring and repair method used for human-machine co-driving experience evaluation is applied to the online data recovery process in cases of severe abnormalities or signal loss in EEG data. For the EEG data segments corresponding to the abnormal labels generated by multidimensional quality evaluation indicators in Specific Implementation One, the system determines that the current data segment cannot be directly used for subsequent driver state analysis. Therefore, a generative data repair mechanism is activated to recover EEG data missing due to sensor detachment, abnormal electrode contact, signal transmission interruption, or other acquisition anomalies online, thus preventing the interruption of the human-machine co-driving experience evaluation process due to partial EEG data loss.

[0048] In real-world driving environments, EEG acquisition devices typically need to operate continuously for extended periods. Drivers inevitably experience head movements, changes in body posture, and driving maneuvers during vehicle operation. When electrodes become loose due to movement or the acquisition device malfunctions due to environmental factors, a particular EEG channel may fail to acquire valid data for a period. For example, an EEG channel might continuously output valid EEG waveforms during normal acquisition, but if the driver's movement causes the electrode to detach momentarily, the output signal amplitude of that channel may rapidly decrease to near-zero microvolts. In this case, traditional filtering methods cannot recover valid information, and simply deleting abnormal data would cause discontinuities in the EEG time series, affecting subsequent psychological state decoding results. Therefore, this implementation method does not directly discard or use simple linear interpolation for the data segments corresponding to abnormal labels. Instead, it uses a generative prediction model to intelligently recover missing data.

[0049] In practice, the system invokes a pre-trained prediction model specifically designed for the target driver. This model can employ an autoregressive model, a lightweight Seq2Seq neural network model, or other generative models capable of handling time-series prediction tasks. Before deployment, the prediction model can be trained on normal EEG data from the target driver's historical driving history. This allows the model to learn the driver's EEG patterns under different driving conditions, including trends in EEG signals at different time points, the relationships between different brain regions, and brain activity characteristics under specific driving tasks. By training the model on individual driver data, the impact of physiological differences between drivers can be reduced, improving the consistency between the recovered missing EEG data and the actual EEG state.

[0050] When a segment of EEG data is identified as abnormal by the dynamic quality label, the system first acquires historical normal data of the target EEG channel prior to the occurrence of the abnormality, and simultaneously acquires data collected from other adjacent normal EEG channels within the current time period. This data is then used as input to the prediction model. The historical normal data describes the temporal variation of the target channel, while data from adjacent normal EEG channels provides spatial correlation information between different brain regions. For example, when data is missing from the Cz channel, the system can acquire continuously acquired normal EEG waveforms from the Cz channel over the past two seconds, while simultaneously acquiring normal waveform data from adjacent EEG channels such as C3, C4, and Fz at the current moment. This time-series information and spatial correlation information are then input into the prediction model.

[0051] After receiving input data, the predictive model analyzes and predicts the EEG trends during the time period when the target channel is missing. Since EEG signals are not simple periodic signals, their changes are influenced by the driver's current psychological state, cognitive load, and driving task. Therefore, the predictive model does not simply connect the waveforms before and after the missing signal using mathematical fitting. Instead, it generates high-fidelity EEG waveforms that match the characteristics of the current state based on learned EEG change patterns during driving. The generation process comprehensively considers the target driver's current brain activity patterns, ensuring that the generated data maintains temporal continuity, frequency characteristics, and spatial correlation similar to real EEG signals.

[0052] For example, when there is a 0.5-second signal loss in the Cz channel, the system acquires normal EEG data from the past 2 seconds in the Cz channel and combines it with normal EEG waveforms collected from the C3, C4, and Fz channels during that time period. A lightweight Seq2Seq neural network model is then used for prediction. The prediction model generates the corresponding EEG waveform for the Cz channel during the 0.5-second loss based on historical trends and activity information from adjacent brain regions contained in the input data. The generated EEG waveform is not a simple sine wave or linear interpolation result, but rather inherits the specific alpha and theta frequency oscillation characteristics of the driver in the current driving state, while maintaining its correlation with the activity of surrounding brain regions. For example, when the driver is under high cognitive load, the data generated by the model can maintain the corresponding EEG rhythm change trend, ensuring that the repaired EEG data still has value for psychological state analysis.

[0053] After generating the missing EEG waveform, the system replaces the original abnormal data segment with the predicted data, restoring the continuity of the time series data for the target EEG channel. During the replacement process, the system can verify the continuity between the generated data and the normal data before and after the missing segment to determine whether the generated waveform meets the time continuity requirements. For example, it can analyze the amplitude and frequency changes before and after the missing data segment, as well as the correlation between adjacent channels, to ensure that the generated data does not produce significant abrupt changes in the time series. This method avoids the evaluation curve breakage problem caused by traditional data deletion methods, allowing the driver's psychological state changes to be continuously preserved.

[0054] Furthermore, after completing the EEG data restoration, the system does not treat the generated data indiscriminately with the actual collected data. Instead, it adds restoration status markers to the restored EEG data segments. These restoration status markers indicate the current data source and the data's reliability, serving as an important reference in the subsequent psychological state decoding process. For example, for directly acquired normal EEG data, a high reliability weight can be maintained; for data restored through generative prediction, corresponding uncertainty weights can be set based on model prediction error, input data quality, and the stability of the generated results. This allows subsequent psychological load identification models and human-computer interaction experience evaluation models to reasonably reduce their reliance on the restored data.

[0055] In the subsequent evaluation of the human-machine co-driving experience, the mental state decoding model can adopt different data credibility strategies for different data segments based on the restored state labels. When the restored data has high predictive consistency, it can participate normally in the driver state analysis; when the restored data has significant uncertainty, the influence of this part of the data in the final evaluation result can be reduced, thereby avoiding bias in the evaluation result due to errors in the generated data. By introducing the restored state labeling mechanism, not only is the continuous recovery of missing EEG data achieved, but the evaluation model's ability to perceive abnormal data is also improved.

[0056] This implementation method ensures that even in suboptimal driving environments where sensor malfunctions, electrode detachment, or prolonged loss of EEG signals occur, the human-machine collaborative driving experience evaluation process can continue to operate continuously, preventing data gaps from hindering the analysis of critical driving events. This method utilizes the target driver's historical EEG information and real-time EEG information from adjacent brain regions for joint prediction, achieving highly faithful generative repair with individual adaptation. Simultaneously, by combining repair status labeling and credibility adjustment mechanisms, the entire process of data quality monitoring, anomaly identification, missing data repair, and evaluation feedback is automated and operates online, meeting the real-time, continuous, and reliable application requirements of in-vehicle edge computing environments in intelligent driving scenarios.

[0057] The foregoing has only described certain exemplary embodiments of this application by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of this application. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of this application.

Claims

1. A method for multimodal data quality monitoring and repair for human-machine co-driving experience evaluation, characterized in that, Includes the following steps: Multimodal data was collected during the human-machine co-driving process. The EEG data in the multimodal data was segmented in real time, and the quality evaluation indicators corresponding to contact impedance, signal amplitude range, signal-to-noise ratio and electromyography artifact power were calculated for each data segment. The quality evaluation indicators are fused and calculated to generate dynamic quality labels for the corresponding EEG data segments. The dynamic quality labels include normal labels, suspicious labels, and abnormal labels. For the EEG data segments corresponding to suspicious tags, the corresponding adaptive filtering parameters are selected according to the artifact type represented by the dynamic quality tag, and the EEG data segments are filtered in real time. For the EEG data segment corresponding to the abnormal label, the historical EEG data of the target channel and the current EEG data of the adjacent normal channel are obtained. The historical EEG data and the current EEG data of the adjacent normal channel are input into the prediction model to perform generative prediction and repair of the missing EEG data. The repaired EEG data segments were marked with a repair status, and the data credibility in the subsequent human-machine co-driving experience evaluation process was adjusted based on the repair status marking.

2. The multimodal data quality monitoring and repair method for human-machine co-driving experience evaluation according to claim 1, characterized in that, The EEG data in the multimodal data is continuously divided according to a preset time window, and each time window corresponds to an independent data quality evaluation unit. For each EEG data segment, electrode contact status information, signal amplitude change information, spectral energy distribution information, and artifact interference information are collected. The reliability of the current EEG data segment is described by multiple dimensions of quality evaluation indicators.

3. The multimodal data quality monitoring and repair method for human-machine co-driving experience evaluation according to claim 2, characterized in that, Dynamic quality labels are generated based on the fusion calculation results of quality evaluation indicators. Dynamic quality labels are used to distinguish the usability status of EEG data segments. Among them, the normal label corresponds to the data segment that can directly participate in the evaluation of human-machine co-driving experience, the suspicious label corresponds to the data segment that contains mild artifacts but still retains effective EEG characteristics, and the abnormal label corresponds to the data segment that has signal loss or abnormal acquisition. The dynamic quality labels are used as the basis for judging the selection of subsequent data processing strategies.

4. The multimodal data quality monitoring and repair method for human-machine co-driving experience evaluation according to claim 3, characterized in that, The quality evaluation indicators include contact impedance evaluation indicators, signal amplitude range evaluation indicators, signal-to-noise ratio evaluation indicators, and electromyography artifact power evaluation indicators. Among them, the contact impedance evaluation indicator is used to determine the connection status between the EEG acquisition electrodes and the human body; the signal amplitude range evaluation indicator is used to identify abnormal signal attenuation or sensor detachment; the signal-to-noise ratio evaluation indicator is used to analyze the proportion of effective EEG components; and the electromyography artifact power evaluation indicator is used to determine the interference caused by the driver's actions.

5. The multimodal data quality monitoring and repair method for human-machine co-driving experience evaluation according to claim 3, characterized in that, The adaptive filtering parameters are adjusted based on the artifact feature information in the dynamic quality label. When the suspicious label corresponding to the EEG data segment contains EMG artifact features, the bandpass filter range is adjusted; when the suspicious label contains power frequency interference features, the corresponding notch filter processing method is selected according to the interference frequency.

6. The multimodal data quality monitoring and repair method for human-machine co-driving experience evaluation according to claim 1, characterized in that, The prediction model employs either an autoregressive model or a lightweight Seq2Seq neural network model. The prediction model receives historical normal data from the target EEG channel and current EEG data from adjacent normal channels. By analyzing the temporal variation patterns of EEG signals and the spatial correlations between different brain regions, it predicts the EEG waveforms of data-missing regions corresponding to abnormal labels.

7. The multimodal data quality monitoring and repair method for human-machine co-driving experience evaluation according to claim 6, characterized in that, The historical EEG data of the target channel includes normal EEG waveforms continuously acquired before the abnormality occurred. The current EEG data of adjacent normal channels includes data from multiple EEG acquisition channels that are spatially related to the target channel. By jointly inputting the prediction model, the generated missing EEG data maintains the brain activity change pattern corresponding to the current driving state of the target driver.

8. The multimodal data quality monitoring and repair method for human-machine co-driving experience evaluation according to claim 1, characterized in that, In the generative predictive repair process, missing data is generated based on the frequency characteristics of the EEG signal and the correlation characteristics between brain regions in the current state of the driver. The generated EEG waveform maintains the alpha band oscillation mode and the theta band oscillation mode of the target driver in the driving state.

9. The multimodal data quality monitoring and repair method for human-machine co-driving experience evaluation according to claim 8, characterized in that, The repaired state markers are used to distinguish between the original acquired EEG data and the predicted generated EEG data, and the data credibility in the subsequent psychological state decoding process is adjusted according to the repaired state markers. The EEG data after generative prediction repair carries the corresponding uncertainty information.

10. The multimodal data quality monitoring and repair method for human-machine co-driving experience evaluation according to claim 5, characterized in that, A correspondence is established between dynamic quality labels and subsequent data processing strategies. When an EEG data segment is identified as a normal label, the original collected data is retained for experience evaluation. When an EEG data segment is identified as a suspicious label, an adaptive filtering processing mechanism is invoked. When an EEG data segment is identified as an abnormal label, a generative prediction and repair mechanism is invoked.

Images