A multi-modal driving distraction information extraction method and device
By accurately capturing driving distraction cycles using multimodal physiological data and nonlinear dynamic models, the distortion problem in distraction information extraction in existing technologies has been solved, enabling accurate identification and early warning of distraction segments and reducing the risk of traffic accidents.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JILIN UNIVERSITY
- Filing Date
- 2026-05-14
- Publication Date
- 2026-07-24
AI Technical Summary
Existing technologies for extracting distraction information during driving suffer from labeling distortion and fixed duration limitations. They neglect the dynamic evolution of distraction states, resulting in severe distortion between the extracted samples and the actual physiological state, making it impossible to accurately capture distraction fragments.
By combining multimodal physiological data (EEG, eye movement, and ECG) with self-reported information, physiological feature sequences are extracted through time windows to construct a nonlinear dynamic model, which determines the initial moment, termination moment, and time envelope interval of the distraction evolution state, thus achieving complete capture of the distraction cycle.
It achieves accurate extraction of driver distraction information, providing a warning 0.5 seconds in advance, significantly reducing the likelihood of traffic accidents and improving driving safety.
Smart Images
Figure CN122454545A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent driving, and in particular relates to a method and device for extracting multimodal driving distraction information. Background Technology
[0002] Distraction while driving is a hidden and dangerous cognitive state, essentially a "decoupling" of the driver's thinking from the driving task. Distraction significantly reduces a driver's ability to perceive danger, leading to untimely and inaccurate judgment of the road environment, resulting in improper operation and ultimately driving accidents. Statistics show that if a driver's reaction time can be improved by just 0.5 seconds, the probability of an accident can be reduced by approximately 60%. Therefore, the accurate extraction of distraction fragments has become a crucial issue that urgently needs to be addressed in this field. However, existing technologies have serious shortcomings in extracting distraction fragments. One limitation lies in the distortion of annotation and the fixed duration. Because there is a significant physiological delay between the occurrence of spontaneous distraction and the awakening of "meta-consciousness" and the triggering of a button (self-report), traditional methods define distraction fragments by only using the self-report time as the endpoint and looking back a fixed duration (e.g., 5s, 10s). This fixed duration ignores the true dynamic start and end points of the distraction state, resulting in severe distortion between the extracted samples and the actual physiological state. Meanwhile, existing classification methods often lack key physiological or eye movement changes during the process of distraction evolution, and ignore the dynamic evolution of multimodal features (such as EEG, eye movement, and ECG) in the complete closed-loop cycle of "dissociation (entry)-maintenance-recoupling (exit)". Summary of the Invention
[0003] To address the aforementioned technical problems, the present invention aims to provide a method and apparatus for extracting multimodal driving distraction information.
[0004] According to one aspect of the present invention, a method for extracting multimodal driving distraction information is provided, comprising the following steps: Multimodal physiological data of the target driver is collected. This multimodal physiological data includes at least EEG characteristic data, eye movement characteristic data, and distraction cognition self-report information data. The distraction cognition self-report information data includes the self-report time, which can be marked as time anchors. ; Based on distraction cognition self-report information data and a preset duration time window, physiological feature sequences of each modality of physiological data are extracted from the multimodal physiological data through the time window, wherein the preset duration... The value can be determined by referring to the prior physiological delay range (e.g., 30 seconds). In actual calculations, the preset duration is... An adaptive search can also be performed based on the fitting results; A mind-distraction evolution morphology model is constructed using the physiological feature sequence and nonlinear dynamics. The constraints of the mind-distraction evolution morphology model are configured, and the mind-distraction evolution morphology domain information data of each modality's physiological features are determined. Based on the aforementioned distraction evolution morphology domain information data, under the aforementioned constraints, a distraction evolution morphology model is run, and the initial time, termination time, and time envelope interval for extracting driving distraction information are determined through a preset statistical significance check condition for multimodal physiological characteristics. This is to achieve the capture of the complete cycle of the mind-distraction evolution state, wherein the time envelope interval is configured by the initial time and the termination time.
[0005] Furthermore, the physiological feature sequences of each modality of physiological data in the multimodal physiological data are extracted through the time window, including: The preprocessing methods corresponding to each modality of the multimodal physiological data are adopted to preprocess each modality of the physiological data. The preprocessing method is sliding window processing, which uses a sliding window with a size of [5,10] seconds and a step size of 10% to clean the data. After preprocessing, physiological features of each modality are extracted using the feature extraction method corresponding to each modality, through the time window, to obtain the physiological feature sequence of each modality. The physiological feature sequence includes the attention dissociation index, apex region... Relative power index, frontal α relative power ratio index in, The attention dissociation index is used to characterize the degree of dissociation of the target driver's attention. The top region The relative power index is used to characterize the changes in the electroencephalogram (EEG) spectrum characteristics during the driver's distraction. The frontoparietal α relative power ratio index is used to characterize the changes in the allocation of attentional resources among different brain regions of the target driver.
[0006] When the preprocessed physiological data of each modality is eye-tracking feature information data, the feature extraction method corresponding to each modality of physiological data is adopted, and physiological features are extracted through the time window to obtain the physiological feature sequence of each modality of physiological data, including: The preprocessed eye movement feature information data is extracted through the time window based on the changes of each eye movement feature during the distraction evolution process. Each eye movement feature includes the average number of fixations, the average number of saccades, and the average fixation duration. The trend of the average fixation duration during the distraction evolution process is opposite to the trend of the average number of fixations and the average number of saccades during the distraction evolution process. When the average number of fixations and the average number of saccades decrease, the average fixation duration will increase. The extracted eye movement feature information data is used to generate the attention dissociation index based on the contribution weight of each eye movement feature information data in the driver's distraction state recognition, the attention dissociation index and the preset relationship between the eye movement feature information data, and the attention dissociation index. The attention dissociation index is attributed to the physiological feature sequence.
[0007] The processing of eye movement feature information data also includes baseline standardization of eye movement features according to Equation 1.
[0008] [1], in, It is the original value of the i-th eye-tracking feature at time t. These are the standardized eigenvalues. It is the mean of this feature at the conscious baseline stage. It is the standard deviation of this feature at the conscious baseline stage.
[0009] The contribution weight of each eye movement feature data in the driver's distraction state recognition, the attention dissociation index, and the preset relationship between the eye movement feature data satisfy Equation 2.
[0010] [2], in, This represents the degree of attentional dissociation at time t. This represents the contribution weight of each eye-tracking indicator to distraction detection, and its value range is... And satisfy The acquisition method is based on a fixed allocation according to prior knowledge (such as equal weight 1 / 3). The process of configuring the contribution weight for driver distraction state recognition also includes the following process (dynamically calculated through the deviation sensitivity adaptive operator), which includes: configuring the sensitivity coefficient of each eye movement feature based on the preset relationship between the feature mean of each eye movement feature in the current window and the mean and standard deviation of the corresponding eye movement feature in the awake baseline stage. The sensitivity coefficient is used to characterize the degree of deviation of each eye movement feature from the awake baseline. The contribution weight for driver distraction state recognition is generated based on the correlation between the contribution weight for driver distraction state recognition, the sensitivity coefficient, and the morphological accommodation coefficient.
[0011] Furthermore, when the preprocessed physiological data of each modality is EEG feature information data, the feature extraction method corresponding to each modality of physiological data is adopted, and physiological features are extracted through the time window to obtain the physiological feature sequence of each modality of physiological data, including: The preprocessed EEG feature data, based on the power of each feature domain corresponding to the top region and the total power of the corresponding top region, and the corresponding top region... The preset relationships of relative power indices, namely the preset relationships between the power of each feature domain corresponding to the frontoparietal region in the EEG feature information data and the total power of the corresponding frontoparietal region, and the preset relationships of the relative power ratio index of the corresponding frontoparietal α region, are configured to generate the corresponding frontoparietal regions. Relative power index, the relative power ratio index of the forehead α; Based on the top region mentioned in the process of divinity evolution The variation characteristics of the relative power index and the frontoparietal α relative power ratio index are extracted from the EEG feature information data through the time window to obtain the EEG feature information data in the physiological feature sequence.
[0012] Among them, the corresponding feature domains of the top region correspond to the top region Pz, P3, and P4 channels, respectively, and the corresponding feature domains of the frontoparietal region correspond to the front region Fz, F3, and F4 channels, and the top region Pz, P3, and P4 channels, respectively.
[0013] The power of each feature domain corresponding to the top region and the total power of the corresponding top region in the EEG feature information data, and the corresponding top region The preset relationship of the relative power index satisfies Equation 3. [3], in, The region below time t represents the region (Pz, P3, P4). Frequency band power; This represents the total power in the apex region (Pz, P3, P4) at time t. Apex region activity is typically associated with internal cognitive processing and attentional distraction; therefore, an increase in this indicator can characterize an increase in the driver's level of inattention.
[0014] The preset relationship between the power of each feature domain corresponding to the frontoparietal region in the EEG feature information data and the total power of the corresponding frontoparietal region and the relative power ratio index of the corresponding frontoparietal α region satisfies Equation 4.
[0015] [4], in, The subfrontal region (Fz, F3, F4) represents time t. Frequency band power; This represents the total power of the lower jaw region (Fz, F3, F4) at time t; The region below time t represents the region (Pz, P3, P4). Frequency band power, The total power of the lower vertex region (Pz, P3, P4) at time t is represented. The "first decreasing and then increasing" trend of the frontal vertex α relative power ratio index can effectively characterize the dynamic process of the driver's attention dissociating from external tasks to internal cognition and finally recoupling to the external environment.
[0016] Furthermore, a morphological model of mental dissociation evolution is constructed using the aforementioned physiological characteristic sequence and nonlinear dynamics. Constraints on this model are configured, and the morphological domain information data of each modality's physiological characteristics in the morphological domain are determined, including: The physiological feature sequence is used to construct the distraction evolution morphology model using a nonlinear regression method. The constraints of the distraction evolution morphology model are configured according to the dynamic structural features and the gain directionality of the physiological features in the distracted state of the target driver, and the distraction evolution morphology domain information data is determined. The distraction evolution morphology domain information data includes the physiological extreme moments in the distraction evolution state.
[0017] Specifically, the attention dissociation index within the time window W. Top area Relative power index and the relative power ratio index of frontoparietal α The dynamic evolution model, as shown in Equation 5, is constructed by performing quadratic polynomial fitting on each model, which is the aforementioned morphological model of the split spirit evolution.
[0018] [5], in, .
[0019] The constraint condition is based on the gain directionality of physiological characteristics in a distracted state, and the coefficients of the second-order terms are set accordingly. The symbolic constraint criteria include those for the attention dissociation index and the vertex region. Positive gain constraints on the relative power index and negative gain constraints on the relative power ratio index of the forehead α. A valid distraction evolution trend is determined to exist in this time domain only if the fitted model satisfies the following corresponding constraints: Forward gain constraint: For and The coefficients of its second-order terms are required to satisfy... This is used to prove that the indicator has a maximum envelope within the window.
[0020] Negative gain constraint: For The coefficients of its second-order terms are required to satisfy... This is used to prove that the indicator has a minimum dip within the window.
[0021] For the indicators that pass the above constraint verification, set the first derivative of the model to zero, and calculate the physiological extreme moments in the process of distraction evolution. The calculation is shown in Equation 6.
[0022] [6], Furthermore, based on the aforementioned distraction evolution morphology domain information data, under the aforementioned constraints, the distraction evolution morphology model is run, and the initial and final moments of the driving distraction evolution state are determined through preset statistical significance verification conditions of multimodal physiological characteristics, including: Based on the distraction evolution morphology domain information data, distraction cognition self-report information data, and preset duration configuration of the driver's distraction evolution state entry search interval. and the termination search interval , Based on the correlation between the entry and termination search intervals and the fitting curve feature information of the distraction evolution morphology model and its first derivative, candidate initial and candidate termination times for the driving distraction evolution state corresponding to each modality are generated using preset screening conditions for each modality's physiological features. The correlation between the entry and termination search intervals and the fitting curve feature information of the distraction evolution morphology model and its first derivative satisfies equations 7 and 8. The preset screening conditions for each modality's physiological features are as follows: within the search interval of the distraction stage... Inside, calculate the fitting model. The first derivative is taken as the starting point for defining the index, with the moment when the absolute value of the first derivative is the largest within the interval; the search interval for recovery to the lucid stage. Within the range, the moment when the absolute value of the first derivative of the fitted curve is the largest is taken as the termination point of the index definition.
[0023] [7], [8], The candidate initial and candidate termination times of the driving distraction evolution state are determined by using preset admission conditions and preset statistical significance verification conditions of multimodal physiological characteristics.
[0024] The preset admission conditions for multimodal physiological characteristics are to ensure that the fragment covers the earliest occurrence of distraction, and the minimum value of the start time identified by the three indicators is taken as the final starting point, as shown in Equation 9; to ensure that the fragment covers the time of complete recovery of physiological state, the maximum value of the termination time identified by the three indicators is taken as the final endpoint, as shown in Equation 10.
[0025] [9],
[10] , Furthermore, the preset statistical significance verification conditions for the multimodal physiological characteristics are configured based on the determination coefficient of the mental evolution morphology model, the determination coefficient of the linear regression model within the same time interval, and the preset relationship between the incremental threshold and the preset threshold, as shown in Equation 11.
[0026] The final extracted distraction fragments are defined as follows: This segment needs to further satisfy statistical significance verification, that is, within this interval, the adjusted coefficient of determination of the quadratic fitting model for each indicator is required. It must satisfy equation 11.
[0027]
[11] , in, The coefficient of determination for the linear regression model within the same interval is used to quantitatively exclude baseline drift of non-kinetic nature. The incremental threshold is greater than a preset threshold (e.g., 0.1), which is an empirical constant preset based on the signal-to-noise ratio characteristics and kinetic evolution intensity of driving physiological signals. Its purpose is to ensure that the extracted segments, on the basis of statistical significance, possess sufficient nonlinear evolution characteristics to effectively distinguish the "dissociation-recoupling" process from conventional physiological baseline fluctuations.
[0028] According to another aspect of the present invention, an apparatus is provided, characterized in that the apparatus comprises: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors perform the method described above.
[0029] According to one aspect of the present invention, a computer-readable storage medium storing a computer program is provided, wherein the program, when executed by a processor, implements the method described above.
[0030] Compared with the prior art, the present invention has the following beneficial effects: 1. The multimodal driving distraction information extraction method of this invention is based on multimodal physiological data. It configures physiological feature sequences for each modality, constructs a distraction evolution morphology model using these physiological feature sequences and nonlinear dynamics, configures its constraints, runs the distraction evolution morphology model based on distraction evolution morphology domain information data, and determines the initial and final times of the driving distraction evolution state and the time envelope interval for driving distraction information extraction through preset statistical significance verification conditions of multimodal physiological features. This achieves the capture of the complete distraction cycle of the distraction evolution state. During the entire process of capturing the distraction evolution state, the method... Throughout the complete closed-loop cycle of "dissociation (entry) - maintenance - recoupling (exit)," the dynamic evolution of multimodal physiological characteristics (such as EEG, eye movement, and ECG) is always involved. This perfectly integrates dissociation dynamics with multimodal physiological characteristics, ensuring that the driver distraction information extracted using the time envelope interval is completely consistent with the actual physiological state of the distraction. This achieves accurate extraction of distraction information. Statistics show that if a driver's reaction time can be reduced by 0.5 seconds, the probability of traffic accidents can be reduced by approximately 60%. The realization of this technology will inevitably have a landmark impact on the field of intelligent driving and is of great significance to it.
[0031] 2. The device and computer-readable storage medium storing computer programs in the present invention employ a driving distraction segment extraction method based on the fusion of dissociation dynamics and multimodal physiological features, which makes the driving distraction information extracted by using time envelope intervals completely consistent with the actual physiological state of distraction. The entire extraction process is simple to operate and the extracted data is accurate. Attached Figure Description
[0032] Figure 1 This is a flowchart of a multimodal driving distraction information extraction method as an example. Figure 2 An eye-tracking data graph; Figure 3 for , and Characteristic curve graph; Figure 4 This is a graph of EEG data; Figure 5 The relative power ratio index of frontoparietal α ( ) and top region Relative power index ( Characteristic curve diagram. Detailed Implementation
[0033] To better understand the technical solution of the present invention, the present invention will be further described below in conjunction with specific embodiments and accompanying drawings.
[0034] This embodiment provides a method for extracting multimodal driving distraction information, such as... Figure 1 As shown, it includes the following steps: S1. Collect multimodal physiological data of the target driver, wherein the multimodal physiological data includes at least EEG feature data, eye movement feature data, and distracted cognition self-report information data, and the distracted cognition self-report information data includes the self-report time, which can be marked as time anchors. This involves simulating driving experiments, simultaneously recording the driver's electroencephalogram (EEG), eye movements, electrocardiogram (PPG), electrical activity of the skin (EDA), and self-reported moments, with the self-reported moments serving as time anchors. .
[0035] S2. Based on the distraction cognition self-report information data and a preset duration time window, extract the physiological feature sequences of each modality of physiological data from the multimodal physiological data through the time window, wherein the preset duration... The value can be determined by referring to the prior physiological delay range (e.g., 30 seconds). In actual calculations, the preset duration is... An adaptive search can also be performed based on the fitting results.
[0036] Furthermore, the physiological feature sequences of each modality of physiological data in the multimodal physiological data are extracted through the time window, including: The preprocessing methods corresponding to each modality of the multimodal physiological data are adopted to preprocess each modality of the physiological data. The preprocessing method is sliding window processing, which uses a sliding window with a size of [5,10] seconds and a step size of 10% to clean the data. After preprocessing, physiological features of each modality are extracted using the feature extraction method corresponding to each modality, through the time window, to obtain the physiological feature sequence of each modality. The physiological feature sequence includes the attention dissociation index, apex region... Relative power index, frontal α relative power ratio index in, The attention dissociation index is used to characterize the degree of dissociation of the target driver's attention. The top region The relative power index is used to characterize the changes in the electroencephalogram (EEG) spectrum characteristics during the driver's distraction. The frontoparietal α relative power ratio index is used to characterize the changes in the allocation of attentional resources among different brain regions of the target driver.
[0037] Furthermore, when the preprocessed physiological data of each modality is eye-tracking feature information data, the feature extraction method corresponding to each modality of physiological data is adopted, and physiological features are extracted through the time window to obtain the physiological feature sequence of each modality of physiological data, including: The preprocessed eye movement feature information data is extracted through the time window based on the changes of each eye movement feature during the distraction evolution process. Each eye movement feature includes the average number of fixations, the average number of saccades, and the average fixation duration. The trend of the average fixation duration during the distraction evolution process is opposite to the trend of the average number of fixations and the average number of saccades during the distraction evolution process. When the average number of fixations and the average number of saccades decrease, the average fixation duration will increase. The extracted eye movement feature information data is used to generate the attention dissociation index based on the contribution weight of each eye movement feature information data in the driver's distraction state recognition, the attention dissociation index and the preset relationship between the eye movement feature information data, and the attention dissociation index. The attention dissociation index is attributed to the physiological feature sequence.
[0038] The processing of eye movement feature information data also includes baseline standardization of eye movement features according to Equation 1.
[0039] [1], in, It is the original value of the i-th eye-tracking feature at time t. These are the standardized eigenvalues. It is the mean of this feature at the conscious baseline stage. It is the standard deviation of this feature at the conscious baseline stage.
[0040] The contribution weight of each eye movement feature data in the driver's distraction state recognition, the attention dissociation index, and the preset relationship between the eye movement feature data satisfy Equation 2.
[0041] [2], in, This represents the degree of attentional dissociation at time t. This represents the contribution weight of each eye-tracking indicator to distraction detection, and its value range is... And satisfy The acquisition method is based on a fixed allocation according to prior knowledge (such as equal weight 1 / 3). The process of configuring the contribution weight for driver distraction state recognition also includes the following process (dynamically calculated through the deviation sensitivity adaptive operator), which includes: configuring the sensitivity coefficient of each eye movement feature based on the preset relationship between the feature mean of each eye movement feature in the current window and the mean and standard deviation of the corresponding eye movement feature in the awake baseline stage. The sensitivity coefficient is used to characterize the degree of deviation of each eye movement feature from the awake baseline. The contribution weight for driver distraction state recognition is generated based on the correlation between the contribution weight for driver distraction state recognition, the sensitivity coefficient, and the morphological accommodation coefficient.
[0042] Furthermore, when the preprocessed physiological data of each modality is EEG feature information data, the feature extraction method corresponding to each modality of physiological data is adopted, and physiological features are extracted through the time window to obtain the physiological feature sequence of each modality of physiological data, including: The preprocessed EEG feature data, based on the power of each feature domain corresponding to the top region and the total power of the corresponding top region, and the corresponding top region... The preset relationships of relative power indices, namely the preset relationships between the power of each feature domain corresponding to the frontoparietal region in the EEG feature information data and the total power of the corresponding frontoparietal region, and the preset relationships of the relative power ratio index of the corresponding frontoparietal α region, are configured to generate the corresponding frontoparietal region. Relative power index, the relative power ratio index of the forehead α; Based on the top region mentioned in the process of divinity evolution The variation characteristics of the relative power index and the frontoparietal α relative power ratio index are extracted from the EEG feature information data through the time window to obtain the EEG feature information data in the physiological feature sequence.
[0043] Among them, the corresponding feature domains of the top region correspond to the top region Pz, P3, and P4 channels, respectively, and the corresponding feature domains of the frontoparietal region correspond to the front region Fz, F3, and F4 channels, and the top region Pz, P3, and P4 channels, respectively.
[0044] The power of each feature domain corresponding to the top region and the total power of the corresponding top region in the EEG feature information data, and the corresponding top region The preset relationship of the relative power index satisfies Equation 3. [3], in, The region below time t represents the region (Pz, P3, P4). Frequency band power; This represents the total power in the apex region (Pz, P3, P4) at time t. Apex region activity is typically associated with internal cognitive processing and attentional distraction; therefore, an increase in this indicator can characterize an increase in the driver's level of inattention.
[0045] The preset relationship between the power of each feature domain corresponding to the frontoparietal region in the EEG feature information data and the total power of the corresponding frontoparietal region and the relative power ratio index of the corresponding frontoparietal α region satisfies Equation 4.
[0046] [4], in, The subfrontal region (Fz, F3, F4) represents time t. Frequency band power; This represents the total power of the lower jaw region (Fz, F3, F4) at time t; The region below time t represents the region (Pz, P3, P4). Frequency band power, The total power of the lower vertex region (Pz, P3, P4) at time t is represented. The "first decreasing and then increasing" trend of the frontal vertex α relative power ratio index can effectively characterize the dynamic process of the driver's attention dissociating from external tasks to internal cognition and finally recoupling to the external environment.
[0047] S3. Construct a mind-distraction evolution morphology model using the physiological feature sequence and nonlinear dynamics, configure the constraints of the mind-distraction evolution morphology model, and determine the mind-distraction evolution morphology domain information data of each modality's physiological features.
[0048] Furthermore, a morphological model of mental dissociation evolution is constructed using the aforementioned physiological characteristic sequence and nonlinear dynamics. Constraints on this model are configured, and the morphological domain information data of each modality's physiological characteristics in the morphological domain are determined, including: The physiological feature sequence is used to construct the distraction evolution morphology model using a nonlinear regression method. The constraints of the distraction evolution morphology model are configured according to the dynamic structural features and the gain directionality of the physiological features in the distracted state of the target driver, and the distraction evolution morphology domain information data is determined. The distraction evolution morphology domain information data includes the physiological extreme moments in the distraction evolution state.
[0049] Specifically, the attention dissociation index within the time window W. Top area Relative power index and the relative power ratio index of frontoparietal α The dynamic evolution model, as shown in Equation 5, is constructed by performing quadratic polynomial fitting on each model, which is the aforementioned morphological model of the split spirit evolution.
[0050] [5], in, .
[0051] The constraint condition is based on the gain directionality of physiological characteristics in a distracted state, and the coefficients of the second-order terms are set accordingly. The symbolic constraint criteria include those for the attention dissociation index and the vertex region. Positive gain constraints on the relative power index and negative gain constraints on the relative power ratio index of the forehead α. A valid distraction evolution trend is determined to exist in this time domain only if the fitted model satisfies the following corresponding constraints: Forward gain constraint: For and The coefficients of its second-order terms are required to satisfy... This is used to prove that the indicator has a maximum envelope within the window.
[0052] Negative gain constraint: For The coefficients of its second-order terms are required to satisfy... This is used to prove that the indicator has a minimum dip within the window.
[0053] For the indicators that pass the above constraint verification, set the first derivative of the model to zero, and calculate the physiological extreme moments in the process of distraction evolution. The calculation is shown in Equation 6.
[0054] [6], S4. Based on the information data of the distraction evolution morphology domain, under the constraints, run the distraction evolution morphology model, and determine the initial time, termination time, and time envelope interval of the driving distraction evolution state and the extraction of driving distraction information through the preset statistical significance verification conditions of multimodal physiological characteristics. This is to achieve the capture of the complete cycle of the mind-distraction evolution state, wherein the time envelope interval is configured by the initial time and the termination time.
[0055] Furthermore, based on the aforementioned distraction evolution morphology domain information data, under the aforementioned constraints, the distraction evolution morphology model is run, and the initial and final moments of the driving distraction evolution state are determined through preset statistical significance verification conditions of multimodal physiological characteristics, including: Based on the distraction evolution morphology domain information data, distraction cognition self-report information data, and preset duration configuration of the driver's distraction evolution state entry search interval. and the termination search interval , Based on the correlation between the entry and termination search intervals and the fitting curve feature information of the distraction evolution morphology model and its first derivative, candidate initial and candidate termination times for the driving distraction evolution state corresponding to each modality are generated using preset screening conditions for each modality's physiological features. The correlation between the entry and termination search intervals and the fitting curve feature information of the distraction evolution morphology model and its first derivative satisfies equations 7 and 8. The preset screening conditions for each modality's physiological features are as follows: within the search interval of the distraction stage... Inside, calculate the fitting model. The first derivative is taken as the starting point for defining the index, with the moment when the absolute value of the first derivative is largest within the interval being the starting point; the search interval for recovery to the lucid stage. Within the range, the moment when the absolute value of the first derivative of the fitted curve is the largest is taken as the termination point of the index definition.
[0056] [7], [8], The candidate initial and candidate termination times of the driving distraction evolution state are determined by using preset admission conditions and preset statistical significance verification conditions of multimodal physiological characteristics.
[0057] The preset admission conditions for multimodal physiological characteristics are to ensure that the fragment covers the earliest occurrence of distraction, and the minimum value of the start time identified by the three indicators is taken as the final starting point, as shown in Equation 9; to ensure that the fragment covers the time of complete recovery of physiological state, the maximum value of the termination time identified by the three indicators is taken as the final endpoint, as shown in Equation 10.
[0058] [9],
[10] , Furthermore, the preset statistical significance verification conditions for the multimodal physiological characteristics are configured based on the determination coefficient of the mental evolution morphology model, the determination coefficient of the linear regression model within the same time interval, and the preset relationship between the incremental threshold and the preset threshold, as shown in Equation 11.
[0059] The final extracted distraction fragments are defined as follows: This segment needs to further satisfy statistical significance verification, that is, within this interval, the adjusted coefficient of determination of the quadratic fitting model for each indicator is required. It must satisfy equation 11.
[0060]
[11] , in, The coefficient of determination for the linear regression model within the same interval is used to quantitatively exclude baseline drift of non-kinetic nature. The incremental threshold is greater than a preset threshold (e.g., 0.1), which is an empirical constant preset based on the signal-to-noise ratio characteristics and kinetic evolution intensity of driving physiological signals. Its purpose is to ensure that the extracted segments, on the basis of statistical significance, possess sufficient nonlinear evolution characteristics to effectively distinguish the "dissociation-recoupling" process from conventional physiological baseline fluctuations.
[0061] The above-mentioned multimodal driver distraction information extraction process specifically includes the following steps: S1. Data Preparation Multimodal synchronous acquisition: Through simulated driving experiments, the driver's electroencephalogram (EEG), eye movements, electrocardiogram (PPG), electrical activity of the skin (EDA), and self-reported moments are recorded simultaneously, with the self-reported moments used as time anchors. .
[0062] Sliding window processing: Data is cleaned using a sliding window with a size of [5,10] seconds and a step size of 10%.
[0063] Physiological characteristic calculation: Table 1: Calculation of Physiological Characteristics S2. Determine the backtracking search domain and time anchor point. Because there is a physiological delay (metaconsciousness awakening delay) between the driver's "distraction occurring" and "awareness of the distraction," the self-reported moment cannot be directly used as the endpoint. Using this as a reference, backtrack a length of... time window : in The value of is determined with reference to the prior physiological delay range (e.g., 30 seconds). In actual calculations, the length... An adaptive search can also be performed based on the fitting results.
[0064] S3, Feature Construction (1) Constructing the attention dissociation index ( ) This study establishes an index to reflect the degree of attentional "dissociation" based on eye-tracking characteristics. According to the dissociation hypothesis, when a person enters a distracted state, visual sampling of the external environment becomes sparse and unstable. (See...) Figure 2-3This invention has discovered that eye movement indicators change during distraction: the average number of fixations and average number of saccades decrease, while the average fixation duration increases. When a driver becomes aware of their distraction, they will self-report it. The purpose of this invention is to accurately identify the distraction segment occurring some time before this self-report. The aforementioned changes in eye movement characteristics indicate the distraction phase. Figure 2 This is a graph of eye-tracking data. Figure 3 for , and Characteristic curve graph Figure 3 for Figure 2 A large local image showing the temporal characteristics of the complete distraction process, extracted from the image. , and Corresponding to the number of fixations ( ), number of scans ( ), fixation duration ( The characteristic curve of ).
[0065] To eliminate individual differences among different drivers, eye movement features are first standardized according to Equation 1.
[0066] [1], in It is the original value of the i-th eye-tracking feature at time t. These are the standardized eigenvalues. It is the mean of this feature at the conscious baseline stage. It is the standard deviation of this feature at the conscious baseline stage.
[0067] Based on the standardized eye movement features, an attention dissociation index is constructed according to Equation 2.
[0068] [2], in, This represents the degree of attention dissociation at time t; This represents the contribution weight of each eye-tracking indicator to distraction detection, and its value range is... And satisfy The weights can be obtained by fixed allocation based on prior knowledge (e.g., equal weight 1 / 3), or by dynamic calculation using the following deviation sensitivity adaptive operator. This includes configuring the sensitivity coefficients of each eye movement feature based on a preset relationship between the feature mean of each eye movement feature within the current window and the mean and standard deviation of the corresponding eye movement feature at the awake baseline stage. These sensitivity coefficients characterize the degree of deviation of each eye movement feature from the awake baseline. The preset relationship between the feature mean of each eye movement feature within the current window and the mean and standard deviation of the corresponding eye movement feature at the awake baseline stage satisfies the following equation 12. The contribution weight for recognizing driver distraction is generated based on the correlation between the contribution weight for driver distraction recognition, the sensitivity coefficient, and the morphological adjustment coefficient. The morphological adjustment coefficient is used to adjust the gain intensity of the weight on feature fluctuations. The correlation between the contribution weight for recognizing driver distraction, the sensitivity coefficient, and the morphological adjustment coefficient satisfies the following equation 13.
[0069] The specific steps for dynamically calculating the weights using the deviation sensitivity adaptive operator are as follows: First, the deviation of the measured characteristic from the conscious baseline is calculated using the following formula.
[12] , in, Let be the feature mean of feature i within the current window; To prevent the denominator from being zero, a preset smoothing constant is used. Weight parameters characterize the contribution of each feature to the distraction state, and then the sensitivity coefficients are nonlinearly mapped and normalized using the following formula.
[13] , In the formula, k is the shape adjustment coefficient, used to adjust the strength of the weight gain on feature fluctuations. Under this model, the more significant the feature deviates from the baseline, the higher its sensitivity coefficient. The higher the value, the higher the corresponding weight. The larger the value, the more dynamic the enhancement of eye features in a distracted state. In practical applications, the weights can be predetermined and kept fixed. Attention dissociation index (ADI) This indicator comprehensively reflects the changing trend of visual sampling behavior. An increase in its value indicates that the driver's attention is gradually disengaging from external driving tasks to internal cognitive activities.
[0070] (2) Top region Relative power index ( ) To characterize the changes in EEG spectral features during distraction, the parietal region (Pz, P3, and P4 channels) was selected. Frequency band power construction top region Relative power index, see Figure 4-5 .
[0071] First, calculate the top region. The ratio of band power to total power is shown in Equation 3.
[0072] [3], in, The region below time t represents the region (Pz, P3, P4). Frequency band power; This represents the total power in the apex region (Pz, P3, P4) at time t. Apex region activity is typically associated with internal cognitive processing and attentional distraction; therefore, an increase in this indicator can characterize an increase in the driver's level of inattention.
[0073] (3) Frontal-Vertical α Relative Power Ratio Index ( ) To reflect the changes in the allocation of attentional resources among different brain regions, Equation 4 constructs a ratio index of the relative power of the α band between the frontal and parietal regions.
[0074] [4], in, The subfrontal region (Fz, F3, F4) represents time t. Frequency band power; This represents the total power of the lower jaw region (Fz, F3, F4) at time t; The region below time t represents the region (Pz, P3, P4). Frequency band power, The total power of the lower vertex region (Pz, P3, P4) at time t is represented. The "first decreasing and then increasing" trend of the frontal vertex α relative power ratio index can effectively characterize the dynamic process of the driver's attention dissociating from external tasks to internal cognition and finally recoupling to the external environment.
[0075] Figure 4 This is a graph of EEG data. Figure 5 The relative power ratio index of frontoparietal α ( ) and top region Relative power index ( Characteristic curve graph, where, Figure 5 for Figure 4 A large local image of the complete temporal features extracted from the image.
[0076] Among them, the top region Relative power index ( The alpha-relative power ratio index first rises and then falls during the entire distraction process; () It first decreases and then increases.
[0077] S4. Evolutionary Modeling and Interval Selection Based on Nonlinear Dynamic Fitting After obtaining the physiological feature sequence within the backtracking window W, the nonlinear regression method is used to macroscopically model the morphology of mind distraction, and the mind distraction evolution domain is initially locked based on the dynamic structural characteristics.
[0078] (1) Construction of the quadratic dynamic fitting model Attention dissociation index within window W Top area Relative power index and the relative power ratio index of frontoparietal α The dynamic evolution model is constructed by performing quadratic polynomial fitting, as shown in Equation 5.
[0079] [5], in, .
[0080] (2) Mathematical constraints on evolutionary forms Based on the gain directionality of physiological characteristics under distraction, the coefficients of the second-order terms are set. The sign constraint criterion. A valid distraction evolution trend exists in the time domain only if the fitted model satisfies the following corresponding constraints: Forward gain constraint: For and The coefficients of its second-order terms are required to satisfy... This is used to prove that the indicator has a maximum envelope within the window.
[0081] Negative gain constraint: For The coefficients of its second-order terms are required to satisfy... This is used to prove that the indicator has a minimum dip within the window.
[0082] (3) Determine the characteristic extreme value time For the indicators that pass the above constraint verification, set the first derivative of the model to zero and calculate the physiological extreme moments in their evolution process. The calculation is shown in Equation 6.
[0083] [6], S5. Precise identification of the start and end points of the rate of change peak of the fitted curve. Having determined the core extreme point of the evolution of the mind-distraction process Then, the instantaneous rate of change of the fitted curve is analyzed to finely identify the boundary moments when the physiological state undergoes drastic shifts and recoveries.
[0084] (1) Candidate start time Extraction During the search interval when the mind is distracted Inside, calculate the fitting model. The first derivative. The moment when the absolute value of the first derivative is the largest within the interval is taken as the starting point for the definition of the index, as shown in Equation 7.
[0085] [7], (2) Candidate time Extraction Search area during the recovery to consciousness stage Within the range, the moment when the absolute value of the first derivative of the fitted curve is the largest is taken as the termination point of the index definition, as shown in Equation 8.
[0086] [8], S6. Multidimensional Feature Temporal Fusion and Optimal Distraction Segment Extraction Considering the differences in response delays of different physiological channels to distraction perception, candidate boundary points are determined by combining multiple indicators to construct the optimal time envelope interval, so as to capture the complete distraction cycle.
[0087] (1) Optimal moment of distraction Fusion determination To ensure that the segment covers the earliest moment of distraction, the minimum starting time identified by the three indicators is taken as the final starting point, as shown in Equation 9.
[0088] [9], (2) Optimal time to terminate distraction Fusion determination To ensure that the fragment covers the moment of complete recovery of the physiological state, the maximum value of the termination time identified by the three indicators is taken as the final endpoint, as shown in Equation 10.
[0089]
[10] , (3) Sample output and statistical verification The final extracted distraction fragments are defined as follows: This segment needs to further satisfy statistical significance verification, that is, within this interval, the adjusted coefficient of determination of the quadratic fitting model for each indicator is required. It must satisfy equation 11.
[0090]
[11] , in The coefficient of determination for the linear regression model within the same interval is used to quantitatively exclude baseline drift of non-kinetic nature. The increment threshold is greater than 0.1, which is an empirical constant preset based on the signal-to-noise ratio characteristics and kinetic evolution intensity of driving physiological signals. The purpose is to ensure that the extracted segments, on the basis of statistical significance, possess sufficient nonlinear evolution characteristics to effectively distinguish the "dissociation-recoupling" process from conventional physiological baseline fluctuations.
[0091] Application value: Distracted driving segments with high physiological consistency and dynamic characteristics extracted using the method described in this paper can provide core data support and technical solutions for intelligent transportation systems and human-machine collaborative driving. Specific applications include, but are not limited to: (1) Training and benchmark construction of high-precision distraction recognition model Construction of deep learning sample library: The fragments extracted by this method accurately cover the entire process of distraction evolution (entry-maintenance-exit), and can be used as gold standard samples to build a high-performance driving distraction recognition model.
[0092] Individualized model tuning: By using extracted fragments with dynamic characteristics, the model can be fine-tuned for specific drivers, improving the system's adaptability to different individual physiological differences.
[0093] (2) Active safety warning and intervention in intelligent cockpit Multi-level early warning mechanism: Based on the distraction start time determined by this invention With extreme moments The intelligent cockpit system can set up a tiered warning logic based on this: visual reminders when a dissociation trend is detected, and strong vibration or voice intervention when entering a period of deep distraction (extreme period).
[0094] Driving task takeover determination: In autonomous driving (L2 / L3 level) scenarios, by real-time monitoring of whether the driver's physiological indicators meet the "re-coupling" mode described in this invention, it is determined whether the driver has successfully returned attention to the driving task, thereby ensuring the safety of the takeover process.
[0095] (3) Comprehensive assessment and health management of drivers Distraction susceptibility assessment: by statistically analyzing the frequency and duration of distraction segments extracted during long-term driving. It can quantitatively assess a driver's ability to allocate attention.
[0096] In another aspect, this embodiment also provides an apparatus suitable for implementing the embodiments of this application. The apparatus includes a computer system comprising a central processing unit (CPU) capable of performing various appropriate actions and processes based on corresponding programs stored in a read-only memory (ROM) for executing the various steps described in the multimodal driving distraction information extraction method, or programs loaded from a storage portion into a random access memory (RAM) for executing the various steps described in the multimodal driving distraction information extraction method. The RAM also stores various programs and data required for system operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.
[0097] The following components are connected to the I / O interface: input sections including keyboards, mice, etc.; output sections including cathode ray tubes (CRTs), liquid crystal displays (LCDs), and speakers; storage sections including hard disks; and communication sections including network interface cards such as LAN cards and modems. The communication sections perform communication processing via networks such as the Internet. Drives are also connected to the I / O interface as needed. Removable media, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on the drive as needed so that computer programs read from them can be installed into the storage section as required.
[0098] In particular, according to embodiments of this disclosure, the processes described in the various steps of the multimodal driving distraction information extraction method described above can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program containing program code for performing the multimodal driving distraction information extraction method described above. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium.
[0099] The flowcharts in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the flowchart, and combinations of blocks in the flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0100] In another aspect, this embodiment also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the system described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores one or more programs, which are used by one or more processors to execute the multimodal driving distraction information extraction method described in this application.
[0101] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, the above-described features have similar functions to (but are not limited to) those disclosed in this application.
Claims
1. A multimodal method for extracting driver distraction information, characterized in that, Includes the following steps: Collect multimodal physiological data of the target driver, wherein the multimodal physiological data includes at least EEG feature data, eye movement feature data, and distracted cognition self-report data; Based on distraction cognition self-report information data and a preset duration time window, the physiological feature sequence of each modality of physiological data in the multimodal physiological data is extracted through the time window; A mind-distraction evolution morphology model is constructed using the physiological feature sequence and nonlinear dynamics. The constraints of the mind-distraction evolution morphology model are configured, and the mind-distraction evolution morphology domain information data of each modality's physiological features are determined. Based on the information data of the distraction evolution morphology domain, under the constraints, the distraction evolution morphology model is run, and the initial time, the termination time, and the time envelope interval for extracting driving distraction information are determined by the preset statistical significance verification conditions of multimodal physiological characteristics, so as to achieve the capture of the complete distraction cycle of the distraction evolution state. The time envelope interval is configured by the initial time and the termination time.
2. The multimodal driving distraction information extraction method according to claim 1, characterized in that, The physiological feature sequences of each modality of physiological data in the multimodal physiological data are extracted through the time window, including: The preprocessing methods corresponding to each modality of the multimodal physiological data are used to preprocess each modality of the physiological data respectively; After preprocessing, physiological features of each modality are extracted using the feature extraction method corresponding to each modality, through the time window, to obtain the physiological feature sequence of each modality. The physiological feature sequence includes the attention dissociation index, apex region... Relative power index, frontal α relative power ratio index in, The attention dissociation index is used to characterize the degree of dissociation of the target driver's attention. The top region The relative power index is used to characterize the changes in the electroencephalogram (EEG) spectrum characteristics during the driver's distraction. The frontoparietal α relative power ratio index is used to characterize the changes in the allocation of attentional resources among different brain regions of the target driver.
3. The multimodal driving distraction information extraction method according to claim 2, characterized in that, When the preprocessed physiological data of each modality is eye-tracking feature information data, the feature extraction method corresponding to each modality of physiological data is adopted, and physiological features are extracted through the time window to obtain the physiological feature sequence of each modality of physiological data, including: The preprocessed eye movement feature information data is extracted through the time window based on the changes of each eye movement feature during the distraction evolution process. Each eye movement feature includes the average number of fixations, the average number of saccades, and the average fixation duration. The trend of the average fixation duration during the distraction evolution process is opposite to the trend of the average number of fixations and the average number of saccades during the distraction evolution process. The extracted eye movement feature information data is used to generate the attention dissociation index based on the contribution weight of each eye movement feature information data in the driver's distraction state recognition, the attention dissociation index and the preset relationship between the eye movement feature information data, and the attention dissociation index. The attention dissociation index is attributed to the physiological feature sequence.
4. The multimodal driving distraction information extraction method according to claim 3, characterized in that, The process of configuring the contribution weights for driver distraction detection includes: The sensitivity coefficient of each eye movement feature is configured based on a preset relationship between the mean value of each eye movement feature within the current window and the mean value and standard deviation of the corresponding eye movement feature at the awake baseline. The sensitivity coefficient is used to characterize the degree of deviation of each eye movement feature from the awake baseline. The contribution weight for recognizing driver distraction is generated based on the correlation between the contribution weight for driver distraction recognition, the sensitivity coefficient, and the morphological adjustment coefficient.
5. The multimodal driving distraction information extraction method according to claim 2, characterized in that, When the preprocessed physiological data of each modality is EEG feature information data, the feature extraction method corresponding to each modality of physiological data is adopted, and physiological features are extracted through the time window to obtain the physiological feature sequence of each modality of physiological data, including: The preprocessed EEG feature data, based on the power of each feature domain corresponding to the top region and the total power of the corresponding top region, and the corresponding top region... The preset relationships of relative power indices, namely the preset relationships between the power of each feature domain corresponding to the frontoparietal region in the EEG feature information data and the total power of the corresponding frontoparietal region, and the preset relationships of the relative power ratio index of the corresponding frontoparietal α region, are configured to generate the corresponding frontoparietal regions. Relative power index, the relative power ratio index of the forehead α; Based on the top region mentioned in the process of divinity evolution The variation characteristics of the relative power index and the frontoparietal α relative power ratio index are extracted from the EEG feature information data through the time window to obtain the EEG feature information data in the physiological feature sequence.
6. The multimodal driving distraction information extraction method according to any one of claims 2-5, characterized in that, A morphological model of mind distraction evolution is constructed using the aforementioned physiological characteristic sequences and nonlinear dynamics. Constraints on this model are configured, and the morphological domain information data of each modality's physiological characteristics in the morphological domain are determined, including: The physiological feature sequence is used to construct the distraction evolution morphology model using a nonlinear regression method. The constraints of the distraction evolution morphology model are configured according to the dynamic structural features and the gain directionality of the physiological features in the distracted state of the target driver, and the distraction evolution morphology domain information data is determined. The distraction evolution morphology domain information data includes the physiological extreme moments in the distraction evolution state.
7. The multimodal driving distraction information extraction method according to claim 6, characterized in that, The constraints include those related to the attention dissociation index and the apex region. Positive gain constraint on the relative power index and negative gain constraint on the relative power ratio index of the top α.
8. The multimodal driving distraction information extraction method according to claim 6, characterized in that, Based on the aforementioned distraction evolution morphology domain information data, under the aforementioned constraints, a distraction evolution morphology model is run, and the initial and final moments of the driving distraction evolution state are determined through a preset statistical significance check condition for multimodal physiological characteristics, including: Based on the distraction evolution morphology domain information data, distraction cognition self-report information data, and preset duration configuration, the driver's distraction evolution state is entered and terminated in the search interval. Based on the correlation between the entry and termination search intervals and the fitting curve feature information of the distraction evolution morphology model and its first derivative, candidate initial time and candidate termination time of driving distraction evolution state corresponding to each modality physiological feature are generated by configuring the preset screening conditions of each modality physiological feature. The candidate initial and candidate termination times of the driving distraction evolution state are determined by using preset admission conditions and preset statistical significance verification conditions of multimodal physiological characteristics.
9. The multimodal driving distraction information extraction method according to claim 8, characterized in that, The preset statistical significance verification conditions for the multimodal physiological characteristics are configured based on the determination coefficient of the divinity evolution morphology model, the determination coefficient of the linear regression model within the same time interval, and the preset relationship between the incremental threshold and the preset threshold.
10. A device, characterized in that, The device includes: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in any one of claims 1-9.