Cabin passenger state monitoring method and device for high-interference environment operation

Abstract

The invention discloses a cabin passenger state monitoring method and device for high-interference environment operation. The method comprises the following steps: acquiring physiological data, behavior data and environment context data of a passenger; performing feature extraction on the physiological data and the behavior data to obtain a multi-modal feature vector; fusing the multi-modal feature vector and the environment context data to generate an initial state vector; correcting the initial state vector through hidden variable coupling modeling to obtain a current state vector; performing time sequence prediction based on the current state vector and the historical state vector to obtain a predicted state vector; performing risk judgment on the predicted state vector, and distinguishing abnormal states such as a normal state and emotional state deterioration, cognitive overload and the like; and generating a man-machine interaction intervention instruction in an abnormal state and controlling the vehicle-mounted system to execute. According to the method, cooperative quantification of emotion and cognitive states is realized, the method has strong anti-interference and prospective early warning capability, and the man-machine cooperative combat effectiveness and the passenger task safety are improved.

Description

Technical Field

[0001] This application relates to the field of human-machine interface and intelligent auxiliary decision-making technology for special vehicles, and in particular to a method and device for monitoring the status of cabin occupants for operation in highly disruptive environments. Background Technology

[0002] With the increasing demands for automotive intelligence and driving safety, the emotions and cognitive states of passengers have a more significant impact on driving safety and the driving experience. In scenarios such as long-distance driving, urban traffic congestion, and driving in inclement weather, passengers are prone to cognitive overload due to prolonged concentration, or negative emotions such as anxiety and irritability due to complex road conditions and travel delays. If these states are not identified and intervened in a timely manner, they may lead to operational errors and increase driving risks.

[0003] However, current technologies for monitoring general emotional and cognitive states have significant limitations: First, the methods of state perception are one-sided. For example, relying solely on cameras to recognize facial expressions to determine fatigue or solely on heart rate sensors to detect stress can lead to significant interference with single-modal signals in real-world scenarios such as vehicle bumps, frequent changes in lighting, and passengers wearing sunglasses or hats, resulting in a substantial decrease in monitoring reliability. Second, existing technologies often assess cognitive fatigue or emotional state separately, neglecting the interaction between the two. For instance, anxiety can significantly impair attention allocation and decision-making abilities. Third, they lack dynamic predictive capabilities, failing to anticipate potential cognitive overload (such as approaching a series of complex intersections) or deterioration in emotional state (such as escalating frustration due to increased congestion) in the near future, making it difficult to support proactive interventions to mitigate risks.

[0004] In particular, within the enclosed cockpits of special-operation platforms such as armored vehicles, engineering machinery, and aircraft, occupants face an extreme and complex environment characterized by continuous strong vibrations, extremely high noise levels, sudden changes in light, and high-intensity tactical mission pressure. Existing civilian vehicle-mounted monitoring technologies rely on sensor signals whose signal-to-noise ratio deteriorates drastically in such environments. Furthermore, their algorithm models fail to consider the strong interference of environmental context on physiological characteristics, and do not model the nonlinear coupling relationship between emotional and cognitive states under high stress, leading to complete system failure or extremely low reliability. Therefore, there is an urgent need for an emotion-cognitive collaborative monitoring and early warning technology that can adapt to extreme physical environments and high psychological loads, and possesses strong anti-interference capabilities. Summary of the Invention

[0005] In view of this, the cockpit occupant status monitoring method and device provided in this application for operation in a highly disruptive environment can achieve coordinated quantification of emotional and cognitive states, possess strong anti-interference capabilities and proactive early warning capabilities, and improve human-machine collaboration efficiency and occupant safety. The cockpit occupant status monitoring method and device provided in this application for operation in a highly disruptive environment is implemented as follows: This application provides a method for monitoring the status of cabin occupants during operations in highly disruptive environments, comprising: Acquire physiological and behavioral data of occupants, as well as environmental context data characterizing cabin physical disturbances and task load; Feature extraction processing is performed on the physiological and behavioral data to obtain multimodal feature vectors; The multimodal feature vector and the environmental context data are fused to obtain an initial state vector; The initial state vector is subjected to latent variable coupling modeling to obtain the current state vector; Based on the current state vector and the historical state vector, a time series prediction is performed to obtain the predicted state vector; The predicted state vector is subjected to state risk assessment to obtain emotional and cognitive states, which include normal and abnormal states. The abnormal states include at least one of cognitive overload and emotional state deterioration. When the emotional and cognitive states are abnormal, a human-computer interaction intervention command is generated, and the vehicle system is controlled to execute the human-computer interaction intervention command.

[0006] In some embodiments, the step of fusing the multimodal feature vector with the environmental context data to obtain an initial state vector includes: The environmental context data is encoded using a fully connected network to obtain an environmental modulation vector, and the multimodal feature vector is subjected to modal linear embedding to obtain an initial embedding vector for the multimodality. The initial embedding vector of the multimodal mode is added element by element to the environment modulation vector to obtain the environment-modulated mode embedding vector; Intramodal self-attention calculation is performed on the modal embedding vector after environmental modulation to obtain the intramodal optimized feature vector; The intra-modal optimized feature vectors are weighted to obtain multi-modal collaborative fusion feature vectors; The initial state vector is obtained by fitting the multimodal collaborative fusion feature vector.

[0007] In some embodiments, the physiological data includes electroencephalogram (EEG) data and peripheral physiological data. The step of performing feature extraction processing on the physiological data and behavioral data to obtain a multimodal feature vector includes: The EEG data is processed by bandpass filtering, power frequency notch filtering, and artifact removal to obtain EEG features; The behavioral data is statistically extracted to obtain eye movement features; The peripheral physiological data are subjected to signal filtering and separation and feature statistical processing to obtain the processed peripheral physiological data. The EEG features, eye movement features, and processed peripheral physiological data are spliced ​​together to obtain the multimodal feature vector.

[0008] In some embodiments, the process of performing latent variable coupling modeling on the initial state vector to obtain the current state vector includes: The initial state vector is subjected to feature extraction processing to obtain the observed variables; Construct a latent variable coupling model by inputting the observed variables into the latent variable coupling model to obtain the latent variables; Partial least squares path analysis is performed on the latent variables and the observed variables to obtain the path coefficients; The initial state vector is corrected based on the path coefficients to obtain the current state vector.

[0009] In some embodiments, the step of performing time-series prediction based on the current state vector and historical state vectors to obtain the predicted state vector includes: The state vectors in the historical time series are cached to construct a time series dataset that includes the current state vector and the historical state vectors within the most recent preset time period; The time series dataset is preprocessed to obtain preprocessed time series input data; The preprocessed time series input data is input into the preset time series prediction model to obtain the initial prediction vector; The initial prediction vector is standardized to obtain the predicted state vector.

[0010] In some embodiments, the process of performing state risk determination on the predicted state vector to obtain emotional and cognitive states includes: The predicted state vector is decomposed to obtain cognitive load and emotional state, which includes emotional arousal and emotional valence. The cognitive load is compared with a preset cognitive safety threshold, and the emotional valence is compared with a preset emotional normality threshold. If the cognitive load does not exceed the cognitive safety threshold and the emotional valence is within the range of the emotional normality threshold, it is determined to be a normal state. Alternatively, if the cognitive load exceeds the cognitive safety threshold, it is determined to be cognitive overload; if the emotional valence is not within the normal emotional threshold range and the emotional arousal deviates from the normal fluctuation range, it is determined to be a deterioration of the emotional state and judged as an abnormal state.

[0011] In some embodiments, the preset time series prediction model is a time series prediction model based on a long short-term memory network.

[0012] This application provides a cockpit occupant status monitoring device for operation in highly disruptive environments, comprising: The acquisition module is used to acquire physiological data, behavioral data, and environmental context data characterizing cabin physical disturbances and task load of the occupants. The processing module is used to perform feature extraction processing on the physiological data and behavioral data to obtain multimodal feature vectors; The processing module is further configured to fuse the multimodal feature vector with the environmental context data to obtain an initial state vector; The processing module is also used to perform latent variable coupling modeling on the initial state vector to obtain the current state vector; The prediction module is used to perform time-series prediction based on the current state vector and historical state vectors to obtain the predicted state vector. The processing module is also used to perform state risk judgment processing on the predicted state vector to obtain emotional and cognitive states, the emotional and cognitive states include normal states and abnormal states, the abnormal states include at least one of cognitive overload and emotional state deterioration. The processing module is also used to generate human-computer interaction intervention commands when the emotional and cognitive states are abnormal, and to control the vehicle system to execute the human-computer interaction intervention commands.

[0013] The computer device provided in this application includes a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements the method described in this application.

[0014] The computer-readable storage medium provided in this application embodiment stores a computer program thereon, which, when executed by a processor, implements the method described in this application embodiment.

[0015] This application provides a method and apparatus for monitoring the state of cockpit occupants in high-interference environments. It acquires physiological and behavioral data of the occupants, as well as environmental context data characterizing cockpit physical disturbances and task load. Features are extracted from the physiological and behavioral data to obtain multimodal feature vectors. These multimodal feature vectors are fused with the environmental context data to generate an initial state vector. The initial state vector is corrected through latent variable coupling modeling to obtain the current state vector. Based on the current and historical state vectors, time-series prediction is performed to obtain a predicted state vector. Risk assessment is performed on the predicted state vector, distinguishing between normal states and abnormal states such as emotional state deterioration and cognitive overload. In abnormal states, human-machine interaction intervention commands are generated and the onboard system is controlled to execute them. This application achieves the coordinated quantification of emotional and cognitive states in high-interference environments, possesses strong anti-interference capabilities and forward-looking early warning capabilities, improves human-machine collaborative combat effectiveness and occupant mission safety, and solves the technical problems mentioned in the background art. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating the implementation of a cockpit occupant status monitoring method for operations in highly disruptive environments, provided in this application embodiment; Figure 2 A schematic diagram illustrating an implementation process for obtaining an initial state vector, provided in an embodiment of this application; Figure 3 This is a schematic diagram of a cabin occupant's emotional and cognitive state monitoring device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0019] The following description of some technologies involved in the embodiments of this application is provided to aid understanding and should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, some descriptions of well-known functions and structures are omitted in the following description.

[0020] Traditional intelligent cockpit emotional interaction technologies generally lack a crew state-combat effectiveness coupling optimization mechanism for high-stress, high-interference mission scenarios. When crew members are in extreme environments such as severe vibration, high noise, and electromagnetic interference on platforms such as armored vehicles, existing technologies cannot quantify the coupling relationship between emotion and cognition and dynamically adjust system strategies, which can easily lead to a decrease in crew combat effectiveness, delay in mission response, and affect the accuracy and safety of tactical execution.

[0021] For example, the invention patent with publication number CN119577557A achieves digital human emotional interaction in various driving environments by integrating multimodal data, providing users with personalized feedback. Although it involves multimodal data application and emotional interaction, its core is to use a large model to achieve a more human-like interactive experience to optimize the comfort of ordinary driving, without addressing anti-interference design and combat effectiveness optimization for high-stress mission scenarios. It does not design anti-interference processing solutions for harsh physical environments, and its output is limited to comfort feedback such as audio and mode adjustments. In the extreme mission environment of armored vehicles, it cannot resist signal interference caused by strong vibrations and high noise, nor can it quantify the interaction between emotion and cognition, let alone support tactical-level dynamic allocation of human-machine functions.

[0022] For example, the invention patent with publication number CN120439964A identifies occupant emotions by integrating multimodal data such as facial images, voice, and physiological signals, and then adjusts cabin environmental parameters such as air conditioning and lighting. Although it involves the relationship between occupant emotions and the cabin system, its core is to improve the comfort of ordinary driving and riding through environmental adjustment, without addressing anti-interference design or combat effectiveness optimization for high-stress mission scenarios. It uses a general data preprocessing method, does not introduce environmental context data for signal decoupling, and the output is only the emotion category for basic environmental adjustment. In the extreme mission environment of armored vehicles, it cannot resist interference to obtain reliable status signals, nor can it quantify the interaction between emotion and cognition, let alone support tactical-level dynamic allocation of human-machine functions.

[0023] For example, the invention patent with publication number CN116101311A uses multimodal perception information from cameras, microphones, etc., for fusion analysis to provide voice or image interaction and reminder functions for ordinary smart cockpits. Although it focuses on the fusion application of multimodal information, its core is to generate basic human-machine interaction content to improve the convenience of ordinary driving and riding, without addressing anti-interference design and combat effectiveness optimization for high-stress mission scenarios. It does not emphasize the targeted overcoming of interference in extreme environments, and the output is only conventional interactive reminder information. In the extreme mission environment of armored vehicles, it cannot cope with the distortion of perception signals caused by strong electromagnetic fields and severe impacts, nor can it quantify the coupling relationship between emotion and cognition to predict effectiveness decay, and it is even more difficult to meet the high-precision requirements of human-machine collaborative decision-making in tactical missions.

[0024] Figure 1 This is a flowchart illustrating the implementation of a cockpit occupant status monitoring method for operations in highly disruptive environments, as provided in this application embodiment, including steps 101 to 107. Figure 1 This is merely one execution order shown in the embodiments of this application and does not represent the only execution order for a cockpit occupant status monitoring method for operations in highly disruptive environments. Where the final result can be achieved, Figure 1 The steps shown can be performed in parallel or in reverse order.

[0025] Step 101: Obtain physiological data, behavioral data, and environmental context data characterizing cabin physical disturbances and task load of the occupants.

[0026] In this embodiment, physiological data is collected by using a vehicle-mounted, non-invasive sensing device, including multi-channel electroencephalogram (EEG) signals, binocular eye movement data, skin conductance signals, and pulse wave signals.

[0027] Behavioral data is acquired synchronously by relying on eye-tracking sensors to obtain visual behavioral data such as occupant gaze, saccades, and blinks.

[0028] Environmental context data is read through the vehicle bus interface to provide information related to cockpit physical disturbances and mission load, including three-axis shock and vibration spectrum data from the vehicle's inertial navigation system, cabin noise and illumination data, and the current mission phase identifier (e.g., maneuver, reconnaissance, approach) and threat level obtained from the tactical mission management system.

[0029] Step 102: Perform feature extraction processing on physiological and behavioral data to obtain multimodal feature vectors.

[0030] In this embodiment of the application, the collected physiological and behavioral data are preprocessed and feature extracted to screen out features that are strongly correlated with emotional state and cognitive load.

[0031] The EEG feature extraction process involves bandpass filtering, power frequency interference removal, and artifact removal of the EEG signals within a preset frequency range. A sliding analysis is performed using a preset time window and step size to extract the power and power ratio of EEG bands reflecting cognitive load, and the power and asymmetry index of EEG bands reflecting emotional state.

[0032] Eye movement feature extraction identifies behaviors such as fixation, saccades, and blinking by performing event detection on eye movement data based on a preset speed threshold. Within a sliding time window, it statistically analyzes pupil change features reflecting cognitive load, saccade speed features reflecting attentional state, and fixation point features reflecting visual attention distribution.

[0033] Peripheral physiological feature extraction involves filtering the skin conductance signal to separate the skin conductance level, which represents the degree of tension, and statistically analyzing its features within a time window. Peak detection is performed on the pulse wave signal to generate a heartbeat interval sequence, and then the time-domain and frequency-domain features reflecting autonomic nervous activity are extracted.

[0034] After feature extraction is completed, different types of features are concatenated according to modality, and standardization processing to eliminate dimensional differences is performed to finally obtain a multimodal feature vector with uniform dimensions that can be directly input into the model.

[0035] Step 103: The multimodal feature vector and environmental context data are fused to obtain the initial state vector.

[0036] In this embodiment, a multimodal fusion model based on an attention mechanism is adopted, and environmental context data modulation feature weights are introduced to weaken the impact of environmental interference on state assessment.

[0037] Through independent feature mapping layers, feature subsets from different modalities in the multimodal feature vectors are mapped to a unified model embedding space to obtain initial embedding vectors for each modality. Simultaneously, environmental context data is encoded into environmental modulation vectors that match the dimensions of the embedding vectors. These are then combined with the initial embedding vectors for each modality, enabling the model to perceive and adapt to environmental disturbances. For example, when environmental context data indicates an extremely high vibration condition, the model automatically reduces the weight of high-frequency tremors in eye-tracking features through modulation, thereby avoiding misinterpreting pupil tremors caused by mechanical vibration as cognitive effort.

[0038] First, intramodal attention is calculated on the embedded vectors of each modality after environmental modulation to extract effective features that are strongly correlated with emotional-cognitive states within a single modality. Then, cross-modal attention weighting is applied to the optimized features of all modalities. Based on the core physiological modality features, the contribution weight of each modality feature is dynamically adjusted to strengthen effective information and suppress interfering information.

[0039] The cross-modal fusion features are input into the regression layer to generate a preliminary evaluation vector, i.e., the initial state vector, which includes three core dimensions: cognitive load level, emotional arousal, and emotional valence.

[0040] Step 104: Perform latent variable coupling modeling on the initial state vector to obtain the current state vector.

[0041] In this embodiment, a latent variable coupling relationship is constructed through a structural equation model to correct the initial state vector deviation and achieve the synergistic quantification of emotional and cognitive states.

[0042] Cognitive load, emotional arousal, and emotional valence in the initial state vector are defined as observed variables. These three are driven by a higher-order latent variable representing the overall psychophysiological state of the occupants, which reflects the interaction between emotion and cognition.

[0043] Partial least squares path analysis was used to fit the structural equation model with sample data in a vehicle-mounted scenario, calculate the path coefficients between latent variables and each observed variable, and verify the coupling strength and correlation between emotion and cognition.

[0044] By combining the latent variable scores obtained from the fitting with the observed variable errors after path coefficient correction, the initial state vector is theoretically corrected and standardized to finally obtain the current state vector that can truly reflect the emotional-cognitive synergistic state.

[0045] Step 105: Perform time series prediction based on the current state vector and historical state vectors to obtain the predicted state vector.

[0046] In this embodiment of the application, a time-series prediction model is used to capture the state evolution pattern, so as to realize the forward prediction of the occupant state in the near future.

[0047] Maintain a fixed-length historical state data buffer to store continuous state vectors within the most recent preset time period. Combine the current state vector with the historical state vector in chronological order to form a time-series input dataset.

[0048] Preprocessing time series datasets can be done by smoothing to suppress short-term random fluctuations or by using differencing to enhance the trend of state evolution and improve the stability of model predictions.

[0049] The preprocessed time series data is input into the time series prediction model (such as a long short-term memory network), and real-time environmental context data can be selectively introduced as auxiliary input to modulate the prediction process. By learning the evolution of historical states, the model outputs the state sequence within a preset time period in the future, i.e., the predicted state vector, and can also include prediction uncertainty information.

[0050] Step 106: Perform state risk assessment on the predicted state vector to obtain the emotional and cognitive states.

[0051] In this embodiment of the application, the normal state and abnormal state of the occupant are distinguished based on preset judgment rules and thresholds.

[0052] Based on a large amount of in-vehicle scenario sample data, a safe threshold for cognitive load, a normal range for emotional valence, and a reasonable fluctuation range for emotional arousal are preset.

[0053] The cognitive load level in the predicted state vector is compared with the safety threshold to determine whether there is a risk of cognitive overload; the emotional valence and emotional arousal are compared with the normal range and the reasonable fluctuation range, respectively, to determine whether there is a risk of emotional state deterioration.

[0054] If there is one or both of the risks of cognitive overload and deterioration of emotional state, it is judged as an abnormal state; if all dimensions are within the normal range, it is judged as a normal state.

[0055] Step 107: When the emotional and cognitive states are abnormal, generate human-computer interaction intervention commands and control the vehicle system to execute the human-computer interaction intervention commands.

[0056] In this embodiment of the application, an appropriate human-machine interaction intervention command is generated based on the specific risk type of the abnormal state, and the vehicle system is driven to execute it.

[0057] To address the risk of cognitive overload, instructions are generated to simplify the main combat situation display interface, suspend secondary information broadcasts, or suggest temporary transfer of mission permissions; to address the risk of worsening negative emotions, instructions are generated to trigger physiological regulation guidance (such as tactical breathing prompts), provide mission certainty feedback, or adjust the response thresholds of auxiliary systems.

[0058] If multiple risks exist, the above intervention instructions can be generated by combining them.

[0059] All commands are transmitted to the corresponding functional modules (such as display module, audio module, and task management module) through the vehicle system's standard interface. The drive module executes intervention operations within a preset time, forming a closed loop of monitoring-prediction-intervention to improve task efficiency and occupant safety.

[0060] This application's embodiments address the fragmentation problem of existing technologies by implementing a closed-loop process for monitoring emotional and cognitive states. It covers the entire chain from data acquisition, feature extraction, fusion modeling, time-series prediction, risk assessment, to intervention execution, avoiding the fragmented stage of existing monitoring methods that only focus on data collection or current state assessment. It forms a complete solution from data to intervention, adapting to the actual needs of common in-vehicle scenarios (such as long-distance driving and urban commuting). By simultaneously acquiring physiological, behavioral, and environmental data, and combining multimodal feature fusion, it solves the problem of one-sided state perception caused by relying on single data in existing technologies. It is particularly effective in dealing with interference such as bumps and changes in lighting in in-vehicle scenarios, making the assessment of emotional and cognitive states more comprehensive. Through latent variable coupling modeling, emotional states (such as anxiety and irritability) and cognitive states (such as cognitive overload and inattention) are quantified collaboratively, avoiding the shortcomings of existing technologies that assess emotions or cognition separately. This approach better reflects the correlation between human psychological and physiological states (e.g., negative emotions weaken cognitive abilities), resulting in more realistic assessment results. Based on time-series prediction, it can anticipate future cognitive overload and emotional deterioration, and generate intervention instructions in advance (such as simplified screen display and voice reminders), allowing passengers time to adjust, effectively reducing the risk of operational errors caused by deterioration of state, and ensuring driving and riding safety.

[0061] In the above Figure 1 Based on the above, this application embodiment also provides a schematic diagram of the implementation process for obtaining the initial state vector, as shown below. Figure 2 As shown, steps 201 to 205 are included: Step 201: Perform fully connected network encoding on the environmental context data to obtain the environmental modulation vector and perform modal linear embedding on the multimodal feature vector to obtain the initial embedding vector of the multimodality.

[0062] In this embodiment, the collected vehicle environment context data (including vehicle vibration data, cabin noise data, ambient illumination data, and driving scene coding and road condition level coding provided by the vehicle information system) is input into a small fully connected network with an adapted vector dimension. The environmental context data is feature-mapped and dimension-unified through a limited number of hidden layers, ultimately outputting an environmental modulation vector with the same dimension as the subsequent multimodal feature embedding vector. The purpose of this encoding process is to transform unstructured or multidimensional environmental information into a numerical vector that can be directly combined with the feature vector, enabling the model to perceive and adapt to environmental interference in advance (such as physiological signal artifacts that may be caused by strong vehicle vibration).

[0063] For the standardized multimodal feature vectors, they are split into EEG feature subsets, eye-tracking feature subsets, and peripheral physiological feature subsets according to modality category. To ensure that the features of different modalities are in a unified data space, three independent linear layers are used to perform linear embedding processing on the three feature subsets respectively. Each linear layer maps the feature dimension of the corresponding modality to the same model embedding space as the environmental modulation vector, and finally obtains the initial embedding vectors of EEG modality, eye-tracking modality, and peripheral physiological modality, which together constitute the multimodal initial embedding vector set.

[0064] Step 202: Add the initial embedding vector of the multimodal mode to the environment modulation vector element by element to obtain the modal embedding vector after environment modulation.

[0065] In this embodiment, the obtained multimodal initial embedding vector is added element-wise to the environmental modulation vector. Specifically, the initial embedding vector of the EEG modality is added element-wise to the environmental modulation vector to obtain the environmentally modulated EEG embedding vector; similarly, the initial embedding vectors of the eye-tracking modality and the peripheral physiological modality are added element-wise to the environmental modulation vector to obtain the environmentally modulated eye-tracking embedding vector and the environmentally modulated peripheral physiological embedding vector, respectively. The core function is to inject environmental information as prior knowledge into each modal feature. For example, when environmental context data shows that the vehicle is in a state of severe vibration, the environmental modulation vector will be pre-corrected for modal features such as eye-tracking and peripheral physiological features that are susceptible to vibration interference by adding them element-wise, enabling the model to actively distinguish feature fluctuations caused by environmental interference from feature fluctuations caused by changes in the actual state of the occupants in subsequent feature processing.

[0066] Step 203: Perform intra-modal self-attention calculation on the modal embedding vector after environmental modulation to obtain the intra-modal optimized feature vector.

[0067] In this embodiment, intramodal self-attention calculation is performed on each modal embedding vector after environmental modulation. Taking the environmentally modulated EEG embedding vector as an example, it is input into a multi-head self-attention layer. By calculating the correlation between different features within the EEG modality, the feature weights that are more valuable for assessing emotional-cognitive states are automatically strengthened, while redundant or interfered feature weights are weakened. Similarly, the same intramodal self-attention calculation is performed on the environmentally modulated eye-tracking embedding vector and peripheral physiological embedding vector. The effective features within each modality are refined and enhanced. For example, the correlation between pupil diameter changes and cognitive load in the eye-tracking modality is strengthened, while abnormal signals caused by accidental blinking are weakened. Finally, the optimized feature vectors within the EEG modality, the optimized feature vectors within the eye-tracking modality, and the optimized feature vectors within the peripheral physiological modality are output, which together constitute the set of optimized feature vectors within the modality.

[0068] Step 204: Weight the intramodal optimized feature vectors to obtain multimodal collaborative fusion feature vectors.

[0069] In this embodiment, the optimized feature vectors within the EEG modality, the optimized feature vectors within the eye-tracking modality, and the optimized feature vectors within the peripheral physiological modality are first concatenated to form a cross-modal feature set.

[0070] The query criterion is the optimized feature vector within the EEG modality, and all feature vectors in the cross-modal feature set are used as key-value pairs.

[0071] By using cross-attention calculation, the contribution weights of each modality feature in the fusion process are dynamically adjusted. For example, when environmental context data shows that sudden changes in cabin lighting lead to a decrease in the reliability of eye-tracking features, the model will automatically reduce the weight of eye-tracking modal features while increasing the weight of modal features with stronger anti-interference capabilities, such as EEG and peripheral physiology. Conversely, when the environment is stable, the model will distribute the weights of each modality evenly.

[0072] Through weighted processing, effective features from different modalities are synergistically integrated, ultimately outputting a multimodal synergistic fusion feature vector.

[0073] Step 205: Fit the multimodal collaborative fusion feature vector to obtain the initial state vector.

[0074] In this embodiment, the multimodal collaborative fusion feature vector is input into a fully connected layer, through which regression fitting is performed. The fully connected layer maps the high-dimensional multimodal collaborative fusion feature vector to a low-dimensional numerical vector based on the requirements of emotional-cognitive state assessment. During the fitting process, the parameters of the fully connected layer are optimized through a preset training objective (such as minimizing the error between the fitted result and the true state label) to ensure that the output vector accurately reflects the passenger's current preliminary emotional-cognitive state. Finally, the fully connected layer outputs a vector containing quantified values ​​of three dimensions: cognitive load level, emotional arousal, and emotional valence; this is the initial state vector.

[0075] This application's embodiments generate modulation vectors by encoding environmental context data through a fully connected network, injecting environmental interference as prior knowledge into multimodal features. This addresses the feature bias problem caused by existing fusion methods neglecting environmental influences, making the fusion results more resistant to dynamic interference in vehicle scenarios. First, effective single-modal features are extracted through intramodal self-attention, and then multimodal information is integrated through cross-modal weighted integration. This avoids the interference of invalid information caused by equal weighting in existing fusion methods, making the multimodal collaborative fusion features more accurate.

[0076] In some embodiments, physiological data includes electroencephalogram (EEG) data and peripheral physiological data. Feature extraction processing is performed on the physiological data and behavioral data to obtain multimodal feature vectors, including: bandpass filtering, power frequency notch filtering, and artifact removal processing on the EEG data to obtain EEG features.

[0077] Specifically, the raw EEG data is input into a bandpass filter module, and the filter frequency range is set to 1-45 Hz. The filter operation removes low-frequency baseline drift below 1 Hz (such as signal shift caused by slight shaking of the occupant's body) and high-frequency noise above 45 Hz (such as high-frequency interference generated by in-vehicle electronic devices) from the EEG signal, while retaining EEG band signals that are strongly correlated with cognitive and emotional states.

[0078] Considering that in-vehicle electrical equipment (such as air conditioning and central control system) is prone to 50 Hz power frequency interference, the EEG signal after bandpass filtering is subjected to 50 Hz power frequency notch processing, and the interference signal of this specific frequency is suppressed by narrowband filtering.

[0079] Independent component analysis was used to separate and remove artifacts from the preprocessed EEG signals. This method can identify and separate electrooculography (EOG) artifacts (such as signal fluctuations caused by blinking) and electromyography (EMG) artifacts (such as signal interference caused by facial muscle activity) mixed in the EEG signals, preserving the pure EEG signal components.

[0080] For the EEG signals after artifact removal, a sliding analysis method with a time window of 4 seconds and a step size of 2 seconds was used to extract two types of core features: one is the feature reflecting cognitive load (such as the average power of the theta band in the prefrontal region and the theta / β power ratio), and the other is the feature reflecting emotional state (such as the power of the alpha band in the frontal region and the frontal alpha asymmetry index). The above features together constitute the EEG features.

[0081] Furthermore, the behavioral data is statistically extracted and processed to obtain eye-tracking features.

[0082] Specifically, noise is first removed from the raw eye-tracking data (e.g., filtering out abnormal coordinate values ​​caused by brief device occlusion), and then event detection is performed based on a velocity threshold. Velocity criteria are set (e.g., eye movement velocity exceeding 30° / s is considered saccade behavior, and velocity below 5° / s with a duration exceeding 100 milliseconds is considered fixation behavior), thereby identifying the three core behaviors in the eye-tracking data: fixation, saccades, and blinking.

[0083] Using a 4-second sliding time window consistent with EEG feature extraction, the detected eye movement behaviors were statistically analyzed. The sliding mean and standard deviation of pupil diameter (reflecting changes in cognitive load, such as increased pupil diameter and increased fluctuations during cognitive overload), the average peak velocity of saccades (reflecting attention shift efficiency), and the information entropy of fixation point distribution (reflecting the degree of visual attention distraction; higher entropy values ​​indicate greater attention distraction) were statistically analyzed. These statistical results together constitute the eye movement features.

[0084] Furthermore, signal filtering and feature statistical processing are performed on the peripheral physiological data to obtain the processed peripheral physiological data.

[0085] Specifically, the raw skin conductance signal is subjected to low-pass filtering to remove high-frequency noise and separate the skin conductance level that can stably reflect the occupant's stress level; the mean of the skin conductance level within a 4-second sliding time window is calculated as a skin conductance-related feature.

[0086] Peak detection is performed on the raw photoplethysmography (PPG) signal. By identifying the peak position of the pulse wave corresponding to each heartbeat, a successive heartbeat interval sequence is generated. Based on this sequence, two types of features are further extracted: one is time-domain features (such as the average heartbeat interval and the standard deviation of the heartbeat interval, reflecting the stability of the heartbeat), and the other is frequency-domain features (such as the high-frequency power and low-frequency power ratio of the sequence, reflecting the balance between the sympathetic and parasympathetic nervous systems).

[0087] By integrating the above-mentioned skin conductance-related features with photoplethysmography-derived features, processed peripheral physiological data were obtained.

[0088] Furthermore, the EEG features, eye movement features, and processed peripheral physiological data are spliced ​​together to obtain a multimodal feature vector.

[0089] Specifically, the obtained EEG features, eye-tracking features, and processed peripheral physiological data are concatenated in modal order from EEG features to eye-tracking features and then to peripheral physiological data to form an initial feature set. Considering the difference in the dimensions of the three types of features, Z-score standardization is performed on the initial feature set. By calculating the mean and standard deviation of each feature, all feature values ​​are converted into standardized values ​​with a mean of 0 and a standard deviation of 1, eliminating the influence of dimensional differences on subsequent model inputs; finally, the concatenated and standardized feature set is the multimodal feature vector.

[0090] This application's embodiments utilize bandpass filtering, power frequency notch filtering, and artifact removal for EEG data to filter out interference from in-vehicle electrical appliances and human artifacts, solving the problem of high feature noise caused by directly using raw data. Filtering and separating peripheral physiological data extracts features that stably reflect stress states and autonomic nervous activity, ensuring that each modality's features have attributes strongly correlated with emotion and cognition. By using a unified sliding time window to statistically analyze eye movement and peripheral physiological features, combined with Z-score standardization, the problem of large differences in feature dimensions across different modalities is addressed, ensuring a unified format and comparable values ​​for multimodal feature vectors. This provides highly adaptable input data for multimodal fusion, avoiding fusion bias caused by inconsistent feature formats.

[0091] In some embodiments, latent variable coupling modeling is performed on the initial state vector to obtain the current state vector, including: feature extraction processing of the initial state vector to obtain the observed variables.

[0092] Specifically, the initial state vector is a preliminary affective-cognitive assessment vector after multimodal fusion, containing three key dimensions: cognitive load level (quantifying the occupant's information processing stress and attention allocation status), emotional arousal (quantifying the occupant's emotional excitement level, such as calm, tension, or irritability), and affective valence (quantifying the occupant's emotional positive or negative tendency, such as pleasure, neutrality, or anxiety). Based on this, when extracting features from the initial state vector, the quantified values ​​of the above three dimensions are directly used as independent features, defined as the observed variables of cognitive load, emotional arousal, and affective valence, respectively. These three together constitute the set of observed variables required for latent variable coupling modeling. The core purpose of the extraction process is to select core indicators that can directly reflect cognitive and affective states.

[0093] Furthermore, a latent variable coupling model is constructed by inputting the observed variables into the latent variable coupling model to obtain the latent variables.

[0094] Specifically, a latent variable coupling model is constructed using structural equation modeling. The core logic of the model is that the observed variables are driven by higher-order latent variables. The observed variables of cognitive load, emotional arousal, and affective valence are not independent but are jointly influenced by a latent variable of the overall psychological and physiological state of the occupants, which cannot be directly measured. This latent variable comprehensively reflects the interaction between emotion and cognition (e.g., when the overall psychological state is tense, it will lead to both increased cognitive load and increased emotional arousal, with affective valence leaning towards the negative). In terms of model structure, the latent variable of the overall psychological and physiological state of the occupants is treated as an exogenous latent variable, and the three observed variables are treated as endogenous observed variables. A mapping relationship is established from the latent variable to each observed variable (i.e., changes in the latent variable will affect the values ​​of each observed variable through specific correlation rules).

[0095] The three observed variables extracted above are input into the constructed latent variable coupling model. Through the model's measurement equations, combined with sample data from in-vehicle scenarios (covering observed variable data for different driving durations, road conditions, and occupant states), a quantitative score of the latent variable representing the overall psychological and physiological state of the occupant is calculated (i.e., the specific numerical representation of the latent variable). This latent variable score is not calculated in isolation, but is derived based on the synergistic relationship of the three observed variables, reflecting the overall synergistic state of the occupant's emotions and cognition, rather than a single-dimensional summation of states.

[0096] Furthermore, partial least squares path analysis is performed on the latent variables and observed variables to obtain the path coefficients.

[0097] Specifically, partial least squares path analysis was used to quantify and verify the relationship between latent variables and observed variables and to calculate the coefficients.

[0098] Collect a large amount of sample data in vehicle scenarios (such as the latent variable scores and corresponding observed variable values ​​of different passengers in scenarios such as long-distance driving, urban congestion, and severe weather), input the data into the latent variable coupling model, and fit the model through the partial least squares algorithm to minimize the error between the predicted value of the observed variable output by the model and the actual sample data.

[0099] During the fitting process, standardized path coefficients from the latent variables of the overall psychophysiological state of the occupants to each observed variable are calculated simultaneously. For example, the path coefficient from the latent variable to the observed variable of cognitive load quantifies the strength of the influence of changes in the overall psychophysiological state on cognitive load; the path coefficient from the latent variable to the observed variable of affective valence quantifies the influence of changes in the overall state on the positive or negative tendency of emotions. The numerical range of the path coefficients is usually [-1, 1], and the larger the absolute value, the stronger the association (e.g., a path coefficient of 0.8 means that for every unit change in the latent variable, the corresponding observed variable changes by approximately 0.8 units).

[0100] Furthermore, the initial state vector is corrected based on the path coefficients to obtain the current state vector.

[0101] Specifically, the residuals of each observed variable in the initial state vector are first calculated (i.e., the difference between the actual value of the observed variable and the value predicted by the model through latent variables). These residuals reflect the random errors that may exist in the initial state vector due to single data fusion (such as the bias of observed variables caused by brief environmental interference with eye-tracking features at a certain moment). The residuals are then weighted and corrected using the path coefficients obtained above. Observed variables with larger absolute values ​​of path coefficients have a greater impact on the overall state assessment, and therefore require higher correction weights to reduce the interference of random errors on the results.

[0102] The corrected residuals of the observed variables are integrated with the latent variable scores of the overall psychological and physiological state of the occupants. This ensures that the final state vector retains the specific quantitative information of each dimension while incorporating the influence of the overall coordinated state. For example, if the latent variable scores indicate overall psychological tension and the path coefficient from the latent variables to emotional arousal is high, the quantitative value of emotional arousal needs to be appropriately adjusted during the correction process to better align with the overall state logic.

[0103] The integrated vector is standardized to eliminate potential dimensional differences that may arise during the correction process, ultimately yielding a current state vector that accurately reflects the emotional-cognitive synergy. Compared to the initial state vector, the current state vector is better able to avoid biases from single data points and better reflects the actual psychological and physiological state of the occupants.

[0104] This application's embodiments construct the association between latent variables and observed variables using structural equation modeling, resolving the potential random biases that may exist in generating initial vectors solely through data fusion. This upgrades state assessment from data-driven to data-plus-theory-driven, resulting in more reliable results. By calculating the association strength between latent and observed variables using path coefficients, the interaction patterns between emotion and cognition are clarified, avoiding the ambiguity of subjective judgments of the relationship between emotion and cognition in existing technologies. This makes the coupled modeling process quantifiable and verifiable, ensuring that the current state vector truly reflects the collaborative state of emotion and cognition. From observed variable extraction and model construction to path coefficient correction, all are based on explicit statistical methods, avoiding subjectivity in the correction process and ensuring the stability of the initial vector correction results under different samples and scenarios.

[0105] In some embodiments, time series prediction is performed based on the current state vector and historical state vectors to obtain the predicted state vector, including: caching the state vectors on the historical time series and constructing a time series dataset including the current state vector and historical state vectors within the most recent preset time period.

[0106] Specifically, the onboard system maintains a fixed-length First-In-First-Out (FIFO) data buffer, the capacity of which is determined by a preset duration and the state vector generation interval. For example, if the system is preset to store historical state vectors from the most recent 3 minutes, and the state vectors are updated at a frequency of one every 2 seconds, then the buffer can store 90 historical state vectors (3 minutes × 60 seconds / 2 seconds = 90), ensuring sufficient coverage of historical evolution cycles.

[0107] Whenever a new current state vector is generated, it is stored at the end of the buffer, while the oldest historical state vector at the beginning of the buffer is removed (if the buffer is full), and the buffer always stores the most recent consecutive state vectors within the preset time. If the buffer is not full, the current state vector is stored directly until the preset capacity is reached.

[0108] All historical state vectors stored in the buffer are arranged in ascending order of generation time. The latest current state vector is then appended to the end of the sequence, forming a time-series dataset with dimensions of (number of historical vectors + 1) × state vector dimension. The time-series dataset fully records the evolution of crew state from the most recent preset start time to the current time.

[0109] Furthermore, the time series dataset is preprocessed to obtain preprocessed time series input data.

[0110] Specifically, if the time-series dataset contains short-term state fluctuations caused by accidental factors (such as a passenger briefly looking down or sudden external noise), a moving average method is used for processing. A sliding window size is set (e.g., 3 consecutive state vectors form one window). The average value is calculated for each dimension feature (cognitive load, emotional arousal, and emotional valence) within the window. This average value replaces the state vector value at the center of the window. The entire dataset is processed by sliding the window sequentially, thereby reducing the masking of state trends by random fluctuations.

[0111] If the time-series dataset exhibits a clear trend of state evolution (such as a gradual increase in cognitive load and a gradual decrease in affective valence during long-distance driving), the first-order difference method is used for processing. The difference between two adjacent state vectors (the latter vector value minus the former vector value) is calculated to obtain a difference sequence that reflects the rate of state change, transforming absolute state values ​​into relative change values, making it easier for the model to capture the upward / downward trend of the state.

[0112] Based on the actual characteristics of the time series dataset, one or a combination of the above processing methods (such as smoothing first and then differencing) are selected to finally obtain preprocessed time series input data that eliminates noise and highlights trends.

[0113] Furthermore, the preprocessed time series input data is input into a preset time series prediction model to obtain the initial prediction vector.

[0114] Specifically, a Long Short-Term Memory (LSTM) network is used as the time series prediction model. The model learns the evolution patterns of historical states and outputs a sequence of future states.

[0115] The pre-defined LSTM model consists of an LSTM layer and a fully connected output layer. The LSTM layer uses a single unidirectional LSTM structure with 128 hidden units to capture long-term dependencies of states in the time-series input data. The fully connected output layer maps the output of the LSTM layer to a sequence of state vectors for a pre-defined future prediction duration (e.g., 1-5 minutes in the future).

[0116] The preprocessed time-series input data is directly input into the LSTM model. At the same time, real-time environmental context data (such as scene information obtained from vehicle navigation indicating that there will be heavy rain in the next 5 minutes before entering a congested section) can be selectively encoded into environmental feature vectors that match the dimension of the time-series input data and injected into the cell state of the LSTM model. By modulating the prediction process through environmental information (such as predicting that congested sections will accelerate the increase in cognitive load), the degree to which the prediction fits the actual scene is improved.

[0117] The model uses parameters formed during early training (using historical time series data plus subsequent real states as training samples, with the goal of minimizing the error between the predicted state and the real state) to calculate the input time series data and output a continuous state vector sequence within a preset time period in the future (e.g., one state vector every 2 seconds in the next 5 minutes). This sequence is the initial prediction vector, which contains preliminary quantitative values ​​of cognitive load, emotional arousal, and emotional valence at each future moment.

[0118] Furthermore, the initial prediction vector is standardized to obtain the predicted state vector.

[0119] Specifically, the Z-score standardization method, consistent with the current state vector of the multimodal feature vector, is adopted. The mean and standard deviation of each feature dimension (cognitive load, emotional arousal, and emotional valence) in the initial prediction vector are calculated in the historical training samples. Each feature value in the initial prediction vector is converted into a standardized value of (feature value - mean) / standard deviation to eliminate the differences in the dimensions of different features.

[0120] Based on the model's output capabilities, a prediction interval or uncertainty index (such as an 85% probability that cognitive load will exceed the threshold in the next 2 minutes) can be attached to the standardized vector. This information comes from the statistical analysis of prediction errors during the LSTM model training process.

[0121] After standardization, a predictive state vector with a unified format and consistent dimensions is obtained, which can be directly used for risk assessment and fully reflects the evolution trend of the occupant's emotional and cognitive state within a preset time period.

[0122] This application's embodiments store historical state vectors of the most recent preset duration in a fixed-length FIFO buffer, addressing the trend capture bias problem caused by incomplete historical data in existing prediction methods. This ensures that the time-series dataset fully reflects the state evolution from the past to the present, providing continuous trend evidence for the prediction model. Smoothing processes suppress short-term fluctuations (such as temporary abnormalities in cognitive load caused by a passenger's occasional head-down movement) and differencing processes enhance long-term trends (such as the gradual increase in cognitive load during long-distance driving), resolving the issues of high prediction noise and unclear trends caused by directly using raw time-series data in existing prediction methods. This makes the preprocessed time-series input data more closely match the requirements of the prediction model. Standardization of the initial prediction vector ensures that the prediction results are in a unified format and have consistent dimensions with the current state vector, addressing the problems of inconsistent prediction result formats in existing prediction methods that make comparison with the current state impossible and risk assessment difficult.

[0123] In some embodiments, the predicted state vector is subjected to state risk determination processing to obtain emotional and cognitive states, including: decomposing the predicted state vector to obtain cognitive load and emotional state, wherein the emotional state includes emotional arousal and emotional valence.

[0124] Specifically, the predicted state vector is decomposed into dimensions in a fixed order from cognitive load to emotional arousal and then to emotional valence, and the quantitative values ​​of the three dimensions are extracted respectively.

[0125] Cognitive load quantification reflects the information processing pressure of occupants at future moments (e.g., 5.2 indicates moderate to high information processing pressure, and the higher the value, the greater the pressure).

[0126] The emotional arousal metric reflects the level of emotional excitement of the occupants at a future moment (e.g., 3.1 indicates mild excitement, while a value that is too low indicates depression and a value that is too high indicates tension).

[0127] The emotional valence quantification value reflects the positive or negative emotional tendency of the occupants at future moments (e.g., 0.6 indicates a positive tendency, values ​​below 0 indicate a negative tendency, and values ​​above 0 indicate a positive tendency).

[0128] The extracted cognitive load quantification value is classified separately as a cognitive state indicator, while the emotional arousal quantification value and the emotional valence quantification value are jointly classified as an emotional state indicator, forming two categories of judgment criteria: cognitive load and emotional state.

[0129] Furthermore, the cognitive load is compared with the preset cognitive safety threshold and the emotional valence is compared with the preset emotional normality threshold. If the cognitive load does not exceed the cognitive safety threshold and the emotional valence is within the range of the emotional normality threshold, it is judged as a normal state.

[0130] Specifically, cognitive load data of occupants under normal driving conditions in different in-vehicle scenarios (such as long-distance constant speed driving, urban traffic congestion, driving in heavy rain, and driving at night with insufficient lighting) will be collected (a total of no less than 1,000 valid samples will be collected). The maximum value of cognitive load in the samples will be counted (e.g., the statistical result is 6.0), and this maximum value will be set as the cognitive safety threshold. That is, when the cognitive load quantification value is ≤6.0, it is determined that the cognitive state is not overloaded.

[0131] Based on the sample data of normal driving conditions mentioned above, the mean (e.g., 0.2) and standard deviation (e.g., 0.3) of affective valence are statistically analyzed. The range of mean ± 1 standard deviation is set as the threshold for normal affective valence (i.e., 0.2-0.3=-0.1 to 0.2+0.3=0.5). That is, when the quantitative value of affective valence is within the range of [-0.1, 0.5], it is judged as normal affective tendency.

[0132] The minimum (e.g., 2.0) and maximum (e.g., 4.0) emotional arousal values ​​were statistically analyzed in samples of normal driving conditions. This range [2.0, 4.0] was defined as the normal fluctuation range of emotional arousal. That is, when the quantified value of emotional arousal falls within this range, it is considered that the level of emotional excitement is reasonable and there is no excessive depression or excessive tension.

[0133] Furthermore, if the cognitive load exceeds the cognitive safety threshold, it is determined to be cognitive overload; if the emotional valence is not within the normal emotional threshold range and the emotional arousal deviates from the normal fluctuation range, it is determined to be a deterioration of the emotional state and judged as an abnormal state.

[0134] Specifically, the normal state is determined as follows: when both conditions are met, the state is considered normal: the cognitive load quantification value is ≤ the cognitive safety threshold (e.g., ≤ 6.0): this indicates that the information processing pressure of the occupant in the future is within a safe range and will not lead to operational errors due to cognitive overload; the emotional valence quantification value is in the emotional normal threshold range (e.g., [-0.1, 0.5]): this indicates that the occupant's emotional tendency in the future is neutral or slightly positive, with no obvious negative emotions.

[0135] Example: If the predicted vector is decomposed into cognitive load of 5.5, emotional arousal of 3.2, and emotional valence of 0.3, then the cognitive load does not exceed the threshold and the emotional valence is within the normal range, and it is judged as a normal state.

[0136] Abnormal State Determination: A state is determined to be abnormal if any of the following conditions are met, and the specific risk type is further distinguished. Cognitive Overload Determination: Only a cognitive load quantification value > cognitive safety threshold (e.g., > 6.0) is required; emotional state is not considered. This is common in scenarios where occupants need to process a large amount of information in the future (e.g., approaching a series of complex intersections, or experiencing attention fatigue in the later stages of long-distance driving). Example: A predicted cognitive load of 6.8 and an emotional valence of 0.4 constitute cognitive overload. Deterioration of Emotional State Determination: Two conditions must be met simultaneously: an emotional valence quantification value < the lower limit of the normal emotional threshold (e.g., < -0.1) or > the upper limit of the normal emotional threshold (e.g., > 0.5; exceeding the upper limit often indicates emotional instability due to over-excitement); and an emotional arousal quantification value < the lower limit of the normal fluctuation range (e.g., < 2.0, indicating depressed mood) or > the upper limit of the normal fluctuation range (e.g., > 4.0, indicating tension / irritability). This situation is common when the scenario deteriorates in the future (such as entering a long-term traffic jam or sudden severe weather). Example: The predicted emotional valence is -0.3 and the emotional arousal is 4.5, which is judged as a deterioration of the emotional state.

[0137] The final judgment result (normal state - abnormal state - cognitive overload - deterioration of emotional state) is used as the emotional and cognitive state output, which is directly used to generate subsequent human-computer interaction intervention instructions.

[0138] This application's embodiments decompose the prediction vector into cognitive load, emotional arousal, and emotional valence, clarifying the dimensions for determining cognitive and emotional states. This solves the problem of ambiguous criteria for judging emotional and cognitive states, making the judgment basis clearer. The cognitive safety threshold, normal emotional threshold, and normal fluctuation range of arousal are all statistically derived from a large amount of sample data in in-vehicle scenarios (such as the maximum cognitive load and the mean ± standard deviation of emotional valence under normal driving conditions), avoiding judgment bias caused by subjectively set thresholds and ensuring that the judgment criteria are consistent and objective for different occupants and different scenarios.

[0139] While this application provides method operation steps as shown in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive labor. The order of steps listed in this embodiment is merely one possible execution order among many and does not represent the only execution order. In actual device or client product execution, the method can be executed sequentially according to this embodiment or the accompanying drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0140] like Figure 3 As shown in the illustration, this application also provides a cockpit occupant status monitoring device 300 for operation in highly disruptive environments. The device includes: The acquisition module 301 is used to acquire physiological data, behavioral data, and environmental context data characterizing cabin physical disturbances and task load of the occupants.

[0141] The processing module 302 is used to perform feature extraction processing on physiological and behavioral data to obtain multimodal feature vectors.

[0142] The processing module 302 is also used to fuse the multimodal feature vectors with the environmental context data to obtain the initial state vector.

[0143] The processing module 302 is also used to perform latent variable coupling modeling on the initial state vector to obtain the current state vector.

[0144] The prediction module 303 is used to perform time-series prediction based on the current state vector and the historical state vector to obtain the predicted state vector.

[0145] The processing module 302 is also used to perform state risk judgment processing on the predicted state vector to obtain emotional and cognitive states, which include normal states and abnormal states. Abnormal states include at least one of cognitive overload and emotional state deterioration.

[0146] The processing module 302 is also used to generate human-computer interaction intervention commands when the emotional and cognitive states are abnormal, and to control the vehicle system to execute the human-computer interaction intervention commands.

[0147] In some embodiments, the processing module 302 is further configured to perform fully connected network encoding on the environmental context data to obtain an environmental modulation vector and perform modal linear embedding on the multimodal feature vector to obtain an initial embedding vector for the multimodality.

[0148] The processing module 302 is also used to perform element-wise addition of the initial embedding vector of the multimodal mode and the environmental modulation vector to obtain the modal embedding vector after environmental modulation.

[0149] The processing module 302 is also used to perform intramodal self-attention calculation on the modal embedding vector after environmental modulation to obtain the intramodal optimized feature vector.

[0150] The processing module 302 is also used to perform weighted processing on the intramodal optimized feature vector to obtain a multimodal collaborative fusion feature vector.

[0151] The processing module 302 is also used to fit the multimodal collaborative fusion feature vector to obtain the initial state vector.

[0152] In some embodiments, the processing module 302 is further configured to perform bandpass filtering, power frequency notch filtering, and artifact removal processing on the EEG data to obtain EEG features.

[0153] The processing module 302 is also used to perform statistical extraction processing on behavioral data to obtain eye movement features.

[0154] The processing module 302 is also used to perform signal filtering and separation and feature statistical processing on peripheral physiological data to obtain processed peripheral physiological data.

[0155] The processing module 302 is also used to splice the EEG features, eye movement features and processed peripheral physiological data to obtain a multimodal feature vector.

[0156] In some embodiments, the processing module 302 is further configured to perform feature extraction processing on the initial state vector to obtain the observed variables.

[0157] The processing module 302 is also used to construct a latent variable coupling model by inputting the observed variables into the latent variable coupling model to obtain the latent variables.

[0158] The processing module 302 is also used to perform partial least squares path analysis on latent variables and observed variables to obtain path coefficients.

[0159] The processing module 302 is also used to correct the initial state vector based on the path coefficient to obtain the current state vector.

[0160] In some embodiments, the processing module 302 is further configured to cache the state vectors on the historical time series and construct a time series dataset including the current state vector and the historical state vectors within the most recent preset time period.

[0161] The processing module 302 is also used to preprocess the time series dataset to obtain preprocessed time series input data.

[0162] The prediction module 303 is also used to input the preprocessed time series input data into a preset time series prediction model to obtain an initial prediction vector.

[0163] The prediction module 303 is also used to standardize the initial prediction vector to obtain the predicted state vector.

[0164] In some embodiments, the processing module 302 is further configured to decompose the predicted state vector to obtain cognitive load and emotional state, wherein the emotional state includes emotional arousal and emotional valence.

[0165] The prediction module 303 is also used to compare the cognitive load with the preset cognitive safety threshold and the emotional valence with the preset emotional normal threshold. If the cognitive load does not exceed the cognitive safety threshold and the emotional valence is within the range of the emotional normal threshold, it is judged as a normal state.

[0166] The prediction module 303 is also used to determine cognitive overload when the cognitive load exceeds the cognitive safety threshold. When emotional valence is not within the normal emotional threshold range and emotional arousal deviates from the normal fluctuation range, it is determined that the emotional state has deteriorated and is classified as an abnormal state.

[0167] Some modules in the apparatus described in this application can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0168] The apparatus or module described in the above embodiments can be implemented by a computer chip or physical entity, or by a product with a certain function. For ease of description, the above apparatus is described by dividing it into various modules according to their functions. When implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware. Of course, a module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0169] The methods, apparatus, or modules described in this application can be implemented in a computer-readable program code manner. The controller can be implemented in any suitable manner, such as a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of a memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code manner, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included within it for implementing various functions can also be considered as structures within the hardware component. Alternatively, the device used to implement various functions can be viewed as either a software module that implements the method or a structure within a hardware component.

[0170] This application also provides an apparatus, the apparatus comprising: a processor; a memory for storing processor-executable instructions; wherein, when the processor executes the executable instructions, it implements the method described in this application.

[0171] This application also provides a non-volatile computer-readable storage medium storing a computer program or instructions thereon, which, when executed, enables the method described in this application embodiment to be implemented.

[0172] Furthermore, in the various embodiments of the present invention, each functional module can be integrated into a processing module, or each module can exist independently, or two or more modules can be integrated into a single module.

[0173] The aforementioned storage media include, but are not limited to, Random Access Memory (RAM), Read-Only Memory (ROM), Cache, Hard Disk Drive (HDD), or Memory Card. The memory can be used to store computer program instructions.

[0174] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, or it can be embodied in the process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0175] The various embodiments described in this specification are presented in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. All or part of this application can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, mobile communication terminals, multiprocessor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.

[0176] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of this application.

Claims

1. A method for monitoring the status of cabin occupants during operations in highly disruptive environments, characterized in that, include: Acquire physiological and behavioral data of occupants, as well as environmental context data characterizing cabin physical disturbances and task load; Feature extraction processing is performed on the physiological and behavioral data to obtain multimodal feature vectors; The multimodal feature vector and the environmental context data are fused to obtain an initial state vector; The initial state vector is subjected to latent variable coupling modeling to obtain the current state vector; Based on the current state vector and the historical state vector, a time series prediction is performed to obtain the predicted state vector; The predicted state vector is subjected to state risk assessment to obtain emotional and cognitive states, which include normal and abnormal states. The abnormal states include at least one of cognitive overload and emotional state deterioration. When the emotional and cognitive states are abnormal, a human-computer interaction intervention command is generated, and the vehicle system is controlled to execute the human-computer interaction intervention command.

2. The method according to claim 1, characterized in that, The process of fusing the multimodal feature vector with the environmental context data to obtain the initial state vector includes: The environmental context data is encoded using a fully connected network to obtain an environmental modulation vector, and the multimodal feature vector is subjected to modal linear embedding to obtain an initial embedding vector for the multimodality. The initial embedding vector of the multimodal mode is added element by element to the environment modulation vector to obtain the environment-modulated mode embedding vector; Intramodal self-attention calculation is performed on the modal embedding vector after environmental modulation to obtain the intramodal optimized feature vector; The intra-modal optimized feature vectors are weighted to obtain multi-modal collaborative fusion feature vectors; The initial state vector is obtained by fitting the multimodal collaborative fusion feature vector.

3. The method according to claim 1, characterized in that, The physiological data includes electroencephalogram (EEG) data and peripheral physiological data. The feature extraction process performed on the physiological and behavioral data to obtain a multimodal feature vector includes: The EEG data is processed by bandpass filtering, power frequency notch filtering, and artifact removal to obtain EEG features; The behavioral data is statistically extracted to obtain eye movement features; The peripheral physiological data are subjected to signal filtering and separation and feature statistical processing to obtain the processed peripheral physiological data. The EEG features, eye movement features, and processed peripheral physiological data are spliced ​​together to obtain the multimodal feature vector.

4. The method according to claim 1, characterized in that, The process of performing latent variable coupling modeling on the initial state vector to obtain the current state vector includes: The initial state vector is subjected to feature extraction processing to obtain the observed variables; Construct a latent variable coupling model by inputting the observed variables into the latent variable coupling model to obtain the latent variables; Partial least squares path analysis is performed on the latent variables and the observed variables to obtain the path coefficients; The initial state vector is corrected based on the path coefficients to obtain the current state vector.

5. The method according to claim 1, characterized in that, The step of performing time-series prediction based on the current state vector and historical state vectors to obtain the predicted state vector includes: The state vectors in the historical time series are cached to construct a time series dataset that includes the current state vector and the historical state vectors within the most recent preset time period; The time series dataset is preprocessed to obtain preprocessed time series input data; The preprocessed time series input data is input into the preset time series prediction model to obtain the initial prediction vector; The initial prediction vector is standardized to obtain the predicted state vector.

6. The method according to claim 1, characterized in that, The process of performing state risk determination on the predicted state vector to obtain emotional and cognitive states includes: The predicted state vector is decomposed to obtain cognitive load and emotional state, which includes emotional arousal and emotional valence. The cognitive load is compared with a preset cognitive safety threshold, and the emotional valence is compared with a preset emotional normality threshold. If the cognitive load does not exceed the cognitive safety threshold and the emotional valence is within the range of the emotional normality threshold, it is determined to be a normal state. Alternatively, if the cognitive load exceeds the cognitive safety threshold, it is determined to be cognitive overload; if the emotional valence is not within the normal emotional threshold range and the emotional arousal deviates from the normal fluctuation range, it is determined to be a deterioration of the emotional state and judged as an abnormal state.

7. The method according to claim 5, characterized in that, The preset time series prediction model is a time series prediction model based on long short-term memory networks.

8. A cockpit occupant status monitoring device for operation in highly disruptive environments, characterized in that, include: The acquisition module is used to acquire physiological data, behavioral data, and environmental context data characterizing cabin physical disturbances and task load of the occupants. The processing module is used to perform feature extraction processing on the physiological data and behavioral data to obtain multimodal feature vectors; The processing module is further configured to fuse the multimodal feature vector with the environmental context data to obtain an initial state vector; The processing module is also used to perform latent variable coupling modeling on the initial state vector to obtain the current state vector; The prediction module is used to perform time-series prediction based on the current state vector and historical state vectors to obtain the predicted state vector. The processing module is also used to perform state risk judgment processing on the predicted state vector to obtain emotional and cognitive states, the emotional and cognitive states include normal states and abnormal states, the abnormal states include at least one of cognitive overload and emotional state deterioration. The processing module is also used to generate human-computer interaction intervention commands when the emotional and cognitive states are abnormal, and to control the vehicle system to execute the human-computer interaction intervention commands.

9. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images