Multi-mode signal synchronous acquisition and fusion calculation method and system

Abstract

The invention discloses a multi-modal signal synchronous acquisition and fusion calculation method and system. The method comprises the following steps: synchronously acquiring multi-modal data of an operator; performing standardization processing on the multi-modal data according to a preset feature splicing sequence to form a unified feature vector; performing time sequence analysis on the unified feature vector based on a dynamic window mechanism; and inputting the normalized feature vectors into a preset multi-modal classification model, calculating to obtain probability values of a plurality of categories, and determining a state label of the current window according to the probability values. According to the invention, high-precision synchronous acquisition and real-time fusion calculation of multi-modal signals can be realized, and the accuracy and stability of individual state recognition in a complex task environment are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of complex human-machine system, and particularly relates to a multi-modal signal synchronous acquisition and fusion calculation method and system. BACKGROUND

[0002] In a complex task environment, electrocardiogram data has important value in evaluating the state of cognition and emotion of a person. Common physiological indicators include electrocardiogram, skin temperature, skin electricity, electroencephalogram, electromyogram, respiration, and cardiopulmonary function, etc. These physiological signals can provide researchers with a multi-dimensional perspective of the individual's physiological state.

[0003] However, the measurement of a single physiological signal has limitations in information dimension, and is easily affected by external environmental interference, affecting the stability and reliability of the data. For example, indicators related to heart activity (such as heart rate and heart rate variability) are sensitive to changes in task demand and individual effort, and can be conveniently collected by low-invasive devices such as infrared photoplethysmography, but the signal quality is easily affected by environmental noise, temperature changes, and individual emotional fluctuations. Similarly, indicators related to eye movement behavior are closely related to high-level cognitive processes such as perception, memory, and thinking, and can reflect an individual's visual attention and information processing strategy, but they cannot provide effective information about physiological activation or emotional arousal. At the same time, eye movement data is also easily affected by environmental factors such as light conditions and screen reflections, affecting the stability and repeatability of the measurement.

[0004] Therefore, in order to more comprehensively and accurately reflect the physiological and cognitive state of an individual in a complex task environment, more and more researches have begun to use multi-modal signal fusion methods in recent years, by integrating physiological and behavioral data in multiple dimensions, to improve the accuracy, robustness and generalization ability of state recognition. However, there are still many challenges in data synchronization, real-time calculation, multi-source feature fusion and result visualization, and it is urgent to develop a multi-modal physiological computing platform that integrates acquisition, processing, analysis and visualization, to support individual state perception in a complex task environment. SUMMARY

[0005] The main purpose of the present application is to provide a multi-modal signal synchronous acquisition and fusion calculation method.

[0006] Another purpose of the present application is to provide a multi-modal signal synchronous acquisition and fusion calculation device.

[0007] To achieve the above purpose, the first aspect of the present application provides a multi-modal signal synchronous acquisition and fusion calculation method, comprising: S1, synchronously acquiring multi-modal data of an operator through a cross-platform data transmission protocol, the multi-modal data comprising eye movement data, electrocardiogram data and task performance indicators; S2, standardizing the multi-modal data according to a preset feature splicing sequence to eliminate dimensional differences of different features to form a uniform feature vector; S3, performing time series analysis on the uniform feature vector based on a dynamic window mechanism, the dynamic window mechanism including a cache window and cooperative processing; S4, inputting the normalized feature vector into a preset multi-modal classification model, calculating probability values of multiple categories, and determining a state label of a current window according to the probability values, the state label including three categories of underload, normal and overload.

[0008] In an embodiment of the present application, the synchronously collecting multi-modal data of the operator through the cross-platform data transmission protocol further includes: S11, constructing a cross-platform data transmission channel using a Lab Streaming Layer protocol to realize microsecond-level timestamp alignment of eye movement data and electrocardiogram data; S12, caching raw data at an update frequency of 50 Hz, and setting a cache window length of 50s to provide time series context information.

[0009] In an embodiment of the present application, the standardizing the multi-modal data according to a preset feature splicing sequence further includes: S21, splicing a pupil diameter, a standard deviation (SDNN) of a sinusoidal heart beat RR interval, a ratio (LF / HF) of low frequency and high frequency components, a heart rate, a task completion time, and a task completion accuracy in this fixed order to form a six-dimensional feature vector; S22, performing z-score normalization processing on the input feature using the mean and standard deviation generated in the model training stage, and the calculation formula is .

[0010] In an embodiment of the present application, the performing time series analysis on the uniform feature vector based on a dynamic window mechanism further includes: S31, caching a historical feature vector using a 50s sliding window to ensure the context integrity of the time series analysis; S32, triggering a feature update cycle once every 10 seconds, and reducing output jitter through a state retention strategy based on a rolling prediction sequence.

[0011] In an embodiment of the present application, the inputting the normalized feature vector into a preset multi-modal classification model further includes: S41, calculating probability values of three categories of states through a softmax function, and the formula is . ​S42, outputting a "to be determined" state when the maximum probability value is lower than a preset threshold, or using the same label at the previous moment to maintain state stability.

[0012] In one embodiment of the present application, further comprising: S5, transmitting data including timestamps, state labels, real-time physiological parameters and task performance results through JSON data packets, and periodically polling the backend results at the front end using JavaScript scripts to dynamically control the display state of the dashboard, trend chart and task performance chart.

[0013] To achieve the above purpose, the second aspect of the embodiment of the present application provides a multi-modal signal synchronous acquisition and fusion computing device, comprising: A cross-platform data transmission protocol synchronous acquisition module is configured to synchronously acquire multi-modal data of an operator through a cross-platform data transmission protocol, wherein the multi-modal data includes eye movement data, electrocardiogram data and task performance indicators. A multi-modal data standardization processing module is configured to standardize the multi-modal data according to a preset feature splicing order, and eliminate the dimensional differences of different features to form a unified feature vector. A dynamic window time series analysis module is configured to perform time series analysis on the unified feature vector based on a dynamic window mechanism, wherein the dynamic window mechanism includes a cache window and a cooperative processing. A multi-modal classification model processing module is configured to input the normalized feature vector into a preset multi-modal classification model, calculate probability values of multiple categories, and determine a state label of the current window according to the probability values, wherein the state label includes three categories of underload, normal and overload.

[0014] The multi-modal signal synchronous acquisition and fusion computing method and device of the embodiment of the present application can realize high-precision synchronous acquisition and real-time fusion calculation of multi-modal signals, and improve the accuracy and stability of individual state recognition in a complex task environment. BRIEF DESCRIPTION OF DRAWINGS

[0015] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which: Figure 1 A flowchart of a multi-modal signal synchronous acquisition and fusion computing method provided by the embodiment of the present application; Figure 2 A module schematic diagram of a multi-modal signal synchronous acquisition and fusion computing system provided by the embodiment of the present application; Figure 3 A physiological indicator real-time dashboard schematic diagram provided by the embodiment of the present application; Figure 4 A mental load trend chart provided by the embodiment of the present application; Figure 5 The task performance graph provided in the embodiments of the present invention; Figure 6 This is a schematic diagram of a multimodal signal synchronous acquisition and fusion computing device provided in an embodiment of the present invention. Detailed Implementation

[0016] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0017] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0018] The following description, with reference to the accompanying drawings, describes a method and system for synchronous acquisition and fusion calculation of multimodal signals according to an embodiment of the present invention.

[0019] This embodiment provides a method for synchronous acquisition and fusion calculation of multimodal signals. For example... Figure 1 As shown, the method includes the following steps: S1, synchronously collects multimodal data of the operator through a cross-platform data transmission protocol, including eye-tracking data, electrocardiogram data and task performance indicators.

[0020] Specifically, this step synchronously collects multimodal data from operators, including eye-tracking data, electrocardiogram data, and task performance indicators, through a cross-platform data transmission protocol. This is a crucial step in enabling the system to perceive operator status in real time and in multiple dimensions. In some implementations, this step uses the Lab Streaming Layer (LSL) protocol as a data synchronization mechanism to ensure that data from different hardware devices are strictly aligned on the timeline, thus providing a high-quality input foundation for subsequent status recognition and analysis.

[0021] The system realizes high sampling rate collection of eye movement data through integration of Tobii Pro Fusion portable eye movement and physiological recorder, including key indicators such as fixation duration, pupil diameter, and pupil dilation amplitude. At the same time, heart rate and heart rate variability data are collected through an infrared photoplethysmograph, and SDNN and LF / HF are extracted to reflect the regulation state of the autonomic nervous system. Task performance indicators are recorded in real time through the task execution system, including task completion time and task completion accuracy, which are used to evaluate the behavior performance of operators when performing tasks.

[0022] The system sets the update frequency of data collection to 1000 Hz. A fixed time window mechanism is used, and feature extraction is performed every 10 seconds. The data buffer window length is 50 seconds to retain sufficient temporal context information and improve the temporal modeling ability and prediction stability of the state recognition model. All collected data are cached locally in field classification mode for efficient calling and processing by subsequent modules.

[0023] This step is widely applicable to complex human-machine systems that require real-time monitoring of operator state, such as aviation piloting, nuclear power operation, and medical monitoring. Through synchronous collection of multi-source data, the system can effectively capture physiological and behavioral changes of operators during task execution, providing reliable data support for subsequent brain load recognition, fatigue monitoring, or emotion evaluation. Through cross-platform data transmission protocols, high-precision synchronization of multi-modal data is achieved, solving the common temporal misalignment problem in traditional multi-source data collection, thereby significantly improving the input quality and output stability of the state recognition model, and laying a solid foundation for building a highly robust physiological computing system.

[0024] Further, S1 includes: S11, using the Lab Streaming Layer protocol to build a cross-platform data transmission channel to achieve millisecond-level timestamp alignment of eye movement data and electrocardiogram data.

[0025] Specifically, in some implementations, this step uses the Lab Streaming Layer (LSL) protocol to build a cross-platform data transmission channel to achieve millisecond-level timestamp alignment of eye movement data and electrocardiogram data. LSL is a lightweight, low-latency, high-precision real-time data stream transmission protocol designed based on IEEE11073 standards, widely used in the fields of neuroscience and biological signal processing, supporting cross-operating system (Windows, Linux, macOS) and multiple programming languages (such as Python, MATLAB, C++) data synchronization collection and transmission.

[0026] The system configures the eye tracking device (e.g., Tobii ProFusion) and the physiological recorder (e.g., a device based on the ErgoLAB platform) as independent data publishing nodes through the publisher-subscriber architecture of LSL. Each node generates a timestamped data packet in real time during data acquisition and broadcasts it through the network interface of LSL. On the subscription side, the system receives data streams from different devices through a unified LSL interface and automatically calibrates and aligns the timestamps of each data source based on its built-in clock synchronization mechanism. This mechanism calculates and compensates for the clock deviation between different devices by periodically exchanging timestamp information, thereby achieving microsecond (μs) precision time synchronization.

[0027] The system sets the data acquisition frequency to , i.e., 80 data points per second. To ensure the stability of timestamp alignment, the system uses a 10-second feature extraction window and performs timestamp matching on eye movement and ECG data within each window. The timestamp alignment error is controlled within , meeting the high requirements of multi-modal data fusion for time consistency.

[0028] This step is suitable for complex human-machine interaction environments that require simultaneous collection of eye movement and physiological signals, such as aviation driving simulation, nuclear power control room operation, medical surgery training, etc. In these scenarios, the cognitive load, attention allocation, and physiological state of the operator are closely related, and microsecond-level timestamp alignment can ensure the accurate correspondence of eye movement behavior and physiological response in the time dimension, providing a reliable data foundation for subsequent multi-modal feature fusion and state recognition. Cross-platform data transmission and timestamp alignment achieved through the LSL protocol significantly improves the synchronization accuracy of multi-source data in the time dimension, effectively avoiding the data misplacement problem caused by device clock deviation. This step provides a key guarantee for the subsequent feature splicing, model input normalization, and state recognition of the system, and is the core technical link to achieve multi-modal data fusion analysis.

[0029] S12, with an update frequency of , caches the raw data and sets a cache window length of 50s to provide timing context information.

[0030] Specifically, in some implementations, the caching step of raw ECG data uses The update frequency is based on the comprehensive consideration of the sampling requirements of multi-modal signals and system resources. Specifically, the frequency can meet the sampling requirements of key physiological signals such as eye movement data (such as pupil diameter, fixation duration) and electrocardiogram (such as HR), ensuring sufficient resolution in the time dimension to capture the dynamic changes of individuals in complex task environments. For example, eye movement data usually requires sampling between and to ensure the continuity and stability of the fixation point, and the setting of reduces the occupation of hardware resources by the system while ensuring data quality.

[0031] Further, the system sets a cache window length of seconds to provide sufficient temporal context information. The window length is set based on the time dependence requirements of the cognitive load state recognition model. In practical applications, the change of cognitive state often has a certain hysteresis and persistence, so it is necessary to accumulate enough data in the time dimension to enhance the recognition ability of the model to state changes. For example, in the logistic regression-ElasticNet classification model, the input features usually include statistical quantities (such as mean, standard deviation, rate of change, etc.) in multiple time windows, and longer window length helps to extract more representative temporal features, thereby improving the robustness and accuracy of classification.

[0032] The system uses a local cache mechanism to store raw data by field, such as eye movement data, heart rate data, and task performance data, which are stored in separate buffer queues. Each buffer queue is updated at a fixed time interval (i.e. ), ensuring the real-time and synchronization of data. At the same time, the system extracts features from the data in the cache window every seconds to provide standardized input for subsequent model calculations.

[0033] This step plays a key role in the system and is the basis for multi-modal data fusion and state recognition. By reasonably setting the update frequency and cache window length, the system can ensure data real-time while providing sufficient temporal information to lay a solid data foundation for subsequent feature calculation and model reasoning S2, according to the preset feature splicing order, the multi-modal data is standardized to eliminate the dimensional differences of different features to form a unified feature vector.

[0034] Specifically, in some implementations, the step of "standardizing the multi-modal data in a preset feature concatenation order and eliminating the dimensional differences of different features to form a unified feature vector" is a key link in the implementation of multi-modal feature fusion in the data calculation module. The core goal of this step is to unify the heterogeneous data from different physiological sensors in numerical scale, thereby providing a consistent input format for the subsequent classification model and improving the training efficiency and prediction stability of the model.

[0035] This step first defines a fixed order for feature concatenation, including six dimensions such as pupil diameter, SDNN, LF / HF, instantaneous heart rate, task completion time, and task completion accuracy. The data of each dimension is extracted with a window length of 10 seconds to ensure the continuity and representativeness of the time series information. In the concatenation process, the system uses Python scientific computing libraries such as NumPy or Pandas to combine the feature values in order into a one-dimensional feature vector , providing a structured input for subsequent normalization processing.

[0036] Subsequently, the system performs z-score standardization on the concatenated feature vector, with the mathematical expression being:

[0037] wherein, represents the original value of the th feature, and are the mean and standard deviation of the feature calculated during the model training phase. Through this standardization method, the system can eliminate the dimensional differences between features, making different physiological signals comparable in numerical distribution, thereby avoiding certain features from dominating model decisions due to their large numerical range.

[0038] The window length of the standardization processing is 10 seconds, and the update frequency is once every 10 seconds, consistent with the feature extraction frequency of the data acquisition module. The standardization parameters and are usually calculated from historical data sets during the model training phase and are fixed in the system as constants during the deployment phase, ensuring consistency and efficiency of real-time processing.

[0039] This step is widely used in real-time state recognition systems in complex task environments, such as aviation driving, nuclear power operation, medical monitoring, and other fields that are highly sensitive to operator state. Through standardization processing, the system can convert multi-source heterogeneous electrocardiogram data into a unified feature representation, providing standardized input for the logistic regression-ElasticNet classification model, thereby achieving accurate recognition and classification of mental workload state.

[0040] Further, the technical value of this step lies in improving the consistency and interpretability of model input, while enhancing the robustness of the system under different individuals and environments. By fixing the feature concatenation order and standardizing the processing, the system can effectively reduce the noise interference in the feature space, improve the generalization ability and prediction stability of the classification model.

[0041] Further, S2 includes: S21, the pupil diameter, the standard deviation of the RR interval of sinus heart beats (SDNN), the ratio of low-frequency and high-frequency components (LF / HF), heart rate, task completion time, and task completion accuracy are concatenated in this fixed order to form a six-dimensional feature vector.

[0042] Specifically, in the feature preprocessing stage of the data calculation module, the pupil diameter, the standard deviation of the RR interval of sinus heart beats (SDNN), the ratio of low-frequency and high-frequency components (LF / HF), heart rate, task completion time, and task completion accuracy are concatenated in a fixed order to form a six-dimensional feature vector, which is a key step in realizing multi-modal signal fusion modeling. This step ensures the consistency and interpretability of model input by structuring the feature arrangement, providing a standardized feature space for subsequent classification models.

[0043] In specific implementation, the system first extracts the mean pupil diameter in the current time window from the eye tracker and physiological recorder, with units in millimeters (mm), which usually fluctuates within . Then, SDNN is extracted from heart rate variability data, with units in milliseconds (ms), and its typical value range is , which is used to reflect the overall heart rate level. The LF / HF ratio is used to assess the balance between sympathetic and parasympathetic nerves in the autonomic nervous system, with a value range usually between . Heart rate (HR) is measured in beats per minute (bpm), with a normal range of . Task completion time is measured in seconds (s), reflecting the timing efficiency of the operator in completing the specified task; task completion accuracy is expressed as a percentage (%), quantifying the accuracy of task execution.

[0044] The above six types of features are extracted within every 10-second time window and concatenated into a six-dimensional vector , where is the mean pupil diameter, is the task completion time, is the task completion accuracy. This vector serves as the input of the logistic regression-ElasticNet classification model for real-time identification of mental workload status.

[0045] This step has a dual role in the system: on the one hand, by unifying the feature splicing order, it ensures the consistency of the input in the model training and inference stages, improving the generalization ability of the classification model; on the other hand, the construction of the six-dimensional feature vector provides a structured basis for multi-modal data fusion, which helps to capture the multi-dimensional physiological responses of the operator during the cognitive load change process. In addition, this step provides a clear feature dimension and order for the subsequent z-score normalization processing, effectively eliminating the dimensional differences between different features and improving the stability and interpretability of the model input.

[0046] S22, using the mean value generated in the model training stage and the standard deviation The z-score normalization processing is performed on the input features, and the calculation formula is .

[0047] Specifically, in the data calculation module, the z-score normalization processing of the input features is one of the key steps to realize multi-modal signal fusion modeling. Based on the feature mean and the standard deviation calculated in the model training stage, the current input feature vector is standardized, and the mathematical expression is This formula ensures the comparability of different physiological signals in numerical scale, thereby improving the input consistency and generalization ability of the classification model.

[0048] In specific implementation, the system first splices the original features from eye movement, electrocardiogram, and task performance modules into a unified feature vector in a fixed order, including pupil diameter, SDNN, LF / HF, heart rate, task completion time, and task completion accuracy. Subsequently, the system calls the mean and standard deviation parameters calculated and stored in the training stage to independently perform z-score normalization operations on each feature dimension. Among them, the mean and the standard deviation are calculated based on the historical training data set in the same time window (10 seconds), ensuring that the standardization parameters are statistically representative.

[0049] During the normalization process, the system uses floating-point precision operations to avoid feature distortion caused by integer truncation. In actual application, this step is usually executed once every feature extraction period (i.e., every 10 seconds) to adapt to the dynamic changes of physiological signals. The normalized feature vector will be sent to the logistic regression-ElasticNet classification model to calculate the linear scores of the three types of mental load states (underload, normal, and overload).

[0050] This step has important technical value in the system. First, by eliminating the dimensional differences of different features, the adaptability of the model to multi-source heterogeneous data is improved; second, standardization processing helps to accelerate model convergence, improve classification accuracy and robustness. In high-risk human-machine interaction scenarios such as aviation and nuclear power, this normalization mechanism provides a stable data basis for real-time recognition of operator state.

[0051] S3, based on a dynamic window mechanism, the unified feature vector is analyzed in time sequence, and the dynamic window mechanism includes a 50s cache window and a 10s update period.

[0052] Specifically, this step is based on a dynamic window mechanism to analyze the unified feature vector in time sequence, which is a key link for state recognition in the data calculation module. In some implementations, this mechanism sets a 50s cache window and a 10s update period to achieve continuous modeling of multi-modal signals and stability enhancement of state prediction.

[0053] After the system receives the raw data output by the data acquisition module at a frequency of 80Hz, it first caches it in time sequence into a sliding window with a length of 50s. This window is used to build the time sequence context at the current time, ensuring that the model can make state judgments based on long enough historical data. Every 10 seconds, the system extracts a feature vector from the cache window, including pupil diameter, SDNN, LF / HF, heart rate, task completion time, task completion accuracy, and other key indicators, and concatenates them in a fixed order to form a unified multi-modal feature vector. The feature vector is then sent to the logistic regression-ElasticNet classification model for processing.

[0054] The 50s cache window length in the dynamic window mechanism is set based on the time sequence characteristics of physiological signals, aiming to capture the persistence characteristics of operator state changes. The 10s update period is matched with the computational complexity of the model and the real-time requirements of the task environment, ensuring that the system maintains high response speed while not losing key time sequence information. In addition, the model input features need to be z-score normalized before entering the classifier, with the formula:

[0055] where, is the original feature value, is the feature mean calculated in the training phase, is the feature standard deviation. This processing method effectively eliminates the dimensional differences between different features, improving the generalization ability of the model.

[0056] The dynamic window mechanism is widely applicable to complex human-machine systems such as aviation, nuclear power, medical treatment and the like which have high requirements for real-time monitoring of the state of operators. By periodically updating the electrocardiogram data in the window, the system can continuously evaluate the mental load state of the operator and output three types of labels of "underload", "normal" and "overload" for adaptive intervention or early warning of the upper system.

[0057] The technical effect of this step is that, by introducing the timing context information, the robustness and accuracy of state recognition are significantly improved. At the same time, the 10-second update period and the 50-second cache window work together to ensure the real-time performance of the system while avoiding misjudgment caused by transient noise or abnormal values, thereby realizing stable and continuous evaluation of the state of the operator.

[0058] Further, S3 comprises: S31, using a 50s sliding window to cache the historical feature vectors to ensure the context integrity of the timing analysis.

[0059] Specifically, in some implementations, the 50s sliding window is used to cache the historical feature vectors to ensure the context integrity and continuity of the physiological state of the operator in the timing analysis process. Based on the time-dependent characteristics of the multi-modal signals, the physiological features at the current time and the historical features within the previous 50s are integrated by constructing a sliding window of fixed length, thereby providing more rich timing context information for the subsequent state recognition model.

[0060] The sliding window mechanism is implemented in a circular buffer structure in the system, the window length is 50s, and the sampling frequency is That is, each window contains sampling points. The system performs feature extraction every 10 seconds, so each window contains 5 feature vectors. These feature vectors are composed of multi-modal data such as eye movement, electrocardiogram and task performance, including but not limited to gaze duration, pupil diameter, SDNN, LF / HF, task completion time and accuracy, etc. After each feature extraction, the system adds the current feature vector to the sliding window cache queue and removes the earliest feature vector to maintain a constant window length.

[0061] The length of the sliding window is set based on the typical time scale of the change of cognitive load state, which can capture the short-term state fluctuations of the operator when performing complex tasks. At the same time, the update frequency of the window is consistent with the feature extraction period, ensuring the balance between the timing consistency and the computational efficiency of the model input. In addition, the system requires that all feature vectors have completed z-score normalization processing before entering the window cache to eliminate the dimensional differences between different modal data.

[0062] The sliding window mechanism is widely used in aviation, nuclear power, medical and other fields that require real-time monitoring of operator state. For example, in flight simulation training, the system continuously caches eye movement and heart rate variability features, combined with task performance indicators, to achieve dynamic recognition and trend analysis of pilot mental load state. In the data calculation module, the window is used as the basis for model input to calculate the classification probability of each time window and finally output the state label.

[0063] By introducing a 50s sliding window, the system can effectively alleviate the recognition error caused by insufficient single-frame feature vector information, and improve the robustness and accuracy of the state recognition model. At the same time, this mechanism provides a data basis for subsequent smoothing strategies and trend visualization, enhancing the system's real-time perception ability and decision support value in complex task environments.

[0064] S32, trigger a feature update cycle every 10 seconds, and reduce output jitter based on a rolling prediction sequence through a state retention strategy.

[0065] Specifically, in some implementations, the step of "triggering a feature update cycle every 10 seconds and reducing output jitter based on a rolling prediction sequence through a state retention strategy" in the present invention is a key mechanism for achieving state recognition stability control in the data calculation module. This step ensures that the system can maintain the smoothness and consistency of state recognition results under high-frequency data acquisition by setting a fixed time window length and a rolling update strategy.

[0066] Specifically, the system acquires the multi-modal physiological feature vector within the current time window from the data acquisition module every 10 seconds. This feature vector is composed of eye movement data (such as average fixation duration, pupil diameter, pupil dilation amplitude), electrocardiogram indicators (such as SDNN, LF / HF), task performance data (such as task completion time, task completion accuracy), etc., and is spliced in a fixed order. Subsequently, the system uses the mean and standard deviation saved during the model training phase to perform z-score normalization on the input features, with the formula being:

[0067] wherein, is the original feature value, is the mean of the training set features, is the standard deviation of the training set features. The normalized feature vector is input into the logistic regression-ElasticNet classification model to calculate the linear scores of the three states ("underload", "normal", "overload"), and convert them into probability distributions through the softmax function:

[0068] wherein, probability that the current window belongs to the state To reduce the instantaneous jitter of state recognition, the system adopts a rolling prediction sequence and state retention strategy. Specifically, the system maintains a rolling prediction sequence of length 5, recording the latest five state recognition results. When the current recognition result is inconsistent with the majority of the sequence, the system will adopt the state with the highest frequency in the sequence as the final output, thereby avoiding state misjudgment caused by transient noise or feature drift.

[0069] This strategy is of great significance in high-risk human-machine interaction scenarios such as aviation and nuclear power, and can effectively improve the robustness and interpretability of state recognition, providing stable input for subsequent adaptive intervention or decision support.

[0070] S4, input the normalized feature vector into the preset multi-modal classification model, calculate the probability values of multiple categories, and determine the state label of the current window according to the probability values, the state label including underload, normal and overload.

[0071] Specifically, in some implementations, inputting the normalized feature vector into the preset multi-modal classification model is a key step in implementing state recognition in the data calculation module. This step is based on a logistic regression-ElasticNet classification model, which realizes real-time classification of the current mental workload state of the operator by fusing multi-source physiological and task performance features. Specifically, the system extracts features from eye movement, electrocardiogram, and task performance modules within a 10-second time window, and concatenates them in a fixed order (pupil diameter, SDNN, LF / HF, heart rate, task completion time, task completion accuracy) to form a 6-length feature vector. This vector is then normalized by z-score to eliminate the dimensional differences between different features, ensuring the consistency and comparability of model input. The normalization formula is:

[0072] wherein, is the original feature value, is the feature mean calculated during the training phase, is the feature standard deviation. The normalized feature vector is sent to the logistic regression-ElasticNet classification model, which is composed of three weight vectors and a bias parameter, corresponding to the "underload", "normal" and "overload" states. The model performs linear weighting calculation on the input feature vector to obtain the linear scores of the three states and further converts them to probability values through the softmax function:

[0073] wherein, represent the probability that the current window belongs to the category . The system selects the category with the maximum probability value as the state label of the current window, and the label encoding is“underload”(0),“normal”(1), and“overload”(2), respectively. To enhance the stability of the recognition result, the system sets a confidence threshold (0.6), and when the maximum probability is lower than the threshold, the“to be determined”state is output or a smoothing strategy is used to keep the label of the previous time to reduce state jitter. This step is of great significance in complex human-machine systems, and can provide real-time and reliable decision basis for operator state monitoring, adaptive task allocation, and early warning intervention.

[0074] The multi-modal signal synchronous acquisition and fusion calculation method of the embodiment of the application can realize high-precision synchronous acquisition and fusion calculation of multi-modal signals, and improve the accuracy and stability of operator state recognition in complex task environments.

[0075] Further, S4 comprises: S41, calculating the probability values of the three categories of states through a softmax function, and the formula is .

[0076] Specifically, in the data calculation module of the application, the probability calculation of the three categories of mental load states (“underload”,“normal”, and“overload”) through the softmax function is one of the key steps to realize state classification. This step is based on the linear score output by the logistic regression-ElasticNet classification model, and maps the original score to a normalized probability distribution, thereby providing a quantitative basis for subsequent state determination.

[0077] In a specific implementation, the model processes the multi-modal feature vector in each 10-second window, and calculates the linear scores of the three categories of states, respectively, wherein correspond to the“underload”,“normal”, and“overload”states, respectively. Then, the scores are converted into probability values through the softmax function, and the mathematical expression is:

[0078] wherein, represents the probability that the input feature vector belongs to the i-th category of state, is the linear prediction score of the model for the i-th category of state, all categories (i.e. ) are traversed. This formula ensures that the output probability value is within the interval [0, 1], and the sum of the three probabilities is always 1, meeting the basic requirements of probability distribution.

[0079] The calculation of the softmax function is usually implemented in Python's NumPy or deep learning frameworks such as TensorFlow and PyTorch. In the present system, since the logistic regression-ElasticNet model is used, the output is three sets of linear scores, so the implementation of the softmax function needs to perform exponential operation and normalization processing on these three sets of scores. To improve computational efficiency, the system uses vectorized operations to avoid element-by-element calculations, ensuring real-time performance under high-frequency data processing (every 10 seconds).

[0080] The softmax function itself does not introduce additional adjustable parameters, but the linear scores that are input into it are affected by the model training parameters such as the weight vector and the bias term. The model optimizes the parameters during the training phase through the cross-entropy loss function to maximize the probability output of the correct class. In addition, the system introduces a confidence threshold mechanism in the classification decision. When the maximum probability is less than the set threshold (such as 0.6), the system outputs the "to be determined" state, thereby avoiding misjudgment under low confidence.

[0081] This step is of great significance in practical applications. In high-risk operating environments such as aviation, nuclear power, and medical treatment, the mental workload state of the operator directly affects the safety and efficiency of task execution. Through the three types of probability values output by the softmax function, the system can quantitatively evaluate the confidence of the current state and improve the stability of the recognition results by using smoothing strategies such as sliding window averaging or state persistence judgment. The final output state code (0, 1, 2) can be used by the front-end visualization module to draw trend charts, or as input signals for external systems such as adaptive human-machine interfaces or warning systems to achieve real-time monitoring and intervention of the operator's state.

[0082] S42, when the maximum probability value is lower than the preset threshold, output the "to be determined" state, or use the same label at the last moment to maintain state stability.

[0083] Specifically, in some implementations, when the maximum probability value output by the logistic regression-ElasticNet classification model is lower than the preset threshold, the system will output the "to be determined" state, or use the same state label at the last moment to maintain the stability of the output. This step is a key mechanism in the data calculation module for enhancing the robustness of state recognition, and its technical implementation is based on probability threshold control and state smoothing strategies.

[0084] This step first obtains the probability values of the three classes at the model output stage ​), respectively corresponding to “underload”, “normal”, “overload” state. The system determines the maximum probability value by comparing the probability of the three , where is a preset confidence threshold (e.g. ), the classification result of the current window is not reliable, and the system will output the “to be determined” state, or select the same state label as the previous time to avoid frequent state jumps. This strategy effectively suppresses misjudgments caused by short-term noise or data fluctuations by introducing a time continuity constraint.

[0085] The key parameters involved in this step include the probability value of the classification model output , the maximum probability value , and the confidence threshold . Among them, The setting needs to be adjusted in combination with the actual application scenario, and the value is usually taken in the range of [0.6, 0.9] to balance the recognition sensitivity and stability. In addition, the state cache window length is 50 seconds, and feature extraction and classification prediction are performed every 10 seconds to ensure that the system has good real-time response capability.

[0086] This step is widely used in fields such as aviation, nuclear power, and medical care that require high operator state monitoring. In these environments, the operator's mental load state needs to be continuously and stably output to support subsequent adaptive intervention or human-machine collaborative decision-making. For example, in flight simulation training, if the current recognition probability is low, the system can maintain the load state of the previous time to avoid the mis-triggering of control strategies due to transient interference.

[0087] The technical effect of this step is to significantly improve the stability and reliability of state recognition. By introducing a probability threshold and a state smoothing mechanism, the system can avoid frequent output jumps when facing noise interference or data anomalies, thereby enhancing the continuity and interpretability of the overall state monitoring. This design is of great significance for building a high-robustness multi-modal physiological state recognition system.

[0088] Also includes: S5, transmit data containing timestamps, state labels, real-time physiological parameters, and task performance results through JSON data packets, and use JavaScript scripts in the front end to periodically poll the backend results to dynamically control the display state of the dashboard, trend chart, and task performance chart.

[0089] ​Specifically, in some implementations, the data containing timestamps, status labels, real-time physiological parameters, and task performance results are transmitted through JSON data packets, and JavaScript scripts are used in the front end to periodically poll the back end results to dynamically control the display state of the dashboard, trend chart, and task performance chart, which is the core mechanism of the system's data visualization module to realize real-time feedback and human-computer interaction. This step is based on the separation of front and back end architecture, uses RESTful API interface for data communication, and ensures efficient data transmission and timely updating of front-end components.

[0090] The back-end system encapsulates the results as structured JSON data packets after feature extraction and state classification every 10 seconds. The data packet contains the following fields: `timestamp` (timestamp, unit is millisecond, precision is 1ms), `status_label` (status label, integer encoding, 0 indicates "underload", 1 indicates "normal", 2 indicates "overload"), `physiological_data` (real-time physiological parameters such as pupil diameter, SDNN, LF / HF, etc., numerical type is float), `task_performance` (task performance results such as task completion time, correct completion rate, numerical type is float or int). The JSON data format follows the standard key-value pair structure, which is convenient for front-end parsing and rendering.

[0091] The front end uses JavaScript scripts to implement periodic polling through `setInterval` or `setTimeout`, usually setting the polling interval to 10 seconds to ensure the real-time nature of data updates and the stability of system responses. The polling request is sent to the back-end API interface through `fetch` or `axios`, and after obtaining the latest JSON data packet, the front end parses the data, extracts the key fields and updates the corresponding buffer array. For example, the status label is used to update the classification sequence of the trend chart, the physiological parameters are used to refresh the numerical display of the dashboard, and the task performance is used to update the completion rate curve of the line chart.

[0092] In specific operations, the front end triggers the re-rendering of local components through DOM operations or the responsive mechanism of Vue / React and other frameworks, avoiding the performance loss caused by full-page refresh. For example, when a new status label is received, the front end dynamically sets the color of the bar chart in the trend chart (blue, green, red) according to its encoding value and updates the status description text. At the same time, to ensure the continuity and smoothness of data display, the front end can use moving average or sliding window strategy to process historical data.

[0093] This step plays a key role in the entire system, as it not only realizes the visualized integration of multi-modal data and task performance, but also improves the real-time and intuitiveness of human-computer interaction through a dynamic refreshing mechanism. In high-risk operation scenarios such as aviation, nuclear power, and medical treatment, this mechanism can provide intuitive and timely decision support for operator state monitoring and adaptive intervention.

[0094] The multi-modal signal synchronous acquisition and fusion calculation method of the embodiment of the present application realizes the real-time visualized linkage of ECG data and task performance through the structured transmission of time stamps, state labels, real-time physiological parameters, and task performance results by JSON data packets, combined with the periodic polling mechanism of the front-end JavaScript script, further improving the explainability and response efficiency of the operator state recognition results.

[0095] Embodiment 2 The present application constructs a multi-modal signal synchronous acquisition and fusion calculation system, which can efficiently process and interpret multi-modal data. This system not only supports the collection and analysis of common ECG data such as ECG and eye movement, but also can realize comprehensive evaluation of operator state combined with task performance indicators. The system platform is open and extensible, and different calculation models can be called or replaced according to different application scenarios. For example, in this embodiment, brain load state recognition is taken as a specific application case for illustration, but this system is also applicable to stress evaluation, fatigue monitoring, emotion recognition, and other physiological indicators extraction and calculation, and can be widely used in fields such as aviation, nuclear power, and medical treatment, which have high requirements for operator state monitoring.

[0096] The system is developed using Python, relying on its powerful open source ecosystem and scientific computing capabilities to realize the modularity, expandability, and high reusability of the data processing process. As shown in Figure 2 The system consists of three core functional modules: data acquisition module, data calculation module, and data visualization module.

[0097] Specifically, the data acquisition module is located at the input end of the system and is responsible for high-frequency and synchronized acquisition of multi-modal physiological information of the operator. The module integrates the Tobii Pro Fusion portable eye movement and physiological recorder based on the ErgoLAB platform, which can synchronously acquire the following key data: (1) eye movement data: fixation duration, pupil diameter, pupil dilation amplitude, etc.; (2) ECG data: heart rate, SDNN, LF / HF, etc.; (3) task performance indicators: including task completion time, task completion accuracy, etc.

[0098] To ensure the time alignment and synchronization between different data sources, the system uses Lab Streaming Layer packets to build a cross-platform data transmission channel, ensuring the time alignment and consistency of multi-source data.

[0099] All raw data is cached locally by field classification, updated at a frequency of 80Hz, and feature extraction is performed every 10 seconds. The data cache window length is set to 50 seconds to provide sufficient temporal context information and improve the timeliness and accuracy of recognition models.

[0100] Specifically, the data computation module is the core unit of the system, responsible for converting the collected multimodal input features into status labels that can be used by subsequent applications. This module can flexibly integrate various machine learning or statistical models, which can be replaced or adjusted according to different target tasks. In this embodiment, the logistic regression-ElasticNet classification model is selected to identify mental workload status, outputting three categories of labels: "underloaded," "normal," and "overloaded."

[0101] In its implementation, the data computation module first receives the feature vectors output by the data acquisition module every 10 seconds and performs model input normalization processing on them. This normalization step includes: concatenating the multimodal features in a fixed order of pupil diameter, SDNN, LF / HF, heart rate, task completion time, and task completion accuracy; then, using the mean and standard deviation generated during the model training phase, z-score normalization is performed on the input features to eliminate differences in the dimensions of different features and ensure the consistency of the model input.

[0102] The normalized multimodal feature vectors are fed into a logistic regression-ElasticNet classification model. This model consists of three sets of weight vectors and bias parameters, corresponding to the three categories of "underload," "normal," and "overload," respectively. For the input feature vector x, the model calculates linear scores for each of the three categories: , k ∈{Underload, Normal, Overload} Subsequently, the data calculation module uses the softmax function to convert the linear score into probability values ​​for the three categories:

[0103] in, The input window represents the category. k The module selects the category with the highest probability as the mental workload recognition result for the current window.

[0104] To enhance output stability, the data calculation module incorporates a confidence threshold and a smoothing strategy. When the maximum class probability falls below the preset threshold, the module outputs a "pending determination" state or uses the same label as the previous time step to reduce state fluctuations. Furthermore, the module caches the recognition results chronologically and forms a rolling prediction sequence, which is used by the data visualization module to plot mental workload trends and can also be used by upper-level applications for state monitoring or adaptive intervention.

[0105] The identified status labels are output to the system's main control channel through encoding, with "underload" corresponding to code 0, "normal" to code 1, and "overload" to code 2. The encoded data can be called by the front-end visualization module and can also be extended for use in the early warning and intervention system.

[0106] Specifically, the data visualization module enables the front-end visualization of the system's collected and analyzed results, facilitating real-time monitoring and interactive operation. The module uses embedded JavaScript to periodically poll the back-end results, dynamically controlling the display status of the front-end components.

[0107] In the data processing workflow, the data visualization module first obtains a JSON data packet from the backend containing timestamps, status labels, real-time physiological parameters, and task performance results. On the frontend, the module parses and extracts fields from the received data, such as parsing eye-tracking metrics into average fixation duration, pupil diameter, and dilation amplitude; parsing ECG metrics into SDNN, LF / HF values, and heart rate waveform sequences; and parsing status results into integer codes for "underload," "normal," or "overload." Subsequently, based on the numerical format and sequence length required by different chart components, the module writes the above content into the corresponding frontend buffer array and triggers a partial refresh event to achieve real-time display.

[0108] The data visualization module mainly includes the following components: (1) Real-time dashboard of physiological indicators ( Figure 3 (1) Real-time display of fixation duration, pupil diameter and dilation amplitude, heart rate variability index and heart rate waveform; (2) Mental workload trend chart ( Figure 4 ): Use a bar chart to show the prediction results of the most recent five mental workload classification models, with blue indicating underload, green indicating normal, and red indicating overload; (3) Task performance chart ( Figure 5 The completion rates of the primary and secondary tasks are displayed in line graph form, with blue for primary tasks and red for secondary tasks.

[0109] Example 3 This invention also provides a multimodal signal synchronous acquisition and fusion computing device 10, such as... Figure 6 As shown, the device 10 includes: The cross-platform data transmission protocol synchronous acquisition module 100 is used to synchronously acquire the operator's multimodal data through the cross-platform data transmission protocol. The multimodal data includes eye-tracking data, electrocardiogram data, and task performance indicators. The multimodal data standardization processing module 200 is used to standardize the multimodal data according to a preset feature splicing order to eliminate the dimensional differences of different features and form a unified feature vector. The dynamic window timing analysis module 300 is configured to perform timing analysis on the unified feature vector based on a dynamic window mechanism, and the dynamic window mechanism comprises a cache window and cooperative processing. The multi-modal classification model processing module 400 is configured to input the normalized feature vector into a preset multi-modal classification model, calculate probability values of multiple categories, and determine a state label of the current window according to the probability values, and the state label comprises three categories of underload, normal and overload.

[0110] Further, the cross-platform data transmission protocol synchronous acquisition module is further configured to: The Lab Streaming Layer protocol is used to construct a cross-platform data transmission channel, and microsecond-level timestamp alignment of the eye movement data and the electrocardiogram data is realized. The original data is cached at an update frequency, and a cache window length of 50s is set to provide timing context information.

[0111] Further, the multi-modal data standardization processing module is further configured to: The pupil diameter, SDNN, LF / HF, heart rate, task completion time and task completion accuracy are spliced in this fixed order to form a six-dimensional feature vector; The mean value and the standard deviation generated in the model training stage are used for z-score normalization processing of the input feature, and the calculation formula is .

[0112] Further, the dynamic window timing analysis module is further configured to: A 50s sliding window is used to cache the historical feature vector, so as to ensure the context integrity of the timing analysis; A feature update cycle is triggered once every 10 seconds, and a state retention strategy is used to reduce output jitter based on a rolling prediction sequence.

[0113] The multi-modal signal synchronous acquisition and fusion computing device can realize high-precision synchronous acquisition and fusion computing of multi-modal signals, and improve the accuracy and stability of operator state recognition in a complex task environment.

[0114] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0115] ​​​In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. Furthermore, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0116] In addition, the terms "first", "second", are used only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Thus, the features defined with "first", "second" can include at least one of the features, explicitly or implicitly. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly specified.

Claims

1. A multi-modal signal synchronous acquisition and fusion calculation method, characterized in that, The method comprises the following steps: S1, synchronously collecting multi-modal data of the operator through a cross-platform data transmission protocol, wherein the multi-modal data comprises eye movement data, electrocardio data and task performance indicators; S2, standardizing the multi-modal data according to a preset feature splicing sequence to eliminate dimensional differences of different features and form a unified feature vector; S3, performing time series analysis on the unified feature vector based on a dynamic window mechanism, wherein the dynamic window mechanism comprises a cache window and cooperative processing; S4, inputting the normalized feature vector into a preset multi-modal classification model, calculating probability values of multiple categories, and determining a state label of the current window according to the probability values, wherein the state label comprises three categories of underload, normal and overload.

2. The method of claim 1, wherein, The method of synchronously collecting multi-modal data of the operator through the cross-platform data transmission protocol further comprises the following steps: S11, constructing a cross-platform data transmission channel by using a Lab Streaming Layer protocol to realize microsecond-level timestamp alignment of the eye movement data and the electrocardio data; S12, to The raw data is cached with an update frequency and a cache window length of 50s is set to provide the timing context information.

3. The method of claim 1, wherein, The method of standardizing the multi-modal data according to the preset feature splicing sequence further comprises the following steps: S21, splicing the pupil diameter, the standard deviation (SDNN) of the sinus rhythm RR interval, the ratio (LF / HF) of the low-frequency component and the high-frequency component, the heart rate, the task completion time and the task completion accuracy in the fixed sequence to form a six-dimensional feature vector; S22, the mean value generated by the model training stage with the standard deviation The input features are z-score normalized, and the calculation formula is .

4. The method of claim 1, wherein, The method of performing time series analysis on the unified feature vector based on the dynamic window mechanism further comprises the following steps: S31, caching the historical feature vector by using a 50s sliding window to ensure the context integrity of the time series analysis; S32, triggering a feature update cycle once every 10 seconds, and reducing output jitter by using a state retention strategy based on a rolling prediction sequence.

5. The method of claim 1, wherein, The method of inputting the normalized feature vector into the preset multi-modal classification model further comprises the following steps: S41, calculate the probability values of the three states by a softmax function, the formula is ; S42, outputting a "to be determined" state when the maximum probability value is lower than a preset threshold, or maintaining the state stability by using the same label at the previous moment.

6. The method of claim 1, wherein, The method further comprises the following step: S5, transmitting data containing a timestamp, a state label, real-time physiological parameters and task performance results by using a JSON data packet, and periodically polling the backend results by using a JavaScript script on the front end to dynamically control the display states of an instrument panel, a trend chart and a task performance chart.

7. A multi-modal signal synchronous acquisition and fusion computing device, characterized in that, The method comprises the following steps: A cross-platform data transmission protocol synchronous collection module is configured to synchronously collect multi-modal data of the operator through a cross-platform data transmission protocol, wherein the multi-modal data comprises eye movement data, electrocardio data and task performance indicators; A multi-modal data standardization processing module is configured to standardize the multi-modal data according to a preset feature splicing sequence to eliminate dimensional differences of different features and form a unified feature vector; A dynamic window time series analysis module is configured to perform time series analysis on the unified feature vector based on a dynamic window mechanism, wherein the dynamic window mechanism comprises a cache window and cooperative processing; A multi-modal classification model processing module is configured to input the normalized feature vector into a preset multi-modal classification model, calculate probability values of multiple categories, and determine a state label of the current window according to the probability values, wherein the state label comprises three categories of underload, normal and overload.

8. The apparatus of claim 7, wherein, The cross-platform data transmission protocol synchronous acquisition module is further configured to: A cross-platform data transmission channel is constructed by using a Lab Streaming Layer protocol to realize microsecond-level timestamp alignment of eye movement data and electrocardiogram data. by The update frequency caches the raw data, and a cache window length of 50 seconds is set to provide timing context information.

9. The apparatus of claim 7, wherein, The multi-modal data standardization processing module is further configured to: The pupil diameter, SDNN, LF / HF, heart rate, task completion time, and task completion accuracy are spliced in this fixed order to form a six-dimensional feature vector. The mean generated by the model training phase With the standard deviation The input features are z-score normalized, and the calculation formula is .

10. The apparatus of claim 7, wherein, The dynamic window time series analysis module is further configured to: A 50s sliding window is used to cache historical feature vectors to ensure the context integrity of time series analysis. A feature update cycle is triggered once every 10 seconds, and a state retention strategy is used to reduce output jitter based on a rolling prediction sequence.

Images