Multi-modal interaction evaluation system and method based on synchronous data acquisition

Abstract

The invention discloses a multi-modal interaction evaluation system and method based on synchronous data acquisition, and relates to the technical field of biomedical engineering and man-machine interaction, and the method comprises the steps: generating a synchronous pulse sequence through a synchronous signal generator according to the same coding time base, and outputting the synchronous pulse sequence to an optical path and an electrical path at the same time; pulse events are recognized in the video and the electrophysiological data respectively, and observation pairs of counting numbers and timestamps are obtained; time drift is continuously estimated and compensated through robust regression in a sliding window, and a unified time axis is generated; furthermore, a dynamic time window of heart beat anchoring is constructed according to an R peak event of the electrocardiosignal, and consistent segmentation and feature extraction of cross-modal data are achieved.

Description

Technical Field

[0001] This invention relates to the fields of biomedical engineering and human-computer interaction technology, specifically a multimodal interaction evaluation system and method based on synchronous data acquisition. Background Technology

[0002] In the field of multimodal human-computer interaction and cognitive assessment, it is often necessary to simultaneously collect multiple signals such as electroencephalogram (EEG), electrocardiogram (ECG), facial video, and voice audio to comprehensively analyze the user's physiological state and behavioral responses. However, because each acquisition device typically has an independent clock system and different sampling rates, buffering mechanisms, and interface latency, it is difficult to achieve millisecond-level high-precision time synchronization by relying solely on software timestamps or a single hardware trigger signal. Especially in long-term acquisition or dynamic interaction scenarios, clock drift between devices gradually accumulates, leading to significant deviations in cross-modal data on the time axis, affecting the effectiveness of subsequent signal fusion and analysis.

[0003] Existing technologies attempt to achieve initial alignment through external triggers or optical markers, such as marking time points in video and EEG using LED flash events, and then fitting event mapping relationships through post-processing. However, these methods often fail to address the dynamic drift problem during real-time acquisition, and the alignment accuracy is limited by the video frame rate, making it difficult to meet the millisecond-level phase consistency requirements of high-frequency physiological signals such as EEG / ECG. Furthermore, traditional methods often use fixed-length time windows for signal segmentation, ignoring the impact of heart rate variability (HRV) on physiological rhythms, leading to a mismatch between the window and the heart cycle under different heart rate states, thereby weakening the correlation between cross-modal features.

[0004] Therefore, there is an urgent need for a time alignment method that can achieve high-precision synchronization of multimodal data in a real-time interactive environment, support dynamic heart rate adaptive windowing, and has strong robustness, so as to improve the reliability and response speed of cognitive assessment and training systems. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal interactive evaluation system and method based on synchronous data acquisition, so as to solve the problems raised in the prior art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multimodal interaction evaluation method based on synchronous data acquisition, the evaluation method comprising: The synchronization signal generator generates a synchronization pulse sequence based on the same encoded time base and simultaneously outputs the synchronization pulse sequence to both the optical path and the electrical path. Video data from the optical path and electrophysiological data from the electrical path are acquired; Identify optical pulse events in video data and obtain the optical observation pairs corresponding to the optical pulse events; Identify electrical pulse events in electrophysiological data and obtain the electrical observation pairs corresponding to the electrical pulse events; Based on the counting number, the mapping relationship between video time and electrophysiological time to the encoded time base is established respectively, and the time drift is continuously estimated and compensated through robust regression and residual monitoring within the sliding window to generate a unified time axis; R-peak events in the electrocardiogram signal are detected on the unified time axis, and a dynamic time window for heartbeat anchoring is constructed based on the interval between adjacent R-peaks. The dynamic time window is mapped to the corresponding intervals of video, audio, and EEG data to achieve consistent segmentation and feature extraction across modal data.

[0007] Furthermore, identifying optical pulse events in video data and obtaining the corresponding optical observation pairs includes: Pulse detection is performed within a preset ROI region of the video frame. The optical pulse event is identified by threshold processing and edge detection algorithm. The timestamp of the video frame where the optical pulse event is located and its vertical position in the video frame are recorded. If the video acquisition device uses a rolling shutter mechanism, then the timing of the identified optical pulse events is corrected using the following formula: ; Among them, t frame N represents the timestamp of the video frame in which the optical pulse event occurs; rows y represents the vertical resolution of the video frame; y represents the vertical position of the optical pulse event in the video frame. This indicates the corrected optical pulse event time; The pulse sequence corresponding to the identified optical pulse event is decoded, including start symbol detection, count field extraction, and check field verification, to obtain an optical counting sequence {c} that is consistent with the global counting sequence emitted by the synchronization signal generator. k}, and record each count number c k Corresponding corrected video timestamp ; The decoded optical counting sequence and the corresponding corrected time are combined to form an optical observation pair. ; The encoded frame structure of the above-mentioned synchronization pulse sequence is output in sequence according to the start symbol, count field, check field, and gap within a preset period; the code pattern of the synchronization pulse sequence adopts any one of Manchester encoding, pulse width encoding, or pulse position encoding.

[0008] Furthermore, identifying electrical impulse events in electrophysiological data and obtaining the corresponding electrical observation pairs for the electrical impulse events includes: The raw electrical signal is obtained from the isolated synchronization channel of the electrophysiological acquisition device, and the raw electrical signal is subjected to bandpass filtering and noise reduction processing. The processed raw electrical signal is subjected to threshold comparison or edge detection to identify the rising or falling edge of the electrical pulse event, and the sample index s corresponding to each pulse edge in the electrophysiological data stream is recorded. k ; Based on the sampling frequency f of the electrophysiological data s , sample index s k Convert to electrophysiological timestamps The conversion formula is: ; The pulse sequence of the identified electrical pulse event is decoded, including start symbol, count field extraction, and check field verification, to obtain an electrical count sequence {c} that is consistent with the global count sequence emitted by the synchronization signal generator. k}, record each count number c k Corresponding corrected electrophysiological timestamp ; The decoded count number c k Together with the corresponding electrophysiological timestamps, they form an electrical observation pair. .

[0009] Furthermore, based on the count number, mapping relationships between video time and electrophysiological time to the encoded time base are established respectively. Through robust regression and residual monitoring within a sliding window, time drift is continuously estimated and compensated for, generating a unified time axis including: Obtain the set of optical observation pairs obtained by decoding optical pulse events and the set of electrical observation pairs obtained by decoding electrical pulse events; Establish affine mapping models from video time and electrophysiological time to encoding time base count c, respectively: ; ; Among them, t e Indicates the time of an electrophysiological event; a v b v For video time mapping parameters; a e b e These are electrophysiological time mapping parameters; All are residual terms; Robust regression analysis was performed on both optical and electrical observation pairs within the sliding time window to continuously estimate and update the mapping parameter a. v b vand a e b e ; Among them, robust regression uses the Huber loss function or L1 norm minimization method to suppress the influence of outliers caused by impulse loss or noise; Calculate the regression residuals and monitor their statistical distribution. If the residuals exceed a preset threshold, trigger a recalibration mechanism or narrow the sliding window to improve estimation sensitivity. Based on the latest estimated mapping parameters, video timestamps and electrophysiological timestamps are uniformly mapped to a global time axis dominated by the coding time base, generating a unified time axis and achieving cross-modal time alignment; In the above steps, when a certain modal pulse event is briefly lost, the most recent mapping parameters are used for extrapolation to maintain the continuity of the time axis, and the confidence level during the extrapolation period is marked with attenuation. After the signal is recovered, it is re-included in the sliding window for regression estimation.

[0010] Furthermore, detecting R-peak events in the electrocardiogram signal on the unified time axis and constructing a dynamic time window for heartbeat anchoring based on the interval between adjacent R-peaks includes: On the unified time axis, R-peak detection is performed on the electrocardiogram signal to identify and locate the R-wave peak event in each cardiac cycle, resulting in the R-peak time series {R i}, where i is the heartbeat number; Calculate the time interval between adjacent R peaks to obtain the RR interval sequence {RR i},in, ; Based on the preset proportional coefficients a and b, and taking each R peak Ri as the center, take forward... Time, take it backward Time, constructing a dynamic time window W for heartbeat anchoring i : ; Where a and b are dimensionless coefficients.

[0011] Furthermore, mapping the dynamic time window to corresponding intervals of video, audio, and EEG data to achieve consistent segmentation and feature extraction across modal data includes: The RR interval sequence is smoothed to remove abnormal intervals. The smoothing process is median filtering or exponential smoothing. The abnormal intervals include premature beats, missed detections, and invalid detections caused by insufficient signal-to-noise ratio. Each of the aforementioned dynamic time windows W i The corresponding intervals mapped to multimodal data include: For EEG signals, extract the dynamic time window W. i Feature extraction is performed on sample segments within a time range; For video data, the dynamic time window W i Convert to the corresponding frame index sequence and perform line timing correction; For audio data, extract the dynamic time window W. i Feature analysis is performed on audio sample segments within a time range; When the ECG signal is unavailable or its quality is below a set threshold, a rollback strategy is activated, using the peak value of the photoplethysmography pulse wave, the peak value of the heart sound, or the phase of the respiratory signal as an alternative anchor point to construct a time window proportionally, or rollback to a fixed duration window.

[0012] Furthermore, to better implement the above method, a multimodal interactive evaluation system based on synchronous data acquisition is also provided. The evaluation system includes: a synchronization signal transmitter, a video acquisition device, an electrophysiological acquisition device, a processing unit, a user interface, and a data storage module. A synchronization signal transmitter is used to generate and output synchronization pulse sequences to optical and electrical paths. Video acquisition equipment is used to acquire video data, including optical pulses; Electrophysiological acquisition equipment is used to acquire electrophysiological data, including electrical pulses and electrocardiogram signals; The processing unit is used to perform pulse identification, time mapping, R-peak detection, window construction, and data alignment. A user interface for displaying alignment status, signal quality, and evaluation results; The data storage module is used to store the original multimodal data and the aligned timeline information.

[0013] Furthermore, the synchronization signal generator includes: a main time base crystal oscillator, an encoding modulation module, an LED driver circuit, and an electrically isolated output circuit; The master time base crystal oscillator is used to generate frequency signals and provide a reference clock source for the synchronization signal generator; The encoding and modulation module is used to generate coded pulse sequences; LED driver circuit, used to drive infrared or visible light; Electrically isolated output circuit, used to output isolated synchronization pulses to electrophysiological acquisition equipment.

[0014] Furthermore, the processing unit includes: Real-time monitoring of synchronization signal quality, ECG signal quality, and mapping residuals; A degradation strategy is initiated when signal quality deteriorates, and rapid recalibration is performed once the signal is restored.

[0015] Compared with the prior art, the beneficial effects of the present invention are: the present invention achieves high-precision time alignment and adaptive segmentation of multimodal data through optical-electrical dual-channel synchronization mechanism and dynamic window strategy of heartbeat anchoring, which significantly improves the accuracy and stability of cross-modal signal fusion, and is especially suitable for real-time interaction and dynamic heart rate change scenarios, with strong practicality and robustness. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the system structure of a multimodal interactive evaluation system and method based on synchronous data acquisition according to the present invention; Figure 2 This is a schematic diagram of the LED / IR optical output of a multimodal interactive evaluation system and method based on synchronous data acquisition according to the present invention; Figure 3 The present invention relates to the encoding frame structure of a synchronization pulse sequence in a multimodal interactive evaluation system and method based on data synchronous acquisition. Figure 4 This is a schematic diagram of video frame sampling, electrical synchronization pulse, and EEG / ECG sampling for a multimodal interactive evaluation system and method based on data synchronous acquisition according to the present invention. Figure 5 This is a schematic diagram of cross-modal clock mapping and sliding estimation of a multimodal interactive evaluation system and method based on data synchronous acquisition according to the present invention; Figure 6 This is a schematic diagram of R-peak detection for a multimodal interactive evaluation system and method based on synchronous data acquisition according to the present invention. In the diagram: S_LED: Optical synchronization pulse sequence; S_TTL: Electrical synchronization pulse sequence. Detailed Implementation

[0017] Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1: As Figures 2-6 As shown, the present invention provides a technical solution, a multimodal interaction evaluation method based on synchronous data acquisition, the evaluation method comprising: The synchronization signal generator generates a synchronization pulse sequence based on the same encoded time base and simultaneously outputs the synchronization pulse sequence to both the optical path and the electrical path. Video data from the optical path and electrophysiological data from the electrical path are acquired; Identify optical pulse events in video data and obtain the optical observation pairs corresponding to the optical pulse events; Identify electrical pulse events in electrophysiological data and obtain the electrical observation pairs corresponding to the electrical pulse events; Based on the counting number, the mapping relationship between video time and electrophysiological time to the encoded time base is established respectively, and the time drift is continuously estimated and compensated through robust regression and residual monitoring within the sliding window to generate a unified time axis; R-peak events in the electrocardiogram signal are detected on the unified time axis, and a dynamic time window for heartbeat anchoring is constructed based on the interval between adjacent R-peaks. The dynamic time window is mapped to the corresponding intervals of video, audio, and EEG data to achieve consistent segmentation and feature extraction across modal data; Among these, identifying optical pulse events in video data and obtaining the corresponding optical observation pairs includes: Pulse detection is performed within a preset ROI region of the video frame. The optical pulse event is identified by threshold processing and edge detection algorithm. The timestamp of the video frame where the optical pulse event is located and its vertical position in the video frame are recorded. If the video acquisition device uses a rolling shutter mechanism, then the timing of the identified optical pulse events is corrected using the following formula: ; Among them, t frame N represents the timestamp of the video frame in which the optical pulse event occurs; rows y represents the vertical resolution of the video frame; y represents the vertical position of the optical pulse event in the video frame. This indicates the corrected optical pulse event time; The pulse sequence corresponding to the identified optical pulse event is decoded, including start symbol detection, count field extraction, and check field verification, to obtain an optical counting sequence {c} that is consistent with the global counting sequence emitted by the synchronization signal generator. k}, and record each count number c k Corresponding corrected video timestamp ; The decoded optical counting sequence and the corresponding corrected time are combined to form an optical observation pair. ; The process of identifying electrical impulse events in electrophysiological data and obtaining the corresponding electrical observation pairs includes: The raw electrical signal is obtained from the isolated synchronization channel of the electrophysiological acquisition device, and the raw electrical signal is subjected to bandpass filtering and noise reduction processing. The processed raw electrical signal is subjected to threshold comparison or edge detection to identify the rising or falling edge of the electrical pulse event, and the sample index s corresponding to each pulse edge in the electrophysiological data stream is recorded.k ; Based on the sampling frequency f of the electrophysiological data s , sample index s k Convert to electrophysiological timestamps The conversion formula is: ; The pulse sequence of the identified electrical pulse event is decoded, including start symbol, count field extraction, and check field verification, to obtain an electrical count sequence {c} that is consistent with the global count sequence emitted by the synchronization signal generator. k}, record each count number c k Corresponding corrected electrophysiological timestamp ; The decoded count number c k Together with the corresponding electrophysiological timestamps, they form an electrical observation pair. ; Specifically, based on the counting number, a mapping relationship is established between video time and electrophysiological time to the encoded time base, and a unified time axis is generated by continuously estimating and compensating for time drift through robust regression and residual monitoring within a sliding window. Obtain the set of optical observation pairs obtained by decoding optical pulse events and the set of electrical observation pairs obtained by decoding electrical pulse events; Establish affine mapping models from video time and electrophysiological time to encoding time base count c, respectively: ; ; Among them, t e Indicates the time of an electrophysiological event; a v b v For video time mapping parameters; a e b e These are electrophysiological time mapping parameters; All are residual terms; Robust regression analysis was performed on both optical and electrical observation pairs within the sliding time window to continuously estimate and update the mapping parameter a. v b v and a e b e ; Calculate the regression residuals and monitor their statistical distribution. If the residuals exceed a preset threshold, trigger a recalibration mechanism or reduce the sliding window. Based on the latest estimated mapping parameters, video timestamps and electrophysiological timestamps are uniformly mapped to a global time axis dominated by the coding time base, generating a unified time axis; The detection of R-peak events in the electrocardiogram signal on the unified time axis, and the construction of a dynamic time window for heartbeat anchoring based on the interval between adjacent R-peaks, includes: On the unified time axis, R-peak detection is performed on the electrocardiogram signal to identify and locate the R-wave peak event in each cardiac cycle, resulting in the R-peak time series {R i}, where i is the heartbeat number; Calculate the time interval between adjacent R peaks to obtain the RR interval sequence {RR i},in, ; Based on the preset proportional coefficients a and b, and taking each R peak Ri as the center, take forward... Time, take it backward Time, constructing a dynamic time window W for heartbeat anchoring i : ; Where a and b are dimensionless coefficients; The process of mapping the dynamic time window to corresponding intervals of video, audio, and EEG data to achieve consistent segmentation and feature extraction across modalities includes: The RR interval sequence is smoothed to remove abnormal intervals. The smoothing process is median filtering or exponential smoothing. The abnormal intervals include premature beats, missed detections, and invalid detections caused by insufficient signal-to-noise ratio. Each of the aforementioned dynamic time windows W i The corresponding intervals mapped to multimodal data include: For EEG signals, extract the dynamic time window W. i Feature extraction is performed on sample segments within a time range; For video data, the dynamic time window W i Convert to the corresponding frame index sequence and perform line timing correction; For audio data, extract the dynamic time window W. i Feature analysis is performed on audio sample segments within a time range; When the ECG signal is unavailable or its quality is below the set threshold, the rollback strategy is activated, using the peak value of the photoplethysmography pulse wave, the peak value of the heart sound, or the phase of the respiratory signal as an alternative anchor point to construct a time window proportionally, or rollback to a fixed duration window. Example 2: Figure 1 As shown, in order to better implement the above method, a multimodal interactive evaluation system based on synchronous data acquisition is also provided. The evaluation system includes: a synchronization signal transmitter, a video acquisition device, an electrophysiological acquisition device, a processing unit, a user interface, and a data storage module. A synchronization signal transmitter is used to generate and output synchronization pulse sequences to optical and electrical paths. Video acquisition equipment is used to acquire video data, including optical pulses; Electrophysiological acquisition equipment is used to acquire electrophysiological data, including electrical pulses and electrocardiogram signals; The processing unit is used to perform pulse identification, time mapping, R-peak detection, window construction, and data alignment. A user interface for displaying alignment status, signal quality, and evaluation results; The data storage module is used to store the original multimodal data and the aligned timeline information; The synchronization signal generator includes: a main time base crystal oscillator, an encoding and modulation module, an LED driver circuit, and an electrically isolated output circuit; The master time base crystal oscillator is used to generate frequency signals and provide a reference clock source for the synchronization signal generator; The encoding and modulation module is used to generate coded pulse sequences; LED driver circuit, used to drive infrared or visible light; Electrically isolated output circuit, used to output isolated synchronization pulses to electrophysiological acquisition equipment; The processing unit includes: Real-time monitoring of synchronization signal quality, ECG signal quality, and mapping residuals; A degradation strategy is initiated when signal quality deteriorates, and rapid recalibration is performed once the signal is restored.

[0019] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-modal interaction evaluation method based on data synchronous acquisition, characterized in that: The evaluation method comprises: The synchronization signal generator generates a synchronization pulse sequence according to the same encoding time base, and simultaneously outputs the synchronization pulse sequence to the optical path and the electrical path; Video data from the optical path and electrophysiological data from the electrical path are collected; Optical pulse events are identified in the video data, and optical observation pairs corresponding to the optical pulse events are obtained; Electrical pulse events are identified in the electrophysiological data, and electrical observation pairs corresponding to the electrical pulse events are obtained; According to the count number, the mapping relationship of video time and electrophysiological time to the encoding time base is established respectively, and the time drift is continuously estimated and compensated through robust regression and residual monitoring in the sliding window to generate a unified time axis; R-peak events in the electrocardiogram signal are detected on the unified time axis, and a dynamic time window anchored by heartbeats is constructed according to adjacent R-peak intervals; The dynamic time window is mapped to the corresponding interval of video, audio and electroencephalogram data to realize consistent segmentation and feature extraction of cross-modal data.

2. The multi-modal interaction evaluation method based on data synchronous acquisition according to claim 1, characterized in that: The identification of optical pulse events in video data and the acquisition of optical observation pairs corresponding to the optical pulse events comprise: Pulse detection is performed in the preset ROI region of the video frame, the optical pulse events are identified through threshold processing and edge detection algorithm, and the timestamp of the video frame where the optical pulse event is located and the vertical position in the video frame are recorded; If the video acquisition device adopts a rolling shutter mechanism, the timing of the identified optical pulse events is corrected, and the correction formula is as follows: ； wherein t frame represents the timestamp of the video frame in which the optical pulse event is located; N rows represents the vertical resolution number of lines of the video frame; y represents the vertical position of the optical pulse event in the video frame; represents the corrected optical pulse event time; decoding the identified optical pulse event corresponding pulse sequence, including start symbol detection, count field extraction, check field verification, obtaining optical count sequence {c k} consistent with the global count sequence generated by the synchronization signal generator, and recording each count number c k the corresponding corrected video timestamp ; composing an optical observation pair from the decoded optical count sequence and the corresponding corrected time group .

3. The multi-modal interaction evaluation method based on data synchronous acquisition according to claim 1, characterized in that: The identification of electrical pulse events in electrophysiological data and the acquisition of electrical observation pairs corresponding to the electrical pulse events comprise: The original electrical signal is obtained from the isolated synchronization channel of the electrophysiological acquisition device, and the original electrical signal is subjected to band-pass filtering and denoising processing; Thresholding or edge detection of the processed raw electrical signal identifies rising or falling edges of electrical pulse events and records the sample index s in the electrophysiological data stream corresponding to each pulse edge k ; According to the sampling frequency f of the electrophysiological data s The sample index s k is converted into an electrophysiological timestamp The conversion formula is: ； decoding the identified pulse sequence of electrical pulse events, including start symbol, count field extraction, check field verification, obtaining an electrical count sequence {c k} consistent with the global count sequence emitted by the sync signal generator k corresponding corrected electro-physiological time stamp ; The decoded count number c k The electrical observation pair consisting of the corresponding electrophysiological time stamp .

4. The multi-modal interaction evaluation method based on data synchronous acquisition according to claim 1, characterized in that: The mapping relationship of video time and electrophysiological time to the encoding time base is established respectively according to the count number, and the time drift is continuously estimated and compensated through robust regression and residual monitoring in the sliding window to generate a unified time axis, which comprises: An optical observation pair set decoded from the optical pulse events and an electrical observation set decoded from the electrical pulse events are obtained; An affine mapping model of video time and electrophysiological time to the encoding time base count number c is established respectively: ； ； where t e represents the electrophysiological event time; a v , b v are video time mapping parameters; a e , b e are electrophysiological time mapping parameters; are residual terms; Within the sliding time window, robust regression analysis is performed on the optical observation pairs and the electrical observation pairs, respectively, to continuously estimate and update the mapping parameters a v , b v and a e , b e ; The regression residual is calculated, and the statistical distribution of the regression residual is monitored. If the residual exceeds the preset threshold, a recalibration mechanism is triggered or the sliding window is reduced; According to the latest estimated mapping parameters, the video timestamp and the electrophysiological timestamp are uniformly mapped to the global time axis dominated by the encoding time base to generate a unified time axis.

5. The multi-modal interaction evaluation method based on data synchronous acquisition according to claim 1, characterized in that: The detection of R-peak events in the electrocardiogram signal on the unified time axis and the construction of a dynamic time window anchored by heartbeats according to adjacent R-peak intervals comprise: On the unified time axis, R-peak detection is performed on the electrocardiosignal, R-wave peak events in each cardiac cycle are identified and located, and an R-peak time sequence {R i} is obtained, where i is a cardiac cycle number. The time intervals between adjacent R-peaks are computed, resulting in a sequence of RR intervals {RR i}, wherein, ; According to the preset proportion coefficients a and b, taking forward time, taking backward time, constructing a dynamic time window W of heart beat anchoring i : ； Wherein, a and b are dimensionless coefficients.

6. The multi-modal interaction evaluation method based on data synchronous acquisition according to claim 1, characterized in that: The mapping of the dynamic time window to the corresponding interval of video, audio and electroencephalogram data to realize consistent segmentation and feature extraction of cross-modal data comprises: The RR interval sequence is subjected to smoothing processing, and abnormal intervals are removed. The smoothing processing is median filtering or exponential smoothing method, and the abnormal intervals include premature beats, missed detection, and invalid detection caused by insufficient signal-to-noise ratio. mapping each of the dynamic time windows W i to a corresponding interval of the multi-modal data, including: For the electroencephalogram signal, a dynamic time window W is intercepted i The feature extraction is performed on the sample segment in the time range; For video data, the dynamic time window W i is converted to a corresponding sequence of frame indices, and line timing correction is performed; For audio data, extracting dynamic time window W i performing feature analysis on the audio sample segment within the time range; When the ECG signal is unavailable or the quality is below a set threshold, fallback strategies are enabled, using the photoplethysmography peak, heart sound peak or respiratory signal phase as a substitute anchor point, scaling the time window, or falling back to a fixed length window.

7. A multi-modal interaction evaluation system based on data synchronous acquisition, configured to perform the multi-modal interaction evaluation method based on data synchronous acquisition of any one of claims 1-6. The evaluation system comprises a synchronization signal transmitter, a video acquisition device, an electrophysiological acquisition device, a processing unit, a user interface, and a data storage module. The synchronization signal transmitter is configured to generate and output a synchronization pulse sequence to the optical path and the electrical path. The video acquisition device is configured to acquire video data comprising optical pulses. The electrophysiological acquisition device is configured to acquire electrophysiological data comprising electrical pulses and ECG signals. The processing unit is configured to perform pulse identification, time mapping, R-peak detection, window construction, and data alignment. The user interface is configured to display alignment status, signal quality, and evaluation results. The data storage module is configured to store multimodal raw data and aligned time axis information.

8. The multi-modal interaction evaluation system based on data synchronous acquisition according to claim 7, characterized in that: The synchronization signal generator comprises a main time base crystal oscillator, an encoding modulation module, an LED driving circuit, and an electrical isolation output circuit. The main time base crystal oscillator is configured to generate a frequency signal to provide a reference clock source for the synchronization signal generator. The encoding modulation module is configured to generate an encoded pulse sequence. The LED driving circuit is configured to drive infrared or visible light. The electrical isolation output circuit is configured to output an isolated synchronization pulse to the electrophysiological acquisition device.

9. The multi-modal interaction evaluation system based on data synchronous acquisition according to claim 7, characterized in that: The processing unit comprises: Real-time monitoring of synchronization signal quality, ECG signal quality, and mapping residuals. Starting a degradation strategy when signal quality decreases and quickly recalibrating when the signal recovers.

Images