Cross-modal time alignment data processing method and system for electroencephalogram and electromyogram signals

Abstract

The present application relates to the technical field of multi-modal physiological signal synchronous processing, and discloses a cross-modal time alignment data processing method and system for electroencephalogram and electromyogram signals, comprising: synchronously collecting double-modal signals and hardware event markers, synchronously sampling and trial intercepting outputting a synchronous data set and double-modal original signals; performing mode-independent direct current removal and zero-phase filtering, outputting a pre-processed double-modal signal set; using full-channel cross-correlation and median aggregation to estimate a trial-level relative time delay, outputting a time delay value, a reliability and a signal set to be corrected; completing alignment correction through directional cutting and equal-length matching, outputting two types of aligned signal sets; performing sliding window slicing and mode-independent normalization, outputting standardized paired samples and a solidified parameter set. The present application does not need to preset a fixed delay, has strong robustness, can eliminate the influence of filtering phase and physiological conduction time delay, guarantees the consistency of cross-batch data distribution, and provides a stable data basis for neuromuscular coupling analysis.

Description

Technical Field

[0001] This invention relates to the field of multimodal physiological signal synchronization processing technology, and in particular to a method and system for cross-modal time-aligned data processing of electroencephalogram (EEG) and electromyogram (EMG) signals. Background Technology

[0002] Electroencephalography (EEG) signals reflect the electrical activity of neuronal populations in the cerebral cortex, corresponding to the central generation process of motor intention; electromyography (EMG) signals reflect the recruitment state of motor units in peripheral muscles, corresponding to the peripheral response process of motor execution. Joint analysis of these two signals can fully present the central-periphery neuromuscular regulatory pathway, possessing core application value in areas such as motor intention decoding, neurorehabilitation training, and human-computer interaction control. However, in actual data acquisition scenarios, even using the same sampling clock from the same acquisition device, the temporal correspondence between the two types of signals is still subject to interference from multiple factors, including event marker transmission, channel scheduling buffering, digital filtering phase shift, and neuromuscular physiological conduction, making it impossible to directly establish a precise sample-level correspondence.

[0003] Existing cross-modal signal processing schemes suffer from several shortcomings: First, most schemes rely solely on hardware sampling clock synchronization, ignoring the combined time delay caused by the superposition of multiple stages. Directly pairing samples by array index introduces systematic time misalignment, leading to incorrect correspondences learned in subsequent analysis. Second, time delay correction often uses manually set fixed delay values, which cannot adapt to time delay differences among different subjects, different exercise modes, different fatigue states, and different acquisition batches, resulting in poor generalization adaptability. Third, some time delay estimation schemes only select a single representative channel to calculate the cross-correlation peak, making them susceptible to the effects of bad leads, transient artifacts, and sudden EMG interference. A single channel anomaly can directly cause the entire estimation to fail, resulting in insufficient robustness. Fourth, conventional bandpass filtering introduces fixed group delay, and the difference in filtering parameters between the two modalities further amplifies the time misalignment. Moreover, this type of systematic error is often misjudged as physiological time delay. Fifth, the inconsistent window partitioning and normalization statistics during the training and inference phases can easily lead to input distribution shifts and performance degradation in online applications. Furthermore, using the current trial data to fit the normalization parameters can introduce data leakage issues. Therefore, there is an urgent need for a cross-modal time-aligned data processing method that features hardware event benchmarks, zero-phase preprocessing, robust latency estimation, and reusable, fixed parameters. Summary of the Invention

[0004] To address the aforementioned technical shortcomings, the purpose of this invention is to propose a cross-modal time-aligned data processing method for EEG and EMG signals. This method aims to solve the technical problems in existing technologies, such as reliance on clock synchronization, poor adaptability to fixed delays, weak robustness of single-channel delay estimation, large filtering phase interference, and data distribution drift caused by inconsistent processing parameters.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: The present invention provides a method for cross-modal time-aligned data processing of electroencephalogram (EEG) and electromyogram (EMG) signals.

[0006] The cross-modal time-aligned data processing method for EEG and EMG signals includes: Step S10: Synchronously acquire EEG signals, EMG signals and hardware event labeling data, and output the synchronous sampling dataset, trial benchmark index set and bimodal raw signal set after sampling clock synchronization and trial truncation preprocessing; Step S20: Based on the synchronous sampling dataset, trial benchmark index set and dual-mode original signal set obtained in step S10, perform submodal DC removal and zero-phase filtering processing to output a preprocessed dual-mode signal set, including the preprocessed brain electron set and the preprocessed muscle electron set. Step S30: Based on the preprocessed dual-mode signal set output in step S20, relative time delay estimation is performed using a full-channel cross-correlation and median aggregation mechanism, and the trial relative time delay estimate, time delay confidence parameter, and dual-mode signal set to be corrected are output. Step S40: Based on the trial relative time delay estimate, time delay confidence parameter and the dual-modal signal set to be corrected output in step S30, time alignment correction is performed using a directional clipping and equal-length matching mechanism, and the aligned EEG signal set, aligned EMG signal set and aligned confidence parameter are output. Step S50: Based on the aligned EEG signal set, aligned EMG signal set, and aligned confidence parameters output in step S40, perform sliding window slicing and modality-independent normalization processing, outputting a standardized paired sample set and a solidified processing parameter set. The solidified processing parameter set is fed back to step S20 to update the filter configuration and to this step to update the normalization benchmark, ensuring consistency across batches.

[0007] Preferably, step S10, which involves simultaneously acquiring EEG signals, EMG signals, and hardware event-labeled data, and performing sampling clock synchronization and trial truncation preprocessing to obtain a synchronized sampling dataset, a trial baseline index set, and a bimodal raw signal set, specifically includes: Step S101: Use the same sampling clock to drive the EEG acquisition channel, EMG acquisition channel and event channel to sample synchronously, and synchronously acquire EEG timing data, EMG timing data and hardware event tag sequence; Step S102: Using the rising edge of the hardware event marker as the zero point of the trial time, extract the dual-mode signal segments of each trial according to the preset time interval to generate a trial reference index set. Step S103: Align the sampling point timestamps of all channel data, remove invalid packet loss segments, and form a synchronous sampling dataset and the corresponding dual-modal raw signal set for each trial.

[0008] Preferably, in step S20, the step of performing mode-specific DC removal and zero-phase filtering processing based on the synchronous sampling dataset, trial reference index set, and dual-mode original signal set obtained in step S10 to obtain a preprocessed dual-mode signal set specifically includes: Step S201: Locate each trial interval based on the trial baseline index set, and perform mean removal processing on the EEG and EMG signals in the bimodal raw signal set by trial and by channel respectively to eliminate DC bias and baseline drift. The mean removal calculation satisfies the following formula:

[0009] in, Let be the original signal amplitude at the t-th sampling point of the c-th channel, T be the total length of a single trial sample, and t be the discrete sampling point index, with a value range of [1, T]. This represents the signal amplitude at the t-th sampling point of the c-th channel after DC removal. Step S202: Perform power frequency notch filtering on both types of signals to suppress power frequency interference; Step S203: Perform zero-phase bandpass filtering on the EEG signal in a preset frequency band and perform zero-phase bandpass filtering on the EMG signal in the corresponding preset frequency band. Both types of filtering adopt a forward and backward bidirectional filtering structure to cancel phase shift, and output a preprocessed dual-modal signal set containing the two types of signals.

[0010] Preferably, step S30, based on the preprocessed dual-mode signal set output in step S20, employs a full-channel cross-correlation and median aggregation mechanism to perform relative delay estimation, obtaining trial-order relative delay estimates, delay reliability parameters, and the dual-mode signal set to be corrected. This step specifically includes: Step S301: Within the preset maximum physiological time delay range, calculate the cross-correlation sequences of all EEG channels and all EMG channels in pairs. The cross-correlation calculation satisfies:

[0011] in, The relative time delay between the i-th EEG channel and the j-th EMG channel The cross-correlation value under the following conditions Let be the signal amplitude at the t-th sampling point of the i-th EEG channel. For the j-th electromyographic channel The signal amplitude at each sampling point Let be the signal mean of the i-th EEG channel. For the j-th electromyographic channel signal, Here, t represents the relative time delay value, and t is the discrete sampling point number. The optimal time delay value corresponding to the cross-correlation peak value of each channel pair is extracted. The optimal time delay satisfies:

[0012] in, The optimal time delay value between the i-th EEG channel and the j-th EMG channel is given. This indicates taking the independent variable that maximizes the function value. This represents the rounding function. The common sampling rate for both types of signals. This is the preset maximum allowable physiological delay, in seconds. This represents the number of sampling points on one side of the time delay search range; Step S302: Remove invalid channel pairs whose relevant peak values ​​are lower than a preset threshold. Take the median of the optimal delay values ​​for the remaining valid channel pairs as the relative delay estimate for the current trial. The trial delay estimate satisfies the following:

[0013] in, This is the overall relative delay estimate for the current trial. This represents the median function. For the effective number of EEG channels, The effective number of electromyographic channels; Step S303: Calculate the time delay reliability parameter based on the effective channel pair ratio and the absolute deviation of the median, and simultaneously encapsulate the preprocessed dual-mode signal into a set of dual-mode signals to be corrected and pass it down, outputting the trial relative time delay estimate, the time delay reliability parameter and the set of dual-mode signals to be corrected.

[0014] Preferably, step S40, based on the trial-level relative time delay estimate, time delay reliability parameter, and the dual-modal signal set to be corrected output in step S30, performs time alignment correction using a directional clipping and equal-length matching mechanism to obtain the aligned EEG signal set, aligned EMG signal set, and aligned reliability parameter, specifically includes: Step S401: Verify whether the delay confidence parameter has reached the qualified threshold. If it has not reached the threshold, mark the current trial as low confidence and do not perform alignment clipping. If it has reached the threshold, proceed to the subsequent clipping process. Step S402: Determine the cropping direction based on the sign of the relative time delay estimate. When the offset is positive, crop the number of samples corresponding to the beginning of the EEG signal and the number of samples corresponding to the end of the EMG signal. When the offset is negative, perform reverse cropping. Step S403: Based on the common shortest length after the two types of signals are cut, perform equal-length truncation to ensure that the number of sample points of EEG and EMG signals are completely consistent, generate the mapping relationship between the original index and the aligned index, and output the aligned EEG signal set, the aligned EMG signal set, and the aligned confidence parameter.

[0015] Preferably, in step S50, the step of performing sliding window slicing and modality-independent normalization processing based on the aligned EEG signal set, aligned EMG signal set, and aligned confidence parameters output in step S40 to obtain the standardized paired sample set and the fixed processing parameter set specifically includes: Step S501: Using a sliding window of preset length and preset step size, synchronously slice the two types of aligned signals. EEG windows and EMG windows at the same time position form pairs of samples. The window slices satisfy the following:

[0016] in, Y is the multi-channel signal matrix output by the k-th window, Y is the aligned single-trial multi-channel signal matrix, and k is the window number. The length of the sampling points in a single window is given by s, where s is the window step size in sampling points. The square brackets indicate the multidimensional array slice index, a Python / NumPy style matrix indexing syntax. The first colon in the square brackets indicates retrieving all channels, preserving all rows of data. ks:ks+w-1 represents a continuous slice in the time dimension, extracting a segment of data from the ks-th sampling point to the ks+w-1-th sampling point. The total length within the window is exactly w sampling points. T is the total sample length of the aligned single-trial signal. This is the floor function; Step S502: EEG and EMG signals are independently normalized using the normalized parameters fitted from the training set. The same batch of normalized parameters is reused in the verification, testing, and online stages without refitting. The normalization uses a minimum-maximum transform, satisfying the following:

[0017] in, The normalized window signal, The original window signal before normalization. This represents the function that takes the minimum value. This represents the function that takes the maximum value. The training set contains all signal data for the corresponding mode. To prevent the denominator from being zero, a smoothing coefficient is used; Step S503: Combine the aligned confidence parameters to label the quality status of each pair of samples, generate a standardized pairwise sample set with labels and quality metadata, and encapsulate the filtering parameters, window parameters, and normalization parameters into a solidified processing parameter set.

[0018] Preferably, step S50 further includes marking the corresponding trial sample as low confidence when the alignment confidence parameter is lower than a preset qualified threshold and not including it in the effective training set. The solidified processing parameter set contains a version identifier, and the same version parameter is called when processing across batches to ensure consistent output data distribution.

[0019] This invention also provides a cross-modal time-aligned data processing system for electroencephalogram (EEG) and electromyogram (EMG) signals, comprising: Synchronous Acquisition and Trial Capture Module: Used to synchronously acquire EEG signals, EMG signals and hardware event-labeled data. After sampling clock synchronization and trial capture preprocessing, it outputs synchronous sampling dataset, trial benchmark index set and bimodal raw signal set. The modal zero-phase preprocessing module is used to perform modal DC removal and zero-phase filtering processing on the synchronous sampling dataset, trial reference index set and dual-modal raw signal set obtained by the synchronous acquisition and trial interception module, and output a preprocessed dual-modal signal set, including the preprocessed brain electron set and the preprocessed muscle electron set. Cross-correlation delay robust estimation module: used to perform relative delay estimation based on the preprocessed dual-mode signal set output by the sub-mode zero-phase preprocessing module, and outputs trial relative delay estimates, delay confidence parameters and dual-mode signal set to be corrected; Directional clipping alignment correction module: Based on the trial relative time delay estimate, time delay confidence parameter and dual-modal signal set to be corrected output by the cross-correlation time delay robust estimation module, it performs time alignment correction by using directional clipping and equal length matching mechanism, and outputs aligned EEG signal set, aligned EMG signal set and aligned confidence parameter. Window normalization and parameter solidification module: Based on the aligned EEG signal set, aligned EMG signal set, and aligned confidence parameters output by the direction clipping alignment correction module, it performs sliding window slicing and modality-independent normalization processing, outputs a standardized paired sample set and a solidified processing parameter set. The solidified processing parameter set is fed back to the modality-specific zero-phase preprocessing module to update the filter configuration and back to this module to update the normalization benchmark, ensuring consistency across batch processing.

[0020] The present invention also provides a cross-modal time-aligned data processing device for electroencephalogram (EEG) and electromyogram (EMG) signals. The cross-modal time-aligned data processing device for EEG and EMG signals includes: a memory, a processor, and a cross-modal time-aligned data processing program for EEG and EMG signals stored in the memory and executable on the processor. When the cross-modal time-aligned data processing program for EEG and EMG signals is executed by the processor, the above-described method is implemented.

[0021] The present invention also provides a computer program product, the computer program product including a cross-modal time-aligned data processing program for electroencephalogram (EEG) and electromyogram (EMG) signals, the cross-modal time-aligned data processing program for EEG and EMG signals implementing the above method when executed by a processor.

[0022] The beneficial effects of this invention are as follows: This invention establishes a common time reference independent of software scheduling and transmission delay by using hardware event markers as the trial time zero point, avoiding time reference offsets caused by thread scheduling and communication jitter, and ensuring the repeatability of trial division. Simultaneously, it employs modal zero-phase filtering, which adapts to the different spectral characteristics of EEG and EMG, and cancels phase offsets through bidirectional filtering, preventing filter group delays from contaminating the true physiological time delay estimation results, thus eliminating system errors from the preprocessing stage. Furthermore, it adopts a time delay estimation mechanism combining full-channel cross-correlation and median aggregation, exhibiting strong robustness to abnormal channels, local artifacts, and sporadic EMG bursts; single-channel anomalies do not affect the overall trial-level estimation results; it also includes a confidence parameter to automatically filter low-quality samples. A directional clipping method is used to achieve time alignment, avoiding the introduction of artificially constructed data such as zero padding and interpolation, preserving the original physiological form of the signal; in addition, a normalization mechanism combining training set parameter fixation ensures the consistency of data distribution during training, validation, and online inference, avoiding cross-batch drift and data leakage. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the first embodiment of the cross-modal time-aligned data processing method for EEG and EMG signals of the present invention. Detailed Implementation

[0024] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0025] Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Example 1: As Figure 1 The diagram shown is a flowchart of the first embodiment of the cross-modal time-aligned data processing method for EEG and EMG signals of the present invention, which presents the first embodiment of the cross-modal time-aligned data processing method for EEG and EMG signals of the present invention.

[0027] In the first embodiment, the cross-modal time-aligned data processing method for EEG and EMG signals includes: Step S10: Synchronously acquire EEG signals, EMG signals and hardware event labeling data, and output the synchronous sampling dataset, trial benchmark index set and bimodal raw signal set after sampling clock synchronization and trial truncation preprocessing; It should be noted that this step uses a common sampling clock to drive all acquisition channels and event channels, ensuring hardware synchronization at the sampling level. Hardware event markers and physiological signals are written to the same buffer, using the rising edge as the zero point of the trial time, rather than the receiving time of the host computer software. This fundamentally avoids millisecond-level jitter caused by thread scheduling, data transmission, and interface refresh. Trial segmentation divides continuous sampling data into individual task segments according to a preset time interval. Timestamp alignment is used to correct sampling offsets between different channels, and packet loss removal is used to remove invalid sampling segments. The three types of output results are input into step S20, providing a unified time reference and data foundation for subsequent preprocessing.

[0028] Understandably, through the synchronous acquisition and trial division in this step, a hardware-level common time reference is established, so that the time starting points of the two types of modal signals have a strict physical correspondence, rather than a simple array index correspondence, providing a reliable reference framework for subsequent accurate time delay estimation.

[0029] It should be understood that traditional synchronization schemes often rely on software labeling or system timestamps, which are easily affected by the operating status of the host computer, resulting in large fluctuations in the trial start point and directly impacting the accuracy of subsequent delay estimations. The hardware event recording mechanism in this step ensures the stability and repeatability of the time base.

[0030] For example, in a certain motor imagery acquisition scenario, a 1000Hz homologous sampling clock drives simultaneous sampling of 6 EEG channels, 6 EMG channels, and 1 event channel. When the subject receives a motor cue, the acquisition device simultaneously writes a hardware event marker. The system uses the rising edge of the marker as the time zero point and extracts a signal segment 2 seconds after the cue as a single trial. After packet loss verification and timestamp alignment, a bimodal raw signal set for each trial is generated.

[0031] Step S20: Based on the synchronous sampling dataset, trial benchmark index set and dual-mode original signal set obtained in step S10, perform submodal DC removal and zero-phase filtering processing to output a preprocessed dual-mode signal set, including the preprocessed brain electron set and the preprocessed muscle electron set. It should be noted that this step locates the signal interval for each trial based on the trial reference index set, and performs mean reduction processing independently by trial and by channel, subtracting the signal mean of that channel within the current trial point by point to eliminate DC drift and baseline fluctuations caused by electrode offset. This operation does not change the time correspondence between channels. Power frequency notch filtering is used to suppress power grid interference and improve the signal-to-noise ratio. Bandpass filtering of the two modes is used for the corresponding frequency bands to match the spectral characteristics of EEG and EMG respectively; all filtering adopts a forward-backward bidirectional zero-phase structure, and the phase shift generated by forward filtering is completely canceled in the reverse filtering process, without introducing additional group delay, thus avoiding misjudging the filtering system error as physiological delay. The output preprocessed dual-mode signal set encapsulates the two types of signals and is passed down to step S30.

[0032] Understandably, this step, while removing noise and preserving effective frequency bands, strictly maintains the original phase relationship between the two types of signals, and will not introduce additional time misalignment due to filtering, thus ensuring the physical authenticity of subsequent time delay estimation.

[0033] It should be understood that conventional unidirectional filtering introduces a fixed group delay. When the parameters of the two types of modal filters are different, the delay difference will be superimposed on the physiological delay, causing the estimation result to deviate from the true value. The zero-phase filtering design in this step eliminates this type of systematic error at the preprocessing level.

[0034] For example: mean-reduction processing is performed on the EEG and EMG signals of each trial to eliminate baseline drift; first, a 50Hz power frequency notch filter is applied to suppress power grid interference; then, a 5-50Hz zero-phase Butterworth bandpass filter is applied to the EEG signal, and a 20-450Hz zero-phase Butterworth bandpass filter is applied to the EMG signal; the bidirectional filtering structure ensures that the phase shift generated by the forward filtering is canceled out in the reverse process, without introducing additional group delay bias.

[0035] Step S30: Based on the preprocessed dual-mode signal set output in step S20, relative time delay estimation is performed using a full-channel cross-correlation and median aggregation mechanism, and the trial relative time delay estimate, time delay confidence parameter, and dual-mode signal set to be corrected are output. It should be noted that this step, within a preset physiological delay range, iterates through all combinations of EEG and EMG channels, calculates the cross-correlation sequence of each corresponding channel pair, and finds the delay corresponding to the cross-correlation peak as the optimal delay for that channel pair. The search range is calculated by combining the preset maximum allowable physiological delay with the sampling rate. Invalid channel pairs with insufficient signal-to-noise ratio are eliminated by using a peak threshold to avoid low-quality data interfering with the estimation results. The median delay value of the remaining valid channel pairs is taken as the overall relative delay estimate for the current trial. The outlier suppression property of the median is used to resist estimation bias caused by single-channel artifacts and sudden EMG interference. The delay reliability comprehensively reflects the proportion of valid channels and the dispersion of delay values, and is used for subsequent sample quality screening. The preprocessed dual-modal signals are synchronously encapsulated into a set of signals to be calibrated and passed down along with the estimation results to avoid subsequent steps calling front-end data across levels.

[0036] Understandably, this step upgrades single-channel estimation to multi-channel robust estimation through full-channel traversal and median aggregation, significantly improving the anti-interference capability of latency estimation; the credibility parameter also provides a quantitative basis for subsequent data quality control.

[0037] It should be understood that single-channel delay estimation has poor fault tolerance; artifacts or missing data in individual channels can cause the entire estimation to fail. The full-channel median aggregation mechanism in this step can still output stable and reliable delay estimation results even if a small number of channels are abnormal, significantly improving robustness.

[0038] For example: within a preset physiological delay range of ±200ms, the normalized cross-correlation of 36 channel pairs (6 EEG and 6 EMG channels) is calculated, and the optimal delay value for each pair is extracted; after removing 3 invalid channel pairs with correlation peaks below the threshold, the median of the remaining 33 delay values ​​is taken as the relative delay estimate for the current trial; the delay confidence level is calculated to be 0.92, which is within the confidence range, by combining the effective channel ratio with the absolute deviation of the median; the preprocessed signal is synchronously encapsulated into a set of signals to be corrected and passed down.

[0039] Step S40: Based on the trial relative time delay estimate, time delay confidence parameter and the dual-modal signal set to be corrected output in step S30, time alignment correction is performed using a directional clipping and equal-length matching mechanism, and the aligned EEG signal set, aligned EMG signal set and aligned confidence parameter are output. It should be noted that this step first verifies the time delay confidence parameter. Trials below the acceptable threshold are directly marked as low confidence and no alignment pruning is performed to avoid introducing larger errors from low-quality estimation. For trials reaching the threshold, the corresponding ends of the two types of signals are pruned according to the positive and negative directions of the time delay. Time alignment is achieved by removing redundant samples. Throughout the process, artificial data is not constructed using methods such as zero padding, cyclic shifting, or interpolation extrapolation, ensuring that the aligned signals all come from the real sampling interval and retain the original physiological morphology. Equal-length pruning ensures that the number of sample points for the two types of signals is completely consistent, ensuring a one-to-one correspondence between the indices of subsequent window slices. The index mapping table records the correspondence between the original position and the aligned position, supporting backtracking and verification of the processing results. The confidence parameter is synchronously passed down as the aligned confidence parameter along with the alignment result for use in subsequent sample labeling. The output two types of aligned signal sets and the confidence parameter are input together into step S50 for standardization processing.

[0040] Understandably, this step uses a pure cropping alignment method to avoid artifacts and boundary abrupt changes introduced by manually constructed data, so that the aligned signal still has real physiological significance and the subsequent analysis results are more reliable; at the same time, the reliability parameter is passed through layer by layer to ensure the continuity of quality control.

[0041] It should be understood that alignment methods such as zero padding and interpolation can introduce artificial abrupt changes at signal boundaries or alter the spectral characteristics of the signal, interfering with subsequent feature extraction and model training. The directional clipping mechanism in this step only removes redundant samples without modifying valid data, thus preserving the original properties of the signal to the greatest extent possible.

[0042] For example: the current trial estimates a relative delay of 30 sampling points, meaning the EMG signal lags behind the EEG signal by 30ms; first, the confidence level is verified to be 0.92, reaching the acceptable threshold, and then alignment and pruning are performed; the first 30 samples of the EEG signal and the last 30 samples of the EMG signal are pruned; then, segments of equal length are truncated based on the common shortest length of the two types of signals to ensure that the number of sample points of the two types of signals is completely consistent, and at the same time, a mapping table from the original index to the aligned index is generated, and the confidence parameter is synchronously transferred to the aligned confidence level.

[0043] Step S50: Based on the aligned EEG signal set, aligned EMG signal set, and aligned confidence parameters output in step S40, perform sliding window slicing and modality-independent normalization processing, outputting a standardized paired sample set and a solidified processing parameter set. The solidified processing parameter set is fed back to step S20 to update the filter configuration and to this step to update the normalization benchmark, ensuring consistency across batches.

[0044] It should be noted that the sliding window simultaneously slices both types of signals, segmenting the signals synchronously according to the preset window length and sliding step size. Each pair of EEG and EMG windows with completely corresponding time positions constitutes a pair of samples, ensuring strict correspondence between time positions. Normalization uses statistical parameters independently fitted to the training set and performs a minimum-maximum scale transformation. In the validation, testing, and online stages, only the parameters are called to perform the transformation without refitting the statistics, which avoids data leakage and ensures the consistency of data distribution across batches. Combined with the aligned confidence parameter, each pair of samples is labeled with its quality status, and low-confidence samples are not included in the effective training set. The solidified parameter set encapsulates all processing parameters such as filtering, windowing, and normalization and assigns version identifiers. Calling the same version of parameters across batches and devices will yield output data with a consistent distribution. The parameters are fed back to step S20 to update the filtering configuration and to this step to update the normalization benchmark, forming a closed-loop solidification mechanism for the processing parameters.

[0045] Understandably, this step completes the conversion from aligned signals to standardized samples, and at the same time, the parameter solidification mechanism enables the reproducibility and reusability of the processing flow, solves the problem of inconsistent training and inference distributions, and ensures the stability of the model in cross-scenario applications.

[0046] It should be understood that if normalized parameters are fitted independently for each batch of data, differences in data distribution across different batches will occur, leading to performance degradation when the model is applied online. The training set parameter fixation and version management mechanism in this step ensures consistency across batches from the data processing level.

[0047] For example: a sliding window with a length of 1000 points and a step size of 500 points is used to synchronously slice the aligned signal, and each pair of windows forms a sample pair; EEG and EMG signals are transformed using the minimum-maximum normalization parameters prefitted from the training set, and the parameters are not refitted during the online testing phase; the alignment confidence is combined to label the quality status of the samples, generating a standardized pairwise sample set with trial labels, time delay values, and confidence; at the same time, the filtering, windowing, and normalization parameters are encapsulated as a reusable parameter set, and the front-end filtering and normalization benchmarks are updated accordingly, and the same version is called across batches to ensure consistent output distribution.

[0048] Example 2: Furthermore, the cross-modal time-aligned data processing system for EEG and EMG signals provided by this invention employs the cross-modal time-aligned data processing method for EEG and EMG signals described in the above embodiments. This system addresses the technical problems in the prior art, such as reliance solely on clock synchronization, poor adaptability to fixed delays, weak robustness of single-channel delay estimation, large filtering phase interference, and data distribution drift caused by inconsistent processing parameters. The beneficial effects of the cross-modal time-aligned data processing system for EEG and EMG signals provided by this invention are the same as those of the cross-modal time-aligned data processing method for EEG and EMG signals provided in the above embodiments. Other technical features of the cross-modal time-aligned data processing system for EEG and EMG signals are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0049] Example 3: This invention provides a cross-modal time-aligned data processing device for electroencephalogram (EEG) and electromyography (EMG) signals. The device includes: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which are executed to enable the at least one processor to perform the cross-modal time-aligned data processing method for EEG and EMG signals described in Example 1. The cross-modal time-aligned data processing device for EEG and EMG signals in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. This cross-modal time-aligned data processing device for EEG and EMG signals is merely an example and should not be construed as limiting the functionality or scope of the invention. A transmodal time-aligned data processing device for EEG and EMG signals may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes based on a program stored in read-only memory or a program loaded from a storage device into random access memory. The random access memory also stores various programs and data required for the operation of the transmodal time-aligned data processing device. The processing unit, read-only memory, and random access memory are interconnected via a bus. An I / O interface is also connected to the bus. Typically, the following systems can be connected to the I / O interface: input devices including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices including, for example, magnetic tapes, hard disks, etc.; and communication devices. The communication device allows the transmodal time-aligned data processing device for EEG and EMG signals to communicate wirelessly or wiredly with other devices to exchange data. While a cross-modal time-aligned data processing device for EEG and EMG signals with various systems has been described, it should be understood that implementation of or possession of all the described systems is not required. Alternatively, more or fewer systems may be implemented.

[0050] Example 4: This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the cross-modal time-aligned data processing method for EEG and EMG signals as described above. The computer program product provided by this invention can solve the technical problems in the prior art, such as reliance on clock synchronization, poor adaptability to fixed delays, weak robustness of single-channel delay estimation, large filtering phase interference, and data distribution drift caused by inconsistent processing parameters. Compared with the prior art, the beneficial effects of the computer program product provided by this invention are the same as those of the cross-modal time-aligned data processing method for EEG and EMG signals provided in the above embodiments, and will not be repeated here.

[0051] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a read-only memory. When the computer program is executed by a processing device, it performs the functions defined in the methods of the embodiments disclosed in this invention.

[0052] It should be understood that the various parts disclosed in this invention can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0053] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the present invention and its equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A method for cross-modal time-aligned data processing of electroencephalogram (EEG) and electromyography (EMG) signals, characterized in that, The methods include: Step S10: Synchronously acquire EEG signals, EMG signals and hardware event labeling data, and output the synchronous sampling dataset, trial benchmark index set and bimodal raw signal set after sampling clock synchronization and trial truncation preprocessing; Step S20: Based on the synchronous sampling dataset, trial benchmark index set and dual-mode original signal set obtained in step S10, perform submodal DC removal and zero-phase filtering processing to output a preprocessed dual-mode signal set, including the preprocessed brain electron set and the preprocessed muscle electron set. Step S30: Based on the preprocessed dual-mode signal set output in step S20, relative time delay estimation is performed using a full-channel cross-correlation and median aggregation mechanism, and the trial relative time delay estimate, time delay confidence parameter, and dual-mode signal set to be corrected are output. Step S40: Based on the trial relative time delay estimate, time delay confidence parameter and the dual-modal signal set to be corrected output in step S30, time alignment correction is performed using a directional clipping and equal-length matching mechanism, and the aligned EEG signal set, aligned EMG signal set and aligned confidence parameter are output. Step S50: Based on the aligned EEG signal set, aligned EMG signal set, and aligned confidence parameters output in step S40, perform sliding window slicing and modality-independent normalization processing, outputting a standardized paired sample set and a solidified processing parameter set. The solidified processing parameter set is fed back to step S20 to update the filter configuration and to this step to update the normalization benchmark.

2. The cross-modal time-aligned data processing method for EEG and EMG signals according to claim 1, characterized in that, In step S10, the steps of simultaneously acquiring EEG signals, EMG signals, and hardware event-labeled data, and obtaining the synchronous sampling dataset, trial baseline index set, and bimodal raw signal set after sampling clock synchronization and trial truncation preprocessing, specifically include: Step S101: Use the same sampling clock to drive the EEG acquisition channel, EMG acquisition channel and event channel to sample synchronously, and synchronously acquire EEG timing data, EMG timing data and hardware event tag sequence; Step S102: Using the rising edge of the hardware event marker as the zero point of the trial time, extract the dual-mode signal segments of each trial according to the preset time interval to generate a trial reference index set. Step S103: Align the sampling point timestamps of all channel data, remove invalid packet loss segments, and form a synchronous sampling dataset and the corresponding dual-modal raw signal set for each trial.

3. The cross-modal time-aligned data processing method for EEG and EMG signals according to claim 1, characterized in that, In step S20, based on the synchronous sampling dataset, trial reference index set, and dual-mode original signal set obtained in step S10, the step of performing mode-specific DC removal and zero-phase filtering to obtain a preprocessed dual-mode signal set specifically includes: Step S201: Locate each trial interval based on the trial baseline index set, and perform mean removal processing on the EEG and EMG signals in the bimodal raw signal set by trial and by channel respectively to eliminate DC bias and baseline drift. The mean removal calculation satisfies the following formula: in, Let be the original signal amplitude at the t-th sampling point of the c-th channel, T be the total length of a single trial sample, and t be the discrete sampling point number, with a value range of [1, T]. This represents the signal amplitude at the t-th sampling point of the c-th channel after DC removal. Step S202: Perform power frequency notch filtering on both types of signals to suppress power frequency interference; Step S203: Perform zero-phase bandpass filtering on the EEG signal in a preset frequency band and perform zero-phase bandpass filtering on the EMG signal in the corresponding preset frequency band. Both types of filtering adopt a forward and backward bidirectional filtering structure to cancel phase shift, and output a preprocessed dual-modal signal set containing the two types of signals.

4. The cross-modal time-aligned data processing method for EEG and EMG signals according to claim 1, characterized in that, In step S30, based on the preprocessed dual-mode signal set output in step S20, relative time delay estimation is performed using a full-channel cross-correlation and median aggregation mechanism to obtain the trial-order relative time delay estimate, time delay reliability parameter, and the dual-mode signal set to be corrected. Specifically, this includes: Step S301: Within the preset maximum physiological time delay range, calculate the cross-correlation sequences of all EEG channels and all EMG channels in pairs. The cross-correlation calculation satisfies: in, The relative time delay between the i-th EEG channel and the j-th EMG channel The cross-correlation value under the following conditions Let be the signal amplitude at the t-th sampling point of the i-th EEG channel. For the j-th electromyographic channel The signal amplitude at each sampling point Let be the signal mean of the i-th EEG channel. For the j-th electromyographic channel signal, Let t be the relative time delay value, and t be the discrete sampling point number. Extract the optimal time delay value corresponding to the cross-correlation peak value of each channel pair. The optimal time delay satisfies: in, The optimal time delay value between the i-th EEG channel and the j-th EMG channel is given. This indicates taking the independent variable that maximizes the function value. This represents the rounding function. The common sampling rate for both types of signals. This is the preset maximum allowable physiological delay, in seconds. This represents the number of sampling points on one side of the time delay search range; Step S302: Remove invalid channel pairs whose relevant peak values ​​are lower than a preset threshold. Take the median of the optimal delay values ​​for the remaining valid channel pairs as the relative delay estimate for the current trial. The trial delay estimate satisfies the following: in, This is the overall relative delay estimate for the current trial. This represents the median function. For the effective number of EEG channels, The effective number of electromyographic channels; Step S303: Calculate the time delay reliability parameter based on the effective channel pair ratio and the absolute deviation of the median, and simultaneously encapsulate the preprocessed dual-mode signal into a set of dual-mode signals to be corrected and pass it down, outputting the trial relative time delay estimate, the time delay reliability parameter and the set of dual-mode signals to be corrected.

5. The cross-modal time-aligned data processing method for EEG and EMG signals according to claim 1, characterized in that, In step S40, based on the trial-level relative time delay estimate, time delay reliability parameter, and the dual-modal signal set to be corrected output in step S30, time alignment correction is performed using a directional clipping and equal-length matching mechanism to obtain the aligned EEG signal set, aligned EMG signal set, and aligned reliability parameter. Specifically, this includes: Step S401: Verify whether the delay confidence parameter has reached the qualified threshold. If it has not reached the threshold, mark the current trial as low confidence and do not perform alignment clipping. If it has reached the threshold, proceed to the subsequent clipping process. Step S402: Determine the cropping direction based on the sign of the relative time delay estimate. When the offset is positive, crop the number of samples corresponding to the beginning of the EEG signal and the number of samples corresponding to the end of the EMG signal. When the offset is negative, perform reverse cropping. Step S403: Based on the common shortest length after the two types of signals are cut, perform equal-length truncation to ensure that the number of sample points of EEG and EMG signals are completely consistent, generate the mapping relationship between the original index and the aligned index, and output the aligned EEG signal set, the aligned EMG signal set, and the aligned confidence parameter.

6. The cross-modal time-aligned data processing method for EEG and EMG signals according to claim 1, characterized in that, In step S50, based on the aligned EEG signal set, aligned EMG signal set, and aligned confidence parameters output in step S40, the steps of performing sliding window slicing and modality-independent normalization processing to obtain the standardized paired sample set and the fixed processing parameter set specifically include: Step S501: Using a sliding window of preset length and preset step size, synchronously slice the two types of aligned signals. EEG windows and EMG windows at the same time position form pairs of samples. The window slices satisfy the following: in, Let Y be the multi-channel signal matrix output by the k-th window, Y be the aligned single-trial multi-channel signal matrix, k be the window index, w be the sampling point length of a single window, and s be the window sliding step size in sampling points. The square brackets indicate multi-dimensional array slice indices, using Python / NumPy style matrix indexing syntax. The first colon (:) within the square brackets indicates retrieving all channels, preserving all rows of data. ks:ks+w-1 represents a continuous slice in the time dimension, extracting a segment of data from the k-th sampling point to the (ks+w-1)-th sampling point, with a total length of exactly w sampling points within the window. T is the total sample length of the aligned single-trial signal. This is the floor function; Step S502: EEG and EMG signals are independently normalized using the normalized parameters fitted from the training set. The same batch of normalized parameters is reused in the verification, testing, and online stages without refitting. The normalization uses a minimum-maximum transform, satisfying the following: in, The normalized window signal, The original window signal before normalization. This represents the function that takes the minimum value. This represents the function that takes the maximum value. The training set contains all signal data for the corresponding mode. To prevent the smoothing coefficient from being zero in the denominator; Step S503: Combine the aligned confidence parameters to label the quality status of each pair of samples, generate a standardized pairwise sample set with labels and quality metadata, and encapsulate the filtering parameters, window parameters, and normalization parameters into a solidified processing parameter set.

7. The cross-modal time-aligned data processing method for EEG and EMG signals according to claim 1, characterized in that, Step S50 further includes marking the corresponding trial sample as a low-confidence state when the alignment confidence parameter is lower than the preset qualified threshold and not including it in the effective training set. The solidified processing parameter set contains a version identifier, and the same version parameter is called when processing across batches to ensure consistent output data distribution.

8. A cross-modal time-aligned data processing system for electroencephalogram (EEG) and electromyogram (EMG) signals, applied to the cross-modal time-aligned data processing method for EEG and EMG signals as described in any one of claims 1 to 7, characterized in that, The cross-modal time-aligned data processing system for EEG and EMG signals includes: Synchronous Acquisition and Trial Capture Module: Used to synchronously acquire EEG signals, EMG signals and hardware event-labeled data. After sampling clock synchronization and trial capture preprocessing, it outputs synchronous sampling dataset, trial benchmark index set and bimodal raw signal set. The modal zero-phase preprocessing module is used to perform modal DC removal and zero-phase filtering processing on the synchronous sampling dataset, trial reference index set and dual-modal raw signal set obtained by the synchronous acquisition and trial interception module, and output a preprocessed dual-modal signal set, including the preprocessed brain electron set and the preprocessed muscle electron set. Cross-correlation delay robust estimation module: used to perform relative delay estimation based on the preprocessed dual-mode signal set output by the sub-mode zero-phase preprocessing module, and outputs trial relative delay estimates, delay confidence parameters and dual-mode signal set to be corrected; Directional clipping alignment correction module: Based on the trial relative time delay estimate, time delay confidence parameter and dual-modal signal set to be corrected output by the cross-correlation time delay robust estimation module, it performs time alignment correction by using directional clipping and equal length matching mechanism, and outputs aligned EEG signal set, aligned EMG signal set and aligned confidence parameter. Window normalization and parameter solidification module: Based on the aligned EEG signal set, aligned EMG signal set, and aligned confidence parameters output by the direction clipping alignment correction module, it performs sliding window slicing and modality-independent normalization processing, outputs a standardized paired sample set and a solidified processing parameter set. The solidified processing parameter set is fed back to the modality-specific zero-phase preprocessing module to update the filter configuration and back to this module to update the normalization benchmark, ensuring consistency across batch processing.

9. A cross-modal time-aligned data processing device for electroencephalogram (EEG) and electromyogram (EMG) signals, characterized in that, The transmodal time-aligned data processing device for EEG and EMG signals includes: a memory, a processor, and a transmodal time-aligned data processing program for EEG and EMG signals stored in the memory and executable on the processor. When the transmodal time-aligned data processing program for EEG and EMG signals is executed by the processor, it implements the transmodal time-aligned data processing method for EEG and EMG signals according to any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a cross-modal time-aligned data processing program for EEG and EMG signals, which, when executed by a processor, implements the cross-modal time-aligned data processing method for EEG and EMG signals as described in any one of claims 1 to 7.

Images