A method and system for extracting high-level semantic information from multiple physiological signals

Abstract

The present invention discloses a method and system for extracting high-level semantic information from a variety of physiological signals, and belongs to the field of time series data processing. Synchronously collected EEG signals and other physiological signals are respectively input into an EEG base model and a physiological encoder, and the semantic representations of the EEG signals and other physiological signals are extracted as high-level semantic information to realize downstream tasks based on physiological signals; the EEG base model is a pre-trained model, and the training process of the physiological encoder includes: given two multi-channel time series of synchronously collected EEG signals and other physiological signals; extracting the semantic representation of the EEG signal through the pre-trained EEG base model; extracting the semantic representation of other physiological signals through the physiological encoder; aligning the semantic representation of the EEG signal with the semantic representation of other physiological signals using a contrastive learning method, thereby training the physiological encoder. The present invention realizes machine understanding of physiological signals and completing a variety of downstream tasks related to medical care.

Description

Technical Field

[0001] The present invention relates to the field of time series data processing, and in particular to a method and system for extracting high-level semantic information from multiple physiological signals. Background Art

[0002] With the maturity of physiological recording technology and the advancement of machine learning methods, physiological signal modeling has attracted many researchers. Initially, researchers focused on model learning of a single type of signal. A large number of works have proposed learning paradigms for various tasks using time series or graph data structures and supervised or self-supervised EEG signals. Recently, some large EEG models have also emerged, breaking through the limitations of EEG for different tasks. Other researchers have proposed methods based on feature engineering or supervised learning to learn the representation of EO, EMG, and ECG signals. In addition, in order to fully explore the potential semantics of physiological data, research has also focused on modeling multiple types of signals. Jia et al. considered the correlation between EEG, EO, and EMG signals to improve the performance on the sleep staging task. Wang et al. fused the features of single-lead EEG and ECG data for emotion recognition. Aly and Youssef proposed a method to integrate EEG with EMG signals to improve the performance of hand and wrist movement control. However, most of the work on physiological signals is task-specific or signal-specific. They neither fail to exploit the underlying model (and are therefore hampered by data scarcity) nor propose a unified framework for various physiological signals across a range of tasks.

[0003] Taking advantage of the information consistency of multimodal data, alignment strategies based on contrastive learning have achieved remarkable results in many fields such as image text. For physiological signal research, there are many works that implement alignment of image features for medical image segmentation. Wang et al. aligned paired medical images and radiology reports (text) for image classification and object detection, etc. Fan et al. proposed a domain adaptation method to bridge the gap between the distribution of EEG data in the source and target domains of sleep staging. Lv et al. enhanced features for emotion recognition by adjusting the visual and auditory modalities in video clips. However, these studies mainly explored the consistency between text, image or audio modalities, and none of them explicitly aligned physiological signals collected simultaneously.

[0004] Time series (TS) analysis has been widely used in fields including finance, meteorology, and healthcare, attracting more and more researchers. Wu et al. proposed TimesNet as a task-general model backbone to adaptively discover the multi-periodicity of TS analysis. Dong et al. proposed to recover masked time points by weighted aggregation of multiple neighbors outside the manifold for TS modeling. Zhou et al. proposed a unified model for TS analysis using language or visual models. Jin et al. proposed a reprogramming framework called Time-LLM to reuse large language models for general TS prediction. However, most TS works cannot adapt well to high-frequency physiological signals and ignore the correlation between physiological signals. Summary of the invention

[0005] In order to solve the problem of lack of data of physiological signals other than EEG signals, the present invention proposes a method for extracting high-level semantic information from multiple physiological signals by utilizing the powerful learning ability of the EEG pedestal model. The present invention extracts semantic information of EEG signals and other physiological signals respectively, aligns the synchronously collected sample pairs, and thus completes the training of physiological signal representation learning, thus realizing the machine's understanding of physiological signal data.

[0006] The present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for extracting high-level semantic information from a variety of physiological signals, wherein synchronously collected EEG signals and other physiological signals are respectively input into an EEG base model and a physiological encoder, and semantic representations of the EEG signals and other physiological signals are extracted as high-level semantic information to implement downstream tasks based on physiological signals; the other physiological signals include any one of electrooculographic signals, electrocardiographic signals, and electromyographic signals;

[0008] The EEG base model is a pre-trained model, and the training process of the physiological encoder includes:

[0009] Given two synchronously collected multi-channel time series of EEG signals and other physiological signals;

[0010] Extract semantic representation of EEG signals through pre-trained EEG pedestal model;

[0011] Extracting semantic representations of other physiological signals through physiological encoders;

[0012] The semantic representation of EEG signals is aligned with that of other physiological signals using contrastive learning to train the physiological encoder.

[0013] Furthermore, the specific method of extracting the semantic representation of EEG signals through the pre-trained EEG pedestal model is:

[0014] The i-th EEG signal sequence x i Split into P continuous segments The segmented segments are fed into the pre-trained EEG pedestal model to generate segment-level semantic representations Among them, p i,j Represents the semantic representation of the jth segment of the ith EEG sequence.

[0015] Furthermore, the specific method of extracting the semantic representation of other physiological signals through the physiological encoder is:

[0016] Other physiological signals collected synchronously with the i-th EEG signal sequence Split into P continuous segments The segmented segments are fed into the physiological encoder to generate segment-level semantic representations. in, Represents the semantic representation of the jth segment of the i-th synchronously collected sequence of other physiological signals.

[0017] Furthermore, the dimension of the semantic representation of the EEG signal is the same as that of the semantic representation of other physiological signals.

[0018] Furthermore, the specific method of aligning the semantic representation of the EEG signal with the semantic representation of other physiological signals by using a contrastive learning method to train the physiological encoder is:

[0019] First, the semantic representation of EEG signals is aligned with the semantic representation of other physiological signals at the segment level: the semantic representation of EEG signal segments p i，j Semantic representation of synchronously acquired physiological signal segments Form a positive sample pair and represent the semantics of the EEG signal fragment p i,j Semantic representation of other physiological signal segments collected asynchronously The negative sample pairs are formed and the segment-level loss is calculated using the contrastive learning method; where P represents the number of segments corresponding to an EEG signal sequence;

[0020] Secondly, the semantic representation of EEG signals is aligned with the semantic representation of other physiological signals at the sequence level:

[0021] The EEG signal sequence x i Linearly map the semantic representation of all the above segments to obtain the EEG signal sequence representation s i ; Other physiological signals collected synchronously with the i-th EEG signal sequence Linearly map the semantic representation of all the above segments to obtain other physiological signal sequence representations The EEG signal sequence is represented by s iThe physiological signal sequence representation collected synchronously Form a positive sample pair and represent the EEG signal sequence as s i Other physiological signal sequences collected asynchronously The negative sample pairs are formed and the sequence-level loss is calculated using the contrastive learning method; where S represents the number of EEG signal sequences;

[0022] Finally, the sum of the segment-level loss and the sequence-level loss is taken as the total loss to train the physiological encoder.

[0023] Furthermore, the dimension of the EEG signal sequence representation is the same as that of other physiological signal sequence representations.

[0024] In a second aspect, the present invention provides a system for extracting high-level semantic information from a variety of physiological signals, for implementing the above method.

[0025] Compared with the prior art, the present invention has the following beneficial effects: the present invention provides a method for extracting high-level semantic information from multiple physiological signals, and aligns the semantic representation of synchronously collected EEG signals with the semantic representation of other physiological signals through comparative learning, thereby realizing the training of physiological encoders. The present invention is not specific to tasks or signals, and can realize subsequent tasks across a series of tasks based on the high-level semantic representation extracted by the physiological encoder from multiple physiological signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the model architecture of the present invention;

[0027] Figure 2 This is a performance comparison of the present invention with existing methods on multiple downstream tasks. DETAILED DESCRIPTION

[0028] The present invention is further described below in conjunction with the accompanying drawings and embodiments. The accompanying drawings are only schematic diagrams of the present invention, and some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities, and these functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0029] like Figure 1 The figure shows a schematic diagram of the model architecture. The forward propagation of this model mainly consists of the following processes:

[0030] 1. Patching.

[0031] Physiological signals require a higher sampling rate to ensure data accuracy and precision, which also brings about the problem of large memory usage and high computational complexity. Therefore, the present invention adopts a segmentation strategy to deal with this problem. Timestamps are aggregated into segments to achieve (1) enhance locality and extract semantic information; (2) reduce computation and memory usage; (3) observe longer time dependencies.

[0032] Specifically, given a multi-channel EEG time series data x i ∈R C×L , where C is the number of EEG signal channels, L is the number of timestamps in the sequence, and the subscript i represents the ith EEG signal sequence. i Split to produce a series of fragments Here x i，j is the jth segment of the i-th EEG signal sequence, is the number of fragments on the channel, It is rounded down.

[0033] 2. Input the segmented EEG signal into the pre-trained EEG pedestal model to extract semantic representation.

[0034] Suppose the i-th EEG signal sequence x i The P consecutive segments of It will be input into the EEG pedestal model, and output the semantic representation of these fragments here represents the representation of the jth segment of the i-th EEG signal sequence, D P is the number of dimensions of the fragment semantic representation.

[0035] In a specific implementation of the present invention, the EEG base model can be implemented using an existing model, which is not limited by the present invention. For example, the Brant-2 model can be used.

[0036] 3. Input other physiological signals into the physiological encoder and obtain semantic representation.

[0037] Assume that the i-th other physiological signal sequence The P consecutive segments of It will be fed into the physiological encoder, which will output the semantic representation of these segments. here represents the representation of the jth segment on the i-th other physiological signal sequence, D P is the number of dimensions of the fragment semantic representation.

[0038] In a specific implementation of the present invention, the physiological encoder can be implemented using an existing model, which is not limited by the present invention. For example, a model of CNN+Transformer architecture can be used.

[0039] 4. The semantic representations collected synchronously are aligned at the segment level using contrastive learning methods.

[0040] EEG signal fragment i,j Physiological signal segments collected synchronously is set as a positive sample pair. The negative sample pair is from other asynchronously collected physiological signal sequences Randomly select from to form a negative sample set The InfoNCE loss function is used to maximize the mutual information between positive sample pairs:

[0041]

[0042] Here, t p Represents the temperature parameter, L p Represents the segment-level InfoNCE loss between EEG signals and other physiological signals. Represents the EEG signal segment p i，j The transpose of , S represents the number of EEG signal sequences.

[0043] 5. Use contrastive learning methods to align the synchronously acquired semantic representations at the sequence level.

[0044] In order to obtain the semantic representation of the sequence from the semantic representation of the fragment, we first i All fragments of Perform a linear mapping Get sequence representation Here D s is the number of dimensions of the sequence semantic representation. The sequence representation s is obtained i Afterwards, the synchronously collected EEG signals and other physiological signals are represented as a sequence (i.e., s i and ) is set as a positive sample. Similarly, all other sample pairs are considered negative samples. The InfoNCE loss function is used to maximize the mutual information between positive sample pairs:

[0045]

[0046] Here, t s Represents the temperature parameter, L s represents the sequence-level InfoNCE loss between EEG signals and other physiological signals, and S represents the number of EEG signal sequences.

[0047] Finally, the segment-level and sequence-level losses are added together, and the overall loss function is:

[0048] L=L p +L s

[0049] This loss function is the optimization objective for training the physiological encoder.

[0050] The process of extracting high-level semantic information from multiple physiological signals using the trained physiological encoder is as follows:

[0051] Synchronous acquisition of multi-channel EEG signal sequence data x i and other physiological signal sequence data The synchronously collected EEG signals and other physiological signals are input into the EEG base model and the physiological encoder respectively, and the semantic representations of the EEG signals and other physiological signals are extracted as high-level semantic information to realize downstream tasks based on physiological signals; the downstream tasks include but are not limited to sleep staging tasks, emotion recognition tasks, gait abnormality detection tasks, eye movement communication tasks, etc.

[0052] The technical effect of the present invention is verified by experiments below.

[0053] (1) Datasets. This embodiment involves two data sets: model alignment training is performed on the alignment training data set, and model effect verification is performed on the downstream task data set.

[0054] a. Alignment training dataset. In order to align the simultaneously recorded EEG and other physiological data, the training data used for unsupervised alignment consists of three datasets: CAP, ISRUC, and HMC, which include EEG, EOG, ECG, and EMG signals. In total, the alignment training data includes 359 recordings from 267 subjects.

[0055] b. Downstream task datasets. This example uses four different datasets to verify the performance of the framework on four different downstream tasks, namely the SleepEDF dataset for sleep staging, the DREAMER dataset for emotion recognition, the FoG dataset for gait abnormality detection, and the ALS dataset for eye movement communication.

[0056] (2) Benchmark. As a method for extracting high-level semantic information from multiple physiological signals, the present invention Brant-X is compared with advanced self-supervised or unsupervised methods designed for general time series, including TF-C and SimMTM. In addition, in order to compare Brant-X with methods that perform time series classification based on pre-trained language models, this embodiment sets OneFitsAll and Time-LLM as baselines. As for supervised methods, MiniRocket is selected as the baseline considering its efficiency and versatility. In addition, this embodiment compares Brant-X with SOTA methods designed specifically for each task to demonstrate the effectiveness of Brant-X in various scenarios. These task-specific or signal-specific supervised methods include: TinySleepNet, XSleepNet, L-SeqSleepNet, SleepPGNN, SleepKD, and SleepDG for sleep staging; MLF-CapsNet, EEG-Conformer, Lin and Wang for emotion recognition; Aly and Youssef, Batool and Javeed, and Goel for freezing gait detection; eyeSay, Adama and Bogdan, and Hossieny for eye communication.

[0057] (3) Experiments on downstream tasks.

[0058] Comparing the test results of the present invention and the baseline model on multiple downstream tasks, as shown in Tables 1-4, the present invention model achieved performance comparable to the optimal baseline in the sleep staging task; all indicators in the emotion recognition, gait freezing detection, and eye movement communication tasks were better than the baseline comparison model. The overall comparison results are shown in the attached Figure 2 shown.

[0059] Table 1: Experimental performance comparison on the sleep staging task

[0060]

[0061] Table 2: Experimental performance comparison on emotion recognition tasks

[0062]

[0063] Table 3: Experimental performance comparison on the gait freezing detection task

[0064]

[0065] Table 4: Comparison of experimental performance on the eye communication task

[0066]

[0067] Based on the same inventive concept, a system for extracting high-level semantic information from multiple physiological signals is also provided in this embodiment, including:

[0068] A physiological signal acquisition module, which is used to synchronously acquire a multi-channel time series of EEG signals and other physiological signals; the other physiological signals include any one of electrooculogram signals, electrocardiogram signals, and electromyography signals;

[0069] The EEG pedestal model module uses a pre-trained EEG pedestal model to extract semantic representations of EEG signals;

[0070] A physiological encoder module for extracting semantic representations of other physiological signals using a physiological encoder;

[0071] A physiological encoder training module, which is used to align the semantic representation of the EEG signal with the semantic representation of other physiological signals using a contrastive learning method, thereby training the physiological encoder;

[0072] A high-level semantic information application module is used to merge the semantic representations of EEG signals and other physiological signals as high-level semantic information to achieve downstream tasks based on physiological signals.

[0073] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment, and the implementation methods of the remaining modules will not be repeated here. The system embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0074] The embodiments of the system of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The system embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the corresponding computer program instructions in the non-volatile memory are read into the memory by the processor of any device with data processing capabilities and run.

[0075] Obviously, the above-described embodiments and drawings are only some examples of the present application. For those of ordinary skill in the art, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that, although the work done in this development process may be complicated and lengthy, for those of ordinary skill in the art, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application. Without departing from the concept of the present application, several variations and improvements can also be made, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A method for extracting high-level semantic information from multiple physiological signals, characterized in that: Inputting the synchronously collected EEG signals and other physiological signals into the EEG base model and the physiological encoder respectively, extracting the semantic representation of the EEG signals and other physiological signals as high-level semantic information to realize downstream tasks based on physiological signals; the other physiological signals include any one of electrooculogram signals, electrocardiogram signals, and electromyography signals; The EEG base model is a pre-trained model, and the training process of the physiological encoder includes: Given two synchronously collected multi-channel time series of EEG signals and other physiological signals; Extract semantic representation of EEG signals through pre-trained EEG pedestal model; Extracting semantic representations of other physiological signals through physiological encoders; The semantic representation of EEG signals and the semantic representation of other physiological signals are aligned using contrastive learning methods to train the physiological encoder; specifically: First, the semantic representation of EEG signals is aligned with the semantic representation of other physiological signals at the segment level: The semantic representation of the EEG signal segment and the semantic representation of the physiological signal segment collected synchronously are combined into a positive sample pair, and the semantic representation of the EEG signal segment and the semantic representation of other physiological signal segments collected asynchronously are combined into a negative sample pair, and the contrastive learning method is used to calculate the segment-level loss; Secondly, the semantic representation of EEG signals is aligned with the semantic representation of other physiological signals at the sequence level: Linearly map the semantic representations of all segments on the EEG signal sequence to obtain the EEG signal sequence representation; linearly map the semantic representations of all segments on other physiological signals collected synchronously with the EEG signal sequence to obtain other physiological signal sequence representations; form positive sample pairs with the EEG signal sequence representation and the physiological signal sequence representation collected synchronously, and form negative sample pairs with the EEG signal sequence representation and the other physiological signal sequence representation collected asynchronously, and use contrastive learning method to calculate sequence-level loss; Finally, the sum of the segment-level loss and the sequence-level loss is taken as the total loss to train the physiological encoder.

2. A method for extracting high-level semantic information from multiple physiological signals according to claim 1, characterized in that: The specific method of extracting the semantic representation of EEG signals through the pre-trained EEG pedestal model is: The i-th EEG signal sequence x i Split into P continuous segments The segmented segments are fed into the pre-trained EEG pedestal model to generate segment-level semantic representations Among them, p i,j Represents the semantic representation of the jth segment of the ith EEG sequence.

3. The method for extracting high-level semantic information from multiple physiological signals according to claim 1, characterized in that: The specific method of extracting the semantic representation of other physiological signals through the physiological encoder is: Other physiological signals collected synchronously with the i-th EEG signal sequence Split into P continuous segments The segmented segments are fed into the physiological encoder to generate segment-level semantic representations. in, Represents the semantic representation of the jth segment of the i-th synchronously collected sequence of other physiological signals.

4. The method for extracting high-level semantic information from multiple physiological signals according to claim 1, characterized in that: The dimension of the semantic representation of the EEG signal is the same as that of the semantic representation of other physiological signals.

5. The method for extracting high-level semantic information from multiple physiological signals according to claim 1, characterized in that: The calculation formula of the segment level loss is as follows: Among them, t p represents the temperature parameter for calculating the segment-level loss, L p represents the segment-level InfoNCE loss between EEG signals and other physiological signals, i.e., segment-level loss; S represents the number of EEG signal sequences, P represents the number of segments corresponding to an EEG signal sequence, and p i,j represents the semantic representation of the jth segment of the ith EEG sequence, represents the semantic representation of the jth segment of the i-th synchronously acquired sequence of other physiological signals, It represents the semantic representation of the nth segment of the mth synchronously collected other physiological signal sequence. The superscript T represents transposition. Representation and semantic representation of EEG signal fragments i,j A set of semantic representations of other physiological signal segments that constitute negative sample pairs.

6. The method for extracting high-level semantic information from multiple physiological signals according to claim 1, characterized in that: The calculation formula of the sequence level loss is as follows: Here, t s represents the temperature parameter for calculating sequence-level loss, L s represents the sequence-level InfoNCE loss between EEG signals and other physiological signals, i.e., sequence-level loss; S represents the number of EEG signal sequences, s i represents the i-th EEG signal sequence, It represents the other physiological signal sequence representation of the ith synchronous acquisition, and the superscript T represents transposition. Representation and EEG signal sequence representation s i Other physiological signal sequences constituting negative sample pairs represent a set.

7. The method for extracting high-level semantic information from multiple physiological signals according to claim 1, characterized in that: The dimension of the EEG signal sequence representation is the same as that of other physiological signal sequence representations.

8. The method for extracting high-level semantic information from multiple physiological signals according to claim 1, characterized in that: The EEG base model adopts the Brant-2 model, and the physiological encoder adopts the CNN+Transformer model.

9. A system for extracting high-level semantic information from multiple physiological signals, characterized in that: include: A physiological signal acquisition module, which is used to synchronously acquire a multi-channel time series of EEG signals and other physiological signals; the other physiological signals include any one of electrooculogram signals, electrocardiogram signals, and electromyography signals; The EEG pedestal model module uses a pre-trained EEG pedestal model to extract semantic representations of EEG signals; A physiological encoder module for extracting semantic representations of other physiological signals using a physiological encoder; The physiological encoder training module is used to align the semantic representation of EEG signals with the semantic representation of other physiological signals using a contrastive learning method, thereby training the physiological encoder; specifically: First, the semantic representation of EEG signals is aligned with the semantic representation of other physiological signals at the segment level: The semantic representation of the EEG signal segment and the semantic representation of the physiological signal segment collected synchronously are combined into a positive sample pair, and the semantic representation of the EEG signal segment and the semantic representation of other physiological signal segments collected asynchronously are combined into a negative sample pair, and the contrastive learning method is used to calculate the segment-level loss; Secondly, the semantic representation of EEG signals is aligned with the semantic representation of other physiological signals at the sequence level: Linearly map the semantic representations of all segments on the EEG signal sequence to obtain the EEG signal sequence representation; linearly map the semantic representations of all segments on other physiological signals collected synchronously with the EEG signal sequence to obtain other physiological signal sequence representations; The EEG signal sequence representation and its synchronously collected physiological signal sequence representation form a positive sample pair, and the EEG signal sequence representation and its asynchronously collected other physiological signal sequence representation form a negative sample pair, and the contrastive learning method is used to calculate the sequence level loss; Finally, the sum of the segment-level loss and the sequence-level loss is taken as the total loss to train the physiological encoder; A high-level semantic information application module is used to merge the semantic representations of EEG signals and other physiological signals as high-level semantic information to achieve downstream tasks based on physiological signals.

Images