Physiological signal self-supervision representation learning method and system based on time-frequency reconstruction

Abstract

The invention provides a physiological signal self-supervision representation learning method and system based on time-frequency reconstruction, and belongs to the technical field of signal processing and mode recognition. Comprising the following steps: acquiring original multi-channel physiological signal data and converting the data into a time-frequency diagram; carrying out segmentation / blocking, random masking and feature conversion on the physiological signal and the time-frequency diagram thereof to obtain a primary time domain representation and a primary frequency domain representation; coding and fusing the preliminary time domain representation and the frequency domain representation in a time dimension and a channel dimension to obtain a time-frequency mixed representation; pre-training an encoder by adopting a reconstruction learning method based on the time-frequency mixed representation; a pre-trained encoder is used for obtaining multi-channel time-frequency representation of to-be-processed original multi-channel physiological signal data, the multi-channel time-frequency representation is mixed to obtain time-frequency mixed representation, and the time-frequency mixed representation serves as learned physiological signal self-supervision representation and is used for a physiological disease classification task. The method has higher generalization ability and adaptability, and an effective solution is provided for a physiological signal classification task.

Description

Technical Field

[0001] The present invention relates to the technical field of signal processing and pattern recognition, and in particular to a physiological signal self-supervisory representation learning method and system based on time-frequency reconstruction. Background Art

[0002] The patterns of physiological signals can greatly help us understand how the human body normally functions and the mechanisms of related diseases. There are many studies on physiological signals, such as cognitive research, emotion recognition, epilepsy detection based on scalp electroencephalogram (EEG); heart rhythm detection based on electrocardiogram (ECG), and motor nerve disease monitoring based on electromyography (EMG). However, it is usually very difficult to obtain labels for these physiological signal data. At the same time, due to individual differences and different disease attack patterns in different patients, models trained with physiological signal data of some patients often perform poorly on other patients. As self-supervised learning has become the dominant learning paradigm in the absence of labels, more and more work is focused on using self-supervised learning to learn effective general representations in massive unlabeled data, which not only reduces label dependence but also makes the model more generalizable.

[0003] Since physiological signals are essentially time series data, general time series self-supervised algorithms can also be used for this task. For example, TimeCLR[Yang, 2022] proposed a method of DTW data enhancement for time series data for contrastive learning. TS2vec[Yue, 2022] designed a cropping and masking mechanism to perform contrastive learning on enhanced context views in a hierarchical manner to provide contextual representations for each timestamp. TS-TCC[Eldele, 2021] learns robust and discriminative representations by applying time contrast modules and context contrast modules. At the same time, there are also some algorithms designed based on specific physiological signals; CLOCS[Kiyasseh, 2021] proposed a method for contrastive learning of ECGs using sample and patient levels. Mixing-up[ 2022] captures the consistency of samples through a hybrid data augmentation scheme. TNC [Tonekaboni, 2021] compares neighboring samples in the same channel signal as positive sample pairs.

[0004] However, most of these methods only focus on the characteristics of time series in the time domain, while ignoring the importance of frequency information for physiological signals: the frequency characteristics of physiological signals are an important means of identifying different physiological states and disease states. For example, in electroencephalograms, different frequency bands are associated with different brain activity states. Alpha waves (8-13Hz) are usually associated with relaxation and closed eyes, while theta waves (4-8Hz) may be associated with deep relaxation or certain types of epileptic activity. By identifying these specific frequency features, doctors can diagnose whether the patient is in a specific mental state or has certain neurological diseases. In electrocardiogram analysis, arrhythmia detection can be achieved by analyzing the frequency changes of heart rate. Filtering and spectrum analysis can help extract useful frequency information from noisy ECG signals. By identifying abnormal frequency patterns of heart rate, such as tachycardia or bradycardia, doctors can diagnose heart disease and take corresponding treatment measures. At the same time, due to the sensitivity of the acquisition environment and data characteristics of physiological signals, using data enhancement methods to do contrastive learning will disrupt the consistent labels of the original sequence, resulting in label confusion and destroying the characteristics of the data itself. Summary of the invention

[0005] In order to solve the problem that the existing physiological signal classification methods ignore the frequency domain information and destroy the original data, the present invention proposes a physiological signal self-supervised representation learning method and system based on time-frequency reconstruction, which realizes the accurate classification of physiological diseases by fully integrating the relevant information of the original signal in the time domain and frequency domain.

[0006] The present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for learning physiological signal self-supervised representation based on time-frequency reconstruction, comprising:

[0008] Obtain original multi-channel physiological signal data and convert it into time-frequency diagram;

[0009] The original multi-channel physiological signal data is segmented, one-dimensionally randomly masked, and feature transformed to obtain a preliminary time domain representation; a time domain encoder is used to encode the data in the time dimension and the channel dimension to obtain the final time domain representation;

[0010] The time-frequency graph is divided into blocks, subjected to two-dimensional random masking and feature conversion operations to obtain a preliminary frequency domain representation; a frequency domain encoder is used to encode in the time dimension and channel dimension respectively to obtain the final frequency domain representation;

[0011] Using a time-frequency hybrid encoder to fuse the final time domain representation and the final frequency domain representation to obtain a time-frequency hybrid representation; based on the time-frequency hybrid representation, reconstructing the original multi-channel physiological signal and its time-frequency graph respectively to obtain a time domain loss and a frequency domain loss; combining the time domain loss and the frequency domain loss, pre-training the time domain encoder, the frequency domain encoder and the time-frequency hybrid encoder;

[0012] A multi-channel time-frequency representation of the original multi-channel physiological signal data to be processed is obtained using a pre-trained time domain encoder and a frequency domain encoder, wherein the multi-channel time-frequency representation includes a final time domain representation and a final frequency domain representation, and a pre-trained time-frequency mixed encoder is used to obtain a time-frequency mixed representation, which is used as a learned self-supervised representation of the physiological signal for the task of physiological disease classification.

[0013] Furthermore, the original multi-channel physiological signal data is any one of electroencephalogram data, electrocardiogram data, electrooculogram data, and electromyogram data.

[0014] Furthermore, the time-frequency diagram is obtained by short-time Fourier transform of original multi-channel physiological signal data.

[0015] Furthermore, the segmentation, one-dimensional random masking and feature conversion operations are performed on the original multi-channel physiological signal data to obtain a preliminary time domain representation, including:

[0016] Given the original multi-channel physiological signal data, for any channel sequence, split it into a series of overlapping fragment combinations and perform one-dimensional random masking; traverse all channels;

[0017] Linearly map the one-dimensional randomly masked fragment combination into a high-dimensional feature vector, and add position encoding as a preliminary time domain representation;

[0018] The one-dimensional random mask is: randomly sampling a certain proportion of fragments from the fragment combination and setting them to zero.

[0019] Furthermore, the time-frequency graph is divided into blocks, subjected to two-dimensional random masking and feature conversion operations to obtain a preliminary frequency domain representation, including:

[0020] Given any channel time-frequency diagram corresponding to the original multi-channel physiological signal data, divide the blocks into frequency domain block sequences and perform two-dimensional random masking; traverse all channels;

[0021] Linearly map the two-dimensional randomly masked frequency domain block sequence into a high-dimensional feature vector, and add position encoding as a preliminary frequency domain representation;

[0022] The two-dimensional random mask adopts a masking method combining time resolution and frequency resolution, that is, a frequency domain block in a row is set to zero along the time axis, and a frequency domain block in a column is set to zero along the frequency axis.

[0023] Furthermore, the time domain encoder adopts a Transformer encoder, and the time domain encoder is used to encode in the time dimension and the channel dimension respectively to obtain the final time domain representation, which is expressed as:

[0024] T′=Transformer(T)

[0025] T rep = Transformer(T time )

[0026] Where T is the initial time domain representation, Transformer(.) is the Transformer encoder, T′ is the representation after encoding in the time dimension, and T time is the two-dimensional vector obtained by expanding T′, T rep is the final time domain representation.

[0027] Furthermore, the frequency domain encoder adopts a multi-layer perceptron, and the frequency domain encoder is used to encode in the time dimension and the channel dimension respectively to obtain the final frequency domain representation, which is expressed as:

[0028] H′=H+GA(MLP(H))

[0029] Z rep =H time +GA(MLP(H time ))

[0030] Among them, H is the preliminary frequency domain representation, MLP(.) is the multi-layer perceptron, GA(.) is the gated attention mechanism, and H time is the frequency domain representation obtained by swapping the first and last dimensions of H′, Z rep is the final frequency domain representation.

[0031] Furthermore, the time-frequency hybrid encoder adopts a pooling operation.

[0032] Furthermore, before being used for the physiological disease classification task, a fine-tuning stage is also included, including:

[0033] Obtain a fine-tuning training set containing multi-channel physiological signal data and its corresponding disease labels;

[0034] The pre-trained time domain encoder, frequency domain encoder, and time-frequency hybrid encoder are used to obtain multi-channel time-frequency fusion representation of multi-channel physiological signal data, the disease type is predicted through the classifier, the loss is calculated based on the disease label, and the classifier and / or encoder are fine-tuned.

[0035] In a second aspect, the present invention provides a physiological signal self-supervised representation learning system based on time-frequency reconstruction, which is used to implement the above-mentioned physiological signal self-supervised representation learning method based on time-frequency reconstruction.

[0036] Compared with the prior art, the present invention has the following beneficial effects: the present invention fully exploits the characteristics of physiological signals in the time domain and frequency domain, effectively combines the time domain and frequency domain representations together, has higher accuracy, generalization ability, and adaptability, and provides an effective solution for the task of physiological signal classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of a physiological signal self-supervised learning method based on time-frequency reconstruction according to an exemplary embodiment.

[0038] Figure 2 is a schematic diagram of a Transformer encoder according to an exemplary embodiment.

[0039] Figure 3 is a schematic diagram of a multi-layer perceptron encoder according to an exemplary embodiment.

[0040] Figure 4 is a schematic diagram of a gated attention mechanism according to an exemplary embodiment. DETAILED DESCRIPTION

[0041] The present invention is further described below in conjunction with the accompanying drawings and embodiments. The accompanying drawings are only schematic diagrams of the present invention, and some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities, and these functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0042] The goal of the present invention is to fully explore the characteristics of physiological signals in the time domain and frequency domain, use the data of these two domains to pre-train the model, and obtain an encoder with strong generalization performance, so as to improve the accuracy and generalization of physiological disease classification during fine-tuning. Specifically, the model used in the physiological signal self-supervised representation learning method is denoted as TFformer, which is a deep learning framework based on self-supervised learning, mask reconstruction, and Transformer. The present invention can achieve a given series of continuous multi-channel physiological signal data of fixed duration, and output an effective representation of each channel through a pre-trained or further fine-tuned encoder, which can be applied to downstream physiological disease classification tasks after aggregation. Specifically, the present invention first performs representation learning on the original physiological signal data; then, the learned representation is used for fine-tuning training of disease classification.

[0043] The overall framework of the present invention (TFformer) consists of three main modules: time domain feature learning module, frequency domain feature learning module, and time-frequency fusion module. The time domain feature learning module is based on the characteristics of physiological signals in the time domain, uses the Transformer encoder to perform encoding modeling in the channel and time dimensions, and performs temporal feature learning through mask reconstruction tasks; the frequency domain feature learning module is based on the characteristics of physiological signals in the frequency domain, uses a multi-layer perceptron to perform encoding modeling in the channel and time dimensions; the time-frequency fusion module combines the time domain features and the frequency domain features together to obtain the final time-frequency mixed representation for performing reconstruction tasks.

[0044] Figure 1 The figure shows a schematic diagram of the framework of the TFformer model proposed in the present invention, including a pre-training stage and a fine-tuning stage. In the pre-training stage, the left side takes the original multi-channel physiological signal data as input, and performs the time domain reconstruction task through the time domain feature learning module; the right side takes the original multi-channel physiological signal data after the conversion of the time-frequency graph data as input, and performs the frequency domain reconstruction task through the frequency domain feature learning module. Through the combination of these two tasks, TFformer can learn effective representations from unlabeled physiological signal data.

[0045] Each module is introduced below.

[0046] (1) Time domain feature learning module

[0047] For any channel of the original multi-channel physiological signal First, we need to divide the original signal into a series of overlapping segments of length P according to a certain step size W, denoted as Where N represents the number of segments after segmentation, Indicates the round-down symbol, and the last segment whose length is less than P is padded with the last value in the segment.

[0048] For each time series x i The segmented fragment combination x pi , using a learnable linear transformation Map it into a set of high-dimensional feature vectors, each segmented fragment corresponds to a feature vector, where D is the dimension of each feature vector; then, add a learnable position code to all fragments Right now in A single physiological signal sequence x i Converted to a preliminary time domain representation t i .

[0049] At this time, for the multi-channel physiological signal A set of preliminary time domain representations were generated C is the number of channels of the physiological signal.

[0050] In a specific implementation of the present invention, a Transformer encoder is used as a time domain encoder to encode the time dimension and the channel dimension:

[0051] T′=Transformer(T)

[0052] T rep = Transformer(T time )

[0053] Among them, T′ is the representation after encoding in the time dimension, which is expanded into a two-dimensional vector and used as the self-representation vector after the Transformer encoder encoding where d h =D×N. It is the representation after encoding the channel dimension.

[0054] The structure of the Transformer encoder is as follows Figure 2 As shown in the figure, it is composed of multiple layers stacked together. A single-layer Transformer encoder consists of a multi-head attention mechanism module and a feedforward neural network, accompanied by residual connections and batch normalization layers.

[0055] The multi-head attention mechanism is composed of multiple attention heads. For each attention head h = 1, 2, ..., H, assuming the input is x i First, three learnable linear transformations are used to transform the feature matrix z i Mapped to query matrix Q i,h , key matrix K i,h , value matrix V i,h ; Then, do a dot product between each query vector and each key vector, and use the result to normalize it using Softmax as the weight to perform weighted summation on each value vector, thus obtaining the feature matrix z i The result obtained after the attention head h

[0056]

[0057] Here the inner product of each query vector and each key vector needs to be divided by the factor Prevent the gradient of the Softmax function from disappearing.

[0058] The multi-head attention mechanism actually uses multiple sets of independent attention mechanisms, which expands the model's ability to focus on different positions and provides multiple representation subspaces for the attention layer. Finally, the results of multiple sets of attention mechanisms are concatenated to obtain zi The final output after the multi-head attention mechanism

[0059]

[0060] Concat(.) represents the concatenation operation, and the default value is d v =D / H, that is, the dimension of each vector is restored to D after the outputs of H attention heads are concatenated.

[0061] The feedforward network module consists of two fully connected layers, which include the GeLU activation function and the Dropout random inactivation operation.

[0062] (2) Frequency domain feature learning module

[0063] For any channel of the original multi-channel physiological signal Apply short-time Fourier transform to convert it into a time-frequency graph s i .

[0064] The specific process of short-time Fourier transform is as follows:

[0065]

[0066] Among them, x i represents the physiological signal sequence of the i-th channel, w(t-τ) represents the window function used to intercept the signal, Represents Fourier transform. s i (ω,τ) represents the spectrum result after Fourier transform of the original physiological signal when the center of the window function is τ. The resulting time-frequency matrix is ​​expressed as m represents the frequency value corresponding to each point after Fourier transform, and n represents the center time of the window.

[0067] Given a physiological signal sequence x i The corresponding time-frequency diagram s i , perform block division and feature conversion operations on it to obtain the preliminary frequency domain representation h of the corresponding slice i :

[0068]

[0069] Among them, W f is a trainable linear projection matrix, is the time-frequency diagram i The frequency domain block sequence after the block operation, W f_posis the learnable position encoding matrix; it is worth noting that the time-frequency diagram here has one more dimension than the original signal. When performing block division, the block division operation in Vision Transformer can be imitated, which will not be repeated here. D f is the dimension of each feature vector, M is the number of frequency domain blocks, a and b are the length and width of the frequency domain block, respectively, where a is divisible by m and b is divisible by n. i The corresponding time-frequency diagram s i Converted into a preliminary frequency domain representation h i .

[0070] At this time, for the multi-channel physiological signal The corresponding time-frequency diagram A set of preliminary frequency domain representations are generated through the frequency domain feature learning module C is the number of channels of the physiological signal.

[0071] In a specific implementation of the present invention, a multi-layer perceptron is used as a frequency domain encoder to encode the time dimension and the channel dimension, and finally the frequency domain representation Z is obtained. rep :

[0072] H′=H+GA(MLP(H))

[0073] Z rep =H time +GA(MLP(H time ))

[0074] Among them, H′ is the frequency domain representation after time dimension modeling and encoding. By exchanging its first and last dimensions, we get Z rep It is the frequency domain representation after channel dimension encoding. MLP is a multi-layer perceptron, and GA is a gated attention mechanism. Since the number of blocks in the two-dimensional time-frequency graph is much larger than that of the one-dimensional time series graph, the use of a multi-layer perceptron can reduce the complexity of modeling and learn the correlation of the time-frequency blocks. At the same time, the application of gated attention can enable the model to filter out unimportant features.

[0075] The structure of the multi-layer perceptron MLP is as follows Figure 3 As shown in , it is composed of multiple linear layers superimposed, accompanied by Dropout random inactivation and activation function. The gated attention mechanism is as follows Figure 4 As shown in Figure 1, it consists of a linear layer, and the attention weight is calculated with a normalized exponential function, and the dot product operation is applied to update the result. The gated attention operation process is as follows:

[0076] Assuming that the output of MLP is Y, the attention weight WA The calculation is as follows:

[0077]

[0078] Among them, softmax is a normalized exponential function, is a linear layer mapping function, then the result GA(Y) is expressed as follows:

[0079] GA(Y)=W A ·Y

[0080] (3) Time-frequency fusion module

[0081] The time-frequency fusion module fuses the output of the time-domain feature learning module and the output of the frequency-domain feature learning module to obtain the time-frequency mixed feature H. rep , and based on the time-frequency mixed feature H rep Use the time-domain linear layer as the time-domain reconstruction head to predict the original time series data Use the frequency domain linear layer as the frequency domain reconstruction head to predict the original time-frequency map The calculation process is as follows:

[0082] H bep =Pooling(T rep ,Z rep )

[0083]

[0084] in, is the time domain linear mapping function, is the time-domain linear mapping function, and Pooling(.) is the pooling operation.

[0085] In the pre-training stage, each multi-channel physiological signal data X in the pre-training set i No label is required. After segmenting / blocking the physiological signal data of a channel and its corresponding time-frequency graph, for each time series x i The segmented fragment combination x pi , further one-dimensional random masking is required, that is, a certain proportion of segments are randomly sampled and set to 0. i The frequency domain block sequence obtained after block division It is also necessary to further perform two-dimensional random masking, that is, to perform two-dimensional random masking on the frequency domain block sequence. Randomly sample a certain proportion of frequency domain blocks and set them to 0. In this embodiment, the two-dimensional random mask adopts a masking method that combines time resolution and frequency resolution, that is, a certain row of frequency domain blocks is set to zero along the time axis, and a certain column of frequency domain blocks is set to zero along the frequency axis. The masked result is further processed to finally obtain the reconstruction result The reconstruction error is calculated as:

[0086]

[0087] L=λL t +(1-λ)L f

[0088] Among them, L is the total loss, L t is the time domain reconstruction loss, L f is the frequency domain reconstruction loss, M is the number of multi-channel physiological signal samples in the pre-training batch, and λ is the regularization parameter.

[0089] Combine the above two losses to perform back-propagation training on the model and update the model parameters.

[0090] In the fine-tuning stage, each multi-channel physiological signal data X in the training set is fine-tuned. i Corresponding to a disease label, the above-mentioned time domain feature learning module, frequency domain feature learning module, and time-frequency fusion module are used to obtain X i The multi-channel time-frequency fusion representation is used to predict the disease type through the classifier, the loss is calculated based on the disease label, and the classifier and / or encoder are fine-tuned.

[0091] Finally, the fine-tuned encoder is used to obtain a multi-channel time-frequency representation of the original multi-channel physiological signal data to be processed, and a multi-channel time-frequency fusion representation is obtained. The multi-channel fusion representation is used as the learned physiological signal self-supervised representation for physiological disease classification tasks.

[0092] The above-mentioned fine-tuning stage can freeze the pre-trained encoder parameters and only train the classifier, or train the encoder and the classifier at the same time, or adopt other fine-tuning methods known to those skilled in the art, which are not limited by the present invention and all belong to the protection scope of the present invention.

[0093] Based on the same inventive concept, a physiological signal self-supervised representation learning system based on time-frequency reconstruction is also provided in this embodiment. The system is used to implement the above-mentioned embodiment, and the descriptions that have been made will not be repeated. The terms "module", "unit", etc. used below can implement a combination of software and / or hardware with predetermined functions. Although the systems described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible.

[0094] The system comprises:

[0095] A data preprocessing module, which is used to obtain the original multi-channel physiological signal data and convert it into a time-frequency diagram;

[0096] The time domain feature learning module is used to segment and transform the original multi-channel physiological signal data to obtain a preliminary time domain representation; the time domain encoder is used to encode the data in the time dimension and the channel dimension to obtain the final time domain representation; the pre-training stage also includes a one-dimensional random masking operation on the segmented sequence;

[0097] The frequency domain feature learning module is used to perform block and feature conversion operations on the time-frequency graph to obtain a preliminary frequency domain representation; the frequency domain encoder is used to encode in the time dimension and channel dimension respectively to obtain the final frequency domain representation; the pre-training stage also includes a two-dimensional random masking operation on the block sequence;

[0098] A time-frequency fusion module, which is used to fuse the final time domain representation and the final frequency domain representation using a time-frequency hybrid encoder to obtain a time-frequency hybrid representation;

[0099] The reconstruction learning module is used to reconstruct the original multi-channel physiological signals and their time-frequency diagrams using time-frequency mixed representations, calculate the losses of the time domain reconstruction task and the frequency domain reconstruction task, and train the encoder.

[0100] A self-supervised representation generation module is used to obtain a multi-channel time-frequency representation of the original multi-channel physiological signal data to be processed using a pre-trained time domain encoder and a frequency domain encoder, wherein the multi-channel time-frequency representation includes a final time domain representation and a final frequency domain representation, and a pre-trained time-frequency mixed encoder is used to obtain a time-frequency mixed representation, and the time-frequency mixed representation is used as the learned physiological signal self-supervised representation for physiological disease classification tasks.

[0101] The implementation process of the functions and effects of each module in the above-mentioned system is specifically detailed in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here. For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The system embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0102] The embodiments of the system of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The system embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the corresponding computer program instructions in the non-volatile memory are read into the memory by the processor of any device with data processing capabilities and run.

[0103] This embodiment verifies the implementation effect of the present invention through a specific experiment.

[0104] (1) Data description

[0105] AD dataset. This dataset includes EEG recordings from 12 patients with Alzheimer's disease and 11 healthy controls. Each patient had an average of 30.0±12.5 trials. Each trial corresponds to a 5-second interval, with 1280 timestamps (sampled at 256Hz), and a total of 16 channels. During the data preprocessing stage, each trial was split into nine semi-overlapping samples, each of which lasted for 256 timestamps. In addition, the training set, validation set, and test set were separated in a patient-independent manner. Samples from patients ID17 and ID18 were used as validation sets, and samples from ID19 and ID20 were used as test sets. All remaining samples were put into the training set.

[0106] PTB dataset. This dataset consists of ECG electrocardiogram records of 290 patients, with 15 channels sampled at a frequency of 1000Hz. There are a total of 8 types of heart disease in the dataset. This experiment focuses on binary classification using a subset of the dataset, which includes 198 myocardial infarction patients and healthy controls. After resampling to a frequency of 250Hz, they are first normalized. In addition, the training set, validation set, and test set are separated in a patient-independent manner. Samples from 28 patients (7 healthy and 21 positive) are used as the validation set, and samples from another 28 patients (7 healthy patients and 21 positive) are used as the test set. All the remaining samples are put into the training set.

[0107] (2) Baseline method.

[0108] This experiment compares the present invention with several existing benchmark methods.

[0109] TS2vec introduces contextual consistency using overlapping subsequences and a hierarchical loss function to capture data consistency at both observation and sample levels.

[0110] Mixing-up proposes a method of mixing enhancement by mixing the ratio of two time series samples. This enhancement involves creating an enhanced sample as a convex combination of two time series samples randomly selected from the dataset.

[0111] TF-C exploits the consistency of time and frequency domains. This method assumes that the time-based and frequency-based representations of the same instance are close in the time-frequency space, and designs the contrastive learning loss based on this.

[0112] SimCLR is the most classic contrastive learning framework first proposed in the CV field. It applies data augmentation techniques to create enhanced views of input samples and constructs contrastive losses based on these views.

[0113] (3) Physiological disease classification experiment

[0114] In the physiological disease classification experiment, the model is first pre-trained on unlabeled data. After the pre-training phase is completed, the multi-channel representation learned by the model is used for the downstream physiological disease classification task. Since there will be interactions between physiological signal channels, the present invention takes into account the correlation between channels. Therefore, the above method / system proposed by the present invention is first used to independently encode the representation of each channel segment, and finally all the channel segment representations are aggregated into one representation, which is then used for the physiological disease classification task. The experimental results are as follows:

[0115] Table 1: Comparison of the effects of the model of the present invention and other models

[0116]

[0117] On the AD (electroencephalogram) and PTB (electrocardiogram) data sets, the classification accuracy of the present invention and the baseline model were compared. The results are shown in Table 2. The verification indicators are the accuracy, precision, recall, and F1 score of the model. The calculation method is as follows:

[0118]

[0119] Among them, TP: true positive, defined as the number of prediction results that are judged as positive samples and are actually positive samples; TN: true negative, defined as the number of prediction results that are judged as negative samples and are actually negative samples; FP: false positive, defined as the number of prediction results that are judged as positive samples but are actually negative samples; FN: false negative, defined as the number of prediction results that are judged as negative samples but are actually positive samples;

[0120] The method of the present invention has excellent performance on the two data sets. Except for the recall rate which is slightly lower than the baseline model, the other indicators are better than the baseline model, especially the F1 score which is an important indicator in the disease monitoring task. This verifies the effectiveness of the present invention in the task of physiological disease classification and its generalizability.

[0121] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A self-supervised representation learning method for physiological signals based on time-frequency reconstruction, It is characterized in that The following steps are involved: Obtain original multi-channel physiological signal data and convert it into time-frequency diagram; The original multi-channel physiological signal data is segmented, one-dimensionally randomly masked, and feature transformed to obtain a preliminary time domain representation; a time domain encoder is used to encode the data in the time dimension and the channel dimension to obtain the final time domain representation; The time-frequency graph is divided into blocks, subjected to two-dimensional random masking and feature conversion operations to obtain a preliminary frequency domain representation; a frequency domain encoder is used to encode in the time dimension and channel dimension respectively to obtain the final frequency domain representation; A time-frequency hybrid encoder is used to fuse the final time domain representation and the final frequency domain representation to obtain a time-frequency hybrid representation; Based on the time-frequency mixed representation, the original multi-channel physiological signals and their time-frequency graphs are reconstructed respectively to obtain the time domain loss and frequency domain loss; Combining the time domain loss and the frequency domain loss, pre-training the time domain encoder, the frequency domain encoder and the time-frequency hybrid encoder; A multi-channel time-frequency representation of the original multi-channel physiological signal data to be processed is obtained using a pre-trained time domain encoder and a frequency domain encoder, wherein the multi-channel time-frequency representation includes a final time domain representation and a final frequency domain representation, and a pre-trained time-frequency mixed encoder is used to obtain a time-frequency mixed representation, which is used as a learned self-supervised representation of the physiological signal for the task of physiological disease classification.

2. The method for learning physiological signal self-supervised representation based on time-frequency reconstruction according to claim 1, It is characterized in that The original multi-channel physiological signal data is any one of electroencephalogram data, electrocardiogram data, electrooculogram data, and electromyogram data.

3. The method for learning physiological signal self-supervised representation based on time-frequency reconstruction according to claim 1, It is characterized in that The time-frequency diagram is obtained by short-time Fourier transform of original multi-channel physiological signal data.

4. The method for learning physiological signal self-supervised representation based on time-frequency reconstruction according to claim 1, It is characterized in that The original multi-channel physiological signal data is segmented, subjected to one-dimensional random masking and feature conversion operations to obtain a preliminary time domain representation, including: Given the original multi-channel physiological signal data, for any channel sequence, split it into a series of overlapping fragment combinations and perform one-dimensional random masking; traverse all channels; Linearly map the one-dimensional randomly masked fragment combination into a high-dimensional feature vector, and add position encoding as a preliminary time domain representation; The one-dimensional random mask is: randomly sampling a certain proportion of fragments from the fragment combination and setting them to zero.

5. The method for learning physiological signal self-supervised representation based on time-frequency reconstruction according to claim 1, It is characterized in that The above-mentioned block division, two-dimensional random masking and feature conversion operations are performed on the time-frequency graph to obtain a preliminary frequency domain representation, including: Given any channel time-frequency diagram corresponding to the original multi-channel physiological signal data, divide the blocks into frequency domain block sequences and perform two-dimensional random masking; traverse all channels; Linearly map the two-dimensional randomly masked frequency domain block sequence into a high-dimensional feature vector, and add position encoding as a preliminary frequency domain representation; The two-dimensional random mask adopts a masking method combining time resolution and frequency resolution, that is, a frequency domain block in a row is set to zero along the time axis, and a frequency domain block in a column is set to zero along the frequency axis.

6. The method for learning physiological signal self-supervised representation based on time-frequency reconstruction according to claim 1, It is characterized in that The time domain encoder adopts a Transformer encoder, and the time domain encoder is used to encode in the time dimension and the channel dimension respectively to obtain the final time domain representation, which is expressed as: T′=TransFormer(T) T rep =Transformer(T time ) Where T is the initial time domain representation, TransFormer(.) is the Transformer encoder, T′ is the representation after encoding in the time dimension, and T time is the two-dimensional vector obtained by expanding T′, T rep is the final time domain representation.

7. The method for learning physiological signal self-supervised representation based on time-frequency reconstruction according to claim 1, It is characterized in that The frequency domain encoder adopts a multi-layer perceptron, and the frequency domain encoder is used to encode in the time dimension and the channel dimension respectively to obtain the final frequency domain representation, which is expressed as: H′=H+GA(MLP(H)) Z rep =H time +GA(MLP(H time )) Among them, t is the preliminary frequency domain representation, MLP(.) is the multi-layer perceptron, GA(.) is the gated attention mechanism, H time is the frequency domain representation obtained by swapping the first and last dimensions of H′, Z rep is the final frequency domain representation.

8. The method for learning physiological signal self-supervised representation based on time-frequency reconstruction according to claim 1, It is characterized in that The time-frequency hybrid encoder adopts a pooling operation.

9. The method for learning physiological signal self-supervised representation based on time-frequency reconstruction according to claim 1, It is characterized in that Before being used for the physiological disease classification task described above, a fine-tuning phase is also included, including: Obtain a fine-tuning training set containing multi-channel physiological signal data and its corresponding disease labels; The pre-trained time domain encoder, frequency domain encoder, and time-frequency hybrid encoder are used to obtain multi-channel time-frequency fusion representation of multi-channel physiological signal data, the disease type is predicted through the classifier, the loss is calculated based on the disease label, and the classifier and / or encoder are fine-tuned.

10. A physiological signal self-supervised representation learning system based on time-frequency reconstruction, used to implement the physiological signal self-supervised representation learning method based on time-frequency reconstruction according to claim 1, It is characterized in that The system comprises: A data preprocessing module, which is used to obtain the original multi-channel physiological signal data and convert it into a time-frequency diagram; The time domain feature learning module is used to segment and transform the original multi-channel physiological signal data to obtain a preliminary time domain representation; the time domain encoder is used to encode the data in the time dimension and the channel dimension to obtain the final time domain representation; the pre-training stage also includes a one-dimensional random masking operation on the segmented sequence; The frequency domain feature learning module is used to perform block and feature conversion operations on the time-frequency graph to obtain a preliminary frequency domain representation; the frequency domain encoder is used to encode in the time dimension and channel dimension respectively to obtain the final frequency domain representation; the pre-training stage also includes a two-dimensional random masking operation on the block sequence; A time-frequency fusion module, which is used to fuse the final time domain representation and the final frequency domain representation using a time-frequency hybrid encoder to obtain a time-frequency hybrid representation; A reconstruction learning module is used to reconstruct the original multi-channel physiological signal and its time-frequency diagram using the time-frequency mixed representation, calculate the loss of the time domain reconstruction task and the frequency domain reconstruction task, and train the encoder; A self-supervised representation generation module is used to obtain a multi-channel time-frequency representation of the original multi-channel physiological signal data to be processed using a pre-trained time domain encoder and a frequency domain encoder, wherein the multi-channel time-frequency representation includes a final time domain representation and a final frequency domain representation, and a pre-trained time-frequency mixed encoder is used to obtain a time-frequency mixed representation, and the time-frequency mixed representation is used as the learned physiological signal self-supervised representation for physiological disease classification tasks.

Images