Physiological signal segment analysis method based on multi-modal multi-scale co-attention

Abstract

The application discloses a physiological signal segment analysis method based on multi-modal multi-scale shared attention, and specifically comprises the following steps: step 1, inputting original ECG and SpO2 signals in a public data set, and dividing the corresponding signals into a training set and a test set after pre-processing operations; step 2, constructing a physiological signal segment analysis model based on multi-modal multi-scale shared attention; step 3, training the model constructed in step 2 using the training set processed in step 1; and step 4, feeding the test set ECG and SpO2 signals pre-processed in step 1 into the model trained in step 3, and finally outputting the results of classification detection. The method uses the structure of multi-head attention to perform multi-scale multi-modal fusion of physiological signal segment analysis, so that the accuracy of analyzing whether a sleep apnea event occurs in a physiological signal segment is improved.

Description

Technical Field

[0001] This invention belongs to the field of medical physiological signal processing technology, and relates to a method for analyzing physiological signal fragments based on multimodal and multiscale co-attention. Background Technology

[0002] Although existing methods for classifying sleep apnea events have achieved some success, they still have some shortcomings in practical applications. Firstly, because they do not consider the potential correlations between different physiological signals, the accuracy of extracting features from single-lead signals to detect sleep apnea syndrome is relatively low. In practice, the recognition effect is not ideal and the generalization performance is poor. For example, ECG-based detection devices are easily affected by motion artifacts, and using multiple electrodes is inconvenient; methods based on blood oxygen saturation (SpO2) detection cannot distinguish mild obstructive sleep apnea syndrome; and tracheal sound monitoring devices can only determine the presence of apnea, not insufficiency. Secondly, traditional machine learning methods for classifying sleep apnea events often rely on prior expert knowledge, and a large amount of work is devoted to feature engineering, resulting in low efficiency.

[0003] To address the aforementioned issues, researchers have recently explored multimodal approaches, effectively integrating information from multiple modalities and leveraging the strengths of different modalities. This allows for the extraction of key information relevant to the target from various modalities, significantly enhancing the model's detection capabilities and proving effective in resolving the low accuracy of single-modal sleep apnea event classification. However, most current methods simply fuse different modalities with fixed weights, primarily focusing on data-level fusion or shallow feature fusion, without considering the potential correlations between different physiological signals and their impact on sleep apnea event classification. Consequently, they fall short in extracting robust target features and suppressing redundant information. Meanwhile, deep learning has been continuously developed and applied across various fields, demonstrating its superiority over traditional machine learning models without requiring domain knowledge. The Transformer architecture, in particular, has made significant strides in various domains and is currently a common method for cross-modal fusion. Furthermore, the overall architecture of convolutional neural networks combined with recurrent neural networks (CNN-RNN) is highly effective for sleep apnea event classification, benefiting from the feature extraction capabilities of CNNs and the temporal modeling capabilities of RNNs for extracted features. However, due to the limitations of the recurrent nature of RNNs, global dependencies cannot be obtained. Summary of the Invention

[0004] The purpose of this invention is to provide a physiological signal segment analysis method based on multimodal and multi-scale co-attention. This method utilizes the structure of multi-head attention (MHA) and co-attention to perform multi-scale and multimodal fusion physiological signal segment analysis, so as to improve the accuracy of analyzing whether a physiological signal segment has caused a sleep apnea event.

[0005] The technical solution adopted in this invention is a method for analyzing physiological signal fragments using multimodal and multi-scale co-attention, specifically including the following steps:

[0006] Step 1: Input the original ECG and SpO2 signals from the public dataset, perform preprocessing operations on the corresponding signals, and divide them into training and test sets;

[0007] Step 2: Construct a physiological signal fragment analysis model based on multimodal and multiscale co-attention;

[0008] Step 3: Train the model built in Step 2 using the training set processed in Step 1;

[0009] Step 4: Input the ECG and SpO2 signals from the test set preprocessed in Step 1 into the model trained in Step 3, and finally output the classification and detection results.

[0010] The invention is further characterized by:

[0011] The preprocessing of ECG and SpO2 signals in step 1 is as follows: segmenting, removing W periods, and normalizing the ECG and SpO2 signals respectively.

[0012] In step 2, the physiological signal fragment analysis model based on multimodal and multiscale co-attention includes a shallow feature extraction module, a cross-modal interaction module, and a classification module.

[0013] The shallow feature extraction module includes a large convolutional kernel neural network, a one-dimensional SE module, and a bidirectional GRU module. The large convolutional kernel neural network includes three one-dimensional convolutional layers and two max pooling layers. Each convolutional layer is followed by a batch normalization layer and uses Gaussian error linear units as activation functions.

[0014] The cross-modal interaction module includes two parallel MM_Transformer modules;

[0015] The classification module includes a statistical layer and a classifier.

[0016] In the classification module, the predicted value of the physiological signal classification of sleep apnea events is obtained using the following formula (1).

[0017]

[0018] In step 3, during the training process, the model constructed in step 2 is constrained using the cross-entropy loss function, i.e., the following formula (2):

[0019]

[0020] Where N is the number of samples, y i p represents the label of sample i. i This represents the probability that sample i is predicted to be of the positive class.

[0021] The beneficial effects of this invention are as follows: It constructs a shallow feature extraction module, a multi-scale cross-modal interaction module, and a classification module; preprocessing uses some relatively simple and common methods; the shallow feature extraction module consists of LKCNN, 1D SEBlock, and BiGRU. LKCNN uses large convolutional kernels to extract shallow features and adaptively optimizes features through the SE network structure to enhance feature learning; finally, BiGRU is used to learn long-term dependencies such as OSA (sleep apnea event) transformation rules; the multi-scale cross-modal interaction module, while fusing multi-modal features, detects local inconsistencies at different scales and learns their interdependencies; in the classification module, the two outputs are concatenated to obtain the global mean and standard deviation; finally, a classifier composed of MLP and Softmax is used to complete the classification of sleep apnea events for physiological signal segments. This invention can improve the accuracy of physiological signal segment classification. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the preprocessing flow of the multimodal, multiscale co-attention physiological signal fragment analysis method of the present invention;

[0023] Figure 2 This is a schematic diagram of the overall structure of the multimodal, multi-scale co-attention physiological signal fragment analysis method of the present invention;

[0024] Figure 3 This is a schematic diagram of the shallow feature extraction module of the multimodal, multi-scale co-attention physiological signal fragment analysis method of the present invention;

[0025] Figure 4 This is a schematic diagram of the cross-modal interaction module of the multimodal, multi-scale co-attention physiological signal fragment analysis method of the present invention. Detailed Implementation

[0026] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0027] This invention relates to a multimodal, multi-scale co-attention physiological signal segment analysis method. This method utilizes a structure combining multi-head attention (MHA) and co-attention to perform multi-scale, multimodal fusion physiological signal segment analysis, thereby improving the accuracy of analyzing whether sleep apnea events have occurred in physiological signal segments.

[0028] The specific steps are as follows:

[0029] Step 1: Input the raw ECG and SpO2 signals from the public dataset. After appropriate preprocessing, divide the dataset into training and testing sets. Use the training set for training and the testing set for testing. The specific preprocessing steps are as follows:

[0030] Step 1.1, Signal Preprocessing. The ECG signal is segmented, de-periodized, filtered, and normalized. The SpO2 signal is also segmented, de-periodized, and normalized. The overall process is as follows: Figure 1 The details of the processing method are as follows:

[0031] Fragmentation: Based on the annotations provided by the dataset, each individual signal is divided into 60-second segments. For example, in the Apnea-ECG dataset, the sampling rate is 100Hz, so each signal segment has 100 × 60 = 6000 sampling points and a corresponding label.

[0032] Trimming the W stage: The W stage at the beginning and end of the signal is trimmed based on the segment and its corresponding sleep stage label. This is because the subject does not fall asleep immediately after wearing the data acquisition device, and is also awake before removing the device. These two situations do not belong to the sleep process, so trimming and deletion can eliminate the influence of some objective factors.

[0033] Filtering: A Butterworth bandpass filter with a bandwidth of 0.5-48Hz was used. This is because the raw ECG signal is susceptible to various types of noise and artifacts, namely lead loosening artifacts, motion artifacts, and power line interference.

[0034] Normalization: Z-score normalization was used (as shown in formula (1)). Before feeding the ECG and SpO2 signals into the model, they needed to be adaptively normalized to achieve dimensionlessness, aiming to eliminate the incomparability between the two signals. X data Let X represent the original data, μ represent the mean of the original data, σ represent the standard deviation of the original data, and X represent the original data. normalization This represents the normalized data. Formula (1) is as follows:

[0035]

[0036] Step 1.2: The preprocessed signals from Step 1.1 are divided into a test set and a validation set. To verify their applicability and robustness to the physiological signal fragment analysis task, the sets are divided proportionally by individual.

[0037] Step 2: The preprocessed ECG and SpO2 signals obtained in Step 1 are sequentially processed through a shallow feature extraction module, a cross-modal interaction module, and a classification module to construct a physiological signal fragment analysis model based on multimodal and multi-scale co-attention (e.g., Figure 2 (As shown). The specific steps are as follows:

[0038] Step 2.1, design a shallow feature extraction module (e.g., Figure 3 As shown, shallow features of ECG and SpO2 signals are extracted and adaptive feature optimization is performed. First, shallow features are extracted using a large convolutional kernel neural network (LKCNN). Next, the obtained features are adaptively filtered for important features using a one-dimensional SE module (1DSE Block). Finally, a bidirectional GRU (BiGRU) is used to learn long-term dependencies such as OSA transformation rules.

[0039] Step 2.1.1, LKCNN consists of three one-dimensional convolutional layers (Conv1D) and two max-pooling layers (MaxPool1d). Each convolutional layer is followed by a batch normalization layer using Gaussian error linear units (GELU) as the activation function. To reduce overfitting, a dropout layer is used after MaxPool1d. The preprocessed signals I1 and I2 are input into LKCNN to extract features F1 and F2.

[0040] Step 2.1.2: Next, the obtained features F1 and F2 are processed by 1D SE Block to extract the channel features F1' and F2' with the highest information content, respectively, to correct and filter the feature sequences F1 and F2 learned by the LKCNN module. Since the operations on the input features F1 and F2 are the same, only F1 will be used as an example in the following explanation. This module first filters the input feature F1 channels through two layers of Conv1D with a kernel size and stride of 1 to obtain the feature sequence, and then uses an adaptive pooling layer (AdaptiveAvgPool) to compress the features in terms of spatial dimension to obtain the channel-level global features c = {c1, c2, ..., c i ,…,cN}∈R N×1 Where i represents the channel number, c i It is the average of d data points for each channel, c i The calculation is shown in formula (2), where d represents the number of data points, j represents the number of each data point, and x j This represents the value of each data point. Then, two fully connected layers are used to form a bottleneck structure to activate the global features, learn the relationship between each channel, and obtain the weights of different channels. The purpose is to reduce the model complexity and improve the generalization performance. The first layer (W1 represents the weight parameter matrix of the first fully connected layer) plays a role in dimensionality reduction. The ReLU activation function δ is used to avoid gradient explosion / vanishing, and at the same time, it makes the calculation faster and easier to converge. The second layer (W2 represents the weight parameter matrix of the second fully connected layer) is used to restore the original dimension. The Sigmoid activation function ρ is used to normalize the channel weights to the range [0,1], and obtain the channel attention weights s (as shown in formula (3)). Finally, the channel attention weights s are multiplied by the input feature F1 to obtain the feature F1', as shown in formula (4).

[0041]

[0042] s=ρ(W2(S(W1(c))) (3);

[0043]

[0044] Step 2.1.3: Finally, input features F1' and F2' into BiGRU to obtain the long-range modeled feature L. F1 and L F2 The purpose is to learn long-term dependencies such as OSA transformation rules.

[0045] Step 2.2, design the cross-modal interaction module, which consists of two parallel MM_Transformers (such as...). Figure 4 (As shown) It is composed of [a model]. The aim is to better achieve multimodal fusion, which aims to improve the accuracy of the overall decision-making results by building a model that can process and correlate information from different modalities, thereby providing more information for classification decisions.

[0046] Step 2.2.1, the MM_Transformer module uses a one-dimensional causal convolution function as the transformation function for multi-head attention to transform the input feature L F1 Generate feature codes Q1, K1, and V1, and then... F2 Generate feature codes Q2, K2, and V2. The calculation process is shown in formulas (5) and (6), where W Q1 WK1 W V1 W Q2 W K2 W V2 It is a linear matrix. Next, the original weights are calculated by multiplying Q1 and K2 and Q2 and K1 point by point, and then activated by the Softmax activation function (see formula (7), where n represents the number of the output node, z n The output value of the nth node (where C is the number of categories) is standardized and then multiplied by its respective V, d. h The dimensions of K1 and K2 are represented to ensure the stability of the gradient. Finally, the outputs Z(Q1,K2,V2) and Z(Q2,K1,V1) are obtained, as shown in formulas (8) and (9).

[0047] Q1 = L F1 W Q1 K1 = L F1 W K1 V1=L F1 W V1 (5);

[0048] Q2 = L F2 W Q2 K2=L F2 W K2 V2=L F2 W V2 (6);

[0049]

[0050]

[0051]

[0052] Step 2.2.2, the MM_Transformer module focuses on multi-scale co-attention. Since the input size is fixed, to achieve multi-scale feature extraction, it utilizes the special structure of Vision Transformer (ViT) and multi-head attention, dividing the input one-dimensional features into patches of different sizes in different heads as input to the attention module. Furthermore, MultiScale_Co-Attention uses the co-attention module for cross-modal interaction, ultimately fusing multi-modal features and learning their interdependencies while detecting local inconsistencies at different scales.

[0053] Step 2.2.1 is essentially the calculation process for one of the heads in a multi-head attention model. Taking Z(Q1,K2,V2) as an example, we have a total of four heads, and the input Q of each head is... 1tK 2t and V 2t Dimensions

[0054] Unlike other values, t takes integer values ​​from 1 to 4. Then, the resulting H... t (See formula (10), Q) 1t ,K 2t V 2t The feature encodings of each head are concatenated to obtain the output result H of MultiScale_Co-Attention (see formula (11)). Finally, the information is integrated through a feedforward neural network (FFN) to obtain the interactive feature Z, the process of which is shown in formula (12). LN is the layer normalization.

[0055] H t =Z(Q 1t ,K 2t V 2t (10);

[0056]

[0057] Z = FFN(LN(H)) (12);

[0058] Step 2.3, the final step, is the final classification module, which concatenates the output features from the two cross-modal interactions to obtain X. cat X is obtained through the statistical layer. cat The mean μ and standard deviation σ of the signal are concatenated and fed into the classifier. The classifier consists of two fully connected layers (represented by Fc1 and Fc2) and a softmax layer, which yields predicted values ​​for the classification of physiological signal segments. The complete formula is shown in formula (13).

[0059]

[0060] Step 3: Train the model using the dataset processed in Step 1. Use the CrossEntropy loss function (see Formula (14)) to constrain the results of the trained network, as shown in Formula (15). Then, perform reverse rebroadcasting to update the parameters. After 150 training iterations (where 1 iteration refers to training the preprocessed signal segment once), the trained network model is finally obtained. Where N is the number of samples, y... i p represents the label of sample i. i This represents the probability that sample i is predicted to be positive, and y is the input label, which is the index value of a certain category.

[0061]

[0062]

[0063] Step 4: The preprocessed ECG and SpO2 signals from Step 1 are fed into the model trained in Step 3, and the final classification and detection results are output. The comparative experimental results on the Apnea-ECG dataset are shown in Table 1 below. It is easy to see that, compared with other methods based on feature engineering and deep learning (DL), this invention achieves better results without requiring complex feature engineering and processing of the original signals. It can better balance various evaluation metrics (including accuracy (Acc), sensitivity (Sen), and specificity (Spec), resulting in higher accuracy and specificity.

[0064] Table 1

[0065]

Claims

1. A method for analyzing physiological signal fragments using multimodal and multiscale co-attention, characterized by: Specifically, the steps include the following: Step 1: Input the original ECG and SpO2 signals from the public dataset, perform preprocessing operations on the corresponding signals, and divide them into training and test sets; Step 2: Construct a physiological signal fragment analysis model based on multimodal and multi-scale co-attention; the physiological signal fragment analysis model based on multimodal and multi-scale co-attention in Step 2 includes a shallow feature extraction module, a cross-modal interaction module, and a classification module; The shallow feature extraction module includes a large convolutional kernel neural network and a one-dimensional SE module. The large convolutional kernel neural network includes three one-dimensional convolutional layers and two max pooling layers. Each convolutional layer is followed by a batch normalization layer and uses Gaussian error linear units as activation functions. The cross-modal interaction module includes two parallel MM_Transformer modules; The classification module includes a statistical layer and a classifier; Step 2.1: Design a shallow feature extraction module to extract shallow features from ECG and SpO2 signals, and perform adaptive feature optimization and filtering. First, shallow features are extracted using a large convolutional kernel neural network; next, the obtained features are adaptively filtered for important features using a one-dimensional SE module; finally, a bidirectional GRU is used to learn the OSA transformation rule. Step 2.1.1: The large convolutional kernel neural network contains three one-dimensional convolutional layers and two max pooling layers. Each convolutional layer is followed by a batch normalization layer and uses Gaussian error linear units as activation functions. The preprocessed signals I1 and I2 are input into the large convolutional kernel neural network to extract features F1 and F2. Step 2.1.2: Extract the channel features F1' and F2' with the highest information content from the obtained features F1 and F2 respectively using a 1D SE Block. The operation on the input features F1 and F2 is the same: first, filter the channels of the input feature F1 by using two one-dimensional convolutional layers with a kernel size and stride of 1 to obtain the feature sequence; then, use an adaptive pooling layer to compress the features in terms of spatial dimension to obtain the channel-level global features. Where i represents the channel number, It is the average of d data points for each channel. The calculation is shown in formula (1), where d represents the number of data points and j represents the number of each data point. This represents the value of each data point; then, two fully connected layers are used to form a bottleneck structure to activate the global features, learn the relationship between each channel, and obtain the channel attention weights s, as shown in formula (2). Finally, the channel attention weights s are multiplied by the input feature F1 to obtain the feature F1', as shown in formula (3): Step 2.1.3: Finally, input features F1' and F2' into the bidirectional GRU to obtain the long-range modeled feature L. F1 and L F2 ; Step 2.2, design the cross-modal interaction module, which consists of two parallel MM_Transformers, specifically: Step 2.2.1, the MM_Transformer module uses a one-dimensional causal convolution function as the transformation function for multi-head attention to transform the input feature L F1 Generate feature codes Q1, K1, and V1, and then... F2 The feature codes Q2, K2, and V2 are generated, and the calculation process is shown in formulas (4) and (5), where W Q1 W K1 W V1 W Q2 W K2 W V2 It is a linear matrix; next, the original weights are calculated by multiplying Q1 and K2 and Q2 and K1 point by point, and then activated by the Softmax activation function, as shown in formula (6), where n represents the number of the output node. The output value of the nth node is given by C, where C is the number of categories. After standardization, the values ​​are multiplied by their respective values ​​V and d. h The dimensions of K1 and K2 are represented, and the final output is obtained. and The process is shown in formulas (7) and (8): （4） （5） （6） （7） （8） Step 2.2.2: The MM_Transformer module uses Vision Transformer and multi-head attention structure to divide the input one-dimensional features into different patch sizes in different heads as input to the attention module. In addition, MultiScale_Co-Attention uses the co-attention module to perform cross-modal interaction and finally fuse multimodal features. Step 2.3: Concatenate the output features from the two cross-modal interactions to obtain X. cat X is obtained through the statistical layer. cat average and standard deviation The concatenated data is then fed into a classifier, which consists of two fully connected layers and a softmax layer, to obtain predicted values ​​for the physiological signal fragment classification. ; Step 3: Train the model built in Step 2 using the training set processed in Step 1; Step 4: Input the ECG and SpO2 signals from the test set preprocessed in Step 1 into the model trained in Step 3, and finally output the classification and detection results.

2. The method for analyzing physiological signal segments using multimodal and multiscale co-attention as described in claim 1, characterized in that: The preprocessing of ECG and SpO2 signals in step 1 is as follows: segmenting, removing W periods, and normalizing the ECG and SpO2 signals respectively.

3. The method for analyzing physiological signal fragments using multimodal and multiscale conattention according to claim 1, characterized in that: In the classification module, the predicted value of the physiological signal classification of sleep apnea events is obtained using the following formula (9). : （9）。 4. The method for analyzing physiological signal fragments using multimodal and multiscale conattention according to claim 1, characterized in that: In step 3, during the training process, the model constructed in step 2 is constrained using the cross-entropy loss function, i.e., the following formula (10): （10） Where N is the number of samples, The label representing sample i, This represents the probability that sample i is predicted to be of the positive class.

Images