Multi-mode unsupervised cross-domain sleep staging method

Abstract

The invention discloses a multi-modal unsupervised cross-domain sleep staging method, which adopts adversarial learning and designs a multi-modal convolution feature extractor module, a domain generalization feature enhancement module and a domain attention module. The method comprises the following steps: firstly, designing a multi-mode convolution feature extractor for physiological signals of two modes of electroencephalogram and electro-oculogram; for the electroencephalogram signals, convolution kernels of different scales are adopted to extract multi-scale features; for the electro-oculogram signal, firstly, the electro-oculogram signal is converted into a two-dimensional frequency spectrum through Fourier transform, and then feature extraction is carried out through two-dimensional convolution, so that unique physiological information of the electro-oculogram signal is fully captured. Thirdly, adaptively adjusting data distribution of a source domain and a target domain by using a domain generalization feature enhancement module, reducing inter-domain differences, and adaptively enhancing high-discrimination-force features; the domain attention module reserves key domain specific features in the adversarial learning process, the classification precision and generalization ability of the model are remarkably improved, and the method shows excellent performance in an unsupervised cross-domain sleep staging task.

Description

Technical Field

[0001] The present invention relates to the cross - technical field of biomedical signal processing and artificial intelligence, focusing on a multi - modal unsupervised cross - domain sleep staging method. In view of the heterogeneity problem of multi - source sleep data and the lack of labels in the target domain in medical monitoring scenarios, this invention can effectively integrate various modal information, narrow the feature distributions between the source domain and the target domain, and combine source - domain knowledge to improve the overall accuracy of sleep staging in the target domain. Background Art

[0002] In recent years, the research of sleep medicine has been continuously deepened, and the innovation of sleep monitoring technology has significantly improved the feasibility and accuracy of sleep staging research. With the popularization of multi - modal biosensor technology and intelligent wearable devices, the acquisition efficiency of high - density sleep monitoring data (including multi - dimensional time - series signals such as polysomnogram electroencephalogram (EEG), electro - oculogram (EOG), electromyogram (EMG), etc.) has been greatly improved, providing a data basis for accurately analyzing sleep structure. Sleep staging is an indispensable tool in the process of sleep quality analysis and has important application values in fields such as clinical sleep disorder diagnosis (such as insomnia, sleep apnea syndrome), health management systems, and the status monitoring of personnel in special environments (aerospace, deep - sea operations).

[0003] Data - driven deep - learning methods usually rely on a large amount of labeled data for training. In recent years, this technology has been widely applied to the field of sleep staging to achieve automated sleep staging. By designing diverse network models, these algorithms have achieved satisfactory results on the test set, demonstrating their potential in sleep data processing. However, existing methods still have some key limitations that restrict their application in practical scenarios. First, there are heterogeneity problems in sleep data itself, such as differences in sampling rates, signal channels, and individual differences caused by factors such as age, gender, and region. This causes the accuracy of the trained model to significantly decline when applied to other specific data, making it difficult to meet the requirements of practical applications. Second, these models generally require a large amount of labeled data as the training basis, and these labeling tasks usually require sleep experts to spend a lot of time and effort, resulting in high data acquisition costs and low efficiency.

[0004] Unsupervised domain adaptation (UDA) has shown great potential in improving the performance of deep learning models in the case of scarce labeled data. Different from traditional methods, UDA does not require a large amount of labeled data. It achieves efficient model training in the target domain by transferring knowledge from the source domain with rich labels to the target domain with scarce labels. By aligning the distributions of source domain and target domain features, UDA effectively alleviates the problem of model performance degradation caused by the distribution differences between the two. In the case of scarce labels in the target domain data, this method trains by transferring source domain knowledge to the target domain, thus ensuring excellent performance of the model in the target domain. In the sleep staging task, the application of UDA provides a new research direction for automated sleep staging. However, there are still some limitations that need to be urgently solved in existing research. First, existing methods fail to fully align the data distributions of the source domain and the target domain, resulting in limited transfer effects and still need further improvement. Second, existing research usually relies on completely shared parameter weights and models to extract features of the source domain and the target domain. Although this way extracts domain-general features, it may ignore domain-specific features, thus affecting the classification performance of the model in the target domain. In addition, existing methods are mainly based on EEG signals and fail to make full use of the multimodal PSG sleep dataset, resulting in poor classification effects in the target domain after migration and may have biases in the recognition of specific classes.

[0005] Aiming at the problems of heterogeneity and lack of labels in sleep data, and at the same time to solve the above existing limitations, the present invention designs a multimodal unsupervised cross-domain sleep staging method, which mainly consists of a multimodal convolutional feature extractor module, a domain generalization feature enhancement module, and a domain attention module. First, a multimodal feature extraction network is designed for the physiological information of two modalities, electroencephalogram (EEG) and electrooculogram (EOG). For EEG signals, a dual-branch convolutional neural network structure is adopted, and multi-scale features are extracted using convolutional kernels of different scales respectively; at the same time, an independent feature extraction branch is designed for EOG signals to fully capture their unique physiological information. Then, the domain generalization feature enhancement module is used to adaptively adjust the feature distributions of the source domain and target domain data, improve the consistency of features between the two, so as to reduce the domain difference, and at the same time further refine the extracted features to adaptively enhance the highly discriminative features. The domain attention module weights and extracts features of different domains, and tries to retain domain-specific features as much as possible during the adversarial learning process, so as to better improve the classification ability and generalization ability of the model. In addition, using adversarial learning, a discriminator is used to distinguish source domain features and target domain features, so as to further narrow the domain difference, enhance the cross-domain adaptation ability of the model, and at the same time calculate the classification loss of the target domain through the generated pseudo-labels to improve the classification performance of the target domain model. Summary of the Invention

[0006] In order to solve the problem of cross-domain sleep staging in the case of heterogeneity of sleep data and lack of labels, the present invention discloses a multimodal unsupervised cross-domain sleep staging method. The present invention adopts an adversarial learning approach and designs a multimodal convolutional feature extractor module, a domain generalization feature enhancement module, and a domain attention module. First, a multimodal convolutional feature extractor is designed for two modal data, EEG and EOG, to fully extract multimodal physiological information. Then, the domain generalization feature enhancement module is used to adaptively adjust the data distribution of the source domain and the target domain, thereby reducing the differences between domains and adaptively enhancing features with high discriminability. The domain attention module retains domain-specific features as much as possible while adversarially learning, thereby better improving the classification and generalization capabilities of the model.

[0007] The main idea of ​​implementing the present invention is: the present invention performs modeling optimization from the perspective of feature space distribution regulation. Based on the adversarial learning framework, a multimodal convolutional feature extractor is first constructed to capture the multi-scale EEG features and the spectral features of the electrooculogram, respectively, and establish a cross-modal joint representation; then, a domain generalization feature enhancement module is designed to dynamically adjust the feature distribution differences between the source domain and the target domain, while enhancing the features with high discrimination and suppressing redundant interference; further, a domain attention module is introduced to adaptively retain domain-specific features related to the classification of this domain during the adversarial learning process. With the iteration of the network, the distribution of the source domain and the target domain in the feature space gradually becomes consistent, and finally the robustness and generalization ability of cross-domain sleep staging are improved.

[0008] A multimodal unsupervised cross-domain sleep staging method based on adversarial learning, comprising the following steps:

[0009] Step 1: Extract multimodal features from sleep data

[0010] In traditional sleep staging tasks, the main focus is on feature extraction of EEG signals. In the present invention, we focus on using the designed multimodal convolutional feature extractor to extract corresponding features from the original EEG and EOG signals. For specific feature extractor details, please refer to step 1 in the implementation method. By using the designed multimodal feature extractor to extract features from EEG and EOG signals respectively, the physiological information of the two modalities can be accurately and fully captured, providing high-quality feature representation for subsequent classification tasks.

[0011] Step 2: Adaptive adjustment of data distribution and enhancement of high discriminative features

[0012] In an actual scenario, it is difficult to comprehensively fit the data distributions of the source domain and the target domain only through adversarial learning, resulting in significant non-overlapping regions in the feature space representations of the two, thereby limiting the extraction ability and discrimination performance of cross-domain common features. To solve the above problems, the present invention designs a domain generalization feature enhancement module, which consists of two sub-modules: a sleep adaptation normalization module and a feature adaptive enhancement module. The former sub-module takes the feature representation output in step 1 as input and is used to dynamically and adaptively adjust the data distribution to reduce feature differences. On this basis, the latter adaptively enhances features with high discriminative power.

[0013] Step 3: Retain domain-specific features through the domain attention module

[0014] During the adversarial learning process, the discriminator makes the model more focused on extracting domain common features to reduce the difference between the feature distributions of the source domain and the target domain. However, this may result in the loss of domain-specific features and affect the performance of the respective domain classifiers. The present invention designs a domain attention module to retain some domain-specific features of each domain during the feature extraction process, improving the classification performance of the model.

[0015] Step 4: Perform adversarial learning and model training

[0016] Inspired by the domain adversarial neural network, the present invention sets up a domain discriminator to determine whether the features after step 3 come from the source domain or the target domain, thereby forcing the feature alignment between the source domain and the target domain. The present invention also uses the model trained on the source domain to generate pseudo-labels in the target domain to solve the problem of the lack of labels in the target domain, thereby performing efficient model training.

[0017] Compared with the prior art, the present invention has the following obvious advantages and beneficial effects:

[0018] The present invention proposes a multi-modal unsupervised cross-domain sleep staging method based on adversarial learning for the cross-domain sleep staging problem under the conditions of sleep data heterogeneity and lack of labels. First, a multi-modal feature extractor is designed to make full use of multi-modal data, enhancing the richness and discriminative ability of feature expression. Second, further considering the alignment of features between the source domain and the target domain, a domain generalization feature enhancement module is used to adaptively adjust the data distribution and enhance features with high discriminative power. Then, the domain attention module is used to retain as much as possible the domain-specific features related to classification within each domain, thereby significantly improving the overall performance of the model. In addition, the unsupervised cross-domain method in the present invention not only achieves good results for sleep staging, but can also be used as a framework for unsupervised cross-domain tasks. For unsupervised cross-domain tasks, the method proposed in the present invention can be tried to be fused, thereby bringing an improvement in model performance. Brief Description of the Drawings

[0019] Figure 1 This is the overall process flowchart of the method involved in the present invention.

[0020] Figure 2 This is the overall architecture diagram of the algorithm involved in the present invention.

[0021] Figure 3 This is the schematic diagram of the structure of the multi-modal convolutional feature extractor involved in the present invention.

[0022] Figure 4 This is the schematic diagram of the structure of the domain generalization feature enhancement module involved in the present invention.

[0023] Figure 5 This is the schematic diagram of the structure of the sleep adaptation normalization sub-module involved in the present invention.

[0024] Figure 6 This is the schematic diagram of the structure of the feature adaptive enhancement sub-module involved in the present invention.

[0025] Figure 7 This is the schematic diagram of the structure of the domain attention module involved in the present invention. Detailed implementation

[0026] To make the objectives, technical solutions, and advantages of the present invention clearer and more understandable, the following further elaborates on the present invention in detail with reference to specific examples and detailed drawings. However, the described implementation examples are only intended to facilitate the understanding of the present invention and do not impose any limitations on it. Figure 1 This is the method flowchart of the present invention. Figure 2 This is the overall architecture diagram of the algorithm designed in the present invention. The method includes the following steps.

[0027] Step 1: Extract multi-modal features from sleep data.

[0028] The present invention designs a multi-modal feature extractor to extract features from EEG and EOG signals respectively. The module details are as Figure 3 shown. For EEG signals, a dual-branch EEG feature extraction architecture is implemented. This architecture contains two independent convolutional branches f small and f big , which use convolutional kernels of different scales respectively to achieve multi-scale extraction of EEG signals. The large-scale convolutional kernel can capture low-frequency information with a longer time step, while the small-scale convolutional kernel focuses on extracting high-frequency information with a short time step. Specifically, it is shown in the following formula:

[0029]

[0030] Among them, X EEG is the EEG (electroencephalogram) data, and are the multi-scale EEG features extracted by the dual convolutional branches.

[0031] For electrooculogram (EOG) signals, an independent two-dimensional convolutional branch is implemented. First, the short-time Fourier transform is performed on the EOG signals to convert them into two-dimensional spectra to capture their frequency-domain characteristics. Subsequently, a two-dimensional convolutional neural network f EOG is used to extract features from the spectrogram to capture the key features related to sleep stages in the EOG signals. This process can be expressed as:

[0032]

[0033] F EOG = f EOG (S EOG (t, f))

[0034] where S EOG (t, f) is the result of the short-time Fourier transform of the EOG signal at time t and frequency f, τ is the time variable, and F EOG is the feature of the EOG signal extracted after two-dimensional convolution.

[0035] Finally, the multi-scale electroencephalogram (EEG) features and EOG features are concatenated to form a multi-modal feature fusion representation to integrate the complementary information of different physiological signals. Subsequently, a Dropout layer is added to randomly mask some features to prevent overfitting and improve the generalization ability of the model, thereby further enhancing the classification performance for complex signals.

[0036]

[0037] F all = Dropout(F concat )

[0038] where Concat(·) is the feature concatenation operation, F concat is the comprehensive vector representation after multi-modal fusion, and F all is the multi-modal feature representation finally obtained after Step 1.

[0039] Step 2: Source domain and target domain data distribution calibration and high discriminative feature enhancement

[0040] The domain generalization feature enhancement module proposed in the present invention is composed of two sub-modules combined, namely the sleep adaptation normalization sub-module and the feature adaptive enhancement sub-module. The details of the overall module are as Figure 4 shown. The details of the two sub-modules are respectively as Figure 5 and Figure 6 shown. The multi-modal feature representation F allFirst, enter the sleep adaptation normalization sub-module. Two one-dimensional convolutional neural networks are used to train the two weight coefficients g and f respectively, so as to adaptively adjust the mean μ and variance σ of the features, reduce the feature differences, and ensure the consistency of the data distributions in the source domain and the target domain. First, calculate the mean and variance of the input features and perform corresponding concatenation:

[0041] μ = Mean(F all ), σ = Std(F all )

[0042] MV-Concat = Concat(μ, σ)

[0043] where Mean(·) represents mean calculation, Std(·) represents variance calculation, and Concat(·) means concatenating the mean and variance to obtain a new statistical vector MV-Concat.

[0044] Next, through a convolutional network g fc and batch normalization g bn , use the Sigmoid function to generate the weight coefficient g for adjusting the input feature distribution. At the same time, use a similar method to generate the coefficient f:

[0045] g = Sigmoid(g bn (g fc (MV-Concat))), f = Sigmoid(f bn (f fc (MV-Concat)))

[0046] Use two adaptive weight coefficients f(·) and g(·) to re-adjust the mean and variance:

[0047] μ' = f(μ, σ)·μ, σ' = g(μ, σ)·σ

[0048] Finally, according to the new mean μ' and variance σ', obtain the new feature F all_1

[0049]

[0050] Next, the feature F all-1 enters the feature adaptive enhancement sub-module. Using the channel attention mechanism, by learning the dependency relationships between features, model the importance of different channels, highlight important features and suppress irrelevant features through the weight mechanism, and finally achieve the purpose of adaptively enhancing highly discriminative features. First, apply two convolutional operations to the feature F all-1 in sequence:

[0051] F = Conv1D(Conv1D(F all-1 ))

[0052] Among them, Conv1D represents a one-dimensional convolution operation.

[0053] Next, the global spatial information is compressed by using adaptive average pooling. Then, two fully connected layers, FC, ReLU, and Sigmoid activation functions are applied to finally obtain the weight matrix. Specifically, it is as follows:

[0054] F1 = ReLU(FC(AvgPooling(F))

[0055] F2 = Sigmoid(FC(F1))

[0056] Finally, the feature F is scaled according to the weight matrix F2 and combined with the original input feature F all-1 combined:

[0057] F all-2 = F·F2 + F all-1

[0058] After this step, the sleep data distribution is adaptively adjusted, high discriminative features are enhanced, and redundant features are suppressed.

[0059] Step 3: Use the domain attention module to retain domain-specific features

[0060] The details of the domain attention module proposed by the present invention are as Figure 7 shown. Its core idea is to obtain an optimized weight matrix through the self-attention mechanism, weight the high-dimensional features to capture domain-specific features, enhance the discriminability of the classifier, and reduce the differences between domains at the same time. Let the feature F obtained through the domain generalization feature enhancement module all-2 be input into this module. where d is the number of channels of the feature and l is the length of the feature. First, two Conv1D are used to perform a linear transformation on the input feature, and batch normalization BatchNorm is applied to improve the efficiency and stability of model training:

[0061] S i = BatchNorm(Conv1D(f i ))

[0062] S j = BatchNorm(Conv1D(f j ))

[0063] where f i and f j represent the feature values at positions i and j in the input feature. Then, the attention weights are calculated. The attention weights are generated by calculating the correlation between feature points:

[0064]

[0065] where W ji represents the attention weight of position j with respect to position i. According to the attention weights, the original features are weighted and combined to generate a weighted output to obtain new features:

[0066]

[0067] where j ∈ [1, l], and all the new feature combinations finally result in F final ={new_f1,..., new_f j}.

[0068] Step 4: Conduct adversarial learning and model training

[0069] The present invention adopts adversarial learning. During the training phase, the main task of the domain discriminator is to distinguish source domain features from target domain features, while the goal of the multi-modal feature extractor is to generate representations that are indistinguishable between the source domain and the target domain. The cross-entropy loss is used to optimize the discriminator:

[0070]

[0071] where N s represents the number of samples in the source domain, and N t represents the number of samples in the target domain, represents the feature vector of the data in the source domain after passing through multiple modules, represents the feature vector from the target domain.

[0072] The adversarial loss is:

[0073]

[0074] where D is the probability output by the discriminator, represents the adversarial loss, and this loss function is used to optimize the features generated after passing through the multi-modal convolutional feature extractor, domain generalization feature enhancement, and domain attention module.

[0075] In addition, the model trained using the source domain generates pseudo-labels for the data lacking labels in the target domain, thereby obtaining the target domain classification loss. Thus, the adversarial loss, source domain classification loss, and target domain classification loss can be integrated into a single objective loss function to obtain the total loss

[0076]

[0077] where, is the classification loss of the source domain, The classification loss obtained for the target domain with the help of pseudo-labels, λ t is the weight of the target domain classification loss, to avoid the overall accuracy from decreasing due to an overly large target domain classification loss.

[0078] Table 1 Precision Comparison

[0079]

[0080] Table 2 F1 Value Comparison

[0081]

[0082] Table 1 shows the precision comparison between the present invention and other different models on multiple datasets.

[0083] Table 2 shows the F1 value comparison between the present invention and other different models on multiple datasets.

[0084] As can be seen from Appendix Table 1 and Appendix Table 2, the method proposed by the present invention shows more excellent performance than the existing methods on multiple datasets.

Claims

1. A multimodal unsupervised cross-domain sleep staging method, characterized in that, The method includes the following steps: Step 1: Extract multi-modal features from sleep data; Feature extraction is performed on EEG and EOG signals respectively through a designed multi-modal convolutional feature extractor module; for EEG signals, multi-scale features are extracted using convolutional kernels of different scales; for EOG signals, they are first transformed into two-dimensional spectra through Fourier transform and then feature extraction is performed using two-dimensional convolution; Step 2: Adaptive adjustment of data distribution and enhancement of highly discriminative features; Adaptive adjustment of data distribution and enhancement of highly discriminative features are performed through a designed domain generalization feature enhancement module; this module consists of two sub-modules, namely a sleep adaptation normalization module and a feature adaptive enhancement module; the features obtained through Step 1 first enter the sleep adaptation normalization sub-module, which is used to dynamically and adaptively adjust the distribution of data and reduce feature differences; the feature adaptive enhancement sub-module adaptively enhances features with high discriminative power and suppresses redundant features; Step 3: Retain domain-specific features through a domain attention module; Through the domain attention module, some domain-specific features of each domain are retained during the feature extraction process to improve the classification accuracy; Step 4: Perform adversarial learning and model training; A domain discriminator is set to determine whether the features after Step 3 come from the source domain or the target domain, so as to achieve feature alignment between the source domain and the target domain; the model trained using the source domain generates pseudo-labels in the target domain to solve the lack of labels in the target domain and perform efficient model training.

2. The multimodal unsupervised cross-domain sleep staging method according to claim 1, wherein Specifically, Step 1 includes: Design a multi-modal feature extractor to extract features from EEG and EOG signals respectively; implement a dual-branch EEG feature extraction architecture for EEG signals, including two independent convolutional branches f small and f big , which use convolutional kernels of different scales respectively to achieve multi-scale extraction of EEG signals; large-scale convolutional kernels can capture low-frequency information with longer time steps, while small-scale convolutional kernels focus on extracting high-frequency information with short time steps; as shown in the following formula: Among them, X EEG is the electroencephalogram (EEG) data, and are the multi-scale EEG features extracted by the double convolutional branches.

3. A multimodal unsupervised cross-domain sleep staging method according to claim 1, characterized in that For EOG signals, an independent two-dimensional convolution branch is implemented; first, the EOG signals are subjected to short-time Fourier transform to convert them into two-dimensional spectra to capture their frequency-domain features; Subsequently, the two-dimensional convolutional neural network f EOG is used to extract features from the spectrogram to capture the key features related to the sleep stage in the electrooculogram signal, expressed as: F EOG = f EOG (S EOG (t, f)) Among them, S EOG (t, f) is the short-time Fourier transform result of the electrooculogram signal EOG at time t and frequency f, τ is the time variable, F EOG is the feature of the electrooculogram signal extracted after two-dimensional convolution; Finally, the multi-scale EEG features and EOG features are concatenated to form a multi-modal feature fusion representation to integrate the complementary information of different physiological signals; subsequently, a Dropout layer is added to randomly mask some features, prevent overfitting and improve the generalization ability of the model, thereby enhancing the classification performance for complex signals; F all = Dropout(F concat ) Among them, Concat(·) is the feature concatenation operation, and F concat is the comprehensive vector representation after multi-modal fusion, and F all is the multi-modal feature representation finally obtained after step 1.

4. A multimodal unsupervised cross-domain sleep staging method according to claim 1, characterized in that, Specifically, Step 2 is: The domain generalization feature enhancement module is composed of two sub-modules, namely the sleep adaptation normalization sub-module and the feature adaptive enhancement sub-module; the multi-modal feature representation F obtained from step 1 all First enters the sleep adaptation normalization sub-module, and two one-dimensional convolutional neural networks are used to train the two weight coefficients g and f respectively, so as to adaptively adjust the mean μ and variance σ of the features, reduce the feature differences, and ensure the consistency of the data distributions in the source domain and the target domain; first, calculate the mean and variance of the input features and perform corresponding splicing: μ = Mean(F all ), σ = Std(F all ) MV-Concat = Concat(μ, σ) where Mean(·) represents mean calculation, Std(·) represents variance calculation, and Concat(·) means concatenating the mean and variance to obtain a new statistical vector MV-Concat; Through a convolutional network g fc and batch normalization g bn , using the Sigmoid function to generate the weight coefficient g for adjusting the input feature distribution, and using a similar method to generate the coefficient f: g = Sigmoid(g bn (g fc (MV - Concat))), f = Sigmoid(f bn (f fc (MV - Concat))) Two adaptive weight coefficients f(·) and g(·) are used to re-adjust the mean and variance: μ = f(μ, σ)·μ, σ = g(μ, σ)·σ Finally, based on the new mean μ' and variance σ', a new feature F is obtained. all_1 Next, feature F all-1 enters the feature adaptive enhancer module. Using the channel attention mechanism, by learning the dependency relationships between features, it models the importance of different channels, highlights important features and suppresses irrelevant features through a weighting mechanism, and finally achieves the purpose of adaptively enhancing highly discriminative features; First, for feature F all-1 successively apply two convolutional operations: F = Conv1D(Conv1D(F all-1 )) where Conv1D represents a one-dimensional convolution operation; Then, the global spatial information is compressed by using adaptive average pooling; then, two fully connected layers FC, ReLU and Sigmoid activation functions are applied to finally obtain a weight matrix; specifically as follows: F1 = ReLU(FC(AvgPooling(F))) F2 = Sigmoid(FC(F1)) Finally, scale the feature F according to the weight matrix F2 and combine it with the original input feature F all-1 : F all-2 = F·F2 + F all-1 Thus, the sleep data distribution is adaptively adjusted, highly discriminative features are enhanced, and redundant features are suppressed.

5. A multimodal unsupervised cross-domain sleep staging method according to claim 1, characterized in that Specifically, Step 3 is: The domain attention module obtains an optimized weight matrix through the self-attention mechanism, weights the high-dimensional features to capture specific intra-domain features, enhances the discriminability of the classifier, and reduces the inter-domain differences at the same time; let the feature F obtained through the domain generalization feature enhancement module all-2 be input into the domain attention module, where d is the number of channels of the feature and l is the length of the feature; first, use two Conv1D to perform a linear transformation on the input feature, and apply batch normalization BatchNorm to improve the efficiency and stability of model training: S i = BatchNorm(Conv1D(f i )) S j = BatchNorm(Conv1D(f j )) where f i and f j represent the feature values at positions i and j in the input features; then attention weight calculation is performed; attention weights are generated by calculating the correlation between feature points: Among which W ji represents the attention weight of position j with respect to position i; According to the attention weights, the original features are weighted and combined to generate a weighted output to obtain new features: where \(j\in[1, l]\), all new feature combinations finally result in \(F\) final =\{new_f1,..., new_f j \}.

Images