Feature extraction method based on multi-mode double-domain consistency

Abstract

The invention discloses a feature extraction method based on multi-modal double-domain consistency, and belongs to the field of feature extraction, and the method comprises the steps: converting time domain data of a multi-modal bio-electricity signal into frequency domain data, and extracting modal specific features through a time domain encoder and a frequency domain encoder. Then, the features are mapped to a shared multi-modal potential space to generate a cross-modal consistency representation, and the cross-modal consistency representation is projected to a dual-domain potential space to generate a dual-domain consistency representation. The two representations are optimized through a contrast learning strategy, so that the time domain and frequency domain features of the same sample are aligned, and meanwhile, the similarity between different samples is inhibited. And finally, fusing cross-modal and double-domain consistency representation to generate final feature output so as to realize effective analysis and processing of the multi-modal bio-electricity signal.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of feature extraction, and particularly relates to a feature extraction method based on multi-modal dual-domain consistency. BACKGROUND

[0002] In the field of bioelectric signal analysis, joint analysis of multi-modal signals (such as EEG, EMG, EOG, etc.) is of great significance to improve the accuracy and robustness of downstream tasks. However, existing analysis methods have the following technical problems:

[0003] Traditional methods usually process signals of different modalities independently, ignoring the inherent correlation between modalities, resulting in features that cannot fully reflect the common information across modalities, thereby affecting the performance of downstream tasks.

[0004] Existing technologies often analyze time-domain or frequency-domain features separately, lacking exploration of dual-domain signal consistency. Time-domain signals contain transient information, while frequency-domain signals reveal periodic characteristics. Combining both can more comprehensively describe signal characteristics. However, current methods fail to effectively integrate dual-domain information, resulting in insufficient comprehensiveness and robustness of feature extraction.

[0005] Bioelectric signals are susceptible to noise interference (such as motion artifacts, device noise, etc.), and there are significant individual differences. Existing methods fail to effectively eliminate noise and reduce the impact of individual differences during feature extraction, resulting in poor model generalization ability. For example, the baseline of EEG signals from different patients varies greatly, and direct use of raw signals for classification may lead to performance degradation. SUMMARY

[0006] To solve the above technical problems, the application provides a feature extraction method based on multi-modal dual-domain consistency, comprising:

[0007] Converting time-domain data of multi-modal bioelectric signals into frequency-domain data;

[0008] Extracting modality-specific features of the time-domain data and frequency-domain data through a time-domain encoder and a frequency-domain encoder, respectively;

[0009] Mapping the modality-specific features to a shared multi-modal latent space to generate cross-modal consistency representation;

[0010] Projecting the features of the time-domain data and the features of the frequency-domain data into a dual-domain latent space, respectively, to generate dual-domain consistency representation;

[0011] Optimizing the cross-modal consistency representation and the dual-domain consistency representation based on a contrastive learning strategy to align the time-domain and frequency-domain features of the same sample and suppress the similarity between different samples;

[0012] The cross-modal consistency representation and the dual-domain consistency representation are fused to generate the final feature output.

[0013] Preferably, the multimodal bioelectrical signals include EEG signals, EMG signals, and EOG signals;

[0014] The time-domain encoder is a neural network based on time dependence, and the frequency-domain encoder is a neural network based on frequency-domain channel dependence.

[0015] Preferably, the process of converting time-domain data into frequency-domain data includes:

[0016] Perform a Fourier transform on the original time-domain signal to generate a frequency-domain power spectrum;

[0017] The dimension expansion module expands the dimension of the frequency domain data from a single channel to multiple channels.

[0018] Preferably, the process of generating cross-modal consistent representations includes:

[0019] The time-domain and frequency-domain features of the same sample are defined as a positive sample group;

[0020] The time-domain and frequency-domain features of different samples are defined as negative sample groups;

[0021] The parameters of the time-domain encoder and the frequency-domain encoder are optimized by comparing the loss functions.

[0022] Preferably, the comparison loss function is a cosine similarity-based loss function, and the calculation formula includes a similarity maximization term for the positive sample group and a similarity minimization term for the negative sample group.

[0023] Preferably, the process of generating a two-domain consistent representation includes:

[0024] Temporal features are mapped to a dual-domain latent space using a fully connected layer projector.

[0025] Frequency domain features are mapped to a dual-domain latent space using a fully connected layer projector.

[0026] We use a normalized temperature-scaled cross-entropy loss function to align time-domain and frequency-domain features.

[0027] Preferably, the fusion process includes:

[0028] The cross-modal consistency representation and the dual-domain consistency representation are concatenated;

[0029] The concatenated features are input into a feedforward network containing two fully connected layers to generate the final feature output.

[0030] Preferably, the contrastive learning strategy includes a self-supervised learning framework that generates pseudo-labels from unlabeled data and optimizes the feature extraction process based on the pseudo-labels.

[0031] On the other hand, the present invention also provides an electronic device including a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein the processor implements the method when executing the computing program.

[0032] On the other hand, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method.

[0033] Compared with the prior art, the present invention has the following advantages and technical effects:

[0034] This invention uses a self-supervised comparative learning method to comprehensively analyze multimodal signals and integrate time-domain and frequency-domain information. It aims to solve problems such as insufficient multimodal fusion, inadequate utilization of dual-domain features, noise interference, and individual differences in existing technologies, thereby improving the accuracy and robustness of downstream tasks. Attached Figure Description

[0035] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0036] Figure 1 This is a diagram of the multimodal dual-domain consistency model architecture according to an embodiment of the present invention. Detailed Implementation

[0037] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0038] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0039] Example 1

[0040] like Figure 1 As shown, this embodiment provides a feature extraction method based on multimodal dual-domain consistency, including:

[0041] Convert time-domain data of multimodal bioelectric signals into frequency-domain data;

[0042] Modal-specific features of the time-domain data and frequency-domain data are extracted using a time-domain encoder and a frequency-domain encoder, respectively.

[0043] The modality-specific features are mapped to a shared multimodal latent space to generate a cross-modal consistent representation;

[0044] The features of the time-domain data and the features of the frequency-domain data are projected onto the dual-domain latent space to generate a dual-domain consistent representation.

[0045] The cross-modal consistency representation and dual-domain consistency representation are optimized based on the contrastive learning strategy, so that the time domain and frequency domain features of the same sample are aligned and the similarity between different samples is suppressed.

[0046] The cross-modal consistency representation and the dual-domain consistency representation are fused to generate the final feature output.

[0047] Specifically, it includes:

[0048] This embodiment utilizes a self-supervised contrastive learning method to comprehensively analyze multimodal signals, including EEG, EMG, and EOG, while integrating time-domain and frequency-domain signals. It explores the relationship between multimodal and dual-domain signals to improve the analysis level of downstream tasks, thereby enhancing the accuracy and robustness of downstream tasks, while reducing the impact of individual differences and eliminating noise interference. Specifically:

[0049] First, in this embodiment, the time-domain signal dataset is defined as follows: ,in The number of samples per Include Modality and Timestamp. Each sample All of these are labels indicating their pain level. Related. Then this embodiment will... Convert to frequency domain data And through the dimension expansion and domain transformation modules, its dimensions are changed from... Expand to For time-domain samples and its frequency domain counterpart The model extracts time-based features. and frequency-based features These features are then integrated and used for specific downstream tasks.

[0050] The goal of multimodal consistency is to extract inherent and common features across modalities from the same sample, reducing the impact of individual differences. This embodiment assumes that in a potential multimodal space, each sample... (or Modality-specific representation of ) (or All of them should be close to each other. To achieve this, this embodiment employs a time-based encoder. and frequency-based encoders This maps modality-specific representations to a shared multimodal space.

[0051] The time-based encoder can capture reliable dependencies, including transient information, as shown in the following expression:

[0052] ;

[0053] ;

[0054] Where "-" indicates the elimination of the trend portion, Indicates the first The output of each encoder layer.

[0055] This embodiment aims to ensure specific modal representation. Converging in the potential multimodal space, making the same sample Closer. While maintaining The distance. This embodiment defines a positive group. and negative pair To enhance the similarity between positive pairs and reduce the similarity between negative pairs, this embodiment uses a time-based contrast encoder for the samples. loss Defined as:

[0056] ;

[0057] ;

[0058] in Representing vectors sum vector cosine similarity, It is a scaling factor; this loss is used to supervise the encoder. This approach aims to make the embedding vectors within positive pairs closer together and reduce the similarity of embedding vectors between negative pairs. This embodiment uses a frequency channel learner and a frequency time learner to characterize channel dependence and temporal patterns in the frequency domain. Specifically, the frequency channel learner uses... The input expression is as follows:

[0059] ;

[0060] in This represents the operation performed in the modal dimension. It is a frequency domain multilayer perceptron, which uses As a complex weight matrix, where , and with As a bias, where , Similarly, the formula for the frequency-time learner is as follows:

[0061] ;

[0062] ;

[0063] in, and This represents operations in the time dimension. yes The corresponding frequency domain representation. It is a complex weight matrix. It is a bias. Ultimately, this embodiment applies a bias to the input. The data is then concatenated and fed into a two-layer feedforward network to obtain the output. Similarly, this embodiment ensures that for the same sample, its frequency embedding... To maintain consistency across different modalities as much as possible. Therefore, this embodiment defines a positive group. and negative pair Frequency-based loss The calculation is as follows:

[0064] ;

[0065] Furthermore, based on the concept of multimodal consistency, this embodiment introduces dual-domain consistency, a method for extracting features from bioelectrical signals from different domains to effectively reduce the influence of noise. To perform dual-domain joint analysis, this embodiment uses a fully connected layer-based projector. and Embedding the time domain and frequency domain embedded projection This leads to a dual-domain latent space. This ensures that the two embeddings are as consistent as possible within this dual-domain space. Therefore, this embodiment designs a dual-domain consistency loss based on NT-Xent (normalized temperature-scaled cross-entropy) loss. To maintain consistency, this embodiment defines "directly opposite" as... Negative pairs are defined as and The loss is calculated as follows:

[0066] ;

[0067] in It is an indicator function. The purpose of this loss function is to make positive pairs of the same sample tend to be consistent, while keeping negative pairs away from each other.

[0068] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A feature extraction method based on multimodal dual-domain consistency, characterized in that, include: Convert time-domain data of multimodal bioelectric signals into frequency-domain data; Modal-specific features of the time-domain data and frequency-domain data are extracted using a time-domain encoder and a frequency-domain encoder, respectively. The modality-specific features are mapped to a shared multimodal latent space to generate a cross-modal consistent representation; The features of the time-domain data and the features of the frequency-domain data are projected onto the dual-domain latent space to generate a dual-domain consistent representation. The cross-modal consistency representation and dual-domain consistency representation are optimized based on the contrastive learning strategy, so that the time domain and frequency domain features of the same sample are aligned and the similarity between different samples is suppressed. The cross-modal consistency representation and the dual-domain consistency representation are fused to generate the final feature output.

2. The method according to claim 1, characterized in that, The multimodal bioelectric signals include EEG signals, EMG signals, and EOG signals; The time-domain encoder is a neural network based on time dependence, and the frequency-domain encoder is a neural network based on frequency-domain channel dependence.

3. The method according to claim 1, characterized in that, The process of converting time-domain data into frequency-domain data includes: Perform a Fourier transform on the original time-domain signal to generate a frequency-domain power spectrum; The dimension expansion module expands the dimension of the frequency domain data from a single channel to multiple channels.

4. The method according to claim 1, characterized in that, The process of generating cross-modal consistent representations includes: The time-domain and frequency-domain features of the same sample are defined as a positive sample group; The time-domain and frequency-domain features of different samples are defined as negative sample groups; The parameters of the time-domain encoder and the frequency-domain encoder are optimized by comparing the loss functions.

5. The method according to claim 1, characterized in that, The contrast loss function is a cosine similarity-based loss function, and the calculation formula includes a similarity maximization term for the positive sample group and a similarity minimization term for the negative sample group.

6. The method according to claim 1, characterized in that, The process of generating a two-domain consistent representation includes: Temporal features are mapped to a dual-domain latent space using a fully connected layer projector. Frequency domain features are mapped to a dual-domain latent space using a fully connected layer projector. We use a normalized temperature-scaled cross-entropy loss function to align time-domain and frequency-domain features.

7. The method according to claim 1, characterized in that, The fusion process includes: The cross-modal consistency representation and the dual-domain consistency representation are concatenated; The concatenated features are input into a feedforward network containing two fully connected layers to generate the final feature output.

8. The method according to claim 1, characterized in that, The contrastive learning strategy includes a self-supervised learning framework that generates pseudo-labels from unlabeled data and optimizes the feature extraction process based on the pseudo-labels.

9. An electronic device comprising a memory, a processor, and a computing program stored in the memory and executable on the processor, characterized in that, When the processor executes the computing program, it implements the method of any one of claims 1-8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1-8.

Images