A Feature Extraction Method Based on Multimodal Bi-domain Consistency
By integrating time-domain and frequency-domain information through a multimodal dual-domain consistent feature extraction method, the problems of modal correlation and noise interference in bioelectric signal analysis are solved, thereby improving the comprehensiveness of feature extraction and the accuracy and robustness of downstream tasks.
Patent Information
- Application Number
- CN202512023830.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-30
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2045-12-30
AI Technical Summary
Existing bioelectric signal analysis methods ignore the intrinsic correlation between modes and fail to effectively integrate time-domain and frequency-domain information, resulting in insufficient feature extraction and susceptibility to noise interference and individual differences, which affects the performance of downstream tasks.
A multimodal dual-domain consistency feature extraction method is adopted, which extracts features through temporal encoder and frequency encoder respectively, and generates cross-modal consistency representation and dual-domain consistency representation in a shared multimodal latent space. The self-supervised contrastive learning strategy is combined to optimize feature alignment and suppress noise interference.
It improves the accuracy and robustness of downstream tasks, reduces the impact of individual differences and noise interference, and enhances the comprehensiveness of feature extraction and the generalization ability of the model.
Smart Images

Figure CN121808342B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of feature extraction, and in particular relates to a feature extraction method based on multimodal dual-domain consistency. Background Technology
[0002] In the field of bioelectrical signal analysis, the joint analysis of multimodal signals (such as EEG, EMG, EOG, etc.) is of great significance for improving the accuracy and robustness of downstream tasks. However, existing analytical methods suffer from the following technical problems:
[0003] Traditional methods typically process signals from different modalities independently, ignoring the inherent correlation between modalities. This results in the extracted features failing to fully reflect common information across modalities, thus affecting the performance of downstream tasks.
[0004] Existing techniques often analyze time-domain or frequency-domain features separately, lacking exploration of the consistency between signals in both domains. Time-domain signals contain transient information, while frequency-domain signals reveal periodic characteristics; combining the two can provide a more comprehensive description of signal properties. However, current methods fail to effectively integrate information from both domains, resulting in insufficient comprehensiveness and robustness in feature extraction.
[0005] Bioelectrical signals are susceptible to noise interference (such as motion artifacts and device noise), and exhibit significant inter-individual differences. Existing methods have failed to effectively eliminate noise and reduce the impact of individual differences during feature extraction, resulting in poor model generalization ability. For example, the baseline EEG signals of different patients vary greatly, and directly using the raw signals for classification may lead to performance degradation. Summary of the Invention
[0006] To address the aforementioned technical problems, this invention provides a feature extraction method based on multimodal dual-domain consistency, comprising:
[0007] Convert time-domain data of multimodal bioelectric signals into frequency-domain data;
[0008] Modal-specific features of the time-domain data and frequency-domain data are extracted using a time-domain encoder and a frequency-domain encoder, respectively.
[0009] The modality-specific features are mapped to a shared multimodal latent space to generate a cross-modal consistent representation;
[0010] The features of the time-domain data and the features of the frequency-domain data are projected onto the dual-domain latent space to generate a dual-domain consistent representation.
[0011] The cross-modal consistency representation and dual-domain consistency representation are optimized based on the contrastive learning strategy, so that the time domain and frequency domain features of the same sample are aligned and the similarity between different samples is suppressed.
[0012] The cross-modal consistency representation and the dual-domain consistency representation are fused to generate the final feature output.
[0013] Preferably, the multimodal bioelectrical signals include EEG signals, EMG signals, and EOG signals;
[0014] The time-domain encoder is a neural network based on time dependence, and the frequency-domain encoder is a neural network based on frequency-domain channel dependence.
[0015] Preferably, the process of converting time-domain data into frequency-domain data includes:
[0016] Perform a Fourier transform on the original time-domain signal to generate a frequency-domain power spectrum;
[0017] The dimension expansion module expands the dimension of the frequency domain data from a single channel to multiple channels.
[0018] Preferably, the process of generating cross-modal consistent representations includes:
[0019] The time-domain and frequency-domain features of the same sample are defined as a positive sample group;
[0020] The time-domain and frequency-domain features of different samples are defined as negative sample groups;
[0021] The parameters of the time-domain encoder and the frequency-domain encoder are optimized by comparing the loss functions.
[0022] Preferably, the comparison loss function is a cosine similarity-based loss function, and the calculation formula includes a similarity maximization term for the positive sample group and a similarity minimization term for the negative sample group.
[0023] Preferably, the process of generating a two-domain consistent representation includes:
[0024] Temporal features are mapped to a dual-domain latent space using a fully connected layer projector.
[0025] Frequency domain features are mapped to a dual-domain latent space using a fully connected layer projector.
[0026] We use a normalized temperature-scaled cross-entropy loss function to align time-domain and frequency-domain features.
[0027] Preferably, the fusion process includes:
[0028] The cross-modal consistency representation and the dual-domain consistency representation are concatenated;
[0029] The concatenated features are input into a feedforward network containing two fully connected layers to generate the final feature output.
[0030] Preferably, the contrastive learning strategy includes a self-supervised learning framework that generates pseudo-labels from unlabeled data and optimizes the feature extraction process based on the pseudo-labels.
[0031] On the other hand, the present invention also provides an electronic device including a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein the processor implements the method when executing the computing program.
[0032] On the other hand, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method.
[0033] Compared with the prior art, the present invention has the following advantages and technical effects:
[0034] This invention uses a self-supervised comparative learning method to comprehensively analyze multimodal signals and integrate time-domain and frequency-domain information. It aims to solve problems such as insufficient multimodal fusion, inadequate utilization of dual-domain features, noise interference, and individual differences in existing technologies, thereby improving the accuracy and robustness of downstream tasks. Attached Figure Description
[0035] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:
[0036] Figure 1 This is a diagram of the multimodal dual-domain consistency model architecture according to an embodiment of the present invention. Detailed Implementation
[0037] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0038] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.
[0039] Example 1
[0040] like Figure 1 As shown, this embodiment provides a feature extraction method based on multimodal dual-domain consistency, including:
[0041] Convert time-domain data of multimodal bioelectric signals into frequency-domain data;
[0042] Modal-specific features of the time-domain data and frequency-domain data are extracted using a time-domain encoder and a frequency-domain encoder, respectively.
[0043] The modality-specific features are mapped to a shared multimodal latent space to generate a cross-modal consistent representation;
[0044] The features of the time-domain data and the features of the frequency-domain data are projected onto the dual-domain latent space to generate a dual-domain consistent representation.
[0045] The cross-modal consistency representation and dual-domain consistency representation are optimized based on the contrastive learning strategy, so that the time domain and frequency domain features of the same sample are aligned and the similarity between different samples is suppressed.
[0046] The cross-modal consistency representation and the dual-domain consistency representation are fused to generate the final feature output.
[0047] Specifically, it includes:
[0048] This embodiment utilizes a self-supervised contrastive learning method to comprehensively analyze multimodal signals, including EEG, EMG, and EOG, while integrating time-domain and frequency-domain signals. It explores the relationship between multimodal and dual-domain signals to improve the analysis level of downstream tasks, thereby enhancing the accuracy and robustness of downstream tasks, while reducing the impact of individual differences and eliminating noise interference. Specifically:
[0049] First, in this embodiment, the time-domain signal dataset is defined as follows: ,in The number of samples per Include Modality and Timestamp. Each sample All of these are labels indicating their pain level. Related. Then this embodiment will... Convert to frequency domain data And through the dimension expansion and domain transformation modules, its dimensions are changed from... Expand to For time-domain samples and its frequency domain counterpart The model extracts time-based features. and frequency-based features These features are then integrated and used for specific downstream tasks.
[0050] The goal of multimodal consistency is to extract inherent and common features across modalities from the same sample, reducing the impact of individual differences. This embodiment assumes that in a potential multimodal space, each sample... (or Modality-specific representation of ) (or All of them should be close to each other. To achieve this, this embodiment employs a time-based encoder. and frequency-based encoders This maps modality-specific representations to a shared multimodal space.
[0051] The time-based encoder can capture reliable dependencies, including transient information, as shown in the following expression:
[0052] ;
[0053] ;
[0054] Where "-" indicates the elimination of the trend portion, Indicates the first The output of each encoder layer.
[0055] This embodiment aims to ensure specific modal representation. Converging in the potential multimodal space, making the same sample Closer. While maintaining The distance. This embodiment defines a positive group. and negative pair To enhance the similarity between positive pairs and reduce the similarity between negative pairs, this embodiment uses a time-based contrast encoder for the samples. loss Defined as:
[0056] ;
[0057] ;
[0058] in Representing vectors sum vector cosine similarity, It is a scaling factor; this loss is used to supervise the encoder. This approach aims to make the embedding vectors within positive pairs closer together and reduce the similarity of embedding vectors between negative pairs. This embodiment uses a frequency channel learner and a frequency time learner to characterize channel dependence and temporal patterns in the frequency domain. Specifically, the frequency channel learner uses... The input expression is as follows:
[0059] ;
[0060] in This refers to operations performed in the modal dimension. It is a frequency domain multilayer perceptron, which uses As a complex weight matrix, where , and with As a bias, where , Similarly, the formula for the frequency-time learner is as follows:
[0061] ;
[0062] ;
[0063] in, and This represents operations in the time dimension. yes The corresponding frequency domain representation. It is a complex weight matrix. It is a bias. Ultimately, this embodiment applies a bias to the input. The data is then concatenated and fed into a two-layer feedforward network to obtain the output. Similarly, this embodiment ensures that for the same sample, its frequency embedding... To maintain consistency across different modalities as much as possible. Therefore, this embodiment defines a positive group. and negative pair Frequency-based loss The calculation is as follows:
[0064] ;
[0065] Furthermore, based on the concept of multimodal consistency, this embodiment introduces dual-domain consistency, a method for extracting features from bioelectrical signals from different domains to effectively reduce the influence of noise. To perform dual-domain joint analysis, this embodiment uses a fully connected layer-based projector. and Embedding the time domain and frequency domain embedded projection This leads to a dual-domain latent space. This ensures that the two embeddings are as consistent as possible within this dual-domain space. Therefore, this embodiment designs a dual-domain consistency loss based on NT-Xent (normalized temperature-scaled cross-entropy) loss. To maintain consistency, this embodiment defines "directly opposite" as... Negative pairs are defined as and The loss is calculated as follows:
[0066] ;
[0067] in It is an indicator function. The purpose of this loss function is to make positive pairs of the same sample tend to be consistent, while keeping negative pairs away from each other.
[0068] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A feature extraction method based on multimodal dual-domain consistency, characterized in that, include: Convert time-domain data of multimodal bioelectric signals into frequency-domain data; Modality-specific features (MDS) of the time-domain and frequency-domain data are extracted using a time-domain encoder and a frequency-domain encoder, respectively. These MDS are time-based features. and frequency-based features ; The modality-specific features are mapped to a shared multimodal latent space to generate a cross-modal consistent representation; The shared multimodal latent space employs a time-based encoder. and frequency-based encoders This maps modality-specific representations to a shared multimodal space. The features of the time-domain data and the features of the frequency-domain data are projected onto the dual-domain latent space to generate a dual-domain consistent representation. The dual-domain potential space is based on a fully connected layer projector. and Embedding the time domain and frequency domain embedded projection To the dual-domain potential space; The cross-modal consistency representation and dual-domain consistency representation are optimized based on the contrastive learning strategy, so that the time domain and frequency domain features of the same sample are aligned and the similarity between different samples is suppressed. The cross-modal consistency representation and the dual-domain consistency representation are fused to generate the final feature output.
2. The method according to claim 1, characterized in that, The multimodal bioelectric signals include EEG signals, EMG signals, and EOG signals; The time-domain encoder is a neural network based on time dependence, and the frequency-domain encoder is a neural network based on frequency-domain channel dependence.
3. The method according to claim 1, characterized in that, The process of converting time-domain data into frequency-domain data includes: Perform a Fourier transform on the original time-domain signal to generate a frequency-domain power spectrum; The dimension expansion module expands the dimension of the frequency domain data from a single channel to multiple channels.
4. The method according to claim 1, characterized in that, The process of generating cross-modal consistent representations includes: The time-domain and frequency-domain features of the same sample are defined as a positive sample group; The time-domain and frequency-domain features of different samples are defined as negative sample groups; The parameters of the time-domain encoder and the frequency-domain encoder are optimized by comparing the loss functions.
5. The method according to claim 4, characterized in that, The contrast loss function is a cosine similarity-based loss function, and the calculation formula includes a similarity maximization term for the positive sample group and a similarity minimization term for the negative sample group.
6. The method according to claim 1, characterized in that, The process of generating a two-domain consistent representation includes: Temporal features are mapped to a dual-domain latent space using a fully connected layer projector. Frequency domain features are mapped to a dual-domain latent space using a fully connected layer projector. We use a normalized temperature-scaled cross-entropy loss function to align time-domain and frequency-domain features.
7. The method according to claim 1, characterized in that, The fusion process includes: The cross-modal consistency representation and the dual-domain consistency representation are concatenated; The concatenated features are input into a feedforward network containing two fully connected layers to generate the final feature output.
8. The method according to claim 1, characterized in that, The contrastive learning strategy includes a self-supervised learning framework that generates pseudo-labels from unlabeled data and optimizes the feature extraction process based on the pseudo-labels.
9. An electronic device comprising a memory, a processor, and a computing program stored in the memory and executable on the processor, characterized in that, When the processor executes the computing program, it implements the method of any one of claims 1-8.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1-8.
Citation Information
Patent Citations
Electroencephalogram signal feature extraction method and system based on adaptive time-frequency feature collaboration
CN119577419A
Fault diagnosis model training method, fault diagnosis method and related equipment
CN120257143A
