A multi-center non-paired multi-modal sample alzheimer's disease early recognition method

Abstract

The disclosure is a multi-center non-paired multi-modal medical data category identification method, comprising: receiving multi-modal data of EEG, sMRI and plasma protein, and preprocessing; extracting multi-dimensional features of the preprocessed multi-modal data; constructing a plurality of same-label pseudo-paired units according to the category label information and the mode identification of the multi-dimensional features; constructing a multi-modal fusion model, the multi-modal fusion model comprising: an encoder, a latent variable representation module, an adaptive weighted fusion module, a lightweight channel gating module and a multi-classification head; model training and prediction. The embodiment realizes adaptive alignment and collaborative discrimination of cross-modal features, provides a feasible method for improving the accuracy, robustness and explainability of early AD classification, and provides a more practical and flexible multi-modal auxiliary diagnosis tool for clinical use.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method for class identification of multicenter unpaired multimodal medical data. Background Technology

[0002] Alzheimer's disease (AD) is an irreversible neurodegenerative disease caused by abnormal brain cell metabolism. It is the most common cause of dementia, and its clinical features mainly include cognitive decline, difficulty in language expression, and behavioral disorders. Currently, there is no cure for AD. The severity of behavioral symptoms and brain atrophy (loss of gray matter) can vary greatly among different patients, and this heterogeneity complicates diagnosis, treatment, and prevention.

[0003] Due to the high heterogeneity and complex pathological mechanisms of Alzheimer's disease, a single biomarker cannot fully reflect the multi-layered damage caused by the disease, limiting the accuracy of its early identification and severity assessment. Electroencephalography (EEG), structural magnetic resonance imaging (sMRI), and plasma protein (PP) provide complementary information on the development of AD from multiple dimensions, including neurophysiological activity, brain structure, and molecular pathology. Constructing a multimodal biomarker system has become an important approach to improve early diagnostic performance.

[0004] However, in clinical practice, these modal data often originate from different samples and have inconsistent data quality, exhibiting problems such as sample misalignment and significant differences in feature distribution. Traditional fusion methods rely on strictly aligned multimodal samples. On the one hand, this alignment requirement limits their application in real-world scenarios; on the other hand, existing methods often simply discard unpaired samples, ignoring their potential value. How to fully utilize unpaired multimodal samples from multiple centers is crucial for achieving adaptive AD prediction. Therefore, designing an AD recognition model that can effectively handle misaligned multimodal data and adaptively fuse cross-modal discriminative features to process modal data and provide a data foundation for AD recognition is urgently needed in clinical practice.

[0005] It should be noted that this section is intended to provide background or context for the technical solutions of this disclosure as set forth in the claims. The description herein does not constitute an admission that it is prior art simply because it is included in this section. Summary of the Invention

[0006] The purpose of this invention is to provide a method for class identification of multicenter unpaired multimodal medical data, thereby overcoming, at least to some extent, one or more problems caused by the limitations and defects of related technologies.

[0007] This invention first provides a method for class identification of multicenter unpaired multimodal medical data, including: S1 receives multimodal data from EEG, sMRI, and plasma proteins, and performs preprocessing. S2, extract multi-dimensional features from the preprocessed multimodal data. The multi-dimensional features include the spectral features, time-domain features, entropy and complexity features, wavelet time-frequency features, Hjorth parameters and statistical features of EEG, gray matter volume measurement and structured feature vector of sMRI, and plasma protein molecular features. S3: Based on the category label information and modality identifier of multi-dimensional features, construct multiple pseudo-pairing units with the same label, and independently divide each modality into training set, validation set and test set. Pre-set during the model training stage, introduce a weighted random sampler for each modality, assign higher sampling weight to categories with lower frequency of occurrence, and concatenate multiple pseudo-pairing units with the same label to form a multimodal training batch. S4. Construct a multimodal fusion model, which includes: an encoder, a latent variable representation module, an adaptive weighted fusion module, a lightweight channel gating module, and a multi-classification head; The encoder extracts the latent representation of each modality in the pseudo-pairing units with the same label, and processes it through two lightweight linear heads to obtain the mean vector and variance vector. The latent variable representation module constructs a diagonal Gaussian posterior distribution on the mean vector and variance vector to obtain the Gaussian distribution representation of the latent space of each modality. The adaptive weighted fusion module adaptively and dynamically weights each modality according to the modality availability mask, and recalibrates the mean vector using the weights to obtain the weighted representation of each modality. The weighted representations of each modality are stacked according to the modality dimension to form a fusion input tensor. The lightweight channel gating module assigns attention weights to the fusion input tensor and performs feature reshaping to obtain the fusion representation. The multi-class head predicts and identifies the fusion representation to obtain the category recognition result. S5 is used to train the multimodal fusion model. S6. The trained multimodal fusion model is used to predict the multidimensional features corresponding to the pseudo-pairing units with the same label, so as to obtain the category recognition results of the multimodal data.

[0008] In this invention, S1 includes preprocessing including: bandpass filtering, noise removal, artifact removal, re-reference, interpolation repair, channel alignment, sample slicing, and normalization.

[0009] In this invention, in S2, the plasma protein molecular characteristics include PLGF, ApoE, TTR, ApoA-II, PYY, SGOT, AlMicro, BNP, Eotaxin-3, and IGM.

[0010] In this invention, in step S5, a comprehensive loss function is used to train the multimodal fusion model. The comprehensive loss includes cross-entropy classification loss, single-modal supervision loss, and regularization loss. The regularization loss includes probability alignment loss and self-information synchronization loss.

[0011] In this invention, the multi-classification head includes a fusion classifier and a single-modality auxiliary classification head. The fusion classifier is used to predict and identify the fusion representation, and output the predicted category probability and the final classification label. The single-modality auxiliary classification head is used to predict the mean vector to obtain the single-modality independent classification prediction value.

[0012] In this invention, the identification method further includes: S7 uses a five-fold cross-validation method to evaluate the model performance.

[0013] The present invention further provides a multimodal medical data category recognition system, comprising: The data preprocessing module is used to receive and preprocess multimodal data from EEG, sMRI, and plasma proteins. The feature extraction module is used to extract multi-dimensional features of the preprocessed multimodal data. The multi-dimensional features include the spectral features, time-domain features, entropy and complexity features, wavelet time-frequency features, Hjorth parameters and statistical features of EEG, gray matter volume measurement and structured feature vector of sMRI, and plasma protein molecular features. The pseudo-pairing unit construction module is used to construct multiple pseudo-pairing units with the same label based on the category label information and modality identifier of multi-dimensional features, and to independently divide each modality into training set, validation set and test set in a hierarchical manner; The weighted sampling module is used to introduce a weighted random sampler for each modality during the model training phase, assign higher sampling weights to categories that appear less frequently, and concatenate multiple pseudo-paired units with the same label to form a multimodal training batch. A multimodal fusion model construction module is used to construct a multimodal fusion model, which includes: an encoder, a latent variable representation module, an adaptive weighted fusion module, a lightweight channel gating module, and a multi-classification head; The encoder extracts the latent representation of each modality in the pseudo-pairing units with the same label, and processes it through two lightweight linear heads to obtain the mean vector and variance vector. The latent variable representation module constructs a diagonal Gaussian posterior distribution on the mean vector and variance vector to obtain the Gaussian distribution representation of the latent space of each modality. The adaptive weighted fusion module adaptively and dynamically weights each modality according to the modality availability mask, and recalibrates the mean vector using the weights to obtain the weighted representation of each modality. The weighted representations of each modality are stacked according to the modality dimension to form a fusion input tensor. The lightweight channel gating module assigns attention weights to the fusion input tensor and performs feature reshaping to obtain the fusion representation. The multi-class head predicts and identifies the fusion representation to obtain the category recognition result. The training module is used to train the multimodal fusion model; The prediction and recognition module is used to predict the multi-dimensional features corresponding to pseudo-pairing units with the same label using the trained multi-modal fusion model, so as to obtain the category recognition results of multi-modal data.

[0014] The technical solution provided by this invention may include the following beneficial effects: The present invention provides a method for class identification of multi-center unpaired multimodal medical data, which designs a pseudo-paired sample construction mechanism with consistent labels: an adaptive modeling based on class conditions Gaussian mixture model is adopted, and diverse and class-balanced pseudo-paired training samples are generated by clustered round sampling and cross-modal matching with the same label. Multimodal fusion training is achieved without introducing alignment information, reducing the difficulty of data collection and preprocessing, and reducing the engineering cost caused by the difficulty of aligning cross-center data. This invention models each modality representation as a diagonal Gaussian posterior distribution, uses the prediction variance to characterize sample-level uncertainty, and achieves adaptive weighted fusion with uncertainty perception: modes with high noise, poor quality, or distribution deviation are automatically downweighted, thereby suppressing the "over-dominance of dominant modes" and the propagation of erroneous information from a mechanism perspective, and improving the robustness and reliability of fusion decisions. A lightweight channel gating module is introduced to perform fine-grained reshaping and redundancy suppression on the fusion representation based on adaptive modality weighting, thereby enhancing the efficiency of cross-modal complementary information utilization and improving the separability and generalization ability of classification boundaries. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0016] Figure 1A flowchart illustrating a method for classifying multicenter unpaired multimodal medical data in an exemplary embodiment of this disclosure; Figure 2 A flowchart illustrating a method for classifying multicenter unpaired multimodal medical data in yet another exemplary embodiment of this disclosure; Figure 3 This diagram illustrates the overall framework of the multimodal fusion model in an exemplary embodiment of this disclosure. Figure 4 A schematic diagram of the network structure of a multimodal fusion model in an exemplary embodiment of this disclosure is shown. Detailed Implementation

[0017] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0018] Furthermore, the accompanying drawings are merely illustrative diagrams of embodiments of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0019] This example implementation first provides a method for class identification of multicenter unpaired multimodal medical data. Please refer to [reference needed]. Figure 1 This method may include: S1-S6, as follows: S1 receives multimodal data from EEG, sMRI, and plasma proteins, and performs preprocessing. S2, extract multi-dimensional features from the preprocessed multimodal data. The multi-dimensional features include the spectral features, time-domain features, entropy and complexity features, wavelet time-frequency features, Hjorth parameters and statistical features of EEG, gray matter volume measurement and structured feature vector of sMRI, and plasma protein molecular features. S3: Based on the category label information and modality identifier of multi-dimensional features, construct multiple pseudo-pairing units with the same label, and independently divide each modality into training set, validation set and test set. Pre-set during the model training stage, introduce a weighted random sampler for each modality, assign higher sampling weight to categories with lower frequency of occurrence, and concatenate multiple pseudo-pairing units with the same label to form a multimodal training batch. S4. Construct a multimodal fusion model, which includes: an encoder, a latent variable representation module, an adaptive weighted fusion module, a lightweight channel gating module, and a multi-classification head; The encoder extracts the latent representation of each modality in the pseudo-pairing units with the same label, and processes it through two lightweight linear heads to obtain the mean vector and variance vector. The latent variable representation module constructs a diagonal Gaussian posterior distribution on the mean vector and variance vector to obtain the Gaussian distribution representation of the latent space of each modality. The adaptive weighted fusion module adaptively and dynamically weights each modality according to the modality availability mask, and recalibrates the mean vector using the weights to obtain the weighted representation of each modality. The weighted representations of each modality are stacked according to the modality dimension to form a fusion input tensor. The lightweight channel gating module assigns attention weights to the fusion input tensor and performs feature reshaping to obtain the fusion representation. The multi-class head predicts and identifies the fusion representation to obtain the category recognition result. S5 is used to train the multimodal fusion model. S6. The trained multimodal fusion model is used to predict the multidimensional features corresponding to the pseudo-pairing units with the same label, so as to obtain the category recognition results of the multimodal data.

[0020] In this embodiment, modeling can be directly performed when multimodal data such as EEG, sMRI, and plasma proteins come from different cohorts, different follow-up times, or different acquisition protocols, and when modality loss is common. This does not require strict cross-modal one-to-one sample alignment, thus significantly improving the usability and applicability in real clinical multicenter scenarios. This invention designs a pseudo-paired sample construction mechanism with consistent labels: it adopts adaptive modeling based on class conditions Gaussian mixture model, and generates diverse and class-balanced pseudo-paired training samples through clustered round sampling and cross-modal matching with the same label. Without introducing alignment information, it realizes multimodal fusion training, reduces the difficulty of data collection and preprocessing, and reduces the engineering cost caused by the difficulty of aligning cross-center data. This invention models each modality representation as a diagonal Gaussian posterior distribution, uses the prediction variance to characterize sample-level uncertainty, and achieves adaptive weighted fusion with uncertainty perception: modes with high noise, poor quality, or distribution deviation are automatically downweighted, thereby suppressing the "over-dominance of dominant modes" and the propagation of erroneous information from a mechanism perspective, and improving the robustness and reliability of fusion decisions. A lightweight channel gating module is introduced to perform fine-grained reshaping and redundancy suppression on the fusion representation based on adaptive modality weighting, thereby enhancing the efficiency of cross-modal complementary information utilization and improving the separability and generalization ability of classification boundaries.

[0021] The following is combined with Figures 2-4 The specific process of each step in the above embodiments will be explained.

[0022] Please refer to Figure 2 and Figure 4 The overall process of this invention includes: inputting multimodal raw data (unequal samples of various modalities such as EEG, sMRI, and plasma proteins), sequentially undergoing data preprocessing, feature extraction, pseudo-pairing construction and weighted random sampling, modality-specific encoding, adaptive weighted fusion and probability alignment and regularization, then performing overall optimization and model training, and finally outputting classification results (i.e. outputting predictions and evaluations) through a unified multimodal fusion classifier.

[0023] S1 receives multimodal data from EEG, sMRI, and plasma proteins and performs preprocessing.

[0024] In this embodiment, data preprocessing is illustrated using electroencephalogram (EEG) data as an example. Preprocessing is a key step in ensuring the quality of EEG signals and the effectiveness of analysis.

[0025] First, the EEG data was preprocessed using the EEGLAB toolbox in MATLAB, including bandpass filtering (to initially limit the frequency band), noise removal, artifact removal (various physiological artifacts, such as eye movement and electromyography interference), rereference (to improve consistency), and interpolation repair of rejected / interfered channels (to maintain data integrity). Nineteen standard channels were selected for alignment with international standards: Fp1, Fp2, F7, F3, Fz, F4, F8, T7, C3, Cz, C4, T8, P7, P3, Pz, P4, P8, O1, and O2.

[0026] Based on this, in order to meet the model input requirements, the experimental data were divided into sample segments with a continuous duration of 4 seconds according to the channel. Then, a bandpass filter of 0.5 to 45 Hz was used to further remove high and low frequency interference and retain the effective frequency bands closely related to brain activity. Finally, each channel was standardized (mean of 0 and standard deviation of 1) to eliminate amplitude differences and thus provide high-quality and uniform input for subsequent model training.

[0027] Next, the 19-channel EEG data from the experiment were independently divided into 4-second non-overlapping samples to increase the sample size. Then, a second preprocessing step was performed: standard normalization on each channel. This was done independently for each EEG channel and calculated separately for each 4-second sample. The normalized EEG signal value at the i-th time point is shown below. It is expressed as follows:

[0028] Where L is the number of time points contained in this channel within the 4-second EEG segment; i is the time point index, with values ​​ranging from 1, 2, ..., L. The arithmetic mean of the voltage values ​​at L time points for each EEG channel. For time points i The voltage value; This is the calculated standard deviation of the signal within the sample of this channel.

[0029] The normalized EEG signal vector of this channel is .

[0030] S2, multi-dimensional feature extraction.

[0031] 1. First, let's explain the three types of data: EEG, sMRI, and plasma protein.

[0032] (1) Electroencephalogram (EEG) Electroencephalography (EEG), as a non-invasive, low-cost, and high-temporal-resolution detection method, has become a highly promising neurophysiological biomarker for the early classification of Alzheimer's disease (AD). EEG records macroscopic superposition signals of electrical activity of neuronal groups in the cerebral cortex, reflecting the dynamic functional state of the brain in real time and effectively complementing static structural imaging. In AD patients, EEG signals show significant changes, including slowed background rhythms, reduced synchronicity, and deterioration in nonlinear dynamic characteristics. These changes are closely related to cognitive decline and disease progression, giving EEG unique value in early diagnosis and disease monitoring.

[0033] In the frequency domain, such as Indicators such as ratio, dominant frequency, spectral centroid, and spectral flux can effectively reflect AD-related rhythm disturbances and band power variations. Among these, , , , These represent the brain signals in... frequency band frequency band frequency band and Frequency bands. Time-domain features such as peak amplitude, skewness, kurtosis, and variance can capture anomalies in waveform morphology distribution characteristics. Entropy features (approximate entropy, permutation entropy, sample entropy) are used to quantify the attenuation of EEG signal complexity; a decrease in these values ​​suggests that brain activity in Alzheimer's disease (AD) patients tends to be simpler and more regular. Wavelet analysis provides time-frequency localization capabilities; indicators such as wavelet packet energy approximation and energy ratio can finely describe the distribution and transfer patterns of energy in different frequency bands. In addition, nonlinear dynamic parameters such as fractal features, Lyapunov exponent, and peak factor provide supplementary evidence for identifying changes in chaotic characteristics and loss of stability in the nervous system. These multidimensional EEG features not only have clear neurophysiological interpretations individually, but also collectively constitute a comprehensive indicator system for characterizing the electrophysiological state of the brain.

[0034] (2) Structural magnetic resonance imaging (sMRI) Structural magnetic resonance imaging (sMRI) also plays a crucial role in the early diagnosis of Alzheimer's disease (AD). sMRI, with its advantages of being non-invasive and having high spatial resolution, is one of the most commonly used neuroimaging techniques for characterizing the neuroanatomical degenerative changes in AD. Through quantitative analysis of brain structures (such as voxel-based morphological analysis), sMRI can reveal characteristic brain atrophy patterns associated with cognitive decline, providing important evidence for understanding the structural basis of disease progression. Among numerous sMRI biomarkers, changes in gray matter volume are particularly prominent: structural changes in AD often manifest as predominantly medial temporal lobe atrophy, with the hippocampus and amygdala being the most distinctive regions, and can further affect interconnected network structures such as the temporal-frontal-parietal cortex, thus becoming an important imaging basis for identifying AD (and its prodromal stages).

[0035] (3) Plasma proteins The plasma protein molecular characteristics include PLGF, ApoE, TTR, ApoA-II, PYY, SGOT, A1Micro, BNP, Eotaxin-3, and IGM.

[0036] In early Alzheimer's disease (AD) diagnostic research, the analysis of multimodal data such as EEG, sMRI, and plasma proteins faces two core challenges: sample misalignment and modal heterogeneity. Sample misalignment stems from the difficulty in obtaining complete multimodal data from all subjects in clinical practice, resulting in significant gaps and mismatches in cross-modal samples. Modal heterogeneity manifests in the fundamental structural and semantic differences between data from different sources (such as electrophysiological signals from EEG, brain structural changes from sMRI, and molecular concentration information of plasma proteins), making direct fusion difficult. These factors collectively limit the effectiveness and generalization ability of traditional multimodal learning models. Therefore, this application develops an analytical framework capable of adaptively handling misaligned and heterogeneous multimodal data, which is crucial for fully exploring the value of clinical data and constructing a robust and interpretable early AD diagnostic system.

[0037] 2. Next, the extraction of multi-dimensional features from the three types of data will be explained.

[0038] Taking electroencephalogram (EEG) signals as an example, this embodiment describes the characteristics of EEG signals from different dimensions (spectrum, entropy, fractal, statistics, time-frequency, etc.), and can be widely used in brain function research and clinical diagnosis (such as epilepsy, Alzheimer's disease detection, etc.). In practical applications, multiple features are usually combined to improve the accuracy of analysis. The following is a detailed introduction to the extracted EEG feature parameters, organized by functional category: (1) Characteristics of the ratio of spectral power to frequency band in EEG The core EEG feature of AD patients is increased slow-wave activity. , (frequency band) and fast wave activity decreased ( , (Frequency band), which directly affects the power ratio of various frequency bands. Specifically, this embodiment extracts... , , , and as well as The dominant frequency of the frequency band.

[0039] The power ratio is an important characteristic indicator in EEG analysis that reflects the balance between excitation and inhibition in brain electrical activity, especially valuable in research on neurodegenerative diseases and cognitive states. For any target frequency band (e.g., ), bandwidth power The calculation formula is:

[0040] in, The total power within the target frequency band; X[k] is the k-th complex spectral coefficient obtained after Fourier transform of the input signal; k is the frequency index; i and j are the spectral indices corresponding to the lower and upper limits of the target frequency band, respectively; |X[k]|² represents the power at frequency index k.

[0041]

[0042] in: This is the ratio value of the final output. yes Power in the frequency band (8-13 Hz), yes Power in the frequency band (13-30 Hz), yes Power in the frequency band (1-4 Hz), yes Power in the frequency band (4-8 Hz). Similarly, relevant characteristics also include... Power ratio and Power ratio, where express Bandwidth power and The ratio of bandwidth power, express Bandwidth power and The ratio of bandwidth power.

[0043] Extract the dominant frequencies of each band in each channel for each time period. The dominant frequency is used to obtain the power spectrum by performing a Fourier transform on the input signal. Find the frequency with the highest power within the band (8-13Hz). The dominant frequency is also obtained by Fourier transforming the power spectrum. Find the frequency with the highest power within the band (30-45Hz). The dominant frequency represents the frequency at which the power spectrum reaches its maximum value within the target band. Dominant frequency representation The frequency corresponding to the point of maximum power within the frequency band; Dominant frequency representation The frequency corresponding to the point of maximum power within the frequency band; In this invention, the frequency band is 30–45 Hz.

[0044] The core function of extracting these EEG features is to provide a set of quantifiable and objective neurophysiological biomarkers to distinguish Alzheimer's disease patients from healthy individuals. These features effectively capture the typical pattern of "enhanced slow waves and diminished fast waves" in AD patients, enabling machine learning models to classify with high accuracy. More importantly, they can not only be used to assist in diagnosis but also to assess disease severity, track progression, and are expected to be applied to the early screening and prediction of the transformation from mild cognitive impairment (MCI) to AD by detecting early subtle changes, providing discriminative features for deep learning models.

[0045] (2) Entropy and complexity characteristics of EEG Entropy and complexity indices reflect the irregularity and information richness of EEG signals, and are a quantification of the irregularity of EEG sequence signals. The entropy features extracted in this embodiment include approximate entropy, permutation entropy, and sample entropy.

[0046] The formula for calculating approximate entropy is as follows:

[0047] in, is the approximate entropy value; m is the embedding dimension, i.e., the number of consecutive sampling points contained in each pattern vector; r is the similarity tolerance threshold, used to determine whether two pattern vectors are similar; N is the time series length; This represents the m-dimensional pattern vector constructed starting from the i-th sampling point; This represents the (m+1)-dimensional pattern vector constructed starting from the i-th sampling point; i is the index of the reference pattern vector; j is the index of the pattern vector being compared with it. This represents the Chebyshev distance between two pattern vectors; This indicates the number of elements that satisfy the conditions within the parentheses.

[0048] Given a time series of length N To calculate its permutation entropy, we first need to reconstruct the phase space to obtain a set of m-dimensional vectors, the expression of which is as follows:

[0049] in, Let be the i-th reconstructed m-dimensional phase space vector; Let be the value of the time series at the i-th sampling point; m be the embedding dimension; τ be the time delay parameter; N be the length of the time series; and i be the starting index of the reconstructed vector, whose value ranges from 1 to 1. .

[0050] Each permutation pattern in all reconstructed vectors The frequency of occurrence is used to calculate the empirical probability. The formula for calculating the empirical probability is as follows:

[0051] in, This represents the k-th permutation pattern; Arrangement mode The empirical probability of occurrence in all reconstructed vectors; Indicates the arrangement pattern Number of times it appears; K represents the total number of reconstructed vectors; K represents the actual number of different permutations, satisfying K≤m!; m! represents the total number of possible permutations of an m-dimensional vector.

[0052] Based on the probability distribution of the permutation patterns calculated above, the permutation entropy... The Shannon entropy, defined as the discrete probability distribution, is calculated as follows:

[0053] in, τ is the permutation entropy value; m is the embedding dimension; τ is the time delay parameter; K is the number of permutation patterns; Let be the empirical probability of the k-th permutation pattern.

[0054] In the important field of neuroscience, Alzheimer's disease research, permutation entropy plays a unique and important role, providing an effective tool for quantifying the decrease in the complexity of electroencephalogram (EEG) signals.

[0055] The sample entropy is defined as the negative natural logarithm of the ratio of the m+1 dimension to the average probability of the m dimension, as shown in the formula:

[0056] in: , is the average probability in m+1 dimensions This reflects the overall similarity of the m+1 dimensional vectors; , is the m-dimensional average probability This reflects the overall similarity of m-dimensional vectors; the larger the value, the more regular the sequence. is the sample entropy value; m is the embedding dimension; r is the similarity tolerance threshold; N is the time series length; Let m+1 dimensional pattern vectors be the average probability that they satisfy the similarity condition. Let A be the average probability that m-dimensional pattern vectors satisfy the similarity condition; i m (r) represents the proportion that satisfies the similarity condition when the i-th (m+1)-dimensional vector is used as a reference; B i m (r) represents the proportion of similarity conditions when the i-th m-dimensional vector is used as a reference; ln represents the natural logarithm.

[0057] In addition to the entropy features mentioned above, this embodiment also extracts Lempel-Ziv complexity, whose core purpose is to quantify the randomness and structure of time series. Its theoretical basis is algorithm complexity theory, and its core idea is to approximate the incomputable Kolmogorov complexity by evaluating the minimum amount of information required to generate a given sequence. For the preprocessed time series... First, the time series is binarized and preprocessed. A common approach is a thresholding method based on the mean. The normalized complexity value is calculated using the following formula:

[0058] Where LZC is the normalized Lempel-Ziv complexity value; c(N) is the original Lempel-Ziv complexity corresponding to a binarized sequence of length N, i.e., the number of new patterns generated during sequence scanning; b(N) is a normalization factor used to eliminate the influence of sequence length on the complexity value; N is the length of the binarized time series; if using If s is the size of the symbol set of the binary sequence, then s is usually 2 in binary sequences.

[0059] Typically, the EEG signals of Alzheimer's disease (AD) patients exhibit higher regularity and predictability (increased structurality and decreased randomness), which is directly reflected in a significant decrease in the LZC value of the EEG sequence (while the EEG signals of cognitively normal elderly people are more random and have higher LZC values).

[0060] (3) Hjorth parameter and statistical characteristics of EEG Hjorth parameters are a set of physiological signal characteristic descriptors based on time-domain analysis. Their core purpose is to characterize the amplitude characteristics, rate of change, and nonlinear structure of electroencephalogram (EEG) signals. This parameter set provides a concise and effective time-domain index for quantifying the dynamic characteristics of neural electrical activity, primarily comprising three core parameters: activity, mobility, and complexity. Hjorth activity is essentially the variance of the signal, used to characterize the total power or amplitude fluctuation range of the signal. The calculation formula is as follows:

[0061] in: Original signal The overall variance For signal The arithmetic mean, This represents the total number of signal sampling points. The Hjorth mobility calculation formula is as follows: where Activity is the Hjorth activity parameter; Original signal The variance; Let be the signal amplitude at the i-th sampling point; is the arithmetic mean of the signal; N is the total number of signal sampling points; i is the sampling point index.

[0062]

[0063] in: Original signal First derivative variance This represents the variance of the original signal. Where Mobility is the Hjorth mobility parameter. The first derivative of the original signal The variance; For signal The first difference or first derivative at the i-th sampling point; Original signal The variance. This value measures how quickly a signal changes over time.

[0064] Statistical characteristics include skewness and kurtosis.

[0065] Skewness is a statistic that measures the asymmetry of a probability distribution, characterizing the degree and direction to which the distribution shape deviates from a symmetrical normal distribution. In EEG studies of Alzheimer's disease, skewness can serve as a time-domain feature, revealing changes in the probability distribution shape of EEG signal amplitudes. This invention calculates the skewness of unbiased samples, using the following formula:

[0066] in: The arithmetic mean of all samples. This represents the skewness value of the unbiased sample. The sample skewness coefficient before correction; This is the arithmetic mean of all samples; Let be the value of the i-th sample; N is the number of samples; This is a small sample bias correction factor, used to reduce the bias in skewness estimation when the sample size is small.

[0067] The formula for calculating kurtosis is as follows:

[0068] in: The mean of the signal. This is the fourth power of the signal standard deviation (i.e., the square of the variance). Kurtosis is the kurtosis characteristic value. Let be the signal amplitude at the i-th sampling point; The mean of the signal; The standard deviation of the signal is the fourth power; N is the total number of signal sampling points; in the formula... 3 is used to convert normal kurtosis to excess kurtosis, so that the kurtosis baseline of the normal distribution is 0.

[0069] Furthermore, Willison amplitude can quantify the frequency or activity of transient changes in a signal, that is, by detecting "rapid fluctuation events" in the signal, it reflects the severity of dynamic changes in the signal. Willison amplitude It is expressed as follows:

[0070] Where WAMP is the Willisson amplitude characteristic value; N is the number of sample points in the time series; n is the sampling point index in the time series; x[n] is the signal amplitude of the nth sampling point; x[n+1] is the signal amplitude of the (n+1)th sampling point; δ(·) is an indicator function, which takes the value 1 when the condition in parentheses is true, and 0 otherwise; d is the absolute difference threshold, which is the mean of the absolute difference sequence in this paper. ; This indicates the magnitude change between two adjacent sampling points.

[0071] (4) Wavelet packet energy approximation Wavelet packet energy approximation is a function used to analyze the frequency domain characteristics of time series signals (especially suitable for EEG signals). Its core function is to calculate the percentage of energy of the approximation coefficients (low-frequency components) after one layer of wavelet packet decomposition relative to the total energy. The final extracted percentage of approximate energy is:

[0072] Wherein, WPEA is the wavelet packet approximate energy percentage; The approximation coefficient represents the proportion of energy to the total energy. The k-th coefficient in the approximate coefficient sequence obtained from the first-level wavelet packet decomposition; The k-th coefficient in the detail coefficient sequence obtained from the first-level wavelet packet decomposition; The energy is the approximation coefficient A1; denoted as the energy of detail coefficient D1; k is the wavelet coefficient index, which directly reflects the relative intensity of low-frequency components in the signal: the higher the value, the more concentrated the signal energy is in the low-frequency band; the lower the value, the higher the proportion of high-frequency components.

[0073] Similarly, there is the wavelet packet approximation and the detail energy ratio, with the following formula:

[0074] in, This is the ratio of the energy of the approximation coefficients to the energy of the detail coefficients; The energy of the approximate coefficients obtained from the first-level wavelet packet decomposition; This represents the energy of the detail coefficients obtained from the first-level wavelet packet decomposition. The larger this value, the stronger the low-frequency approximation components of the signal relative to the high-frequency detail components.

[0075] The wavelet energy approximation formula is as follows:

[0076] in, denoted as the percentage of wavelet approximation energy; N is the sequence length corresponding to the wavelet coefficients; These are the approximate coefficients obtained from the first-level wavelet decomposition. denoted as the detail coefficients obtained from the first-level wavelet decomposition; k is the wavelet coefficient index; the numerator represents the energy of the approximation coefficients, and the denominator represents the sum of the energy of the approximation coefficients and the energy of the detail coefficients.

[0077] (5) Other characteristics For EEG signals, other features were also extracted, including Hjorth complexity and correlation dimension. Through spectral analysis, the frequency corresponding to the maximum energy component in the output signal spectrum was determined. The Hjorth complexity is defined as:

[0078] Among them, amplitude , amplitude spectrum The index position of the maximum value. Extracted for this article Complexity-related feature values; f[k] is the frequency corresponding to the frequency index where the maximum value of the amplitude spectrum is located; f[k] is the actual frequency corresponding to the k-th frequency index; amplitude spectrum The index position corresponding to the maximum value; X[k] is the amplitude at the k-th frequency index; X[k] is the k-th complex spectral coefficient after the Fourier transform of the signal; M is the Fourier transform length or the number of spectral sampling points.

[0079] In Alzheimer's disease research, the correlation dimension is an extremely important nonlinear feature used to assess the complexity of the brain's dynamic system; its changes directly reflect the functional decline of neural networks caused by AD. The correlation dimension is defined as:

[0080] in: This represents the total number of points after the phase space is reconstructed. This is a Heaviside step function (returns 1 if the condition is met, otherwise returns 0), used to count pairs of points with a distance less than r; For two points in phase space and The distance (usually Euclidean distance or Chebyshev distance); is the neighborhood radius. Let be the correlation integral when the embedding dimension is m and the neighborhood radius is r; m is the embedding dimension of the phase space reconstruction; r is the neighborhood radius. This represents the total number of points after the phase space is reconstructed. and Let i and j be the i-th and j-th reconstruction points in the phase space, respectively; i and j are the indices of the reconstruction points. Θ is the distance between two points, usually Euclidean distance or Chebyshev distance; Θ(·) is the Heaviside step function, which takes the value 1 when the condition in parentheses is met, and 0 otherwise.

[0081] Furthermore, for structural magnetic resonance imaging features, this invention uses voxel-based morphological structural information (VBM-MRI) as the input for sMRI. Based on the AAL brain region template, the whole brain is divided into 116 ROIs, and the corresponding region measurement values ​​are extracted for each ROI using MarsBaR, thereby constructing a 116-dimensional sMRI structural representation for each subject for subsequent classification and multimodal fusion learning.

[0082] For plasma protein biomarkers, this invention selected plasma protein indicators closely related to the pathological mechanism of Alzheimer's disease (AD). These indicators showed significant differences between AD patients and normal individuals, specifically as follows: PLGF (placental growth factor) levels in AD patients are often higher than in normal individuals, and it is associated with neuroinflammation and vascular dysfunction; ApoE (apolipoprotein E), especially ApoE... Carriers of the 4 alleles have abnormal plasma ApoE levels compared to the general population, and this abnormality is significantly associated with the risk of Alzheimer's disease (AD). TTR (transthyretin) levels in AD patients are often lower than in the general population; this reduction may be related to ApoE levels. It is associated with decreased clearance capacity; the expression level of ApoA-II (apolipoprotein A-II) is commonly abnormal in AD patients compared to the normal population, and this abnormality is associated with lipid metabolism disorders and inflammatory responses; the level of PYY (peptide YY) may be altered in AD patients compared to the normal population, and this alteration is related to appetite regulation and neuroprotective mechanisms; elevated levels of SGOT (aspartate aminotransferase) are seen in some AD patients, while this abnormality is usually not observed in the normal population, and its elevation may reflect liver dysfunction or a systemic inflammatory state; A1Micro (A1 microglobulin) acts as... As biomarkers of inflammation and kidney function, their levels differ between AD patients and healthy individuals, a change potentially related to systemic inflammatory responses. BNP (brain natriuretic peptide) levels may be higher in AD patients than in healthy individuals, reflecting cardiovascular stress or neuroendocrine imbalances. Eotaxin-3 (eosinophil chemokine 3) levels are commonly elevated in AD patients compared to healthy individuals, associated with neuroinflammation and immune activation. IGM (immunoglobulin M) levels may be abnormal in AD patients compared to healthy individuals, suggesting an autoimmune or chronic inflammatory state. These plasma protein characteristics not only provide complementary information on the pathophysiological processes of AD at the molecular level but also form a multi-dimensional, cross-scale biomarker system with functional indicators such as EEG and sMRI, collectively enhancing the accuracy and robustness of early AD classification.

[0083] S3, Construction of pseudo-pairs with consistent labels and weighted random sampling strategy.

[0084] Each sample carries both category label information and modality identity information during the data preparation phase. The system uses this information to construct "same-label pseudo-pairing" and organize training batches. Specifically, the three modalities—EEG, structural magnetic resonance imaging (sMRI), and plasma protein—are each assigned a unique modality identifier: Modality 1 corresponds to EEG feature data, Modality 2 to sMRI feature data, and Modality 3 to plasma protein feature data. This identifier is consistently used during the training and inference phases, ensuring that the identity of each modality sample is clear and the path is traceable throughout the pseudo-pairing, fusion, and alignment processes.

[0085] Regarding data sampling strategies, this invention addresses the common problems of inconsistent sample sizes and class imbalance in multimodal data by designing a refined weighted sampling process. First, the dataset for each modality is independently stratified to ensure that the class ratios of the training, validation, and test sets are as consistent as possible with the original data, thus avoiding additional distribution bias introduced during the partitioning process. Then, during the training phase, a weighted random sampler is introduced for each modality: lower-frequency classes are assigned higher sampling weights, allowing the model to see minority class samples more fully during training, thereby mitigating the performance degradation caused by class bias. Sampling with replacement is permitted during the sampling process to ensure more comprehensive and stable class coverage within each training cycle.

[0086] In a single training iteration, the system retrieves one batch of samples from each of the three modalities in parallel. Within each batch, pseudo-pairing is performed based on label consistency: EEG samples, sMRI samples, and plasma protein samples of the same category are grouped into a "pseudo-pairing unit." Subsequently, multiple pseudo-pairing units are concatenated to form a multimodal training batch. This strategy enables the model to simultaneously learn complementary information from the three heterogeneous modalities with each parameter update and complete cross-modal semantic alignment and fusion learning within a unified shared representation space.

[0087] By employing the combined strategy of “label-consistent pseudo-pairing + weighted random sampling”, this invention achieves stable training on unpaired multimodal data without relying on real subject-level alignment. This ensures semantic consistency across modal training units and effectively mitigates the adverse effects of class imbalance and differences in modal sample size on the model optimization process, thereby improving the robustness and generalization performance of the multimodal fusion classification system.

[0088] S4 is the process of building a multimodal fusion model. Please refer to [link / reference]. Figure 4 The multimodal fusion model includes: an encoder (i.e., a probabilistic feature extractor, with quantitative features as input), a latent variable representation module, an adaptive weighted fusion module, a lightweight channel gating module, and a multi-classification head. The data processing procedure of the multimodal fusion model is as follows: 1. Modality-specific coding and Gaussian latent variable representation After the pseudo-pairing is constructed, a modality-specific encoder is used to extract hidden representations for each modality, and these representations are further parameterized into a diagonal Gaussian posterior distribution. Specifically, independent feature extraction networks are set up for the three modalities of EEG, sMRI, and plasma proteins to adapt to the statistical distribution and feature representation of different modalities. Each encoder outputs a set of hidden features, which are then passed through two lightweight linear heads to obtain the mean vector and log-variance vector, thus obtaining the probability representation of that modality in the latent space.

[0089] For the m-th modal input, Its latent variable posterior writing:

[0090] The above probabilistic representation enables the model to explicitly characterize the representation uncertainty of each sample and each modality, providing a foundation for subsequent adaptive fusion and robust alignment. Here, m is the modality index; in this paper, m=1,2,3 correspond to EEG, sMRI, and plasma protein modalities, respectively. The input features are for the m-th modality; Let m be the latent variable representation of the m-th mode in the latent space; Given input Latent variables The approximate posterior distribution of ; N(·) represents the Gaussian distribution; This is the latent variable mean vector output by the m-th modal encoder; Let be the standard deviation vector of the latent variables output by the m-th modal encoder; ) represents the variance vector The resulting diagonal covariance matrix.

[0091] 2. Adaptive weighted fusion and channel gating based on uncertainty perception To avoid the problem of dominant modality in unpaired training, this application uses the variance of the latent variable's posterior as a measure of sample-level uncertainty: the larger the variance, the less reliable the representation of that modality on the current sample. During training and inference, this is achieved using a modality availability mask. (1 for available modalities, 0 for missing ones) Dynamically weight each modality to obtain an adaptive modal weight for each sample, and recalibrate the mean features before participating in the fusion. Weights The calculation uses a masked softmax form:

[0092] in, This represents the prediction variance (uncertainty) of the mode. Wherein, The adaptive fusion weights are for the m-th modality; This is the availability mask for the m-th modality, when that modality is available. =1, missing =0; The prediction variance or uncertainty estimate for the m-th mode can be obtained from... Therefore, softman(-log ); Modal indexes participating in normalization calculations; Let m' be the availability mask for the m-th mode; Let be the prediction variance of the m′-th mode; exp(·) is the exponential function; ⊙ represents element-wise multiplication; Let be the latent variable mean vector of the m-th mode; this formula gives smaller weights to modes with greater uncertainty, and prevents missing modes from participating in the fusion.

[0093] Taking a three-modal model as an example, the weighted modal representations are stacked according to modal dimensions to form a fusion input tensor. Then, a lightweight channel gating module is introduced to perform fine-grained noise suppression and renormalization on the fusion input tensor, thereby further suppressing unreliable modal cues and strengthening complementary information, ultimately obtaining the fusion representation. The result is fed into the fusion classifier and outputs the prediction. It is a multimodal fusion feature representation after processing by adaptive modal weighting and channel gating modules.

[0094] S5. The multimodal fusion model is trained using a comprehensive loss function. The comprehensive loss includes cross-entropy classification loss, single-modal supervision loss, and regularization loss. The regularization loss includes probability alignment loss and self-information synchronization loss.

[0095] Pseudo-pairing allows completely unpaired multimodal data to form cross-modal combinations with the same label, but residual cross-modal mismatches may still exist. To address this, this application introduces two types of regularization constraints based on probabilistic representation: (1) Probabilistic Alignment: For any two modalities' Gaussian posterior distributions, the Bhattacharyya distance is used to measure the distribution difference, and a contrast structure of "positive for alignment, negative for the rest" is constructed within a mini-batch. This encourages the alignment of cross-modal distributions of pseudo-pairing positions within the latent space, suppressing representation drift caused by mismatch noise. The alignment loss can be written in symmetric InfoNCE form:

[0096] in, Let be the probability alignment loss between the m-th mode and the n-th mode; m and n are the indices of two different modes; B is the number of samples in a mini-batch; i is the position index of the positive sample pair in the batch; j is the position index of the candidate sample in the batch; The similarity score between the i-th sample in the m-th modality and the j-th sample in the n-th modality can be obtained by the inverse of the Bhattacharyya distance or its monotonic transformation. represents the similarity of positive samples at the same-label pseudo-pairing positions; τ is a temperature coefficient used to adjust the smoothness of the contrastive learning distribution; exp(·) is an exponential function; and log is a logarithmic function. This loss, through bidirectional contrastive constraints, makes pseudo-paired samples with the same label closer together in the latent space, while unpaired samples are further apart.

[0097] (2) Self-information synchronization: Within the same modality, two latent variable representations are obtained by reparameterizing the same Gaussian posterior twice and used as positive sample pairs; the sampled representations of different samples in the batch are used as negative sample pairs, thereby constraining random sampling under the same posterior to maintain consistency, reducing randomness, and stabilizing training. The SIS loss also adopts the InfoNCE structure:

[0098] in, B represents the self-information synchronization loss; B represents the number of samples in the mini-batch. and This represents two latent variable representations obtained from the same Gaussian posterior distribution of the same sample through reparameterized sampling, which constitute a positive sample pair; sim(·,·) represents the latent variable representation of the j-th sample in the batch; sim(·,·) represents the similarity function between the two latent variable representations, which can be cosine similarity or dot product similarity. is the temperature coefficient in the self-information synchronization loss; i and j are the sample indices within the batch; exp(·) is the exponential function.

[0099] The final regularization term consists of cross-modal alignment and intra-modal synchronization:

[0100] in, This represents the overall regularization loss; The cross-modal probability alignment loss can be derived from the probability of all mode pairs. The summation or average is obtained; This is the intramodal self-information synchronization loss. This regularization term is used to simultaneously enhance cross-modal semantic consistency and the stability of sampled representations within the same modality.

[0101] The model training objectives of this application include: fused classification supervision, single-modal auxiliary supervision, and the aforementioned regularization terms. Specifically, in addition to the fused classifier, a single-modal auxiliary classification head is configured for each modality to apply single-modal supervision to the representations of each modality, thereby enabling each modality encoder to learn discriminative representations and mitigating instability in fused training.

[0102] In addition, to further suppress the "dominant mode effect", this application introduces lightweight gradient modulation in the backpropagation stage: when the single-modal loss of a certain mode is larger (indicating that the mode is more difficult or underlearned), the encoder gradient is moderately amplified to make multimodal training more balanced and alignment and fusion more stable (without changing the loss form, only adjusting the gradient magnitude).

[0103] Writing the comprehensive loss function:

[0104] in, This is the overall loss function for model training; Output the corresponding cross-entropy classification loss for the fusion classifier; The single-modal supervision loss is the corresponding to each single-modal auxiliary classification head; The regularization loss consists of probability alignment loss and self-information synchronization loss; These are the weighting coefficients for the single-modal auxiliary supervision loss; represents the weighting coefficients for the regularization loss. Through this objective function, the model simultaneously optimizes fusion classification performance, single-modal discriminative ability, and cross-modal latent space alignment.

[0105] To support missing modal inputs commonly encountered in real-world clinical scenarios, a random modality masking strategy is employed during the training phase: an availability mask is randomly generated for each sample. Only use The modalities involved in fusion and prediction enable the model to cover different input combinations such as "single-modal / bimodal / trimodal" during training. This represents the availability status of the three modalities. This indicates that the m-th mode is available. 0 indicates that the m-th modality is missing. The inference phase directly uses adaptive weighted fusion based on the actual available modalities to output the results, thereby achieving robust inference for the case of missing modalities.

[0106] S6. The trained multimodal fusion model is used to predict the multidimensional features corresponding to the pseudo-pairing units with the same label, so as to obtain the category recognition results of the multimodal data.

[0107] S7, based on the performance evaluation of classifier output, robustness verification of missing modalities, and characterization interpretation analysis.

[0108] After completing model training and inference, this step evaluates the model performance in a unified manner based on the output of the fusion classifier (and the single-modal auxiliary classifier head). This is used to quantitatively verify the discriminative ability of this method in the Alzheimer's disease classification task, its robustness under the missing modality condition, and to provide interpretable evidence by combining the visualization of the shared representation space and the feature attribution results.

[0109] This embodiment uses five-fold cross-validation for evaluation: in each fold, the training / validation / testing division is completed first. During the training phase, only the training set is used to construct pseudo-pairs and learn model parameters. The test set does not participate in pseudo-pair generation or any statistical estimation throughout the process to avoid leakage of training-test information. After obtaining the test results for each fold, the mean and standard deviation are summarized and reported.

[0110] The basic evaluation metrics in this embodiment directly reflect the model's ability to distinguish sample categories, including accuracy, precision, recall, F1 score, and area under the ROC curve (AUC). It is applicable to binary classification and scalable multi-class classification scenarios.

[0111] For binary classification tasks, we define true positive (TP), true negative (TN), false positive (FP), and false negative (FN), and calculate the accuracy as follows: Precision calculation is Recall rate is calculated as follows: The F1 score is calculated as follows: AUC is used to measure the overall discriminative power of a classifier at different thresholds, where the true positive rate is the most significant factor. False positive rate AUC is defined as the area under the ROC curve, and can be written as: In multi-class scenarios, macro-average or weighted average can be used to summarize Precision, Recall and F1 to obtain an overall performance metric.

[0112] To verify the applicability of the model in the real clinical scenario of "incomplete examination items", this embodiment further evaluates the robustness of missing modalities: during the training phase, a random modality masking mechanism is used to cover different available modal combinations such as single modality, dual modality and trimodality. During the inference phase, adaptive weighted fusion is performed based on the actual available modalities and the prediction results are output. Accuracy / AUC / F1 under different modal combinations are reported to measure the stability and performance degradation of the model when missing modal inputs.

[0113] Where Accuracy is the percentage of correctly classified samples out of the total number of samples; Precision is the percentage of true positives among samples predicted as positive by the model; Recall is the percentage of true positives correctly identified by the model; F1 is the harmonic mean of precision and recall; TP is the number of true positives, i.e., the number of samples that are both true positives and predicted as positive; TN is the number of true negatives, i.e., the number of samples that are both true negatives and predicted as negative; FP is the number of false positives, i.e., the number of samples that are true negatives but predicted as positive; FN is the number of false negatives, i.e., the number of samples that are true positives but predicted as negative; TPR is the true positive rate, equivalent to recall; FPR is the false positive rate, i.e., the percentage of true negatives that are incorrectly predicted as positive; AUC is the area under the ROC curve, TPR(FPR) represents the true positive rate function with FPR as the horizontal axis, and the integration interval [0,1] represents the area under the ROC curve for all classification thresholds. In addition, ensemble gradients are used to attribute the importance of input features, the importance distribution of EEG output channels / feature dimensions, and the ranking of the contribution of image ROI feature outputs, thereby providing interpretable evidence for the model's decision-making basis and indirectly verifying the consistency between the discriminative biomarkers learned by the model and the task objectives.

[0114] This application proposes an uncertainty-aware adaptive weighted multimodal fusion network, aiming to achieve adaptive alignment and collaborative discrimination of cross-modal features. This provides a feasible method to improve the accuracy, robustness, and interpretability of early AD classification, and offers a more practical and flexible multimodal auxiliary diagnostic tool for clinical use.

[0115] Furthermore, this application addresses the residual cross-modal mismatch problem that may be introduced by pseudo-pairing by probabilistic alignment constraints and self-information synchronization regularization: by using distributional similarity measurement and contrastive learning objectives, pseudo-paired samples with the same label are aligned in the latent space, while sampling consistency is constrained within a single modality to stabilize representation learning, thereby significantly reducing the negative impact of pseudo-pairing noise on training and improving training stability. This invention supports missing modality reasoning: through a random modality masking mechanism during the training phase, the model can output diagnostic results even with only one modality / two modalities / three modalities available, adapting to the common situation in clinical practice where examination items are incomplete, and significantly improving the practicality and robustness of the system. This invention can combine attribution methods such as integrated gradients to output an explanation of the importance of brain region ROI and EEG features, which can provide some interpretable evidence for assisted diagnosis and improve the auditability and credibility of the model in medical applications.

[0116] This disclosure also provides a multimodal medical data category recognition system, including: The data preprocessing module is used to receive and preprocess multimodal data from EEG, sMRI, and plasma proteins. The feature extraction module is used to extract multi-dimensional features of the preprocessed multimodal data. The multi-dimensional features include the spectral features, time-domain features, entropy and complexity features, wavelet time-frequency features, Hjorth parameters and statistical features of EEG, gray matter volume measurement and structured feature vector of sMRI, and plasma protein molecular features. The pseudo-pairing unit construction module is used to construct multiple pseudo-pairing units with the same label based on the category label information and modality identifier of multi-dimensional features, and to independently divide each modality into training set, validation set and test set in a hierarchical manner; The weighted sampling module is used to introduce a weighted random sampler for each modality during the model training phase, assign higher sampling weights to categories that appear less frequently, and concatenate multiple pseudo-paired units with the same label to form a multimodal training batch. A multimodal fusion model construction module is used to construct a multimodal fusion model, which includes: an encoder, a latent variable representation module, an adaptive weighted fusion module, a lightweight channel gating module, and a multi-classification head; The encoder extracts the latent representation of each modality in the pseudo-pairing units with the same label, and processes it through two lightweight linear heads to obtain the mean vector and variance vector. The latent variable representation module constructs a diagonal Gaussian posterior distribution on the mean vector and variance vector to obtain the Gaussian distribution representation of the latent space of each modality. The adaptive weighted fusion module adaptively and dynamically weights each modality according to the modality availability mask, and recalibrates the mean vector using the weights to obtain the weighted representation of each modality. The weighted representations of each modality are stacked according to the modality dimension to form a fusion input tensor. The lightweight channel gating module assigns attention weights to the fusion input tensor and performs feature reshaping to obtain the fusion representation. The multi-class head predicts and identifies the fusion representation to obtain the category recognition result. The training module is used to train the multimodal fusion model; The prediction and recognition module is used to predict the multi-dimensional features corresponding to pseudo-pairing units with the same label using the trained multi-modal fusion model, so as to obtain the category recognition results of multi-modal data.

[0117] Please refer to Figure 3The core model framework of this invention comprises three modality-specific encoding networks, respectively processing EEG, sMRI, and plasma protein features. Each modality feature is mapped to a latent space by its corresponding encoder, where probability alignment and regularization are performed. Simultaneously, the fused features from all modalities are input into a unified classifier for decision-making, supervised by a weighted cross-entropy loss.

[0118] Regarding the system in the above embodiments, the specific ways in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0119] It should be noted that although several modules of the system for executing actions are mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules for embodiment. Components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0120] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

[0121] It should be noted that the installation of image acquisition and personal identification equipment in public places involved in this application is necessary for maintaining public safety, complies with relevant national regulations, and is accompanied by prominent warning signs. The collected personal images and identification information can only be used for the purpose of maintaining public safety and not for other purposes; or the images, personal identification data, etc. in this application are all legally and compliantly obtained or collected with the individual's separate consent.

Claims

1. A method for class identification of multicenter unpaired multimodal medical data, characterized in that, include: S1 receives multimodal data from EEG, sMRI, and plasma proteins, and performs preprocessing. S2, extract multi-dimensional features from the preprocessed multimodal data. The multi-dimensional features include the spectral features, time-domain features, entropy and complexity features, wavelet time-frequency features, Hjorth parameters and statistical features of EEG, gray matter volume measurement and structured feature vector of sMRI, and plasma protein molecular features. S3: Based on the category label information and modality identifier of multi-dimensional features, construct multiple pseudo-pairing units with the same label, and independently divide each modality into training set, validation set and test set. Pre-set during the model training stage, introduce a weighted random sampler for each modality, assign higher sampling weight to categories with lower frequency of occurrence, and concatenate multiple pseudo-pairing units with the same label to form a multimodal training batch. S4. Construct a multimodal fusion model, which includes: an encoder, a latent variable representation module, an adaptive weighted fusion module, a lightweight channel gating module, and a multi-classification head; The encoder extracts the latent representation of each modality in the pseudo-pairing units with the same label, and processes it through two lightweight linear heads to obtain the mean vector and variance vector. The latent variable representation module constructs a diagonal Gaussian posterior distribution on the mean vector and variance vector to obtain the Gaussian distribution representation of the latent space of each modality. The adaptive weighted fusion module adaptively and dynamically weights each modality according to the modality availability mask, and recalibrates the mean vector using the weights to obtain the weighted representation of each modality. The weighted representations of each modality are stacked according to the modality dimension to form a fusion input tensor. The lightweight channel gating module assigns attention weights to the fusion input tensor and performs feature reshaping to obtain the fusion representation. The multi-class head predicts and identifies the fusion representation to obtain the category recognition result. S5 is used to train the multimodal fusion model. S6. The trained multimodal fusion model is used to predict the multidimensional features corresponding to the pseudo-pairing units with the same label, so as to obtain the category recognition results of the multimodal data.

2. The method for classifying multi-center unpaired multimodal medical data according to claim 1, characterized in that, In S1, preprocessing includes: bandpass filtering, noise removal, artifact removal, rereference, interpolation repair, channel alignment, sample slicing, and normalization.

3. The method for classifying multi-center unpaired multimodal medical data according to claim 1, characterized in that, In S2, the plasma protein molecular characteristics include PLGF, ApoE, TTR, ApoA-II, PYY, SGOT, AlMicro, BNP, Eotaxin-3, and IGM.

4. The method for classifying multi-center unpaired multimodal medical data according to claim 1, characterized in that, In S5, a comprehensive loss function is used to train the multimodal fusion model. The comprehensive loss includes cross-entropy classification loss, single-modal supervision loss, and regularization loss. The regularization loss includes probability alignment loss and self-information synchronization loss.

5. The method for classifying multi-center unpaired multimodal medical data according to claim 1, characterized in that, The multi-classification head includes a fusion classifier and a single-modality auxiliary classification head. The fusion classifier is used to predict and identify the fused representation, and outputs the predicted category probability and the final classification label. The single-modality auxiliary classification head is used to predict the mean vector to obtain the single-modality independent classification prediction value.

6. The method for classifying multi-center unpaired multimodal medical data according to claim 5, characterized in that, The identification method further includes: S7 uses a five-fold cross-validation method to evaluate the model performance.

7. A multimodal medical data category recognition system, characterized in that, include: The data preprocessing module is used to receive and preprocess multimodal data from EEG, sMRI, and plasma proteins. The feature extraction module is used to extract multi-dimensional features of the preprocessed multimodal data. The multi-dimensional features include the spectral features, time-domain features, entropy and complexity features, wavelet time-frequency features, Hjorth parameters and statistical features of EEG, gray matter volume measurement and structured feature vector of sMRI, and plasma protein molecular features. The pseudo-pairing unit construction module is used to construct multiple pseudo-pairing units with the same label based on the category label information and modality identifier of multi-dimensional features, and to independently divide each modality into training set, validation set and test set in a hierarchical manner; The weighted sampling module is used to introduce a weighted random sampler for each modality during the model training phase, assign higher sampling weights to categories that appear less frequently, and concatenate multiple pseudo-paired units with the same label to form a multimodal training batch. A multimodal fusion model construction module is used to construct a multimodal fusion model, which includes: an encoder, a latent variable representation module, an adaptive weighted fusion module, a lightweight channel gating module, and a multi-classification head; The encoder extracts the latent representation of each modality in the pseudo-pairing units with the same label, and processes it through two lightweight linear heads to obtain the mean vector and variance vector. The latent variable representation module constructs a diagonal Gaussian posterior distribution on the mean vector and variance vector to obtain the Gaussian distribution representation of the latent space of each modality. The adaptive weighted fusion module adaptively and dynamically weights each modality according to the modality availability mask, and recalibrates the mean vector using the weights to obtain the weighted representation of each modality. The weighted representations of each modality are stacked according to the modality dimension to form a fusion input tensor. The lightweight channel gating module assigns attention weights to the fusion input tensor and performs feature reshaping to obtain the fusion representation. The multi-class head predicts and identifies the fusion representation to obtain the category recognition result. The training module is used to train the multimodal fusion model; The prediction and recognition module is used to predict the multi-dimensional features corresponding to pseudo-pairing units with the same label using the trained multi-modal fusion model, so as to obtain the category recognition results of multi-modal data.

Images