Biomedical data analysis method based on adaptive multi-modal data fusion

Abstract

The invention discloses a biomedical data analysis method based on adaptive multi-modal data fusion, and belongs to the technical field of multi-modal feature data processing. The method comprises the following steps: acquiring multi-modal data, and preprocessing the multi-modal data; extracting the preprocessed multi-modal data by adopting a modal specific neural network model; performing primary fusion on each extracted modal feature to generate a primary fusion feature; introducing an adaptive fusion module based on an attention mechanism to obtain final fusion features; and performing classification prediction on the final fusion features by using an FCNN model to complete classification of the multi-modal data. According to the invention, through dual mechanisms of primary average fusion and adaptive fusion, high-precision identification is maintained under the condition of multi-modal data missing, and performance reduction caused by incomplete data is avoided; meanwhile, the modal weight is dynamically adjusted through an attention mechanism, the fusion process can be optimized according to the quality and integrity of each modal data, and the feature expression ability is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal feature data processing, and in particular relates to a biomedical data analysis method based on adaptive multimodal data fusion. Background Art

[0002] With the rapid development of artificial intelligence and medical image processing technologies, the application of multimodal data in biomedical data recognition is gaining increasing attention. Biomedical data recognition methods that rely on single-modality data often struggle to accurately capture the progression of disease. To this end, researchers are attempting to combine multimodal data, including imaging data (such as MRI), genetic data (such as SNPs), and clinical data (such as cognitive tests and demographic information), to improve diagnostic accuracy. At the data level, neuroimaging projects have provided rich multimodal datasets, driving the development of multimodal biomedical data recognition.

[0003] In the existing technology, multimodal fusion methods mainly include the following categories: (1) Feature splicing-based methods, which directly splice the features of different modalities and input them into the deep learning model for classification; (2) Modality completion-based methods, which use generative models to synthesize data of missing modalities; (3) Attention mechanism-based methods, which achieve feature fusion by learning the relationship between modalities. For example, some studies have proposed using attention mechanisms to capture the interaction between modalities, thereby improving the representation ability of multimodal data. However, these methods still face significant challenges in practical applications. In particular, since the collection of clinical data is limited by factors such as equipment, cost or patient compliance, some patients may lack data of certain modalities (such as missing MRI scans or genetic test results), resulting in data imbalance between modalities. This imbalance not only reduces the generalization ability of the model, but also may make the model overly dependent on a certain modality, thereby affecting the accuracy and robustness of diagnosis.

[0004] Furthermore, while existing attention-based fusion methods can alleviate the problem of missing modalities to some extent, they often lack the ability to adaptively adjust for differences in contributions between modalities. For example, when data from a particular modality is missing or of low quality, the model struggles to dynamically adjust the weights of each modality, resulting in a decrease in the expressive power of the fused features.

[0005] Therefore, how to effectively deal with the problem of data imbalance between modalities in multimodal feature data recognition, especially the adaptive fusion of modality missing data sets, is the technical problem that the present invention aims to solve. Summary of the Invention

[0006] The purpose of the present invention is to provide a biomedical data analysis method based on adaptive multimodal data fusion to solve the problems raised in the above background technology.

[0007] The object of the present invention is achieved by: a biomedical data analysis method based on adaptive multimodal data fusion, characterized in that the method comprises the following steps:

[0008] Step S1: Acquire multimodal data and preprocess the multimodal data;

[0009] Multimodal data include imaging modality data, genetic modality data, and clinical modality data;

[0010] Step S2: extracting the preprocessed multimodal data using a modality-specific neural network model;

[0011] Step S3: Perform primary fusion on the extracted modal features to generate preliminary fusion features;

[0012] Step S4: Introduce an adaptive fusion module based on the attention mechanism to obtain the final fusion features;

[0013] Step S5: Use the FCNN model to classify and predict the final fusion features to complete the classification of multimodal data.

[0014] Preferably, the multimodal data is preprocessed in step S1, specifically:

[0015] The preprocessing of the imaging modality data is as follows: brain extraction is performed on the input imaging modality data, the MNI template is aligned, the central three-dimensional slice is extracted, and the data is normalized to the [0,1] interval;

[0016] The preprocessing of gene modality data is as follows: input 15,000-dimensional gene modality features, screen key features using the random forest algorithm, normalize to the [0,1] interval, and fill in missing values;

[0017] The clinical modality data were preprocessed as follows: 29-dimensional clinical features were input, quantitative features were normalized to [0, 1], categorical features were one-hot encoded, and missing values ​​were filled with the mean.

[0018] Preferably, the modality-specific neural network model in step S2 includes a residual convolutional neural network (CNN), a GeneFCNN network, and a Transformer-enhanced fully connected network. The residual convolutional neural network (CNN) includes three layers of residual blocks, each layer of residual blocks is followed by a multi-scale spatial-channel attention module (CBAM). Each layer of residual blocks consists of two 3×3×3 convolutional layers, ReLU activation, and batch normalization. The residual connection transfers features across layers.

[0019] The multi-scale spatial-channel attention module CBAM includes a 1×1×1 convolution kernel, a 3×3×3 convolution kernel, a 5×5×5 convolution kernel, a global average pooling and a self-attention layer;

[0020] The multi-scale spatial-channel attention module (CBAM) uses 1×1×1, 3×3×3, and 5×5×5 convolution kernels to dynamically adjust feature map weights, employs global average pooling and self-attention layers, and outputs high-dimensional feature maps.

[0021] The GeneFCNN network consists of three fully connected layers, each of which uses ReLU activation, Dropout and batch normalization to output a low-dimensional feature vector;

[0022] The Transformer-enhanced fully connected network adopts a three-layer structure. The first layer is the embedding layer, and the second and third layers use Transformer encoders. Each layer contains 4-head multi-head self-attention, a feedforward network, a residual connection and layer normalization, and outputs a feature vector through the fully connected layer.

[0023] Preferably, in step S2, a modality-specific neural network model is used to extract the preprocessed multimodal data, specifically:

[0024] Step S2-1: The first residual block in the three-layer residual block passes through two 3×3×3 convolutional layers, ReLU activation and batch normalization, residual connection to add input features, and output feature map 32×72×72×72;

[0025] Multi-scale spatial-channel attention module (CBAM): applies 1×1×1, 3×3×3, and 5×5×5 convolution kernels to the feature map to generate spatial attention weights:

[0026] Spatial Attention=σ(Conv 1×1×1 (F)+Conv 3×3×3 (F)+Conv 5×5×5 (F));

[0027] Channel attention is generated through global average pooling and a two-layer fully connected network, and the weights are multiplied element-wise with the feature map;

[0028] The second and third residual blocks in the three-layer residual block repeat convolution, activation, normalization and CBAM operations, gradually downsample (stride 2), and the feature map size is reduced to 36×36×36;

[0029] Global processing: Global average pooling reduces the dimensionality of the feature map, and the self-attention layer calculates the global dependency between features. The output feature map has 32 channels and a spatial size of 36×36×36.

[0030] Step S2-2: The first fully connected layer in the three fully connected layers inputs 15,000-dimensional features and reduces them to 1024 dimensions through linear transformation. The formula is:

[0031] h1=ReLU(W1·x+b1);

[0032] Apply Dropout and Batch Normalization;

[0033] The second fully connected layer in the three-layer fully connected layer: 1024-dimensional features are reduced to 512 dimensions, and ReLU, Dropout and batch normalization are repeated;

[0034] The third fully connected layer in the three-layer fully connected layer: 512-dimensional features are reduced to 256 dimensions and the feature vector is output;

[0035] Step S2-3: Embedding layer: Quantitative features are mapped to 64 dimensions through a linear layer, and categorical features are mapped to 32 dimensions to generate a unified embedding vector;

[0036] The second layer's Transformer encoder: The embedding vector is input into a 4-head multi-head self-attention module, with a hidden layer dimension of 128. The calculation formula is:

[0037] MultiHead(Q,K,V)=Concat(head1,…,head4)·W O ;

[0038] where, head i =Attention(QW i Q ,KW i K ,VW i V );

[0039] Followed by a feed-forward network, residual connections and layer normalization;

[0040] The third layer's Transformer encoder repeats self-attention and feedforward operations to enhance feature interaction modeling.

[0041] Fully connected layer: Transformer output is mapped to 32 dimensions through a linear layer and outputs a feature vector.

[0042] Preferably, in step S3, the extracted modal features are initially fused, and the mean of the modal features is calculated using an average fusion strategy, specifically:

[0043] Step S3-1: Perform usability testing on multimodal data, specifically:

[0044] For imaging data, the number of MRI slices was tested to see if it met the preset threshold. For genetic data, the number of single nucleotide polymorphism (SNP) features was tested to see if it reached approximately 15,000 dimensions. For clinical data, the number of missing values ​​in the 29-dimensional features was tested.

[0045] Mark the modes that meet the conditions as available modes, and record the number of available modes as N m , 1<=N m <=3;

[0046] Step S3-2: Average the features of the available modalities to generate primary fusion features

[0047]

[0048] Among them, F m Represents the characteristics of the mth state; when N m = 1, the average fusion feature directly takes the feature F of the only available modality m , ensuring the robustness of the fusion process to modality loss;

[0049] Step S3-3: Pass the primary fusion features to the adaptive fusion module.

[0050] Preferably, the adaptive fusion module includes an attention weight generation unit: for each available modality, the primary fusion feature and the modality feature are concatenated, and an initial attention weight is generated through a convolution layer and a Sigmoid function;

[0051] Weight normalization unit: The initial attention weights are normalized by Softmax to ensure that the sum of the weights is 1;

[0052] Weighted fusion unit: Use normalized weights to perform weighted summation on modal features to generate adaptive fusion features.

[0053] Preferably, in step S4, an adaptive fusion module based on the attention mechanism is introduced to obtain the final fusion features, specifically:

[0054] Step S4-1: Attention weight calculation:

[0055] For each mode m, the modal feature F m With primary fusion features After concatenation, the initial attention weights are generated through the convolutional layer:

[0056]

[0057] Among them, F m is a modality-specific convolutional layer, σ is the Sigmoid function; θ m As its parameters, [F; Fm] represents feature splicing;

[0058] Step S4-2: Perform Softmax normalization on the attention weights of all available modalities to ensure that the sum of the weights is 1:

[0059]

[0060] Among them, W m represents the initial attention weight of the m-th modality; N m Indicates the number of modes; represents the normalized attention weight;

[0061] Step S4-3: Use normalized weights to perform weighted summation on each modality feature to generate the final fusion feature.

[0062] in, Represents element-wise multiplication.

[0063] Preferably, the FCNN model includes a multi-layer fully connected network, a regularization and stabilization unit, and an output layer, and the multi-layer fully connected network includes three fully connected layers, each layer adopts a ReLU activation function;

[0064] The regularization and stabilization unit introduces the Dropout mechanism after each fully connected layer and applies batch normalization to stabilize the training process;

[0065] The output layer converts the final layer output into a probability distribution through the Softmax function for classification prediction.

[0066] Preferably, in step S5, the FCNN model is used to perform classification prediction on the final fusion features, specifically:

[0067] Step S5-1: Fusion features Input the fully connected layer, map it to the classification space, and generate predicted logits, specifically:

[0068] The first layer maps the fused features to 128 dimensions: h1 = ReLU (W1·F fusion +b1); where W1∈R (128×256) ;

[0069] The second layer maps the 128-dimensional features to 64 dimensions: h2 = ReLU(W2·h1+b2); where W2∈R (64×128) ;

[0070] The third layer maps the 64-dimensional features to 3 dimensions: z = W3*h2+b3; where W3∈R (3×64) ;

[0071] Step S5-2: Use the Softmax function to convert logits into probability distribution and output three types of analysis results:

[0072]

[0073] Among them, z i is the logits of the i-th category; z j is the logits of the j-th category.

[0074] Compared with the prior art, the present invention has the following improvements and advantages:

[0075] 1. Through the dual mechanisms of primary average fusion and adaptive fusion, high-precision recognition can be maintained even in the absence of multimodal data, avoiding performance degradation caused by incomplete data. At the same time, the attention mechanism dynamically adjusts the modal weights, optimizing the fusion process based on the quality and completeness of each modal data, significantly improving feature expression capabilities.

[0076] 2. By designing a modality-specific neural network model, it supports any number of modal inputs and has higher versatility and practicality compared to the method of training independent models for each modality combination. At the same time, by capturing the interaction information between modalities, it improves the ability to distinguish between MCI and AD, providing more reliable support for early identification and intervention of clinical data. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 Schematic diagram of the process of the present invention.

[0078] Figure 2 Schematic diagram of the structure of the modality-specific neural network model of the present invention.

[0079] Figure 3 Schematic diagram of the feature fusion process. DETAILED DESCRIPTION

[0080] The present invention is further summarized below with reference to the accompanying drawings.

[0081] like Figure 1 As shown, a biomedical data analysis method based on adaptive multimodal data fusion includes the following steps:

[0082] Step S1: Acquire multimodal data and preprocess the multimodal data, specifically:

[0083] Multimodal data include imaging modality data, genetic modality data, and clinical modality data;

[0084] In the method of the present invention, based on the ADNI dataset of the Neuroimaging Initiative, imaging data (MRI), genetic data (SNPs) and clinical data are used to demonstrate how to deal with inter-modality data imbalance and achieve high-precision recognition;

[0085] Using data from the ADNI database, we selected patient data from ADNI1, ADNI2, and ADNI GO stages, including the following modalities:

[0086] Imaging data (MRI): Baseline MRI scans of 551 patients were extracted. Central three-dimensional slices (transverse, coronal, and sagittal) were obtained for each patient and normalized to 72 × 72 × 72 pixels, totaling 3674 MRI samples.

[0087] Genetic data (SNPs): extracted from the whole genome sequencing data of 805 patients, and approximately 15,000 key SNPs were retained after preprocessing.

[0088] Clinical data: 29 features were extracted from neurological examinations, cognitive assessments, and demographic information of 2384 patients.

[0089] To simulate intermodal data imbalance, 239 patients with trimodal data were selected as the complete dataset. Genetic data from 50% of the patients and MRI data from 30% of the patients were randomly removed to construct an incomplete dataset. Diagnostic labels included healthy controls (CN, 165 patients), mild cognitive impairment (MCI, 39 patients), and Alzheimer's disease (AD, 35 patients), based on the latest MRI and clinical assessments.

[0090] The imaging modality data were preprocessed as follows: T1-weighted MRI scans were downloaded from the ADNI in the original format of NifTI. The input imaging modality data were subjected to brain extraction, aligned to the MNI152 standard template, and the central three-dimensional slice (transverse axis x = 36, coronal axis y = 36, sagittal axis z = 36) was extracted and resized to a resolution of 72 × 72 × 72. The pixel values ​​were normalized to [0, 1] using the normalization formula:

[0091] The preprocessing of gene modality data is as follows: input 15,000-dimensional gene modality features, screen key features using the random forest algorithm, normalize to the [0,1] interval, and fill in missing values;

[0092] The original VCF file contained approximately 3 million SNPs. Quality control was performed using PLINK (SNPs with a missingness rate >5% were removed, and SNPs with a minor allele frequency >0.01 were retained). This yielded approximately 547,863 SNPs. To reduce dimensionality, a random forest classifier (100 trees and a maximum depth of 10) was trained on the training set (excluding the test set). The top 15,000 SNPs were retained based on feature importance. The encoding scheme was 0 (no allele), 1 (1 allele), and 2 (2 alleles), generating a sparse matrix (805 × 15,000).

[0093] The clinical modality data were preprocessed as follows: 29-dimensional clinical features were input, quantitative features were normalized to [0, 1], categorical features were one-hot encoded, and missing values ​​were filled with the mean;

[0094] Extract 29 features from the ADNI CSV file, including quantitative features (such as memory test scores Mini-Mental State Examination, ranging from 0 to 30) and categorical features (such as gender, male / female);

[0095] Use one-hot encoding for categorical features (gender is converted to [0,1] or [1,0]), and normalize quantitative features to [-1,1]. The normalization formula is: Features directly related to AD were removed to ensure that the model did not rely on explicit identification information.

[0096] Step S2: Use a modality-specific neural network model to extract the preprocessed multimodal data, specifically:

[0097] like Figure 2 As shown in the figure, the modality-specific neural network model includes a residual convolutional neural network (CNN), a GeneFCNN network, and a Transformer-enhanced fully connected network. The residual convolutional neural network (CNN) contains three layers of residual blocks. Each layer of residual blocks is followed by a multi-scale spatial-channel attention module (CBAM). Each layer of residual blocks consists of two 3×3×3 convolutional layers, ReLU activation, and batch normalization. The residual connection transfers features across layers.

[0098] The three-layer residual block includes the first layer residual block, the second layer residual block and the third layer residual block. The first layer residual block has a 3×3×3 convolution kernel (stride 1, padding 1), ReLU activation followed by maximum pooling (2×2×2, stride 2), and the output size is 36×36×36;

[0099] The second layer of residual block has a 3×3×3 convolution kernel and repeats the same operation;

[0100] The third layer residual block has a 3×3×3 convolution kernel, no pooling, and outputs a feature map F img ∈R 36×36×36 ;

[0101] The multi-scale spatial-channel attention module (CBAM) includes 1×1×1 convolution kernel, 3×3×3 convolution kernel, 5×5×5 convolution kernel, global average pooling and self-attention layer.

[0102] The multi-scale spatial-channel attention module (CBAM) uses 1×1×1, 3×3×3, and 5×5×5 convolution kernels to dynamically adjust feature map weights, employs global average pooling and self-attention layers, and outputs high-dimensional feature maps.

[0103] The GeneFCNN network consists of three fully connected layers, each of which uses ReLU activation, Dropout, and batch normalization to output a low-dimensional feature vector;

[0104] The three fully connected layers include the first fully connected layer, the second fully connected layer, and the third fully connected layer. The first fully connected layer reduces the input dimension from 15,000 to 1,024 (the weight matrix size is 15000×1024, and the bias vector is 1024). It uses the ReLU activation function and Dropout (dropout probability 0.3) to prevent overfitting.

[0105] The second fully connected layer reduces the dimension from 1024 to 512;

[0106] The third fully connected layer is reduced from 512 dimensions to 256 dimensions, and the output F gen ∈R 256 ;

[0107] A batch normalization (BN) layer is added after each fully connected layer to stabilize the training process, and the weight initialization uses the Xavier method.

[0108] The Transformer-enhanced fully connected network adopts a three-layer structure. The first layer is the embedding layer, and the second and third layers use Transformer encoders. Each layer contains 4-head multi-head self-attention, feedforward network, residual connection and layer normalization, and outputs the feature vector through the fully connected layer.

[0109] Step S2-1: The first residual block in the three-layer residual block passes through two 3×3×3 convolutional layers, ReLU activation and batch normalization, residual connection to add input features, and output feature map 32×72×72×72;

[0110] Multi-scale spatial-channel attention module (CBAM): applies 1×1×1, 3×3×3, and 5×5×5 convolution kernels to the feature map to generate spatial attention weights:

[0111] Spatial Attention=σ(Conv 1×1×1 (F)+Conv 3×3×3 (F)+Conv 5×5×5 (F));

[0112] Among them, F is the input feature map, σ is the Sigmoid function, Conv k×k×k Represents a convolution operation with a kernel size of k×k×k;

[0113] Channel attention is generated through global average pooling and a two-layer fully connected network, and the weights are multiplied element-by-element with the feature map. The channel attention weight calculation formula is:

[0114] A c =σ(MLP(GAP(F)));

[0115] GAP is global average pooling, and MLP is a two-layer fully connected network (the first layer has a dimension of 32, and the second layer has a dimension of 16). Global average pooling and self-attention are added to the last layer of the network to capture the global dependencies of the feature map. The self-attention calculation formula is:

[0116]

[0117] Among them, Q, K, V are query, key, and value matrices respectively; d k =32 is the dimension of the key;

[0118] The second and third residual blocks in the three-layer residual block repeat convolution, activation, normalization and CBAM operations, gradually downsample (stride 2), and the feature map size is reduced to 36×36×36;

[0119] Global processing: Global average pooling reduces the dimensionality of the feature map, and the self-attention layer calculates the global dependency between features. The output feature map has 32 channels and a spatial size of 36×36×36.

[0120] Step S2-2: The first fully connected layer in the three fully connected layers inputs 15,000-dimensional features and reduces them to 1024 dimensions through linear transformation. The formula is:

[0121] h1=ReLU(W1·x+b1);

[0122] Apply Dropout and Batch Normalization;

[0123] The second fully connected layer in the three-layer fully connected layer: 1024-dimensional features are reduced to 512 dimensions, and ReLU, Dropout and batch normalization are repeated;

[0124] The third fully connected layer in the three-layer fully connected layer: 512-dimensional features are reduced to 256 dimensions and the feature vector is output;

[0125] Step S2-3: Embedding layer: Quantitative features are mapped to 64 dimensions through a linear layer, and categorical features are mapped to 32 dimensions to generate a unified embedding vector;

[0126] The second layer's Transformer encoder: The embedding vector is input into a 4-head multi-head self-attention module, with a hidden layer dimension of 128. The calculation formula is:

[0127] MultiHead(Q,K,V)=Concat(head1,…,head4)·W O ;

[0128] where, head i =Attention(QW i Q ,KW i K ,VW i V );

[0129] Followed by a feed-forward network, residual connections and layer normalization;

[0130] The third layer's Transformer encoder repeats self-attention and feedforward operations to enhance feature interaction modeling.

[0131] Fully connected layer: Transformer output is mapped to 32 dimensions through a linear layer and outputs a feature vector.

[0132] In step S3, the extracted modal features are initially fused, and the mean of the modal features is calculated using the average fusion strategy, specifically:

[0133] Step S3-1: Perform usability testing on multimodal data, specifically:

[0134] For imaging data, the number of MRI slices was tested to see if it met the preset threshold. For genetic data, the number of single nucleotide polymorphism (SNP) features was tested to see if it reached approximately 15,000 dimensions. For clinical data, the number of missing values ​​in the 29-dimensional features was tested.

[0135] Mark the modes that meet the conditions as available modes, and record the number of available modes as N m , 1<=N m <=3;

[0136] Step S3-2: Average the features of the available modalities to generate primary fusion features

[0137]

[0138] Among them, F m Represents the characteristics of the mth state; when N m = 1, the average fusion feature directly takes the feature F of the only available modality m , ensuring the robustness of the fusion process to modality loss;

[0139] Step S3-3: Pass the primary fusion features to the adaptive fusion module.

[0140] In step S4, an adaptive fusion module based on the attention mechanism is introduced to obtain the final fusion features, specifically:

[0141] The adaptive fusion module includes an attention weight generation unit: for each available modality, it concatenates the primary fusion features and modality features, and generates the initial attention weights through the convolution layer and the Sigmoid function;

[0142] Weight normalization unit: The initial attention weights are normalized by Softmax to ensure that the sum of the weights is 1;

[0143] Weighted fusion unit: Use normalized weights to perform weighted summation on modal features to generate adaptive fusion features.

[0144] Step S4-1: Attention weight calculation:

[0145] For each mode m, the modal feature F m With primary fusion features After concatenation, the initial attention weights are generated through the convolutional layer:

[0146]

[0147] Among them, F m is a modality-specific convolutional layer, σ is the Sigmoid function; θ m As its parameters, [F; Fm] represents feature splicing;

[0148] Step S4-2: Perform Softmax normalization on the attention weights of all available modalities to ensure that the sum of the weights is 1:

[0149]

[0150] Among them, W m represents the initial attention weight of the m-th modality; N m Indicates the number of modes; represents the normalized attention weight;

[0151] Step S4-3: Use normalized weights to perform weighted summation on each modality feature to generate the final fusion feature.

[0152]

[0153] in, Represents element-wise multiplication.

[0154] In step S5, the FCNN model is used to classify and predict the final fusion features, specifically:

[0155] The FCNN model consists of a multi-layer fully connected network, a regularization and stabilization unit, and an output layer. The multi-layer fully connected network contains three fully connected layers, each of which uses the ReLU activation function.

[0156] The regularization and stabilization unit introduces the Dropout mechanism after each fully connected layer and applies batch normalization to stabilize the training process;

[0157] The output layer converts the final layer output into a probability distribution through the Softmax function for classification prediction.

[0158] Step S5-1: Fusion features Input the fully connected layer, map it to the classification space, and generate predicted logits, specifically:

[0159] The first layer maps the fused features to 128 dimensions: h1 = ReLU (W1·F fusion +b1); where W1∈R (128×256) ;

[0160] The second layer maps the 128-dimensional features to 64 dimensions: h2 = ReLU(W2·h1+b2); where W2∈R (64×128) ;

[0161] The third layer maps the 64-dimensional features to 3 dimensions: z = W3*h2+b3; where W3∈R (3×64) ;

[0162] Step S5-2: Use the Softmax function to convert logits into probability distribution and output three types of analysis results:

[0163]

[0164] Among them, z i is the logits of the i-th category; z j is the logits of the j-th category.

[0165] The modality-specific neural network model was trained using the cross-entropy loss function, the optimizer was Adam, and the learning rate was set to 0.0001.

[0166] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A biomedical data analysis method based on adaptive multimodal data fusion, characterized by: The method comprises the following steps: Step S1: Acquire multimodal data and preprocess the multimodal data; Multimodal data include imaging modality data, genetic modality data, and clinical modality data; Step S2: extracting the preprocessed multimodal data using a modality-specific neural network model; Step S3: Perform primary fusion on the extracted modal features to generate preliminary fusion features; Step S4: Introduce an adaptive fusion module based on the attention mechanism to obtain the final fusion features; Step S5: Use the FCNN model to classify and predict the final fusion features to complete the classification of multimodal data.

2. The biomedical data analysis method based on adaptive multimodal data fusion according to claim 1, characterized in that: In step S1, the multimodal data is preprocessed, specifically: The preprocessing of the imaging modality data is as follows: brain extraction is performed on the input imaging modality data, the MNI template is aligned, the central three-dimensional slice is extracted, and the data is normalized to the [0,1] interval; The preprocessing of gene modality data is as follows: input 15,000-dimensional gene modality features, screen key features using the random forest algorithm, normalize to the [0,1] interval, and fill in missing values; The clinical modality data were preprocessed as follows: 29-dimensional clinical features were input, quantitative features were normalized to [0, 1], categorical features were one-hot encoded, and missing values ​​were filled with the mean.

3. The biomedical data analysis method based on adaptive multimodal data fusion according to claim 1, characterized in that: In step S2, the modality-specific neural network model includes a residual convolutional neural network (CNN), a GeneFCNN network, and a Transformer-enhanced fully connected network. The residual convolutional neural network (CNN) includes three layers of residual blocks, each of which is followed by a multi-scale spatial-channel attention module (CBAM). Each residual block consists of two 3×3×3 convolutional layers, ReLU activation, and batch normalization. The residual connection transfers features across layers. The multi-scale spatial-channel attention module CBAM includes a 1×1×1 convolution kernel, a 3×3×3 convolution kernel, a 5×5×5 convolution kernel, a global average pooling and a self-attention layer; The multi-scale spatial-channel attention module (CBAM) uses 1×1×1, 3×3×3, and 5×5×5 convolution kernels to dynamically adjust feature map weights, employs global average pooling and self-attention layers, and outputs high-dimensional feature maps. The GeneFCNN network consists of three fully connected layers, each of which uses ReLU activation, Dropout and batch normalization to output a low-dimensional feature vector; The Transformer-enhanced fully connected network adopts a three-layer structure. The first layer is the embedding layer, and the second and third layers use Transformer encoders. Each layer contains 4-head multi-head self-attention, a feedforward network, a residual connection and layer normalization, and outputs a feature vector through the fully connected layer.

4. The biomedical data analysis method based on adaptive multimodal data fusion according to claim 3, characterized in that: In step S2, a modality-specific neural network model is used to extract the pre-processed multimodal data, specifically: Step S2-1: The first residual block in the three-layer residual block passes through two 3×3×3 convolutional layers, ReLU activation and batch normalization, residual connection to add input features, and output feature map 32×72×72×72; Multi-scale spatial-channel attention module (CBAM): applies 1×1×1, 3×3×3, and 5×5×5 convolution kernels to the feature map to generate spatial attention weights: Spatial Attention=σ(Conv 1×1×1 (F)+Conv 3×3×3 (F)+Conv 5×5×5 (F)); Channel attention is generated through global average pooling and a two-layer fully connected network, and the weights are multiplied element-wise with the feature map; The second and third residual blocks in the three-layer residual block repeat convolution, activation, normalization and CBAM operations, gradually downsample (stride 2), and the feature map size is reduced to 36×36×36; Global processing: Global average pooling reduces the dimensionality of the feature map, and the self-attention layer calculates the global dependency between features. The output feature map has 32 channels and a spatial size of 36×36×36. Step S2-2: The first fully connected layer in the three fully connected layers inputs 15,000-dimensional features and reduces them to 1024 dimensions through linear transformation. The formula is: h1=ReLU(W1·x+b1); Apply Dropout and Batch Normalization; The second fully connected layer in the three-layer fully connected layer: 1024-dimensional features are reduced to 512 dimensions, and ReLU, Dropout and batch normalization are repeated; The third fully connected layer in the three-layer fully connected layer: 512-dimensional features are reduced to 256 dimensions and the feature vector is output; Step S2-3: Embedding layer: Quantitative features are mapped to 64 dimensions through a linear layer, and categorical features are mapped to 32 dimensions to generate a unified embedding vector; The second layer's Transformer encoder: The embedding vector is input into a 4-head multi-head self-attention module, with a hidden layer dimension of 128. The calculation formula is: MultiHead(Q,K,V)=Concat(head1,…,head4)·W O ; where,head i =Attention(QW i Q ,KW i K ,VW i V ); Followed by a feed-forward network, residual connections and layer normalization; The third layer's Transformer encoder repeats self-attention and feedforward operations to enhance feature interaction modeling. Fully connected layer: Transformer output is mapped to 32 dimensions through a linear layer and outputs a feature vector.

5. The biomedical data analysis method based on adaptive multimodal data fusion according to claim 1, characterized in that: In step S3, the extracted modal features are initially fused, and the mean of the modal features is calculated using an average fusion strategy, specifically: Step S3-1: Perform usability testing on multimodal data, specifically: For imaging data, the number of MRI slices was tested to see if it met the preset threshold. For genetic data, the number of single nucleotide polymorphism (SNP) features was tested to see if it reached approximately 15,000 dimensions. For clinical data, the number of missing values ​​in the 29-dimensional features was tested. Mark the modes that meet the conditions as available modes, and record the number of available modes as N m , 1<=N m <=3; Step S3-2: Average the features of the available modalities to generate primary fusion features Among them, F m Represents the characteristics of the mth state; when N m = 1, the average fusion feature directly takes the feature F of the only available modality m , ensuring the robustness of the fusion process to modality loss; Step S3-3: Pass the primary fusion features to the adaptive fusion module.

6. The biomedical data analysis method based on adaptive multimodal data fusion according to claim 1, characterized in that: The adaptive fusion module includes an attention weight generation unit: for each available modality, the primary fusion feature and the modality feature are concatenated, and the initial attention weight is generated through a convolutional layer and a Sigmoid function; Weight normalization unit: The initial attention weights are normalized by Softmax to ensure that the sum of the weights is 1; Weighted fusion unit: Use normalized weights to perform weighted summation on modal features to generate adaptive fusion features.

7. The biomedical data analysis method based on adaptive multimodal data fusion according to claim 6, characterized in that: In step S4, an adaptive fusion module based on the attention mechanism is introduced to obtain the final fusion features, specifically: Step S4-1: Attention weight calculation: For each mode m, the modal feature F m With primary fusion features After concatenation, the initial attention weights are generated through the convolutional layer: Among them, F m is a modality-specific convolutional layer, σ is the Sigmoid function; θ m As its parameters, [F; Fm] represents feature splicing; Step S4-2: Perform Softmax normalization on the attention weights of all available modalities to ensure that the sum of the weights is 1: Among them, W m represents the initial attention weight of the m-th modality; N m Indicates the number of modes; represents the normalized attention weight; Step S4-3: Use normalized weights to perform weighted summation on each modality feature to generate the final fusion feature. in, Represents element-wise multiplication.

8. The biomedical data analysis method based on adaptive multimodal data fusion according to claim 1, characterized in that: The FCNN model includes a multi-layer fully connected network, a regularization and stabilization unit, and an output layer. The multi-layer fully connected network contains three fully connected layers, each of which uses a ReLU activation function. The regularization and stabilization unit introduces the Dropout mechanism after each fully connected layer and applies batch normalization to stabilize the training process; The output layer converts the final layer output into a probability distribution through the Softmax function for classification prediction.

9. The biomedical data analysis method based on adaptive multimodal data fusion according to claim 8, characterized in that: In step S5, the FCNN model is used to classify and predict the final fusion features, specifically: Step S5-1: Fusion features Input the fully connected layer, map it to the classification space, and generate predicted logits, specifically: The first layer maps the fused features to 128 dimensions: h1 = ReLU (W1·F fusion +b1); where W1∈R (128×256) ; The second layer maps the 128-dimensional features to 64 dimensions: h2 = ReLU(W2·h1+b2); where W2∈R (64×128) ; The third layer maps the 64-dimensional features to 3 dimensions: z = W3*h2+b3; where W3∈R (3×64) ; Step S5-2: Use the Softmax function to convert logits into probability distribution and output three types of analysis results: Among them, z i is the logits of the i-th category; z j is the logits of the j-th category.

Images