Multi-modal data fusion processing method and system based on deep learning

Abstract

The invention relates to the technical field of intelligent medical image auxiliary diagnosis, and discloses a multi-modal data fusion processing method and system based on deep learning, and the method comprises the steps: obtaining multi-modal medical data, carrying out the preprocessing of the multi-modal medical data, carrying out the feature extraction, and recording the quality meta-information; constructing positive and negative sample pairs, mapping the positive and negative sample pairs to a unified semantic space, and performing parameter optimization and soft alignment processing; generating a multi-scale time code, calculating a time sequence association weight matrix, and carrying out time sequence weighted fusion; quality evaluation is carried out, a preliminary quality score is calculated, and misjudgment is corrected through a clinical rule base to obtain a self-adaptive fusion weight; judging modal integrity and determining a fusion strategy, calculating a cross-modal attention interaction weight and carrying out weighted fusion; predicting disease categories, calculating the contribution degree of each modal feature and generating a multi-modal visualization result; according to the method, the problems of cross-modal semantic alignment, sequential relation modeling and data quality difference are solved.

Description

Technical Field

[0001] This invention relates to the field of intelligent medical image-assisted diagnosis technology, and more specifically, to a multimodal data fusion processing method and system based on deep learning. Background Technology

[0002] In the field of intelligent medical image-assisted diagnosis, doctors need to comprehensively analyze various medical data from patients to make accurate disease diagnoses. This data includes heterogeneous information from multiple sources, such as CT images, MRI images, electronic medical records, electrocardiogram signals, and blood test data, originating from different medical devices and examination methods. However, these data exhibit significant differences in format, semantic expression, acquisition time, and data quality, posing a substantial challenge to data fusion and processing.

[0003] Existing multimodal data fusion methods mainly suffer from the following problems: different modalities of data reside in their own independent feature spaces, with significant differences in semantic expression. Simple feature concatenation or weighted averaging cannot effectively capture cross-modal semantic relationships; medical data collection often spans different time points, and existing methods lack in-depth modeling of temporal relationships, failing to fully utilize dynamic information during disease progression; medical data from different sources vary in quality, and existing methods typically use fixed weights for fusion, failing to dynamically adjust the fusion strategy based on data quality, resulting in low-quality data introducing noise that affects diagnostic accuracy.

[0004] Furthermore, in actual clinical applications, due to factors such as examination costs, time constraints, or patient conditions, certain modal data are frequently missing. Existing methods exhibit poor robustness in scenarios with missing modalities, resulting in a significant decrease in diagnostic performance. Simultaneously, existing methods lack interpretability, failing to provide physicians with clear diagnostic evidence and contribution analysis of each modality's data, thus limiting their widespread application in clinical practice. Summary of the Invention

[0005] This invention provides a multimodal data fusion processing method and system based on deep learning, which solves the technical problems of large differences in semantic expression, lack of deep modeling of temporal relationships, and inconsistent quality of medical data from different sources in related technologies.

[0006] This invention provides a deep learning-based multimodal data fusion processing method, comprising:

[0007] Acquire multimodal medical data and preprocess it; extract features from the preprocessed data and record quality metadata to obtain a standardized feature vector set.

[0008] Positive and negative sample pairs are constructed based on a standardized feature vector set, mapped to a unified semantic space, and then parameter optimization and soft alignment are performed to obtain a semantically aligned feature vector set.

[0009] Multi-scale temporal codes are generated based on semantically aligned feature vector sets. Temporal correlation weight matrices are calculated and temporal weighted fusion is performed to obtain temporally enhanced feature vector sets.

[0010] Based on the temporal enhanced feature vector set and quality meta-information, quality assessment is performed and a preliminary quality score is calculated. Adaptive fusion weights are obtained by correcting misjudgments through a clinical rule base.

[0011] Based on the temporal enhancement feature vector set and adaptive fusion weights, modal integrity is judged and fusion strategy is determined. Cross-modal attention interaction weights are calculated and weighted fusion is performed to obtain fused feature vectors.

[0012] Disease categories are predicted based on fused feature vectors, the contribution of each modality feature is calculated and multimodal visualization results are generated to obtain a structured diagnostic report.

[0013] In a preferred embodiment, obtaining the standardized feature vector set includes:

[0014] Identify the format of multimodal medical data and output the format type identifier. Multimodal medical data includes medical image data, electronic medical record text data, physiological signal data and laboratory test data. Input the format feature vector into the format classifier for identification. When the confidence level is lower than the first threshold, the manual confirmation process is initiated.

[0015] Based on the format type identifier, the corresponding processing flow is retrieved from the preset standardization rule base for data standardization preprocessing;

[0016] Feature vectors for each modality are extracted using a deep network. For medical image modalities, a convolutional neural network is used as the feature extractor, and a global average pooling layer is added at the end of the network to obtain image feature vectors.

[0017] Record the quality metadata of each modality of data. For image data, calculate the signal-to-noise ratio as a quality indicator; for text data, evaluate the integrity score; for signal data, count the artifact ratio; and for inspection data, record the number of missing items.

[0018] In a preferred embodiment, obtaining the semantically aligned feature vector set includes:

[0019] Combine the feature vectors of any two different modalities of the same patient into a positive sample pair;

[0020] Calculate the cosine similarity between the feature vector and the corresponding modal feature vectors of other patients in the dataset, select the number of feature vectors with the highest similarity as candidate negative samples, and randomly sample feature vectors from the candidate negative samples and combine them with the feature vectors of another modality to form a negative sample pair;

[0021] For each modality, a projection network is designed. The projection network contains two fully connected layers. The feature vectors of each modality are input into the corresponding projection network and mapped to a unified semantic space to obtain the projected feature vectors.

[0022] The InfoNCE loss function is used to optimize the projection network parameters, maximizing the similarity of projected features for positive sample pairs and minimizing the similarity for negative sample pairs.

[0023] Set a target similarity threshold; alignment is considered successful when the similarity of positive samples is higher than the target similarity threshold.

[0024] In a preferred embodiment, the obtained temporal enhancement feature vector set includes:

[0025] For any two modal data, calculate the acquisition time difference, decompose the time difference into five time scales: hour, day, week, month, and grade. For each time scale, use the sinusoidal position coding method to generate a time embedding vector, and concatenate them to obtain a multi-scale time embedding vector.

[0026] The temporal association weights between modalities are calculated. The temporal association weights consist of temporal proximity and feature similarity. Temporal proximity is calculated using a temporal attention network based on multi-scale temporal embedding vectors, and feature similarity is calculated using cosine similarity based on semantically aligned feature vector sets.

[0027] The temporal context features are obtained by weighting and summing the features of other modalities using temporal correlation weights. The original feature vector is then concatenated with the temporal context features and input into the feature fusion network to obtain the temporal enhanced feature vector.

[0028] In a preferred embodiment, obtaining the adaptive fusion weights includes:

[0029] The quality of each modality of data is evaluated from four dimensions: data integrity, signal-to-noise ratio, consistency and reliability. The scores of the four dimensions are weighted and averaged to obtain a preliminary quality score.

[0030] The quality score is corrected by using a clinical rule base. The rule base contains condition-action pairs, feature patterns of condition description data, and adjustment methods for action description quality scores. The rule base is traversed to check whether the rule conditions are met. If they are met, the corresponding action is executed to adjust the quality score.

[0031] The softmax function is used to normalize the corrected quality score to generate adaptive fusion weights.

[0032] In a preferred embodiment, obtaining the fused feature vector includes:

[0033] The number of modalities is counted based on the modal missing indicator vector. If all modalities have been collected, the complete modal fusion strategy is adopted. If there are missing modalities but the number of collected modalities is greater than or equal to the preset threshold, the modal missing fusion strategy is adopted, and the adaptive fusion weights of the collected modalities are renormalized.

[0034] The temporal enhancement feature vector of each modality is mapped to a unified interaction space through a linear transformation. The dot product of the feature vectors after the transformation of the two modalities is calculated and normalized to cross-modal attention interaction weights through the softmax function.

[0035] The temporal augmentation feature vector is weighted and summed using cross-modal attention interaction weights to obtain the interaction feature. The temporal augmentation feature vector and the interaction feature are concatenated, and the initial fusion feature vector is obtained by weighted summing using adaptive fusion weights. The final fusion feature vector is obtained by nonlinear transformation through a fully connected network.

[0036] In a preferred embodiment, the structured diagnostic report includes:

[0037] The fused feature vector is input into the disease classification network, and the probability distribution of disease categories is calculated through forward propagation. The category with the highest probability is taken as the predicted disease category.

[0038] Calculate the gradient of the predicted probability of the disease category with respect to the fused feature vector, backpropagate the gradient to each modality feature, and calculate the sum of the absolute values ​​of the gradients of all dimensions of each modality feature vector as the contribution score.

[0039] For image modalities, feature maps of the convolutional layers of the image feature extraction network are extracted, the gradient of the predicted disease category with respect to the feature map is calculated, the gradient is used to weight the feature map to obtain the class activation map, and the class activation map is upsampled to generate a class activation heatmap.

[0040] A structured diagnostic report is generated based on predicted disease categories, contribution scores, and multimodal visualization results.

[0041] In a preferred embodiment, the temporal correlation weight is calculated as follows:

[0042] The temporal association weight is obtained by weighted average of temporal proximity and feature similarity. Temporal proximity is calculated by using a temporal attention network to obtain an attention score based on multi-scale temporal embedding vectors. Feature similarity is calculated by using cosine similarity on semantically aligned feature vector sets and then linearly mapped to the range of zero to one.

[0043] In a preferred embodiment, the conditional action pairs in the clinical rule base include:

[0044] When the signal-to-noise ratio of image data is lower than the second threshold and the text data integrity score is higher than the third threshold, the image modal quality score is reduced and the text modal quality score is increased.

[0045] When the number of missing data items exceeds the fourth threshold, the data quality score will be set to the preset minimum value.

[0046] This invention provides a deep learning-based multimodal data fusion processing system for executing the aforementioned deep learning-based multimodal data fusion processing method, comprising:

[0047] The data preprocessing module is used to acquire multimodal medical data and preprocess it, extract features from the preprocessed data and record quality metadata to obtain a standardized feature vector set.

[0048] The semantic alignment module constructs positive and negative sample pairs based on a standardized feature vector set, maps them to a unified semantic space, and performs parameter optimization and soft alignment processing to obtain a semantically aligned feature vector set.

[0049] The temporal modeling module generates multi-scale temporal codes based on semantically aligned feature vector sets, calculates temporal correlation weight matrices and performs temporal weighted fusion to obtain temporal enhanced feature vector sets.

[0050] The quality assessment module performs quality assessment and calculates a preliminary quality score based on the temporal enhanced feature vector set and quality meta-information. It then obtains adaptive fusion weights by correcting misjudgments through a clinical rule base.

[0051] The feature fusion module, based on the temporal enhanced feature vector set and adaptive fusion weights, determines modal integrity and fusion strategy, calculates cross-modal attention interaction weights and performs weighted fusion to obtain the fused feature vector;

[0052] The diagnostic output module predicts disease categories based on fused feature vectors, calculates the contribution of each modality feature, generates multimodal visualization results, and obtains a structured diagnostic report.

[0053] The beneficial effects of this invention are as follows:

[0054] This method effectively addresses key challenges in multimodal medical data fusion by employing techniques such as cross-modal semantic alignment, temporal relationship modeling, adaptive quality assessment, and hierarchical attention fusion. It maps features from different modalities to a unified semantic space, achieving deep semantic alignment and eliminating semantic gaps between modalities. Through multi-scale temporal coding and temporal correlation weight calculation, it fully utilizes the temporal information of medical data to better reflect the dynamic changes in disease processes.

[0055] By employing multi-dimensional quality assessment and clinical rule base correction, adaptive fusion weights are dynamically generated, ensuring that high-quality modalities occupy a greater weight in the fusion process and effectively suppressing the negative impact of low-quality data. In modality-deficient scenarios, this invention compensates for information loss through a dynamic weight adjustment mechanism, guaranteeing the stability and reliability of the diagnostic system. Furthermore, this invention calculates the contribution of each modality feature through gradient backpropagation, generating multimodal visualization results such as class activation heatmaps and vocabulary importance annotations, providing doctors with clear diagnostic evidence and decision support, and enhancing the system's interpretability and clinical applicability. Attached Figure Description

[0056] Figure 1 This is a flowchart of the main process of a deep learning-based multimodal data fusion processing method of the present invention;

[0057] Figure 2 This is a detailed flowchart of a deep learning-based multimodal data fusion processing method according to the present invention;

[0058] Figure 3 This is a block diagram of a deep learning-based multimodal data fusion processing system according to the present invention. Detailed Implementation

[0059] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0060] At least one embodiment of the present invention discloses a multimodal data fusion processing method based on deep learning, such as... Figure 1 - Figure 2 As shown, it includes:

[0061] Step 1: Acquire multimodal medical data and preprocess it. Extract features from the preprocessed data and record quality metadata to obtain a standardized feature vector set.

[0062] Specifically, the following steps are included:

[0063] Step 1.1: Identify the multimodal data format and output the format type identifier;

[0064] It receives raw multimodal medical data from different medical institutions as input, including medical imaging data, electronic medical record text data, physiological signal data, and laboratory test data. At the same time, it acquires the metadata of each data, including data source, acquisition device model, acquisition time, data format identifier, etc.

[0065] For input medical image data, format features such as file header information, pixel arrangement, bit depth, and color space are extracted to construct a format feature vector. This feature vector is then input into a format classifier for recognition. This classifier, based on a decision tree algorithm, can identify common medical image formats such as DICOM, NIfTI, JPEG, and PNG. The classifier outputs a format type identifier and a confidence score. When the confidence score is lower than a first threshold, a manual verification process is initiated. For electronic medical record text data, features such as encoding format, degree of structure, and field naming conventions are analyzed to identify format types such as HL7, CDA, and plain text. For physiological signal data, parameters such as sampling rate, number of channels, and data precision are detected to identify formats such as EDF, MIT-BIH, and CSV. For laboratory test data, information such as data structure, unit labeling, and reference range is analyzed to identify types such as LOINC encoding and custom formats.

[0066] Step 1.2: Perform data standardization preprocessing according to the format type;

[0067] Based on the format type identifier identified in step 1.1, the corresponding processing flow is retrieved from the preset standardization rule base. The standardization rule base stores standardized operation sequences for different formats, including operations such as format conversion, size adjustment, and numerical normalization, as well as their parameter configurations.

[0068] For medical imaging data, if identified as DICOM format, the pixel array is extracted and converted into a standard 3D tensor representation, the spatial resolution is uniformly adjusted to 512×512 pixels, and the grayscale values ​​are normalized to the 0-1 range. If identified as NIfTI format, the volume data is read and resampled to uniformly set the voxel spacing to 1 mm × 1 mm × 1 mm. For electronic medical record text data, if it is structured HL7 format, the XML structure is parsed to extract key fields; if it is unstructured plain text, sentence segmentation, word segmentation, and medical entity recognition are performed to convert the text into a structured representation. For physiological signal data, a uniform sampling rate of 250 Hz is used, the number of channels is standardized to 12 leads, and the numerical range is normalized. For laboratory test data, test values ​​in different units are converted to international standard units, missing values ​​are filled with the median of the clinical normal value of the test item, and outliers are detected and marked using a box plot method.

[0069] Step 1.3: Extract feature vectors for each modality using a deep network;

[0070] The unified format preprocessed data obtained in step 1.2 is input into the feature extraction network of each modality, and the feature vector is calculated through forward propagation.

[0071] For the medical image modality, a convolutional neural network (CNN) is used as the feature extractor. The network contains five convolutional blocks, each consisting of a convolutional layer, a batch normalization layer, and a pooling layer. The first convolutional block contains 64 3×3 convolutional kernels, the second contains 128 3×3 convolutional kernels, the third contains 256 3×3 convolutional kernels, the fourth contains 512 3×3 convolutional kernels, and the fifth contains 512 3×3 convolutional kernels. All convolutions have a stride of 1, and the padding method is same padding to preserve the feature map size. Each convolutional block is followed by a 2×2 max-pooling layer with a stride of 2. The parameters are initialized using ResNet or DenseNet models pre-trained on ImageNet or large-scale medical image datasets, and the network is adapted to the target diagnostic task through transfer learning. The preprocessed 512×512 pixel image is input into the network. After feature extraction through 5 convolutional blocks, the feature map size is reduced to 16×16×512. A global average pooling layer is added at the end of the network to aggregate the spatial features into a one-dimensional vector, resulting in a 512-dimensional image feature vector.

[0072] For the text modality, a pre-trained language model based on the Transformer architecture is used as the feature extractor. The model, pre-trained on medical text corpora, is capable of understanding medical terminology and clinical expressions. The segmented text sequence is input into the model, and the hidden states corresponding to the special tags output by the model are taken as the feature representation of the entire text, resulting in a 768-dimensional text feature vector.

[0073] For physiological signal modalities, a temporal convolutional network was used as the feature extractor. The network contained six temporal convolutional layers with a kernel size of 3. Different dilation rates were used to capture the multi-scale temporal patterns of the signal, with dilation rates of 1, 2, 4, 8, 16, and 32, respectively. Each layer had 128 output channels. The preprocessed signal sequence with a length of 3000 sampling points was input into the network. After six temporal convolutional layers, a 3000×128 feature representation was obtained. A temporal global average pooling layer was added at the end of the network to perform average pooling on the time dimension, resulting in 128-dimensional intermediate features. These intermediate features were then mapped to a 512-dimensional signal feature vector through a fully connected layer.

[0074] For laboratory test data modalities, a three-layer fully connected network is used as the feature extractor. The detection values ​​are encoded into high-dimensional feature representations through nonlinear transformation. The network structure is as follows: the input layer receives the standardized detection value vector (the dimension is determined according to the number of detection items, commonly 20 to 50 dimensions); the first fully connected layer maps the input to 128 dimensions and uses the ReLU activation function; the second fully connected layer maps to 256 dimensions and uses the ReLU activation function; and the third fully connected layer outputs a 256-dimensional feature vector of the test data. A dropout layer with a dropout probability of 0.3 is added after the first and second layers to prevent overfitting.

[0075] Step 1.4: Record the quality metadata of each modal data;

[0076] During feature extraction, quality metadata for each modality is recorded simultaneously. For image data, the signal-to-noise ratio is calculated as a quality metric; for text data, an integrity score is evaluated; for signal data, the artifact ratio is statistically analyzed; and for inspection data, the number of missing items is recorded. This quality metadata will be used in subsequent quality assessment steps to generate adaptive fusion weights.

[0077] Through the above steps, this step outputs a standardized feature vector set and corresponding quality metadata, specifically including: a 512-dimensional image feature vector, a 768-dimensional text feature vector, a 512-dimensional signal feature vector, a 256-dimensional inspection data feature vector, and quality assessment indicators for each modality.

[0078] Step 2: Construct positive and negative sample pairs based on the standardized feature vector set, map them to a unified semantic space, and perform parameter optimization and soft alignment processing to obtain a semantically aligned feature vector set;

[0079] Specifically, the following steps are included:

[0080] Step 2.1: Construct cross-modal positive sample pairs for the same patient;

[0081] The standardized feature vector sets for each modality output from step 1 are received as input. Although these feature vectors have been standardized, they still reside in the feature space specific to their respective modalities, and semantic gaps exist between different modalities. For example, image feature vectors encode visual morphological information, while text feature vectors encode linguistic descriptive information. Their numerical distributions and semantic meanings are completely different, making direct comparison and fusion impossible.

[0082] For different modalities of data from the same patient, their feature vectors should express the same or related medical semantics, and therefore they are treated as positive sample pairs. Iterating through all patient samples in the dataset, for each patient, the feature vectors of all acquired modalities are extracted, and any two feature vectors from different modalities are combined into a positive sample pair. For example, if patient A has data in three modalities: image, text, and signal, then three positive sample pairs can be constructed: image-text, image-signal, and text-signal.

[0083] To increase the diversity of positive samples and improve the robustness of the model, data augmentation is performed on the feature vectors. Augmentation methods include: adding small-amplitude Gaussian noise to the feature vectors, with the noise standard deviation preferably one-tenth of the feature vector standard deviation; performing random dropout on the feature vectors, setting some dimensions to zero with a certain probability, preferably 0.1; and performing linear interpolation on the feature vectors, using the weighted average of the original feature vector and itself as the augmented feature, with weights preferably uniformly sampled between 0.8 and 1.2. Different augmentation operations are applied to the same modality feature vector of the same patient, generating multiple augmented versions, which also form positive sample pairs.

[0084] Step 2.2: Construct negative sample pairs using the hard negative sample strategy;

[0085] Negative sample pairs consist of feature vectors from different patients, representing semantically unrelated cross-modal data. Simple random sampling may select negative samples that are too easily distinguishable, offering limited help to model training. Therefore, a difficult negative sample sampling strategy is adopted, prioritizing negative samples that have similar features to positive samples but are semantically different.

[0086] Specifically, for each positive sample pair, the cosine similarity between one of the feature vectors and the corresponding modality feature vectors of all other patients in the dataset is calculated. The top K feature vectors with the highest similarity are selected as candidate negative samples, where K is preferably between 16 and 64, and in one embodiment, K is set to 32. A feature vector is randomly sampled from the candidate negative samples and combined with the other modality feature vector from the positive sample pair to form a negative sample pair. This sampling strategy ensures that the negative samples are close to the positive samples in the feature space, increasing the difficulty of differentiation and prompting the model to learn more refined semantic representations. To avoid the model overfitting to a specific negative sample distribution, the candidate negative sample set is dynamically updated during training. After a certain number of training rounds, the feature similarity is recalculated and the candidate set is updated.

[0087] Step 2.3: Project the features of each modality onto a unified semantic space;

[0088] Since the feature vectors of different modalities have different dimensions (image features are 512-dimensional, text features are 768-dimensional, signal features are 512-dimensional, and inspection data features are 256-dimensional), they need to be projected into a semantic space of a unified dimension.

[0089] A bidirectional mapping network is designed, with an independent projection network for each modality, mapping the feature vectors of that modality to a unified semantic space. The projection network employs a multilayer perceptron structure, containing two fully connected layers. The first layer maps the input features to a 1024-dimensional hidden space, introducing non-linearity using the ReLU activation function. The second layer maps the hidden features to a 512-dimensional semantic space without using an activation function to maintain the linear additive nature of the features. For image features, the input and output dimensions of the projection network are both 512; for text features, the input and output dimensions are both 768; for signal features, the input and output dimensions are both 512; and for inspection data features, the input and output dimensions are both 256. The feature vectors of each modality are input into the corresponding projection network, and a unified-dimensional projected feature vector is calculated through forward propagation. The projected feature vectors are then L2 normalized to a magnitude of 1 for easier subsequent similarity calculations.

[0090] Step 2.4: Optimize the projection network parameters through comparative learning;

[0091] The InfoNCE loss function of contrastive learning is used to optimize the projection network parameters, maximizing the similarity of projected features for positive sample pairs and minimizing the similarity for negative sample pairs.

[0092] For a positive sample pair, the cosine similarity of the two projected feature vectors is calculated and denoted as the positive sample similarity. For all corresponding negative sample pairs, the cosine similarity of the projected feature vectors is calculated and denoted as the negative sample similarity. The InfoNCE loss is defined as the negative logarithm of the ratio of the positive sample similarity index to the sum of the positive sample similarity index and the similarity indices of all negative samples. A temperature coefficient is introduced to scale the similarity; the preferred temperature coefficient is 0.05 to 0.1, and in one embodiment, it is 0.07. A smaller temperature coefficient makes the model more sensitive to differences in similarity. The InfoNCE loss is calculated for all positive sample pairs in the dataset, and the average value is taken as the total loss. The Adam optimizer is used to minimize the total loss. The preferred learning rate is 0.0001 to 0.001, the preferred batch size is 128 to 512, and in one embodiment, the learning rate is 0.0001, the batch size is 256, and the preferred number of training epochs is 50 to 200, and in one embodiment, it is 100 epochs. During training, the model performance is evaluated on the validation set at regular intervals, and the model parameters with the minimum loss on the validation set are selected as the final projection network parameters.

[0093] Step 2.5 introduces a soft alignment mechanism to tolerate semantic offsets;

[0094] In real-world medical data, semantic shifts can occur between different modalities of the same patient due to data noise, annotation uncertainty, and individual differences. For example, the size of lesions shown in images may differ slightly from the size described in the medical record text, and abnormal patterns of physiological signals may not be completely synchronized with imaging findings. Strict alignment constraints can lead to overfitting of the model to specific patterns in the training data, reducing generalization ability. Therefore, a soft alignment mechanism is introduced to allow for a certain degree of semantic shift between modalities.

[0095] Specifically, when calculating the similarity of positive samples, a similarity of 1 is not required. Instead, a target similarity threshold is set, and alignment is considered successful as long as the similarity of positive samples is higher than this threshold. The target similarity threshold is preferably between 0.7 and 0.9, and in one embodiment, it is set to 0.8, indicating that some differences in the projected features of positive sample pairs are allowed. In the loss function, no additional optimization pressure is applied to positive sample pairs whose similarity is already higher than the target threshold, avoiding over-alignment. Simultaneously, a lower bound threshold is set in the constraint on negative sample similarity. Distinction is considered successful as long as the similarity of negative samples is lower than this threshold. The lower bound threshold is preferably between 0.2 and 0.4, and in one embodiment, it is set to 0.3. This soft alignment mechanism allows the model to maintain a certain degree of feature diversity while ensuring basic alignment performance, improving robustness to data noise and distribution shifts. The optimized projection network from step 2.4 is applied to the modal feature vectors output from step 1 to obtain a semantic alignment feature vector set.

[0096] Step 3: Generate multi-scale temporal codes based on the semantically aligned feature vector set, calculate the temporal correlation weight matrix and perform temporal weighted fusion to obtain the temporal enhanced feature vector set;

[0097] Specifically, the following steps are included:

[0098] Step 3.1: Generate multi-scale temporal embedding vectors;

[0099] The semantic alignment feature vector set output from step 2, along with the timestamp information of each modality's data, is received as input. In clinical practice, medical data from different modalities are often collected at different points in time, with time spans ranging from hours to years. For example, a patient may undergo a CT scan upon admission, an MRI examination several days later, pathological analysis results several weeks later, and a follow-up examination several months later. These data exhibit temporal relationships, reflecting the progression of the disease and the effectiveness of treatment.

[0100] For any two modal datasets, calculate their acquisition time difference, expressed in hours. Decompose the time difference into multiple time scales, including hourly, daily, weekly, monthly, and grade-level scales.

[0101] The hourly time difference is the result of taking the original time difference modulo 24, representing the time shift within a day; the daily time difference is the result of dividing the original time difference by 24 and taking the modulo 7, representing the day shift within a week; the weekly time difference is the result of dividing the original time difference by 168 and taking the modulo 4, representing the week shift within a month; the monthly time difference is the result of dividing the original time difference by 720 and taking the modulo 12, representing the month shift within a year; and the yearly time difference is the result of dividing the original time difference by 8760, representing the year shift. For each time scale, a time embedding vector is generated using a sinusoidal position coding method.

[0102] Sine positional encoding maps time differences to a high-dimensional vector space. Even-numbered dimensions of the vector are encoded using a sine function, while odd-numbered dimensions are encoded using a cosine function, with the frequency decreasing with increasing dimension. During encoding, the time difference values ​​are converted into values ​​for each dimension of the vector using sine and cosine functions. Different dimensions employ different frequency parameters, enabling the encoding to capture temporal patterns at different scales. The embedding dimension for each time scale is preferably between 32 and 128; in one embodiment, it is set to 64. Embedding vectors from multiple time scales are concatenated to obtain a multi-scale temporal embedding vector.

[0103] Step 3.2: Calculate the temporal correlation weights between modes;

[0104] For any two modal feature vectors, calculate the temporal correlation weight between them. This weight reflects the correlation between the two modal data in the temporal dimension. The temporal correlation weight consists of two parts: temporal proximity and feature similarity.

[0105] Temporal proximity is calculated based on multi-scale temporal embedding vectors, using an attention mechanism to learn the mapping relationship between temporal embedding vectors and associated weights. A temporal attention network is designed, containing multiple fully connected layers. This network maps temporal embedding vectors to the hidden space using the tanh activation function, then maps the hidden vectors to a 1-dimensional attention score and normalizes it to between 0 and 1 using the sigmoid activation function. The attention score represents temporal proximity; the smaller the temporal difference, the higher the proximity.

[0106] Feature similarity is calculated based on semantically aligned feature vectors. Cosine similarity is used to measure the similarity between two feature vectors. The similarity value is between -1 and 1, and is linearly mapped to between 0 and 1.

[0107] The temporal association weight is a weighted average of temporal proximity and feature similarity. The weight of temporal proximity is preferably 0.5 to 0.7, and the weight of feature similarity is preferably 0.3 to 0.5. In one embodiment, the weight of temporal proximity is 0.6 and the weight of feature similarity is 0.4, indicating that temporal information plays a dominant role in association judgment.

[0108] For all modality pairs in the dataset, calculate their temporal association weights and construct a temporal association weight matrix. The rows and columns of the matrix correspond to different modal data, and the matrix elements are the temporal association weights of the corresponding modality pairs.

[0109] Step 3.3: Perform feature weighted fusion based on temporal weights;

[0110] Based on the temporal correlation weight matrix calculated in step 3.2, the features of each modality are weighted and fused. For each modality's feature vector, its temporal correlation weight with all other modality feature vectors is calculated. These weights are then used to weight and sum the features of other modalities to obtain the temporal context features of that modality. The original feature vector and the temporal context features are concatenated and input into the feature fusion network for integration. The feature fusion network contains two fully connected layers. The first layer maps the concatenated features to a 1024-dimensional hidden space using the ReLU activation function. The second layer maps the hidden features back to 512 dimensions, resulting in a temporally enhanced feature vector. The temporally enhanced features not only contain information about the modality itself but also integrate information from other temporally related modalities, enabling a better reflection of the dynamic changes in the disease.

[0111] Through the above steps, step 3 outputs the temporal enhancement feature vector set for each modality, including temporally enhanced image feature vectors, text feature vectors, signal feature vectors, and inspection data feature vectors.

[0112] Step 4: Based on the temporal enhanced feature vector set and quality metadata, perform quality assessment and calculate preliminary quality scores. Obtain adaptive fusion weights by correcting misjudgments through the clinical rule base.

[0113] Specifically, the following steps are included:

[0114] Step 4.1: Evaluate the quality of each modality of data from multiple dimensions;

[0115] The step receives the temporal enhancement feature vector set output from step 3 and the data quality metadata recorded in step 1 as input. The quality of data from different modalities directly affects the fusion effect; low-quality data may introduce noise and erroneous information, reducing diagnostic accuracy. This step dynamically generates fusion weights by evaluating the quality of each modality, giving higher weight to high-quality modalities and appropriately suppressing the influence of low-quality modalities, thereby improving the reliability of the fused features and the diagnostic accuracy.

[0116] Design a multi-dimensional quality assessment index system to evaluate data quality from four dimensions: data integrity, signal-to-noise ratio, consistency, and reliability.

[0117] Data integrity assessment checks for missing values ​​or corruption. For image data, it checks the integrity of the pixel matrix and the presence of abnormal areas that are entirely black or entirely white. For text data, it checks for missing key fields and abnormal text lengths. For signal data, it checks the continuity of sampling points and the presence of signal loss. For inspection data, it calculates the proportion of missing items. The integrity score is the percentage of complete data items out of the total number of data items.

[0118] The signal-to-noise ratio (SNR) is the ratio of effective information to noise in data. For image data, the SNR is calculated as the ratio of the mean gradient magnitude of the image to the standard deviation of the noise. For text data, the information density is calculated as the proportion of medical entity words to the total vocabulary. For signal data, the signal power is calculated as the ratio of signal power to noise power. For examination data, the percentage of items whose values ​​are within the normal reference range is calculated. The SNR score is normalized to between 0 and 1 using the sigmoid function.

[0119] Consistency assessment evaluates the degree of mutual corroboration between different modalities of data from the same patient by calculating the similarity of feature vectors from different modalities; higher similarity indicates better consistency. The consistency score is the average similarity of a modal feature to features from all other modalities.

[0120] The reliability assessment considers the authority of the data source and the standardization of the collection process. Reliability scores are calculated based on information such as equipment model, operator qualifications, and collection protocols within the data metadata. High-end equipment, experienced personnel, and standard protocols correspond to high reliability. The scores from the four dimensions are then weighted and averaged with weights of 0.3, 0.3, 0.2, and 0.2 to obtain a preliminary quality score.

[0121] Step 4.2: Correct the quality score using the clinical rule base;

[0122] Automated quality assessment may result in misjudgments. For example, the imaging features of certain special cases may differ greatly from those of routine cases and may be misjudged as low-quality data.

[0123] A clinical rule base is introduced to correct the automated assessment results. This rule base, compiled by medical experts based on clinical experience, includes rules for handling various special cases. The rules are in the form of condition-action pairs, where the condition describes the characteristic patterns of the data, and the action describes how the quality score is adjusted. For example, Rule 1: If the contrast of image data is below the second threshold but the texture features are rich, the signal-to-noise ratio score is increased by 0.2; Rule 2: If the text data is short but contains key diagnostic terms, the integrity score is increased by 0.15; Rule 3: If the signal data has baseline drift but the waveform is clear, the signal-to-noise ratio score remains unchanged instead of decreasing.

[0124] The system iterates through all rules in the rule base, checking if the conditions for each rule are met. If so, it executes the corresponding action and adjusts the quality score. Multiple rules may trigger simultaneously; they are executed in order of priority, set by experts. The corrected quality score better reflects clinical reality, reducing misjudgments in automated assessments.

[0125] Step 4.3: Normalize and generate adaptive fusion weights;

[0126] Based on the corrected quality scores obtained in step 4.2, fusion weights for each modality are generated. The quality scores are normalized using the softmax function so that the sum of the weights for all modalities is 1. The temperature parameter of the softmax function is preferably between 0.3 and 0.7; in one embodiment, it is set to 0.5. A lower temperature makes the weight distribution more concentrated, amplifying quality differences, giving higher weights to high-quality modalities, and appropriately reducing the weights of low-quality modalities. The normalized weights serve as the final adaptive fusion weights for subsequent multimodal feature fusion steps.

[0127] Step 5: Based on the temporal enhancement feature vector set and adaptive fusion weights, determine modal integrity and fusion strategy, calculate cross-modal attention interaction weights and perform weighted fusion to obtain fused feature vectors;

[0128] Specifically, the following steps are included:

[0129] Step 5.1: Determine modal integrity and fusion strategy;

[0130] In medical diagnosis, data from different modalities provide complementary diagnostic information. For example, CT images show the morphology and location of lesions, medical records describe the patient's symptoms and medical history, and blood tests reflect abnormal biochemical indicators. This information needs to be comprehensively analyzed to make an accurate diagnosis. In clinical practice, due to factors such as examination costs, time constraints, or the patient's physical condition, certain modalities of data are often missing. This step employs an appropriate fusion strategy based on the completeness of the modalities to ensure the generation of high-quality fusion features regardless of whether the modalities are complete or missing.

[0131] The modality missing indicator vector is a binary vector with a length equal to the number of modalities. Each element corresponds to a modality, with a value of 1 indicating that the modality data has been collected and a value of 0 indicating that the modality data is missing. The indicator vector is traversed to count the number of collected modalities. If all modalities have been collected, a full modality fusion strategy is adopted, using all temporal augmentation feature vectors and their corresponding fusion weights. If there are missing modalities but the number of collected modalities is greater than or equal to 2, a modality missing fusion strategy is adopted, extracting the feature vectors of the collected modalities to construct a feature list, and re-normalizing the fusion weights of the collected modalities to ensure that the sum of the weights is 1. If the number of collected modalities is less than 2, multimodal fusion cannot be performed, the system outputs a warning message, and switches to single-modality diagnostic mode.

[0132] Step 5.2, calculate the cross-modal attention interaction weights;

[0133] For the temporal augmentation feature vectors of the acquired modalities, calculate the interaction weights between them. Map the feature vector of each modality to a unified interaction space through a linear transformation; the parameters of the transformation matrix are learned through training. Calculate the dot product of the transformed feature vectors of two modalities to obtain the original interaction score. For each modality, calculate its original interaction score with all other acquired modalities, and normalize it to attention weights using the softmax function; the sum of the weights is 1. The attention weights reflect the correlation between different modalities; a larger weight indicates that the information of the two modalities is more complementary, and more attention should be given to them during fusion. Calculate the attention weights for all acquired modal pairs, and construct a cross-modal attention weight matrix.

[0134] Step 5.3: Perform adaptive weighted fusion;

[0135] Based on the cross-modal attention weight matrix calculated in step 5.2 and the fusion weight determined in step 5.1, the collected modal features are weighted and aggregated.

[0136] For each feature vector of a collected modality, the feature vectors of all other collected modalities are weighted and summed using their corresponding attention weights to obtain the interaction features of that modality. The original temporal enhancement features of that modality are concatenated with the interaction features to obtain the enhanced feature representation. The enhanced features of all collected modalities are weighted and summed using the fusion weights determined in step 5.1 to obtain the preliminary fused feature vector. The preliminary fused features undergo nonlinear transformation through a fully connected network. The network contains multiple fully connected layers and introduces nonlinear transformation capability through a nonlinear activation function, mapping the features to a high-dimensional hidden space and then back to the target dimension to obtain the final fused feature vector.

[0137] Through the above steps, step 5 outputs a 512-dimensional fusion feature vector. This feature vector integrates information from all collected modalities and their interaction relationships. When the modalities are complete, it has high semantic expressive power and discriminative power. When the modalities are missing, it compensates for information loss through a dynamic weight adjustment mechanism to ensure the stability and reliability of the diagnostic system.

[0138] Step 6: Predict the disease category based on the fused feature vector, calculate the contribution of each modality feature and generate multimodal visualization results to obtain a structured diagnostic report;

[0139] Specifically, the following steps are included:

[0140] Step 6.1: Predict the disease category using a classification network;

[0141] The fused features are input into a disease classification network, which is a multilayer perceptron structure containing four fully connected layers with dimensions of 512, 256, 128, and the number of disease categories, respectively. The first three layers use the ReLU activation function to introduce non-linearity, and the last layer uses the softmax activation function to normalize the output to a probability distribution. The number of disease categories is set according to the specific application scenario; for example, in tumor diagnosis tasks, it may include categories such as benign, malignant, and uncertain, while in multi-disease classification tasks, it may include dozens of disease categories.

[0142] The classification network is trained through supervised learning, using the cross-entropy loss function to measure the difference between the predicted probability distribution and the true labels, and optimizing the network parameters through backpropagation. The training data contains a large number of labeled samples, with annotations provided by professional doctors to ensure the accuracy and consistency of diagnostic criteria. During training, techniques such as data augmentation, regularization, and early stopping are used to prevent overfitting and improve the model's generalization ability.

[0143] After training, for new test samples, their fused features are input into the classification network, and the disease category probability distribution is calculated through forward propagation. The category with the highest probability is used as the predicted diagnosis result, and the probability value is used as the diagnosis confidence.

[0144] Step 6.2: Calculate the contribution score of each modal feature;

[0145] To analyze the contribution of each modality feature to the diagnostic results, the gradient backpropagation method is used to calculate feature attribution. For a predicted disease category, the gradient of the predicted probability of that category with respect to the fused feature is calculated. The gradient, calculated using the backpropagation algorithm, represents the sensitivity of each dimension of the fused feature to the predicted probability. Since the fused feature is obtained by weighted fusion of features from various modalities, the gradient of the fused feature is backpropagated to each modal feature to calculate its gradient. For each modality, the sum of the absolute values ​​of the gradients of all dimensions of its feature vector is calculated as the contribution score for that modality. The contribution score is normalized to between 0 and 1; a higher score indicates a greater contribution of the modality to the diagnostic results. The contribution scores of each modality are ranked to identify the modalities most important for diagnosis, providing decision-making reference for physicians.

[0146] Step 6.3: Generate multimodal visualization results;

[0147] For image modalities, class activation heatmaps are generated to visually demonstrate which regions in the image contribute the most to the diagnostic results. Class activation heatmaps are generated by mapping feature contribution values ​​back to the original image space.

[0148] Specifically, the feature map of the last convolutional layer of the image feature extraction network is extracted. This feature map preserves spatial location information. The gradient of the predicted class with respect to the feature map is calculated, and the feature map is weighted using the gradient to obtain a class activation map. The value at each position in the class activation map represents the contribution of that position to the diagnostic result; a larger value indicates a greater contribution. The class activation map is upsampled to the resolution of the original image, and color mapping is used to convert the values ​​into colors, generating a heatmap. The heatmap is overlaid on the original image, with high-contribution areas displayed in warm colors and low-contribution areas displayed in cool colors, allowing doctors to intuitively see the lesion areas of interest to the system.

[0149] For the text modality, a word importance visualization is generated, annotating key words in the text that contribute significantly to diagnosis. The gradient of the predicted category with respect to text features is calculated, and this gradient is backpropagated to the word embedding layer to obtain the importance score for each word. In the original text, words are annotated using different colors or font sizes; words of high importance are marked with bright colors and larger fonts, while words of low importance are marked with light colors and smaller fonts.

[0150] For each signal mode, a temporal importance curve is generated, marking the time periods in the signal that contribute significantly to diagnosis. The gradient of the predicted category with respect to the signal features is calculated, and the gradient is mapped back to the time axis of the original signal to obtain the importance score for each time point. The importance curve is then overlaid on the original signal graph, or color-coded to indicate the importance of different time periods.

[0151] Step 6.4: Integrate and generate a structured diagnostic report;

[0152] Based on the diagnostic results from step 6.1, the contribution score from step 6.2, and the visualization results from step 6.3, a structured diagnostic report is generated. The report uses a predefined template format, which includes fields such as basic patient information, diagnostic results, confidence level, main evidence, and visualization. These fields are arranged in the standard order for clinical diagnostic reports.

[0153] The diagnosis result field is populated with the disease category predicted in step 6.1, and the confidence field is populated with the corresponding predicted probability value. The main basis field is sorted according to the contribution score calculated in step 6.2, and the top three modalities with the highest contribution are selected. Key features of each modality are extracted and described. For the imaging modality, the location, size, and morphological features of high-contribution areas are identified from the class activation heatmap to describe the imaging manifestations of lesions. For the text modality, the keywords and phrases with the highest importance scores are extracted from lexical importance analysis, and key symptom descriptions and medical history information are extracted from the medical records. For the examination data modality, the names of abnormal detection indicators, measured values, normal reference ranges, and degrees of deviation are listed.

[0154] The visualizations embed the class activation heatmap, vocabulary annotation text, and time-series importance curves generated in step 6.3. Each visualization is accompanied by a brief explanation, indicating the location of key areas or key information. A system suggestion field is added at the end of the report, providing further examination suggestions or treatment directions based on the diagnostic results and confidence levels. The generated diagnostic report has a clear structure, complete content, conforms to the standards of clinical diagnostic reports, and is easy for doctors to read and understand.

[0155] A deep learning-based multimodal data fusion processing system is used to execute the aforementioned deep learning-based multimodal data fusion processing method, such as... Figure 3 As shown, it includes:

[0156] The data preprocessing module is used to acquire multimodal medical data and preprocess it, extract features from the preprocessed data and record quality metadata to obtain a standardized feature vector set.

[0157] The semantic alignment module constructs positive and negative sample pairs based on a standardized feature vector set, maps them to a unified semantic space, and performs parameter optimization and soft alignment processing to obtain a semantically aligned feature vector set.

[0158] The temporal modeling module generates multi-scale temporal codes based on semantically aligned feature vector sets, calculates temporal correlation weight matrices and performs temporal weighted fusion to obtain temporal enhanced feature vector sets.

[0159] The quality assessment module performs quality assessment and calculates a preliminary quality score based on the temporal enhanced feature vector set and quality meta-information. It then obtains adaptive fusion weights by correcting misjudgments through a clinical rule base.

[0160] The feature fusion module, based on the temporal enhanced feature vector set and adaptive fusion weights, determines modal integrity and fusion strategy, calculates cross-modal attention interaction weights and performs weighted fusion to obtain the fused feature vector;

[0161] The diagnostic output module predicts disease categories based on fused feature vectors, calculates the contribution of each modality feature, generates multimodal visualization results, and obtains a structured diagnostic report.

[0162] In one embodiment of the present invention, a specific example is provided:

[0163] A six-month field test was conducted in the oncology department of a top-tier hospital. During the test, the multimodal data fusion processing system of this invention was deployed and integrated with the hospital's image archiving and communication system, electronic medical record system, electrocardiogram monitoring system, and laboratory information system. The test area covered the oncology outpatient and inpatient departments, and a total of 328 patients with lung diseases' multimodal medical data were processed.

[0164] Patient A's basic information: The patient is a 58-year-old male, presenting with a 3-month history of persistent cough and chest pain. He has a 20-year smoking history and a family history of lung cancer. Upon admission, the patient underwent a chest CT scan, followed by electronic medical record keeping, electrocardiogram (ECG), and blood tumor marker testing. An example of multimodal medical data acquisition for Patient A is shown in Table 1.

[0165] Table 1: Example of obtaining multimodal medical data for patient A;

[0166]

[0167] Patient B's basic information: The patient is a 62-year-old female with a chief complaint of recurrent cough with hemoptysis for 2 months. She has no history of smoking and a history of chronic bronchitis. The patient underwent a chest CT scan and electronic medical record entry at the outpatient clinic, followed by pulmonary function tests and blood tumor marker testing. An example of multimodal medical data acquisition for Patient B is shown in Table 2.

[0168] Table 2: Example of obtaining multimodal medical data for Patient B;

[0169]

[0170] Taking patient A as an example, the system processing flow is as follows:

[0171] The system standardizes and extracts features from various modalities of data. CT images are formatted as DICOM, pixel arrays are automatically extracted and adjusted to standard resolution, and image features are extracted using a pre-trained convolutional neural network, resulting in a feature vector dimension of 512. Electronic medical record text is parsed in HL7 format, structured fields are extracted, and text features are extracted using a pre-trained medical language model, resulting in a feature vector dimension of 768. Electrocardiogram (ECG) signals are read in EDF format with a uniform sampling rate of 250 Hz, and signal features are extracted using a temporal convolutional network, resulting in a feature vector dimension of 512. Blood test data is parsed using LOINC encoding, converting various indicators to international standard units, and examination data features are extracted using a fully connected network, resulting in a feature vector dimension of 256.

[0172] The system performs cross-modal semantic alignment. A projection network trained using a contrastive learning method maps features from each modality to a unified 512-dimensional semantic space. The semantic similarity between image features and text features is 0.82, indicating a high correlation between the nodule features displayed in the image and the symptoms described in the medical record. The semantic similarity between image features and examination data features is 0.76, indicating that the image abnormalities and elevated tumor markers corroborate each other.

[0173] The system performs temporal relationship modeling. CT images, medical records, and electrocardiograms were acquired on January 5, 2025, while blood tests were acquired on January 6, with a time difference of 24 hours. The system calculates multi-scale time codes and establishes a temporal correlation weight matrix. Due to the close acquisition time, the temporal correlation weights between each modality are all above 0.85, indicating strong temporal consistency of the data.

[0174] The system performs a data quality assessment. Image data has an integrity score of 1.0, a signal-to-noise ratio score of 0.88, a consistency score of 0.82, a reliability score of 0.92, and an overall quality score of 0.90. Text data has a quality score of 0.85, signal data has a quality score of 0.78, and inspection data has a quality score of 0.88. Fusion weights are generated based on these quality scores: image modality weight is 0.32, text modality weight is 0.28, signal modality weight is 0.22, and inspection data modality weight is 0.18.

[0175] The system performs multimodal feature fusion. Using a hierarchical attention mechanism, features are first aggregated within each modality, then cross-modal interaction is performed, and finally all information is integrated through adaptive gating to obtain a 512-dimensional fused feature vector.

[0176] The system generates a diagnostic result. The fused features are input into the classification network, which outputs the following disease category probability distributions: lung adenocarcinoma probability 0.78, lung squamous cell carcinoma probability 0.15, small cell lung cancer probability 0.04, and benign nodule probability 0.03. The system predicts the diagnosis as lung adenocarcinoma with a confidence level of 0.78.

[0177] The system generated an interpretability analysis. Feature contribution analysis showed that the image modality contributed 0.42, the text modality contributed 0.28, the examination data modality contributed 0.20, and the signal modality contributed 0.10. The image heatmap highlighted the nodular region in the right upper lobe, which contributed the most to the diagnosis. Text annotations highlighted keywords such as cough, chest pain, smoking history, and family history. The system retrieved typical features of lung adenocarcinoma from the medical knowledge base, including peripheral nodules, spiculated margins, vascular convergence, and elevated tumor markers, which highly matched patient A's actual presentation. The system-generated diagnostic report indicated that the patient's imaging findings were consistent with lung adenocarcinoma, and the elevated tumor markers supported a malignant diagnosis, recommending a pathological biopsy for confirmation.

[0178] Patient A subsequently underwent bronchoscopy and pathological biopsy, and the pathological results confirmed lung adenocarcinoma, consistent with the system's prediction.

[0179] For patient B, the system also performed complete multimodal data fusion processing. The system identified a ground-glass nodule with solid components in the lower lobe of the left lung. Combined with the patient's clinical symptoms and a slight increase in tumor markers, the system predicted a diagnosis of early-stage lung adenocarcinoma with a confidence level of 0.72. Patient B's pathological biopsy confirmed adenocarcinoma in situ, consistent with the system's prediction, thus verifying the accuracy and reliability of the method of this invention in different case types.

[0180] As demonstrated by the application examples above, the deep learning-based multimodal data fusion processing method proposed in this invention can effectively integrate heterogeneous medical data from different examination methods. Through semantic alignment, temporal modeling, quality assessment, and adaptive fusion, it generates accurate diagnostic results and provides detailed interpretable analysis, offering strong decision support for clinicians. This method has shown good performance and practical value in actual clinical applications.

[0181] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A multimodal data fusion processing method based on deep learning, characterized in that, Includes the following steps: Acquire multimodal medical data and preprocess it. Extract features from the preprocessed data and record quality metadata to obtain a standardized feature vector set. Positive and negative sample pairs are constructed based on a standardized feature vector set, mapped to a unified semantic space, and then parameter optimization and soft alignment are performed to obtain a semantically aligned feature vector set. Multi-scale temporal codes are generated based on semantically aligned feature vector sets. Temporal correlation weight matrices are calculated and temporal weighted fusion is performed to obtain temporally enhanced feature vector sets. Based on the temporal enhanced feature vector set and quality meta-information, quality assessment is performed and a preliminary quality score is calculated. Adaptive fusion weights are obtained by correcting misjudgments through a clinical rule base. Based on the temporal enhancement feature vector set and adaptive fusion weights, modal integrity is judged and fusion strategy is determined. Cross-modal attention interaction weights are calculated and weighted fusion is performed to obtain fused feature vectors. Disease categories are predicted based on fused feature vectors, the contribution of each modality feature is calculated and multimodal visualization results are generated to obtain a structured diagnostic report.

2. The method for multimodal data fusion processing based on deep learning according to claim 1, characterized in that, The obtained standardized feature vector set includes: Identify the format of multimodal medical data and output the format type identifier. Multimodal medical data includes medical image data, electronic medical record text data, physiological signal data and laboratory test data. Input the format feature vector into the format classifier for identification. When the confidence level is lower than the first threshold, the manual confirmation process is initiated. Based on the format type identifier, the corresponding processing flow is retrieved from the preset standardization rule base for data standardization preprocessing; Feature vectors for each modality are extracted using a deep network. For medical image modalities, a convolutional neural network is used as the feature extractor, and a global average pooling layer is added at the end of the network to obtain image feature vectors. Record the quality metadata of each modality of data. For image data, calculate the signal-to-noise ratio as a quality indicator; for text data, evaluate the integrity score; for signal data, count the artifact ratio; and for inspection data, record the number of missing items.

3. The method for multimodal data fusion processing based on deep learning according to claim 1, characterized in that, The obtained semantic alignment feature vector set includes: Combine the feature vectors of any two different modalities of the same patient into a positive sample pair; Calculate the cosine similarity between the feature vector and the corresponding modal feature vectors of other patients in the dataset, select the number of feature vectors with the highest similarity as candidate negative samples, and randomly sample feature vectors from the candidate negative samples and combine them with the feature vectors of another modality to form a negative sample pair; For each modality, a projection network is designed. The projection network contains two fully connected layers. The feature vectors of each modality are input into the corresponding projection network and mapped to a unified semantic space to obtain the projected feature vectors. The InfoNCE loss function is used to optimize the projection network parameters, maximizing the similarity of projected features for positive sample pairs and minimizing the similarity for negative sample pairs. Set a target similarity threshold; alignment is considered successful when the similarity of positive samples is higher than the target similarity threshold.

4. The method for multimodal data fusion processing based on deep learning according to claim 1, characterized in that, The obtained time-series enhanced feature vector set includes: For any two modal data, calculate the acquisition time difference, decompose the time difference into five time scales: hour, day, week, month, and grade. For each time scale, use the sinusoidal position coding method to generate a time embedding vector, and concatenate them to obtain a multi-scale time embedding vector. The temporal association weights between modalities are calculated. The temporal association weights consist of temporal proximity and feature similarity. Temporal proximity is calculated using a temporal attention network based on multi-scale temporal embedding vectors, and feature similarity is calculated using cosine similarity based on semantically aligned feature vector sets. The temporal context features are obtained by weighting and summing the features of other modalities using temporal correlation weights. The original feature vector is then concatenated with the temporal context features and input into the feature fusion network to obtain the temporal enhanced feature vector.

5. The method for multimodal data fusion processing based on deep learning according to claim 1, characterized in that, The obtained adaptive fusion weights include: The quality of each modality of data is evaluated from four dimensions: data integrity, signal-to-noise ratio, consistency and reliability. The scores of the four dimensions are weighted and averaged to obtain a preliminary quality score. The quality score is corrected by using a clinical rule base. The rule base contains condition-action pairs, feature patterns of condition description data, and adjustment methods for action description quality scores. The rule base is traversed to check whether the rule conditions are met. If they are met, the corresponding action is executed to adjust the quality score. The softmax function is used to normalize the corrected quality score to generate adaptive fusion weights.

6. The multimodal data fusion processing method based on deep learning according to claim 1, characterized in that, The obtained fused feature vector includes: The number of modalities is counted based on the modal missing indicator vector. If all modalities have been collected, the complete modal fusion strategy is adopted. If there are missing modalities but the number of collected modalities is greater than or equal to the preset threshold, the modal missing fusion strategy is adopted, and the adaptive fusion weights of the collected modalities are renormalized. The temporal enhancement feature vector of each modality is mapped to a unified interaction space through a linear transformation. The dot product of the feature vectors after the transformation of the two modalities is calculated and normalized to cross-modal attention interaction weights through the softmax function. The temporal augmentation feature vector is weighted and summed using cross-modal attention interaction weights to obtain the interaction feature. The temporal augmentation feature vector and the interaction feature are concatenated, and the initial fusion feature vector is obtained by weighted summing using adaptive fusion weights. The final fusion feature vector is obtained by nonlinear transformation through a fully connected network.

7. The method for multimodal data fusion processing based on deep learning according to claim 1, characterized in that, The structured diagnostic report includes: The fused feature vector is input into the disease classification network, and the probability distribution of disease categories is calculated through forward propagation. The category with the highest probability is taken as the predicted disease category. Calculate the gradient of the predicted probability of the disease category with respect to the fused feature vector, backpropagate the gradient to each modality feature, and calculate the sum of the absolute values ​​of the gradients of all dimensions of each modality feature vector as the contribution score. For image modalities, feature maps of the convolutional layers of the image feature extraction network are extracted, the gradient of the predicted disease category with respect to the feature map is calculated, the gradient is used to weight the feature map to obtain the class activation map, and the class activation map is upsampled to generate a class activation heatmap. A structured diagnostic report is generated based on predicted disease categories, contribution scores, and multimodal visualization results.

8. The method for multimodal data fusion processing based on deep learning according to claim 4, characterized in that, The calculation method for the time-series correlation weight is as follows: The temporal association weight is obtained by weighted average of temporal proximity and feature similarity. Temporal proximity is calculated by using a temporal attention network to obtain the attention score based on multi-scale temporal embedding vectors. Feature similarity is calculated by using cosine similarity on semantically aligned feature vector sets and then linearly mapped to the range of zero to one.

9. The method for multimodal data fusion processing based on deep learning according to claim 5, characterized in that, The conditional action pairs in the clinical rule base include: When the signal-to-noise ratio of image data is lower than the second threshold and the text data integrity score is higher than the third threshold, the image modal quality score is reduced and the text modal quality score is increased. When the number of missing data items exceeds the fourth threshold, the data quality score will be set to the preset minimum value.

10. A multimodal data fusion processing system based on deep learning, characterized in that, A method for performing a deep learning-based multimodal data fusion processing method according to any one of claims 1-9 includes: The data preprocessing module is used to acquire multimodal medical data and preprocess it, extract features from the preprocessed data and record quality metadata to obtain a standardized feature vector set. The semantic alignment module constructs positive and negative sample pairs based on a standardized feature vector set, maps them to a unified semantic space, and performs parameter optimization and soft alignment processing to obtain a semantically aligned feature vector set. The temporal modeling module generates multi-scale temporal codes based on semantically aligned feature vector sets, calculates temporal correlation weight matrices and performs temporal weighted fusion to obtain temporal enhanced feature vector sets. The quality assessment module performs quality assessment and calculates a preliminary quality score based on the temporal enhanced feature vector set and quality meta-information. It then obtains adaptive fusion weights by correcting misjudgments through a clinical rule base. The feature fusion module, based on the temporal enhanced feature vector set and adaptive fusion weights, determines modal integrity and fusion strategy, calculates cross-modal attention interaction weights and performs weighted fusion to obtain the fused feature vector; The diagnostic output module predicts disease categories based on fused feature vectors, calculates the contribution of each modality feature, generates multimodal visualization results, and obtains a structured diagnostic report.

Images