A chronic disease comorbidity risk prediction method based on multi-modal multi-task learning

Abstract

The application discloses a chronic disease comorbidity risk prediction method based on multi-modal multi-task learning, comprising the following steps: collecting multi-modal data such as physiological time series, biochemical time series, pathological section images and three-dimensional medical images of patients, and constructing corresponding comorbidity label vectors; encoding the features of each modal data respectively, extracting time series correlation, key and high-level semantic features; mapping the encoded features to a shared subspace, and realizing cross-modal feature deep fusion through a cross-attention mechanism and a learnable gating parameter; constructing a multi-task learning framework based on the fused features, training the model end-to-end through a joint loss function including classification, auxiliary consistency and regularization loss; optimizing the parameters by using an optimizer and a learning rate scheduling strategy, calculating the feature contribution degree by means of gradient analysis and feature attribution, and outputting the chronic disease incidence probability, median incidence time, feature confidence score and comorbidity burden index to complete risk grading, so as to realize accurate and comprehensive prediction of comorbidity risk.

Description

Technical Field

[0001] This invention relates to the field of chronic disease risk prediction technology, and in particular to a method for predicting the risk of chronic disease comorbidity based on multimodal multitask learning. Background Technology

[0002] Comorbid chronic diseases have become a significant challenge in global public health. Their pathogenesis is complex and influenced by multi-dimensional factors, including physiological, biochemical, and imaging factors, making it difficult for traditional single-modal prediction methods to comprehensively capture disease association characteristics. With the advancement of medical informatization, hospital information systems, laboratory management systems, and image archiving systems have accumulated massive amounts of multimodal data, providing a data foundation for comorbidity risk prediction. However, effectively integrating heterogeneous information such as time-series data and image data to uncover cross-modal associations and disease incidence patterns, and achieving accurate comorbidity risk stratification and onset time prediction, has become a pressing technical challenge in the field of clinical decision support. Simultaneously, clinical practice increasingly demands interpretability and comprehensive risk assessment from prediction models, requiring models not only to output prediction results but also to clearly define the contribution of each factor to the occurrence of comorbidity, providing a basis for personalized intervention program development.

[0003] Existing technologies for predicting comorbidity risk in chronic diseases have two significant shortcomings: First, the multimodal data fusion methods lack deep adaptability. Most methods simply splice or weighted sum the features of each modality without fully considering the dynamic interaction and information complementarity between modalities. This leads to the loss of cross-modal correlation features and makes it difficult to form a unified feature vector that can comprehensively represent the patient's health status. Second, the task design and interpretability of the prediction models are insufficient. Existing methods mostly focus on single disease prediction or simple comorbidity combination judgment, without constructing a multi-task collaborative learning framework to explore potential associations between diseases. Furthermore, they lack an attribution analysis mechanism for the prediction results, making it impossible to clarify the degree of influence of each modality's data and key features on comorbidity risk. This makes it difficult for clinicians to trust and apply the prediction results, limiting the practical application of the models in real-world diagnosis and treatment scenarios. Summary of the Invention

[0004] In order to overcome the shortcomings and deficiencies of existing technologies, this invention provides a method for predicting the risk of chronic disease comorbidity based on multimodal multitask learning.

[0005] The technical solution adopted in this invention is a method for predicting the risk of chronic disease comorbidity based on multimodal multitask learning, comprising the following steps: S1, collecting multimodal patient data, including physiological time-series data, biochemical time-series data, RGB three-channel images of pathological slides, and three-dimensional medical images; S2, constructing a patient comorbidity label vector, wherein different elements in the label vector correspond to the disease status identifiers of different chronic diseases; S3, performing feature encoding processing on the different modal data respectively, extracting the temporal correlation features of physiological time-series data through a temporal encoding network, extracting the label features of biochemical time-series data through one-dimensional convolution and global pooling operations, and extracting the high-level semantic features of pathological slide images and three-dimensional medical images respectively through a pre-trained deep convolutional network; S4, converting the features encoded by different modalities into... The features are mapped to a shared subspace, and the interaction weights between modalities are calculated through a cross-attention mechanism. Learnable gating parameters are introduced to control the information flow between modalities. The cross-modal interaction features and the original modal features are aggregated to obtain fused features. S5, a multi-task shared representation layer and multiple task calibration output layers are constructed based on the fused features. The model is trained end-to-end through a joint loss function, which includes classification loss, auxiliary consistency loss, and regularization loss. S6, the model parameters are optimized using an optimizer and learning rate scheduling strategy. The contribution of different modalities and features to the prediction results is calculated through gradient analysis and feature attribution methods. The predicted values ​​of the incidence probability of different chronic diseases, median onset time, image feature confidence scores, and comorbidity burden index are output. Risk classification is performed based on the comorbidity burden index.

[0006] Furthermore, the temporal correlation feature encoding of the physiological time-series data in S3 adopts the following model: ,in Let be the weight matrix at time t, and BiLSTM be a bidirectional long short-term memory network. for The hidden state at all times The data represents the physiological time series data at time t. For bias terms, It is the Sigmoid activation function. This is the temporal attention weight vector. For time steps, This represents the final temporal correlation feature.

[0007] Furthermore, the intermodal interaction weights calculation in S4 adopts the following model: ,in Let m be the query vector for the m-th mode. Let n be the key vector of the nth mode. This is the gradient adjustment coefficient. Let be the gradient of the nth modal feature with respect to the mth modal feature. For feature dimension, For gating adjustment parameters, The first Modal projection characteristics This is the intermodal interaction weight matrix.

[0008] Furthermore, the joint loss function in S5 is specifically as follows: ,in For loss weighting coefficients, For classifying losses, To mitigate consistency loss, For regularization loss, , These are shared representations corresponding to different modalities. It is the square of the Frobenius norm.

[0009] Furthermore, the S6 comorbidity burden index is calculated using the following model: ,in Let the weight of health loss for the kth chronic disease be denoted as . The first The incidence rate of certain chronic diseases Let represent the correlation strength between diseases k and j. The parameter is used to adjust for the probability difference between diseases k and j. For the set of all association strengths, This represents the number of types of chronic diseases.

[0010] Furthermore, the learning rate scheduling in S6 adopts the following model: ,in Let be the learning rate at time t. These are the minimum and maximum learning rates, respectively. This represents the current number of training steps. Total training steps This is the attenuation adjustment coefficient.

[0011] Further, S3 includes the following sub-steps: S31, encoding temporal features of physiological time-series data, capturing the dependency relationship between consecutive time steps through a bidirectional recurrent network, introducing a temporal attention mechanism to assign differentiated weights to features at different time steps, and aggregating the weighted hidden states to obtain a physiological time-series feature vector; S32, performing one-dimensional convolution operation on biochemical time-series data, extracting local features using convolution kernels of a specified size, introducing nonlinear transformation through the ReLU activation function, performing global max pooling operation after multiple layers of convolution stacking, compressing the feature dimension and retaining the calibration information. S33: Input the RGB three-channel image of the pathological slide into the pre-trained ResNet-50 network. Through the convolutional and pooling layers of the network, the low-level visual features and high-level semantic features of the image are extracted step by step. The feature map before the fully connected layer of the network is truncated as the pathological image feature vector. S34: Input the three-dimensional medical image into the 3DResNet network. Through the three-dimensional convolutional kernel, the feature correlation in the spatial dimension is captured. After multi-scale feature fusion and downsampling operations, the deep structural features of the image are extracted, and the three-dimensional image feature vector consistent with other modal dimensions is output.

[0012] Further, S4 includes the following sub-steps: S41, performing layer normalization on the features encoded in different modalities, mapping the modal features of different dimensions to a shared subspace of the same dimension through a fully connected layer to obtain the normalized modal projection features; S42, constructing query vectors and key vectors for each pair of modal projection features, calculating the initial interaction weights through dot product operation, scaling by dividing by the square root of the feature dimension to obtain the cross-attention matrix before normalization; S43, concatenating the projection features of the pairs of modalities, inputting them into the gating network to generate gating weights through the Sigmoid activation function, multiplying the weights element-wise with the product of the cross-attention matrix and the value vector to control the transmission intensity of cross-modal information; S44, summing all cross-modal interaction features, performing residual connections with the original projection features of different modalities, and obtaining the final multimodal fusion feature vector after layer normalization.

[0013] Further, S5 includes the following sub-steps: S51, inputting multimodal fusion features into a shared fully connected layer, performing nonlinear transformation through the ReLU activation function, and extracting high-level abstract shared representations that can support multiple prediction tasks; S52, constructing an independent task labeling output layer for each chronic disease prediction task, mapping the shared representations to the output dimension of the task through the fully connected layer, and obtaining the original task score; S53, applying the Sigmoid activation function to the original scores of different tasks, converting the scores into disease probability values ​​between 0 and 1; S54, constructing a classification loss function to calculate the error between the predicted probabilities of different tasks and the true labels, designing an auxiliary consistency loss to constrain the consistency of different modal representations, adding L2 regularization loss to suppress model overfitting, summing the three types of losses according to their weights to obtain the total loss function, and updating all parameters of the model through backpropagation.

[0014] A method for predicting the risk of chronic disease comorbidities based on multimodal multitask learning is proposed. This method is implemented through different units, including: a multimodal data integration unit, used to collect physiological time-series data, biochemical time-series data, pathological slide images, and 3D medical images from hospital information systems, laboratory information management systems, and medical image archiving systems, establishing a correspondence between patient data and comorbidity labels; a cross-modal feature encoding unit, connected to the output of the multimodal data integration unit, which extracts targeted features from different types of data through a time-series encoding module, a convolutional pooling module, and a pre-trained deep convolution module, outputting high-dimensional feature vectors of different modalities; and a dynamic gating fusion unit, connected to the output of the cross-modal feature encoding unit, which performs deep interaction and self-regulation of multimodal features through a feature projection module, a cross-attention calculation module, a gating adjustment module, and a feature aggregation module. The system integrates various modules to generate a unified patient representation. A multi-task learning modeling unit, connected to the output of the dynamic gating fusion unit, includes a shared representation extraction layer and multiple task-calibrated output layers. It constructs multi-objective optimization objectives through a joint loss calculation module to complete end-to-end model training. An interpretable analysis unit, also connected to the multi-task learning modeling unit, calculates modal contribution through a gradient attribution module, outputs calibrated influencing factors through a feature importance assessment module, and presents cross-modal feature correlations through a correlation visualization module. A comprehensive risk assessment unit, connected to the output of the interpretable analysis unit, integrates the predicted incidence rates, onset times, and feature confidence scores of different chronic diseases, calculates the comorbidity burden index, performs risk grading, and outputs a structured prediction report. These different units are sequentially connected via a data transmission interface, enabling fully automated processing from data input to risk assessment.

[0015] Beneficial Effects: This invention proposes a method for predicting the risk of chronic disease comorbidity based on multimodal and multi-task learning. Addressing the problem of insufficient adaptability in multimodal data fusion, it employs a targeted feature encoding strategy to extract features from physiological time series, biochemical time series, pathological images, and 3D medical images. A cross-attention mechanism is used to mine the dynamic interaction relationships between modalities, and learnable gating parameters are combined to precisely control the information flow transmission. This deeply aggregates cross-modal interaction features with original modal features, avoiding the loss of associated features and forming a unified feature vector that comprehensively represents the patient's health status. To address the shortcomings of existing models, such as single task design and lack of interpretability, a multi-task learning framework is constructed. A joint loss function is used to collaboratively optimize multiple chronic disease prediction tasks, fully exploring the potential correlations between diseases. Gradient analysis and feature attribution methods are introduced to clarify the contribution of each modality and key feature to the prediction results, improving the model's credibility. This invention enables simultaneous prediction of the incidence probability and median time of onset of multiple chronic diseases, completes risk stratification through the comorbidity burden index, and outputs imaging feature confidence scores to enhance the reference value of the results. The various technical links form a closed loop, which not only solves the problem of heterogeneous data integration, but also makes up for the shortcomings of traditional methods in clinical applicability, provides reliable support for the formulation of personalized intervention plans, and significantly improves the scientificity and clinical translational value of comorbidity risk prediction. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the overall process of the method of the present invention.

[0017] Figure 2 This is a flowchart of method step S3 of the present invention;

[0018] Figure 3 This is a flowchart of method step S4 of the present invention;

[0019] Figure 4 This is a flowchart of step S5 of the method of the present invention;

[0020] Figure 5 This is a diagram showing the system unit composition of the present invention. Detailed Implementation

[0021] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] like Figure 1 As shown, a method for predicting the risk of chronic disease comorbidity based on multimodal multitask learning includes the following steps:

[0023] S1, collect patient multimodal data, including physiological time-series data, biochemical time-series data, pathological slide RGB three-channel images and three-dimensional medical images;

[0024] Specifically, step S1 involves the comprehensive collection and screening of multimodal patient data. During implementation, the system connects with the hospital information system, laboratory information management system, and medical imaging archive system to obtain four types of core data of the target patient group in batches. Physiological time-series data includes 12 key physiological indicators such as heart rate, blood pressure, blood glucose, and blood oxygen saturation. The acquisition frequency is set to once every 5 minutes, and the data is continuously recorded for 90 days to form a time-series sequence with a uniform time step. Each sequence includes no less than 25,920 data points. Biochemical time-series data selects 18 biochemical indicators such as liver function, kidney function, blood lipids, and blood glucose. The data is collected once a week for 12 months, and each sequence includes the test results at 60 time points. Pathological slide RGB three-channel images are acquired by a digital pathology scanner with a 40x objective resolution. The image pixel size is uniformly adjusted to 2048×2048. 3-5 representative slide images are selected from each patient, including the lesion area and the normal tissue control area. Three-dimensional medical imaging includes modalities such as chest CT and brain MRI. The slice thickness of CT images is set to 1mm, and the MRI images use T1-weighted sequences with a slice thickness of 1.5mm. The scanning range covers the entire area of ​​the target organ. The original image data format is converted to the DICOM standard format to ensure data compatibility. During the data collection process, time-series data with a missing rate exceeding 5%, image data containing artifacts, or incomplete scans are removed using data verification rules, ultimately forming a structured multimodal dataset that provides high-quality data support for subsequent feature extraction and model training.

[0025] S2, construct a patient comorbidity label vector, wherein different elements in the label vector correspond to the disease status identifiers of different chronic diseases;

[0026] Specifically, step S2 involves constructing patient comorbidity label vectors. First, 15 common chronic diseases are identified as constituent elements of the label vectors, including hypertension, type 2 diabetes, coronary heart disease, chronic obstructive pulmonary disease, stroke, chronic kidney disease, and cirrhosis. The label vectors are constructed based on patients' electronic medical records, inpatient records, outpatient follow-up records, and diagnostic reports. Two physicians with over 5 years of clinical experience independently label each disease, including two states: "not diseased" and "diseased," corresponding to 0 and 1 in the label vector, respectively. For cases with ambiguous diagnoses, multidisciplinary consultation is used to determine the final labeling results, ensuring label accuracy. The label vectors are set to 15 dimensions, with the elements arranged in a fixed order from highest to lowest chronic disease incidence, facilitating label matching and loss calculation during subsequent model training. Simultaneously, a dynamic label update mechanism is established. For patients newly diagnosed with chronic diseases during follow-up, the values ​​of the corresponding elements in their label vectors are updated promptly and synchronously linked to the original dataset. The label vectors constructed through this step can accurately represent the patient's current comorbidity, provide clear supervision signals for the multi-task learning model, ensure the relevance and effectiveness of model training, and the Kappa coefficient of the label annotation consistency test is not less than 0.85, which meets the label quality requirements for model training.

[0027] S3 performs feature encoding processing on different modal data, extracts temporal correlation features of physiological time series data through temporal coding network, extracts labeling features of biochemical time series data through one-dimensional convolution and global pooling operations, and extracts high-level semantic features of pathological slide images and three-dimensional medical images through pre-trained deep convolutional networks.

[0028] Specifically, step S3 implements differentiated feature encoding based on the characteristics of different modalities. For physiological time-series data, a bidirectional long short-term memory network is used as the temporal encoding network for temporal correlation feature extraction. The network includes three hidden layers, with each layer containing 256 hidden units and a dropout rate of 0.3. Forward and backward propagation paths are used to capture feature dependencies of future and past time steps, respectively, outputting a 512-dimensional temporal correlation feature vector. For biochemical time-series data, a three-layer one-dimensional convolutional network is used for feature extraction. The first layer has 64 kernels with a kernel size of 3, the second layer has 128 kernels with a kernel size of 5, and the third layer has 256 kernels with a kernel size of 7. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. Finally, global average pooling is used to compress the feature dimension to 512 dimensions, realizing the key biochemical features. Effective feature extraction: High-level semantic feature extraction of pathological slide images adopts a pre-trained ResNet-50 network with weight parameters pre-trained on the ImageNet dataset. The parameters of the first 10 convolutional layers are frozen, and the feature extraction layers before the subsequent 24 convolutional layers and fully connected layers are fine-tuned. The input image is pre-processed to 224×224 pixels before being input into the network. The 2048-dimensional feature vector output by the average pooling layer is extracted as the pathological image feature. Feature extraction of 3D medical images adopts the 3DResNet-18 network, which includes 8 3D convolutional blocks with a kernel size of 3×3×3. The number of channels is gradually increased from 64 to 512. Downsampling is performed through the max pooling layer, and finally, 512-dimensional high-level semantic features are obtained through global average pooling. The dimension of each modality feature vector is unified to 512, laying the foundation for subsequent cross-modal fusion.

[0029] S4 maps the features encoded by different modalities to a shared subspace, calculates the intermodal interaction weights through a cross-attention mechanism, introduces learnable gating parameters to control the information flow between modalities, and aggregates the cross-modal interaction features with the original modal features to obtain fused features;

[0030] Specifically, step S4 implements deep fusion of multimodal features. First, the 512-dimensional feature vectors of each modality output by step S3 are subjected to layer normalization. During the normalization process, a batch processing method with a batch size of 32 is adopted to calculate the mean and variance of the features in each batch and standardize the feature values ​​to the range of [-1,1]. Then, the features of each modality are mapped to a 1024-dimensional shared subspace through a fully connected layer. The weights of the fully connected layer are initialized using a He normal distribution, and the bias term is initialized to 0. In the implementation of the cross-attention mechanism, a query vector, a key vector, and a value vector are constructed for each modality feature. The vector dimensions are all 1024-dimensional. The initial interaction weight matrix is ​​calculated by dot product operation. The matrix dimension is 4×4 (corresponding to the four modalities) and is scaled by dividing by the square root (32) of the feature dimension to avoid gradient explosion. The learnable gating parameters are generated by a two-layer fully connected network. The first layer contains 256 hidden units, and the second layer outputs a 4×4 dimensional gating weight matrix. The gating weights are constrained to the [0,1] interval using a sigmoid activation function to control the transmission intensity of information flow between different modalities. Cross-modal interaction features are calculated by multiplying the attention weight matrix with the value vector, then performing a residual connection with the original projection features of each modality element-wise, followed by a layer normalization process to finally generate a 1024-dimensional multimodal fusion feature vector. Throughout the fusion process, the weights of each modality feature are adaptively allocated through an attention mechanism, and the gating parameters dynamically adjust the degree of interaction between cross-modal information, ensuring that the fused features retain the core information of each modality while fully exploring the complementary correlations between modalities.

[0031] S5, construct a multi-task shared representation layer and multiple task calibration output layers based on fusion features, and perform end-to-end training of the model through a joint loss function, which includes classification loss, auxiliary consistency loss and regularization loss;

[0032] Specifically, step S5 constructs a multi-task learning framework and implements end-to-end training. The multi-task shared representation layer consists of three fully connected network layers: the first layer has 2048 hidden units, the second layer has 1024, and the third layer has 512. Each layer incorporates a ReLU activation function and a dropout layer (dropout rate 0.3), transforming the 1024-dimensional fused features output from step S4 into a 512-dimensional high-level shared representation. The task-specific output layer constructs independent output branches for each of the 15 chronic diseases. Each branch includes two fully connected network layers: the first layer has 256 hidden units, and the second layer has 1. The output layer uses a Sigmoid activation function to directly output the predicted incidence probability of the corresponding chronic disease. In constructing the joint loss function, the classification loss uses the binary cross-entropy loss function to calculate the error between the predicted probability and the true label; the auxiliary consistency loss constrains the consistency of representations between modalities by calculating the cosine similarity between shared representations corresponding to different modalities; the regularization loss uses L2 regularization with a regularization coefficient set to 0.001 to suppress model overfitting; the weight coefficients of the three types of losses are set to γ=0.6, β=0.2, and δ=0.1, respectively, and an additional modal consistency loss term is introduced with a weight coefficient ε=0.1, which constrains the differences between modalities by calculating the squared Frobenius norm of shared representations of different modalities. During model training, the batch size is set to 64, the training epochs are 100, and an early stopping strategy is adopted. Training stops when the validation set loss does not decrease for 15 consecutive epochs. The gradient of each layer parameter is calculated through the backpropagation algorithm to achieve synchronous update of all model parameters, ensuring multi-task collaborative optimization and improving the model's predictive performance for each chronic disease.

[0033] S6 employs an optimizer and learning rate scheduling strategy to optimize model parameters. It calculates the contribution of different modalities and features to the prediction results through gradient analysis and feature attribution methods, and outputs the predicted probability of different chronic diseases, median onset time, image feature confidence score, and comorbidity burden index. Based on the comorbidity burden index, risk classification is performed.

[0034] Specifically, step S6 optimizes model parameters and outputs prediction results. During parameter optimization, the AdamW optimizer is used, with an initial learning rate of 0.001 and a weight decay coefficient of 0.0001. Cosine annealing is used for learning rate scheduling, with a minimum learning rate of 0.00001. The total training steps are 8000, with the learning rate adjusted every 1000 steps. The learning rate decay rate is controlled by an exponential decay factor τ=0.8. Gradient analysis uses the gradient values ​​of each layer's parameters during backpropagation to calculate the gradient norm corresponding to each modality feature. Feature attribution uses the SHAP value calculation method to quantify the contribution of each feature to the prediction result, outputting the contribution percentage of each modality and the contribution values ​​of the top 20 key features. The prediction results output includes four core pieces of information: the incidence probability of each chronic disease, ranging from 0 to 1 and rounded to four decimal places; the predicted median time to onset, calculated based on a survival analysis model, in months and rounded to one decimal place; the confidence score for imaging features, using softmax-normalized probability values ​​as the confidence score for pathological slide images and 3D medical image features, ranging from 0 to 1 and rounded to three decimal places; and the comorbidity burden index, calculated by integrating the incidence probability of each chronic disease, disease association strength, and health loss weights, ranging from 0 to 10, and divided into five risk levels: 0-2, 2-4, 4-6, 6-8, and 8-10, corresponding to low risk, low-to-medium risk, medium risk, medium-to-high risk, and high risk, respectively. The output results are presented in a structured report format, including data identification, prediction time, values ​​of each indicator, and risk classification conclusions, supporting clinicians in quickly obtaining key information and providing data support for diagnostic and treatment decisions.

[0035] Preferably, the temporal correlation feature encoding of the physiological time series data in S3 adopts the following model: ,in Let be the weight matrix at time t, and BiLSTM be a bidirectional long short-term memory network. for The hidden state at all times The data represents the physiological time series data at time t. For bias terms, It is the Sigmoid activation function. This is the temporal attention weight vector. For time steps, This represents the final temporal correlation feature.

[0036] Specifically, in step S3, the temporal correlation feature encoding of physiological time-series data achieves accurate feature extraction through dynamic weight adjustment and bidirectional temporal modeling. During implementation, the number of time steps is determined based on the collection cycle and duration of the physiological time-series data, typically set to 25920 steps (corresponding to a scenario of collection every 5 minutes for 90 consecutive days). The bidirectional long short-term memory network has three hidden layers, each with 256 hidden units, and a dropout rate of 0.3. The forward propagation path captures the feature dependencies of future time steps, while the backward propagation path captures the feature dependencies of past time steps, ensuring no temporal information is missed. The weight matrix at time t is initialized using a Xavier normal distribution, with the dimension consistent with the output dimension of the hidden layers, and the bias term initialized to 0, dynamically updated during training. The temporal attention weight vector is generated through a two-layer fully connected network. The first layer includes 128 hidden units, and the output dimension of the second layer is consistent with the number of time steps. After normalization using the Softmax function, differentiated weights in the 0-1 range are assigned to features at different time steps, with the sum of the time step weights being 1. The Sigmoid activation function maps the output of a bidirectional long short-term memory network to the 0-1 interval, suppressing the influence of outliers and strengthening the contribution of key time-step features through the product operation of outputs at each time step. This encoding method preserves the long-term dependencies of time-series data while optimizing feature expression through dynamic weights and activation functions. This makes the extracted time-series associated features more closely match the pathogenesis of chronic disease comorbidities, improving the accuracy of subsequent fusion and prediction. It is suitable for extracting features of dynamically changing physiological indicators such as heart rate and blood pressure.

[0037] Preferably, the intermodal interaction weights calculation in S4 adopts the following model: ,in Let m be the query vector for the m-th mode. Let n be the key vector of the nth mode. This is the gradient adjustment coefficient. Let be the gradient of the nth modal feature with respect to the mth modal feature. For feature dimension, For gating adjustment parameters, The first Modal projection characteristics This is the intermodal interaction weight matrix.

[0038] Specifically, the calculation logic for inter-modal interaction weights in step S4 achieves adaptive inter-modal interaction by fusing attention mechanisms and gradient information. During implementation, the feature dimension is uniformly set to 1024 dimensions (corresponding to the dimension of the shared subspace). The gradient adjustment coefficient is set according to the heterogeneity of the modal data, ranging from 0.1 to 0.3, typically defaulting to 0.2, to balance the contribution ratio of feature interaction and gradient information. The gating adjustment parameters are generated through a 3-layer fully connected network. The network input is a concatenated vector of paired modal projection features. The first layer has 512 hidden units, the second layer has 256, and the third layer output dimension is consistent with the modal logarithm (6 modal pairs corresponding to 4 modalities). The parameter is initialized to 0.5 and dynamically adjusted during training, with a value range limited to 0-1, used to control the information transfer intensity between specific modal pairs. The query vector and key vector are generated through independent fully connected layers. The weight matrix is ​​initialized using a He normal distribution, with a bias term of 0, and the dimensions are consistent with the shared subspace dimension. When calculating the initial interaction weights, the linear correlation between modalities is captured by dot product operation, and the weights are scaled by dividing by the square root of the feature dimension (32) to avoid overflow of weight values ​​due to excessive feature dimension. Then, gradient information and gating adjustment parameters are superimposed, and finally the weights are normalized to the 0-1 range by the Softmax function to ensure that the sum of the interaction weights of each modal pair is 1. This calculation method considers both the original feature correlation between modalities and reflects the dynamic influence between modalities through gradient information. The gating parameters further precisely regulate the information flow, enabling the interaction weights to adapt to the characteristics of different modal data, solve the fusion problem caused by the heterogeneity of multimodal data, and strengthen the mining of cross-modal complementary information.

[0039] Preferably, the joint loss function in S5 is specifically: ,in For loss weighting coefficients, For classifying losses, To mitigate consistency loss, For regularization loss, , These are shared representations corresponding to different modalities. It is the square of the Frobenius norm.

[0040] Specifically, the composition and calculation of the joint loss function in step S5 improves the model's generalization ability and prediction consistency through multi-loss collaborative optimization. During implementation, the loss weight coefficients are set according to task priority and data characteristics. The classification loss weight coefficient is set to 0.6, serving as the core loss to drive the model's classification performance optimization; the auxiliary consistency loss weight coefficient is set to 0.2, used to constrain the consistency of shared representations across different modalities; the regularization loss weight coefficient is set to 0.1; and the modality consistency loss term weight coefficient is set to 0.1. The sum of the four weight coefficients is 1, which can be fine-tuned within ±0.1 based on the actual data distribution and prediction requirements. The classification loss employs binary cross-entropy loss to calculate the error between the predicted probability of each chronic disease and the true label, focusing on the accuracy of the core prediction task. The auxiliary consistency loss quantifies representation differences by calculating the cosine similarity of shared representations across different modalities, constraining information alignment between modalities. The regularization loss uses L2 regularization, penalizing the sum of squares of all trainable parameters with a penalty coefficient of 0.001 to suppress overfitting of parameters to the training data. The modal consistency loss term measures the differences in the representation space by calculating the square of the Frobenius norm of shared representations across different modalities, further strengthening cross-modal consistency. The loss function is calculated in batches of 32, with the average loss value calculated for all samples in each batch. The loss signal is then propagated to each layer of the model via backpropagation, driving synchronous parameter updates. This joint loss function ensures performance for the core prediction task while improving the model's stability, consistency, and generalization ability through multiple constraints, avoiding model bias caused by a single loss, making it suitable for complex training scenarios involving multiple modalities and tasks.

[0041] Preferably, the comorbidity burden index in S6 is calculated using the following model: ,in Let the weight of health loss for the kth chronic disease be denoted as . The first The incidence rate of certain chronic diseases Let represent the correlation strength between diseases k and j. The parameter is used to adjust for the probability difference between diseases k and j. For the set of all association strengths, This represents the number of types of chronic diseases.

[0042] Specifically, step S6, the calculation of the disease burden index, achieves comprehensive risk quantification by integrating disease weights, incidence probabilities, and disease association strength. In implementation, the number of chronic diseases is set to 15 based on clinical needs, including high-incidence chronic diseases such as hypertension and type 2 diabetes. The health loss weight for each chronic disease is determined based on clinical guidelines and epidemiological data, ranging from 0.1 to 1.0. Diseases with greater health impact after onset have higher weights; for example, stroke is set at 1.0, and hypertension at 0.6, with the sum of all disease weights being 10. The disease association strength is determined by statistically analyzing the co-occurrence probability of two diseases in the patient population, ranging from 0 to 1. Higher co-occurrence probabilities indicate stronger associations; for example, the association strength between hypertension and coronary heart disease is set at 0.8. The association strength of all disease pairs is stored in matrix form. The disease probability difference adjustment parameter is set based on the similarity of the pathogenesis of the two diseases, ranging from 0.5 to 2.0; the more similar the mechanisms, the smaller the parameter value. For example, the parameter for type 2 diabetes and kidney disease is set at 0.5, used to adjust the impact of probability differences on the association contribution. The incidence probability is the 0-1 range value output by the model in step S5, rounded to four decimal places. During the calculation, an exponential function is first used to suppress the influence of excessive probability differences on the association terms. The association terms are then normalized by dividing by the maximum value of all association strengths (1.0), and finally multiplied by the incidence probability and health loss weights, and summed to obtain the comorbidity burden index in the 0-10 range. This calculation method considers both the incidence risk of a single disease and integrates the association effects between diseases and differences in health loss, enabling the comorbidity burden index to comprehensively and accurately reflect the overall risk level of patients and provide a scientific basis for risk classification.

[0043] Preferably, the learning rate scheduling in S6 adopts the following model: ,in Let be the learning rate at time t. These are the minimum and maximum learning rates, respectively. This represents the current number of training steps. Total training steps This is the attenuation adjustment coefficient.

[0044] Specifically, in step S6, learning rate scheduling achieves efficient optimization of model parameters through a combination of cosine annealing and exponential decay. In practice, the maximum learning rate is set to 0.001, and the minimum learning rate is set to 0.00001 to ensure rapid convergence in the early stages of training and precise fine-tuning in the later stages. The total number of training steps is determined based on the dataset size and model complexity, typically set to 8000 steps (corresponding to a batch size of 32 and 100 training epochs). The current number of training steps automatically increases with each batch completion. The decay adjustment coefficient is set to 0.8 to control the rate of exponential decay, balancing the stability and adaptability of the learning rate. The learning rate scheduling is updated once after each batch of training. The calculation first uses a cosine function to achieve periodic fluctuations in the learning rate, with the fluctuation period matching the total number of training steps. The cosine function output value is in the range of 0-1, causing the learning rate to exhibit a periodic change of first decreasing and then increasing as training progresses, avoiding local optima. Then, an exponential function is used to achieve overall decay of the learning rate. The exponential part increases linearly with the ratio of the current training steps to the total training steps, resulting in an overall decreasing trend in the learning rate and improving stability in later training stages. The final learning rate is the result of the minimum learning rate plus the cosine fluctuation term and the exponential decay term, always maintaining between the minimum and maximum learning rate. This scheduling strategy combines the advantages of periodic fluctuations and overall decay, enabling it to escape local optima while ensuring the accuracy of parameter updates in later training stages. It effectively avoids gradient vanishing or exploding, accelerates model convergence, and improves the model's generalization ability and prediction accuracy, making it suitable for training complex models in multimodal and multi-task learning.

[0045] Preferred, such as Figure 2 As shown, step S3 includes the following sub-steps: S31, encoding the physiological time-series data into temporal features, capturing the dependencies between consecutive time steps through a bidirectional recurrent network, introducing a temporal attention mechanism to assign differentiated weights to features at different time steps, and aggregating the weighted hidden states to obtain the physiological time-series feature vector; S32, performing a one-dimensional convolution operation on the biochemical time-series data, extracting local features using a convolution kernel of a specified size, introducing a nonlinear transformation through the ReLU activation function, and performing a global max pooling operation after multiple layers of convolution stacking to compress the feature dimension and retain the calibration information. S33: Input the RGB three-channel image of the pathological slide into the pre-trained ResNet-50 network. Through the convolutional and pooling layers of the network, the low-level visual features and high-level semantic features of the image are extracted step by step. The feature map before the fully connected layer of the network is truncated as the pathological image feature vector. S34: Input the three-dimensional medical image into the 3DResNet network. Through the three-dimensional convolutional kernel, the feature correlation in the spatial dimension is captured. After multi-scale feature fusion and downsampling operations, the deep structural features of the image are extracted, and the three-dimensional image feature vector consistent with other modal dimensions is output.

[0046] Specifically, the feature encoding process in step S3 achieves accurate feature extraction for each modality of data through four sub-steps. In S31, when processing physiological time-series data, a bidirectional recurrent network with three hidden layers (256 hidden units per layer) is used, with a dropout rate of 0.3. The forward and backward paths capture the dependencies of future and past time steps, respectively. A temporal attention mechanism generates a weight vector matching the number of time steps through a two-layer fully connected network. After normalization, differentiated weights in the 0-1 range are assigned to different time steps. The weighted hidden states are then summed and aggregated into a 512-dimensional physiological time-series feature vector. In S32, when processing biochemical time-series data, three layers of one-dimensional convolutions are stacked, with kernel sizes of 3, 5, and 7, and numbers of kernels of 64, 128, and 256, respectively. Batch normalization and Regression are applied after each convolutional layer. The LU activation function is used, and finally, global max pooling is applied to compress the features to 512 dimensions, preserving key biochemical information. In S33, when processing pathological slide images, a pre-trained ResNet-50 network is input. The image preprocessing is 224×224 pixels, the parameters of the first 10 convolutional layers are frozen, and the subsequent 24 layers are fine-tuned. The 2048-dimensional feature vector output from the average pooling layer is extracted and unified to 512 dimensions after dimensionality compression. In S34, when processing 3D medical images, a 3DResNet-18 network is used. The kernel size of the eight 3D convolutional blocks is 3×3×3, and the number of channels is gradually increased from 64 to 512. Through multi-scale feature fusion and downsampling operations, 512-dimensional deep structural features are extracted. These four steps design differentiated encoding strategies for the characteristics of each modality's data, ensuring that the extracted features retain both the core modality information and achieve dimensionality uniformity, laying the foundation for cross-modal fusion.

[0047] Preferred, such as Figure 3 As shown, step S4 includes the following sub-steps: S41, performing layer normalization on the features encoded in different modalities, mapping the modal features of different dimensions to a shared subspace of the same dimension through a fully connected layer to obtain the normalized modal projection features; S42, constructing query vectors and key vectors for each pair of modal projection features, calculating the initial interaction weights through dot product operation, scaling by dividing by the square root of the feature dimension to obtain the cross-attention matrix before normalization; S43, concatenating the projection features of the pairs of modalities, inputting them into the gating network to generate gating weights through the Sigmoid activation function, multiplying the weights element-wise with the product of the cross-attention matrix and the value vector to control the transmission intensity of cross-modal information; S44, summing all cross-modal interaction features, performing residual connections with the original projection features of different modalities, and obtaining the final multimodal fusion feature vector after layer normalization.

[0048] Specifically, the multimodal fusion process in step S4 consists of four sub-steps to achieve deep interaction and aggregation of features. In S41, the 512-dimensional encoded features of each modality are first normalized by layer, and the mean and variance are calculated based on the batch size of 32. The features are standardized to the [-1,1] interval, and then mapped to a 1024-dimensional shared subspace through a fully connected layer (the weights are initialized using a He normal distribution) to obtain standardized projection features. In S42, a 1024-dimensional query vector and key vector are constructed for each pair of modal projection features. The initial interaction weight matrix is ​​calculated by dot product operation, and scaled by dividing by the square root (32) of the feature dimension to avoid gradient explosion. Before normalization, a cross-attention matrix is ​​generated. In S43, paired modal projection features are concatenated and input into a gating network (2 fully connected layers, 256 hidden units). After Sigmoid activation, gating weights in the 0-1 interval are generated and multiplied element-wise with the product of the cross-attention matrix and the value vector to precisely control the intensity of cross-modal information transmission. In S44, all cross-modal interaction features are summed and residually connected with the original projection features of each modality. After another layer normalization process, a 1024-dimensional multimodal fusion feature vector is finally generated. These four steps, through progressive operations of projection, attention calculation, gating adjustment, and aggregation, not only preserve the original information of each modality but also deeply explore cross-modal correlations, solving the problem of heterogeneous fusion of multimodal data.

[0049] Preferred, such as Figure 4 As shown, S5 includes the following sub-steps: S51, inputting multimodal fusion features into a shared fully connected layer, performing nonlinear transformation through the ReLU activation function, and extracting high-level abstract shared representations that can support multiple prediction tasks; S52, constructing an independent task labeling output layer for each chronic disease prediction task, mapping the shared representations to the output dimension of the task through the fully connected layer, and obtaining the original task score; S53, applying the Sigmoid activation function to the original scores of different tasks, converting the scores into disease probability values ​​between 0 and 1; S54, constructing a classification loss function to calculate the error between the predicted probabilities of different tasks and the true labels, designing an auxiliary consistency loss to constrain the consistency of different modal representations, adding L2 regularization loss to suppress model overfitting, summing the three types of losses according to their weights to obtain the total loss function, and updating all parameters of the model through backpropagation.

[0050] Specifically, step S5, multi-task learning modeling, includes four sub-steps to achieve end-to-end model training. In S51, a 1024-dimensional multimodal fusion feature is input into a three-layer shared fully connected layer with hidden units of 2048, 1024, and 512 respectively. Each layer incorporates a ReLU activation function and a dropout layer (dropout rate 0.3), extracting high-level abstract shared representations supporting multiple tasks through nonlinear transformation. In S52, independent task-specific output layers are constructed for each of the 15 chronic diseases. Each output layer includes two fully connected layers (the first layer has 256 hidden units), mapping the 512-dimensional shared representations to a one-dimensional task raw score. In S53, Si is applied to the raw scores of each task. The gmoid activation function is used to convert the incidence probability value into a 0-1 interval, intuitively reflecting the risk of each chronic disease. In S54, a binary cross-entropy classification loss is constructed to calculate the error between the predicted probability and the true label. A cosine similarity-assisted consistency loss is designed to constrain the consistency of representations between modes. An L2 regularization loss (regularization coefficient 0.001) is added to suppress overfitting. The three types of losses are summed with weights of 0.6, 0.2, and 0.1, and then a mode consistency loss term with a weight of 0.1 is added to form the total loss function. The gradient of the parameters of each layer is calculated through the backpropagation algorithm to achieve synchronous updates of all model parameters. The four steps construct a complete multi-task learning process. Through the collaborative design of shared representations and task-specific output layers, combined with multi-constraint loss functions, the predictive performance and generalization ability of the model for various chronic diseases are improved.

[0051] like Figure 5As shown, a method for predicting the risk of chronic disease comorbidity based on multimodal multitask learning is characterized by being implemented through different units, including: a multimodal data integration unit, used to collect physiological time-series data, biochemical time-series data, pathological slide images, and three-dimensional medical images from hospital information systems, laboratory information management systems, and medical image archiving systems, and establish a correspondence between patient data and comorbidity labels; a cross-modal feature encoding unit, connected to the output of the multimodal data integration unit, which performs targeted feature extraction on different types of data through a time-series encoding module, a convolutional pooling module, and a pre-trained deep convolution module, and outputs high-dimensional feature vectors of different modalities; and a dynamic gating fusion unit, connected to the output of the cross-modal feature encoding unit, which performs deep gating fusion of multimodal features through a feature projection module, a cross-attention calculation module, a gating adjustment module, and a feature aggregation module. The system integrates interactive and adaptive approaches to generate a unified patient representation. A multi-task learning modeling unit, connected to the output of the dynamic gating fusion unit, includes a shared representation extraction layer and multiple task-calibrated output layers. It constructs multi-objective optimization objectives through a joint loss calculation module to complete end-to-end model training. An interpretable analysis unit, connected to the output of the multi-task learning modeling unit, calculates modal contribution through a gradient attribution module, outputs calibrated influencing factors through a feature importance assessment module, and presents cross-modal feature correlations through a correlation visualization module. A comprehensive risk assessment unit, connected to the output of the interpretable analysis unit, integrates the predicted incidence rates, onset times, and feature confidence scores of different chronic diseases, calculates the comorbidity burden index, performs risk grading, and outputs a structured prediction report. These different units are sequentially connected via a data transmission interface, enabling fully automated processing from data input to risk assessment.

[0052] This paper presents a method for predicting the comorbidity risk of chronic diseases based on multimodal and multitask learning. Addressing the limitations of traditional multimodal data fusion methods, which often suffer from superficiality and loss of associated features, this method employs differentiated feature encoding strategies for physiological time series, biochemical time series, pathological images, and 3D medical images. It leverages a cross-attention mechanism to deeply mine dynamic correlations between modalities and utilizes learnable gating parameters to flexibly control information flow. This effectively aggregates cross-modal interactive features with original modal features, forming a unified representation that comprehensively depicts the patient's health status, fundamentally changing the crude fusion approach of simple splicing or weighted summation. Furthermore, addressing the shortcomings of existing models in terms of single task design and lack of interpretability, this method utilizes a collaborative architecture of a multi-task shared representation layer and a task-specific output layer to fully explore the potential correlations between different chronic diseases. By combining a joint loss function including classification loss, auxiliary consistency loss, and regularization loss, it achieves synergistic optimization of multiple disease prediction tasks, significantly improving prediction accuracy and generalization ability.

[0053] Furthermore, this method introduces gradient analysis and feature attribution methods to clarify the contribution of each modality and key feature to the prediction results, solving the "black box" problem of traditional models and providing interpretable evidence for clinical decision-making. It designs a scientific comorbidity burden index calculation method and risk grading mechanism, integrating multi-dimensional outputs such as incidence probability, median onset time, and imaging feature confidence levels to achieve comprehensive and refined risk assessment. An optimizer and learning rate scheduling strategy ensure model training stability and convergence efficiency, with each technical link forming a closed loop. These designs not only effectively overcome the shortcomings of existing technologies in data integration, task adaptation, and interpretability, but also achieve full automation from data processing to risk assessment, significantly improving the scientific rigor, accuracy, and clinical applicability of comorbidity prediction, providing strong support for the development of personalized intervention plans.

[0054] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," "link," and "fix" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0055] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various equivalent changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for predicting the risk of chronic disease comorbidity based on multimodal multi-task learning, characterized in that, Includes the following steps: S1. Collect multimodal patient data, including physiological time-series data, biochemical time-series data, RGB three-channel images of pathological slides, and three-dimensional medical images; S2. Construct a patient comorbidity label vector, where different elements in the label vector correspond to the disease status identifiers of different chronic diseases; S3. Perform feature encoding processing on the different modalities of data, extracting temporal correlation features of physiological time-series data through a temporal encoding network, extracting labeling features of biochemical time-series data through one-dimensional convolution and global pooling operations, and extracting high-level semantic features of pathological slide images and three-dimensional medical images through a pre-trained deep convolutional network; S4. Map the features encoded by different modalities to a shared subspace, and calculate the modality through a cross-attention mechanism. Intermodal interaction weights are introduced, and learnable gating parameters are used to control the information flow between modalities. Cross-modal interaction features and original modal features are aggregated to obtain fusion features; S5, a multi-task shared representation layer and multiple task calibration output layers are constructed based on the fusion features. The model is trained end-to-end through a joint loss function, which includes classification loss, auxiliary consistency loss, and regularization loss; S6, the model parameters are optimized using an optimizer and learning rate scheduling strategy. The contribution of different modalities and features to the prediction results is calculated through gradient analysis and feature attribution methods. The predicted values ​​of the incidence probability of different chronic diseases, median onset time, image feature confidence scores, and comorbidity burden index are output, and risk classification is performed based on the comorbidity burden index.

2. The method for predicting the risk of chronic disease comorbidity based on multimodal multi-task learning according to claim 1, characterized in that, The temporal correlation feature encoding of the physiological time series data in S3 adopts the following model: ,in Let be the weight matrix at time t, and BiLSTM be a bidirectional long short-term memory network. for The hidden state at all times The data represents the physiological time series data at time t. For bias terms, It is the Sigmoid activation function. This is the temporal attention weight vector. For time steps, This represents the final temporal correlation feature.

3. The method for predicting the risk of chronic disease comorbidity based on multimodal multi-task learning according to claim 1, characterized in that, The intermodal interaction weights in S4 are calculated using the following model: ,in Let m be the query vector for the m-th mode. Let n be the key vector of the nth mode. This is the gradient adjustment coefficient. Let be the gradient of the nth modal feature with respect to the mth modal feature. For feature dimension, For gating adjustment parameters, The first Modal projection characteristics This is the intermodal interaction weight matrix.

4. The method for predicting the risk of chronic disease comorbidity based on multimodal multi-task learning according to claim 1, characterized in that, The joint loss function in S5 is specifically as follows: ,in For loss weighting coefficients, For classifying losses, To mitigate consistency loss, For regularization loss, , These are shared representations corresponding to different modalities. It is the square of the Frobenius norm.

5. The method for predicting the risk of chronic disease comorbidity based on multimodal multi-task learning according to claim 1, characterized in that, The S6 comorbidity burden index is calculated using the following model: ,in Let the weight of health loss for the kth chronic disease be denoted as . The first The incidence rate of certain chronic diseases, Let represent the correlation strength between diseases k and j. The parameter is used to adjust for the probability difference between diseases k and j. For the set of all association strengths, This represents the number of types of chronic diseases.

6. The method for predicting the risk of chronic disease comorbidity based on multimodal multi-task learning according to claim 1, characterized in that, The learning rate scheduling in S6 adopts the following model: ,in Let be the learning rate at time t. These are the minimum and maximum learning rates, respectively. This represents the current number of training steps. Total training steps This is the attenuation adjustment coefficient.

7. The method for predicting the risk of chronic disease comorbidity based on multimodal multi-task learning according to claim 1, characterized in that, S3 includes the following sub-steps: S31, encoding the physiological time-series data into temporal features, capturing the dependencies between consecutive time steps through a bidirectional recurrent network, introducing a temporal attention mechanism to assign differentiated weights to features at different time steps, and aggregating the weighted hidden states to obtain the physiological time-series feature vector; S3 2. Perform one-dimensional convolution on biochemical time-series data, extract local features using convolution kernels of specified size, introduce nonlinear transformation through the ReLU activation function, and perform global max pooling after multiple layers of convolution to compress the feature dimension and retain the calibration information to obtain the biochemical feature vector; S33. Input the RGB three-channel image of the pathological slide into a pre-trained ResNet-50 network, and gradually extract the low-level visual features and high-level semantic features of the image through the network's convolutional layers and pooling layers, and truncate the feature map before the fully connected layer of the network as the pathological image feature vector; S34. Input the three-dimensional medical image into the 3DResNet network, capture the feature correlation in the spatial dimension through three-dimensional convolution kernels, and extract the deep structural features of the image through multi-scale feature fusion and downsampling operations, and output a three-dimensional image feature vector consistent with other modal dimensions.

8. The method for predicting the risk of chronic disease comorbidity based on multimodal multi-task learning according to claim 1, characterized in that, The S4 includes the following sub-steps: S41, performing layer normalization processing on the features encoded in different modalities, and mapping the modal features of different dimensions to a shared subspace of the same dimension through a fully connected layer to obtain the standardized modal projection features; S42, construct query vector and key vector for the projection features of each pair of modalities respectively, calculate the initial interaction weights by dot product operation, and scale by dividing by the square root of the feature dimension to obtain the cross-attention matrix before normalization; S43, concatenate the projection features of the pair of modalities, input them into the gating network and generate gating weights through the Sigmoid activation function, and multiply the weights element-wise with the product of the cross-attention matrix and the value vector to control the intensity of cross-modal information transmission; S44 sums all cross-modal interaction features, performs residual connections with the original projection features of different modalities, and obtains the final multimodal fusion feature vector after layer normalization.

9. The method for predicting the risk of chronic disease comorbidity based on multimodal multi-task learning according to claim 1, characterized in that, S5 includes the following sub-steps: S51, inputting multimodal fusion features into a shared fully connected layer, performing a nonlinear transformation using the ReLU activation function, and extracting a high-level abstract shared representation that can support multiple prediction tasks; S52, constructing an independent task labeling output layer for each chronic disease prediction task, mapping the shared representation to the output dimension of the task through the fully connected layer, and obtaining the original task score; S53, applying the Sigmoid activation function to the original scores of different tasks, converting the scores into disease probability values ​​between 0 and 1; S54, constructing a classification loss function to calculate the error between the predicted probabilities of different tasks and the true labels, designing an auxiliary consistency loss to constrain the consistency of different modal representations, adding L2 regularization loss to suppress model overfitting, summing the three types of losses according to their weights to obtain the total loss function, and updating all parameters of the model through backpropagation.

10. A method for predicting the risk of chronic disease comorbidity based on multimodal multitask learning according to any one of claims 1-9, characterized in that, This method is implemented through different units, including: a multimodal data integration unit, used to collect physiological time-series data, biochemical time-series data, pathological slide images, and 3D medical images from hospital information systems, laboratory information management systems, and medical image archiving systems, and establish a correspondence between patient data and comorbidity labels; a cross-modal feature encoding unit, connected to the output of the multimodal data integration unit, which performs targeted feature extraction on different types of data through a time-series encoding module, a convolutional pooling module, and a pre-trained deep convolution module, and outputs high-dimensional feature vectors of different modalities; and a dynamic gating fusion unit, connected to the output of the cross-modal feature encoding unit, which performs deep interaction and adaptive fusion of multimodal features through a feature projection module, a cross-attention calculation module, a gating adjustment module, and a feature aggregation module, and generates a unified patient table. The system comprises four main components: a multi-task learning modeling unit, connected to the output of the dynamic gating fusion unit, including a shared representation extraction layer and multiple task calibration output layers; a joint loss calculation module to construct a multi-objective optimization objective and complete end-to-end model training; an interpretability analysis unit, connected to the output of the multi-task learning modeling unit, calculating modal contribution through a gradient attribution module, outputting calibrated influencing factors through a feature importance assessment module, and presenting cross-modal feature correlations through a correlation visualization module; and a risk comprehensive assessment unit, connected to the output of the interpretability analysis unit, integrating the predicted incidence probabilities, onset times, and feature confidence scores of different chronic diseases, calculating the comorbidity burden index and performing risk classification, and outputting a structured prediction report. These different units are sequentially connected through a data transmission interface, enabling fully automated processing from data input to risk assessment.

Images