Dynamic prediction method for rheumatoid arthritis patient based on multi-modal data fusion and deep learning

Abstract

The invention discloses a dynamic prediction method for a rheumatoid arthritis patient based on multi-modal data fusion and deep learning, and the method is characterized in that a cross-modal attention fusion module is adopted, and the module does not simply splice all features, but learns the mutual dependency relationship between different modal features, and is used for predicting the dynamic prediction of the rheumatoid arthritis patient. Generating an enhanced feature vector capable of representing the synergistic effect; then, the enhanced feature vector and each original modal feature are sent into a gating fusion unit together to adaptively screen and integrate the most critical information for the current prediction task to form a comprehensive'patient current state snapshot '; and finally, inputting a series of state snapshots arranged according to a time sequence into an upper-layer time sequence LSTM model, thereby realizing dynamic prediction of a future disease activity trajectory and a recurrence risk of the patient. According to the method, through a refined fusion strategy and time sequence dynamic deep modeling, the prediction accuracy and interpretability are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, specifically a dynamic prediction method for rheumatoid arthritis patients based on multimodal data fusion and deep learning. Background Technology

[0002] Rheumatoid arthritis (RA) is an autoimmune disease characterized primarily by erosive arthritis. The disease has a prolonged course and is prone to relapse. If left untreated and untreated, it can lead to joint deformities and loss of function, severely impacting patients' quality of life. The core of RA clinical management lies in controlling disease activity as early and consistently as possible through a "treat-to-target" strategy to prevent irreversible joint damage. High-quality disease monitoring and prospective prediction are crucial for developing individualized treatment plans, improving treatment success rates, and enhancing patient prognosis.

[0003] Clinical assessment of rheumatoid arthritis (RA) relies on multiple sources of information, including patient subjective reports, physician physical examinations, laboratory test results (such as C-reactive protein (CRP) and erythrocyte sedimentation rate (ESR), and imaging studies (such as X-rays, ultrasound, and MRI). Physicians use this comprehensive analysis to determine disease activity and adjust treatment plans. However, this traditional assessment model faces several challenges:

[0004] First, data assessment suffers from bias and lag. Clinical decisions often rely on "snapshot" data from a single follow-up visit, but RA is a chronic, dynamically changing disease where sudden deterioration (i.e., "relapse" or "flare-up") can occur between two follow-ups, making it difficult for traditional assessment models to provide effective early warnings. Furthermore, various data modalities (such as blood test indicators reflecting systemic inflammation, imaging findings reflecting local joint lesions, and questionnaires reflecting patient subjective feelings) are often interpreted in isolation, making it difficult to fully uncover their inherent complex interrelationships.

[0005] Secondly, data fusion analysis lacks depth and precision. Although doctors consider various information comprehensively, this fusion analysis relies heavily on individual clinical experience and is a qualitative, superficial fusion. The complex, non-linear, deep correlations between multimodal data, such as the intrinsic link between specific imaging features and future changes in laboratory indicators, are difficult to quantify and model effectively.

[0006] Finally, existing clinical tools lack prospective predictive capabilities. Current assessment methods primarily describe a patient's "current" state, failing to accurately predict disease progression over the coming weeks or months. Treatment selection often requires a trial-and-error process, potentially delaying effective disease control and increasing patient suffering and financial burden. While some studies have attempted to use machine learning to predict RA, most are based on single data sources or simple feature aggregation, failing to effectively address the aforementioned problems. Summary of the Invention

[0007] To address the aforementioned issues, this invention proposes a dynamic prediction method for rheumatoid arthritis (RA) patients based on multimodal data fusion and deep learning. This method deeply integrates multi-source heterogeneous data from patients, including imaging, clinical indicators, and medical records. Through an innovative hierarchical temporal network architecture, it can not only accurately assess the current state but also dynamically predict future disease trajectories, providing strong technical support for achieving precise, forward-looking, and personalized management of RA.

[0008] A dynamic prediction method for rheumatoid arthritis patients based on multimodal data fusion and deep learning includes the following steps:

[0009] Step 1: Obtain the patient's multimodal data, which includes at least imaging data, structured clinical indicator time series data, and unstructured medical record text data. Perform data preprocessing to construct a training dataset.

[0010] Step 2: Input the data of each modality into the corresponding modality-specific encoder network to extract the initial imaging features, initial clinical indicator features and initial medical record text features respectively;

[0011] Step 3: Enhance the interaction of the multiple initial features through a cross-modal attention module to learn and generate enhanced fusion features that characterize the intermodal synergistic effect;

[0012] Step 4: Input the enhanced fusion features and the initial features into a gated fusion unit to adaptively filter and integrate information to generate a fusion state vector that represents the patient's overall state at a specific time point;

[0013] Step 5: The multiple fused state vectors arranged in chronological order are combined to form a state sequence, and the sequence is input into an upper-level time series prediction model to capture the evolution pattern of the disease and output dynamic prediction results.

[0014] Step 6: Perform end-to-end training and optimization of the overall network model involved in the method based on a composite loss function to improve the accuracy and robustness of prediction;

[0015] Step 7: Train the overall network model built in steps 2, 3, 4, 5, and 6 so that it can generate dynamic predictions of the patient's future disease status based on the input multimodal data.

[0016] Furthermore, in step 1, acquiring and preprocessing the patient's multimodal data to construct a training dataset specifically includes the following steps:

[0017] (1) Collect imaging data, blood test results, rating scales and physician diagnosis records of the same patient at different follow-up time points through the hospital information system (HIS), image archiving and communication system (PACS), laboratory information system (LIS) and electronic medical record (EMR) system;

[0018] (2) Timestamp alignment of the collected data to ensure that all data points correspond to specific follow-up time points, and to screen or imput samples with severe data loss.

[0019] (3) Normalize or standardize the structured data, crop the region of interest (ROI) and unify the size of the image data, and segment, remove stop words and tokenize the medical record text to construct a formatted training dataset.

[0020] Furthermore, in step 2, the modal data is input into the corresponding modal-specific encoder network to extract their respective feature information, specifically including the following steps:

[0021] (1) Input imaging data (such as joint ultrasound images) into a convolutional neural network (CNN) encoder to extract spatial pathological features reflecting synovial hyperplasia, bone erosion, etc.

[0022] (2) Input the time series data of structured clinical indicators (such as the values ​​of CRP, ESR and DAS28 in previous years) into a recurrent neural network (RNN) encoder to extract time-dependent features that reflect the dynamic change trend of the indicators;

[0023] (3) Input unstructured medical record text data (such as doctors' outpatient records) into a pre-trained language model encoder based on the Transformer architecture to extract deep semantic features that reflect information such as patients' chief complaints and physical signs.

[0024] Furthermore, in step 3, the multiple initial features are interactively enhanced through a cross-modal attention module, specifically including the following steps:

[0025] (1) Construct a cross-modal attention module, which uses the features of one modality as a query and the features of one or more other modalities as keys and values ​​to calculate the association weights between modalities;

[0026] (2) It is preferable to adopt a multi-head attention mechanism, which enables the model to learn multiple complex correlation patterns between modalities in parallel from different representation subspaces;

[0027] (3) The calculated attention weights are applied to the value features and weighted summation is performed to generate one or more new fusion feature representations enhanced by other modal context information.

[0028] Furthermore, in step 4, information is adaptively filtered and integrated through a gating fusion unit, specifically including the following steps:

[0029] (1) Design a gated fusion unit that receives all the initial features generated in step 2 and the enhanced fusion features generated in step 3 as joint input;

[0030] (2) A gating vector is calculated based on the joint input using a small fully connected network and a sigmoid or softmax activation function. Each element of the vector represents the importance or pass rate of the corresponding information channel.

[0031] (3) Perform element-wise multiplication on the gate vector and the joint input features to achieve adaptive weighting and filtering of information, thereby suppressing redundant information, enhancing key information, and finally outputting a highly refined fusion state vector.

[0032] Furthermore, in step 5, a higher-level time-series prediction model is used to capture the evolutionary patterns of the disease and output dynamic prediction results. This specifically includes the following steps:

[0033] (1) Collect the fusion state vectors generated in step 4 at multiple consecutive time points (such as t-3, t-2, t-1, t) of the patient and arrange them into a state vector sequence in chronological order;

[0034] (2) Input the state vector sequence into an upper-level time series prediction model consisting of a long short-term memory network (LSTM) or its variant (such as GRU);

[0035] (3) The output layer of the time series prediction model decodes and generates a sequence of predicted values ​​for disease activity scores (such as DAS28) at one or more future time points, or the probability value of disease recurrence risk within a specific future window period.

[0036] Furthermore, in step 6, the overall network model is trained and optimized based on a composite loss function, specifically including the following steps:

[0037] (1) Construct a composite loss function for multi-task learning to simultaneously optimize different prediction objectives of the model;

[0038] (2) The composite loss function includes a regression loss term, preferably mean squared error (MSE) or smoothed L1 loss, which is used to measure the difference between the predicted disease activity trajectory and the actual trajectory.

[0039] (3) The composite loss function includes a classification loss term, preferably a binary cross-entropy loss, which measures the difference between the predicted recurrence risk probability and the actual recurrence situation (yes / no);

[0040] (4) The regression loss term and the classification loss term are combined by weighted summation, and all learnable parameters of the overall network model are updated end-to-end through backpropagation algorithm and optimizers such as Adam.

[0041] Furthermore, in step 7, the overall network model constructed from steps 2, 3, 4, 5, and 6 is trained and applied to actual prediction, specifically including the following steps:

[0042] (1) Input the multimodal raw data of the patient to be predicted at multiple time points in the present and past into the trained and optimized prediction model;

[0043] (2) The model automatically performs feature extraction in step 2, interaction enhancement in step 3, and gated fusion in step 4, generating a fusion state vector for each time point;

[0044] (3) The model automatically executes step 5, inputs the generated state vector sequence into the upper-level time series prediction model, and analyzes the dynamic evolution law;

[0045] (4) Output the dynamic prediction results of the patient's future disease trend in real time, including the disease activity trajectory curve and the probability of recurrence risk, and present them to the clinician.

[0046] Furthermore, the method also includes system deployment and application operation:

[0047] (1) After the model training is completed, it is deployed on the hospital’s internal server or cloud platform and securely connected with the hospital’s existing information systems such as HIS, PACS, and EMR.

[0048] (2) Provide a visual user interface to intuitively display the prediction results in the form of dashboards, trend charts and risk warning lights;

[0049] (3) Integrate an interpretability analysis module (such as based on SHAP or LIME algorithm) to highlight the key clinical indicators or features that contribute the most to the current prediction results, so as to enhance the transparency of the model decision-making process and help doctors understand it.

[0050] (4) Provides one-click generation and export of prediction reports to facilitate case discussions, patient communication and electronic medical record archiving.

[0051] Compared with the closest existing technology, the technical solution provided by this invention has the following beneficial effects:

[0052] Significantly improving the depth and accuracy of data fusion: Existing technologies often employ simple feature concatenation, resulting in "information piling up" that fails to uncover deep correlations between modalities. This invention's "cross-modal attention module" achieves a leap from "independent" to "interactive," proactively discovering and utilizing synergistic effects such as those between "textual descriptions" and "image representations." Building upon this, the "gated fusion unit" addresses potential redundancy and conflict issues in multimodal information through adaptive filtering. This two-stage strategy of "enhanced interaction first, then intelligent filtering" ensures that the final generated "fusion state vector" has higher information density and representational accuracy than simply concatenated features.

[0053] A breakthrough from "static point prediction" to "dynamic trajectory prediction": Most existing technologies rely on "snapshot" predictions based on single-point-of-time data, failing to reflect the disease's evolution. This invention constructs a time series of "fused state vectors" and models them using upper-level time series models (such as LSTM), essentially learning the "evolutionary dynamics" of disease states. This allows the invention not only to predict the state at a future point but also to provide a possible future disease activity trajectory curve, showcasing the continuous development trend of the disease to doctors, achieving a qualitative leap in predictive capabilities.

[0054] The predictive power and clinical applicability are enhanced: Because this invention enables a deeper understanding of the dynamic patterns of diseases, it can provide more reliable early warnings of both short-term sudden deterioration (relapse) and long-term trends. This forward-looking capability allows clinicians to shift from reactive, "after-the-fact" treatment to proactive, "prevention-oriented" management, providing strong quantitative decision-making support for achieving individualized "target-reaching treatment" strategies.

[0055] The system enhances the interpretability and credibility of the model: By integrating an interpretability analysis module, it not only provides a predicted "outcome" but also the "reasons," clearly identifying the key factors influencing the prediction. This significantly increases doctors' trust in the model's output and promotes the practical application of artificial intelligence technology in clinical decision-making.

[0056] Possessing strong systematization and engineering capabilities: This invention provides an end-to-end solution from data integration to result presentation, which can be seamlessly embedded into existing hospital information systems to realize automated prediction processes, and provides a user-friendly visual interactive interface and reporting functions, with broad clinical adaptability and potential for widespread application.

[0057] This invention aims to address the limitations of existing prediction models in terms of accuracy and timeliness when fusing multi-source heterogeneous data. These models fail to fully exploit deep correlations between modalities and lack sufficient capture of the temporal dynamics of diseases. The core of this invention lies in constructing a hierarchical and temporally enhanced fusion prediction network. This method first processes patient multimodal data in parallel, including using convolutional neural networks to extract lesion spatial features from imaging data, using a pre-trained BERT language model to extract semantic features from unstructured medical record text, and using recurrent neural networks to process the time-series features of structured clinical indicators. Its innovation lies in employing a cross-modal attention fusion module. This module doesn't simply concatenate all features; instead, it learns the interdependencies between features from different modalities, generating an enhanced feature vector that represents synergistic effects. This enhanced feature vector, along with the original modal features, is then fed into a gated fusion unit to adaptively filter and integrate the most critical information for the current prediction task, forming a comprehensive "snapshot of the patient's current state." Finally, a series of chronologically ordered "snapshots" are input into a higher-level temporal LSTM model, enabling dynamic prediction of the patient's future disease activity trajectory and relapse risk. This invention significantly improves the accuracy and interpretability of predictions through a refined fusion strategy and deep modeling of temporal dynamics. Clinically, this invention can assist physicians in shifting from "passive response treatment" to "proactive management," providing strong quantitative decision-making support for achieving individualized "target-reaching treatment" strategies for rheumatoid arthritis patients by providing early warnings of disease relapse risk and simulating possible outcomes under different interventions. Attached Figure Description

[0058] Figure 1 This invention provides a dynamic prediction method for rheumatoid arthritis patients based on multimodal data fusion and deep learning.

[0059] Figure 2 A schematic diagram of a multimodal data feature extraction network provided in an embodiment of the present invention;

[0060] Figure 3 A schematic diagram of a two-stage fusion core provided in an embodiment of the present invention;

[0061] Figure 4 A schematic diagram of the upper-layer time-series dynamic prediction module provided in an embodiment of the present invention;

[0062] Figure 5 This is a schematic diagram of the user interface of the result display module provided in an embodiment of the present invention. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] Step 1: Obtain the patient's multimodal data, which includes at least imaging data, structured clinical indicator time series data, and unstructured medical record text data. Perform data preprocessing to construct a training dataset.

[0065] Step 2: Input the data of each modality into the corresponding modality-specific encoder network to extract the initial imaging features, initial clinical indicator features and initial medical record text features respectively;

[0066] Step 3: Enhance the interaction of the multiple initial features through a cross-modal attention module to learn and generate enhanced fusion features that characterize the intermodal synergistic effect;

[0067] Step 4: Input the enhanced fusion features and the initial features into a gated fusion unit to adaptively filter and integrate information to generate a fusion state vector that represents the patient's overall state at a specific time point;

[0068] Step 5: The multiple fused state vectors arranged in chronological order are combined to form a state sequence, and the sequence is input into an upper-level time series prediction model to capture the evolution pattern of the disease and output dynamic prediction results.

[0069] Step 6: Perform end-to-end training and optimization of the overall network model involved in the method based on a composite loss function to improve the accuracy and robustness of prediction;

[0070] Step 7: Train the overall network model built in steps 2, 3, 4, 5, and 6 so that it can generate dynamic predictions of the patient's future disease status based on the input multimodal data.

[0071] Example 1

[0072] Figure 1 A flowchart of a dynamic prediction method for rheumatoid arthritis patients based on multimodal data fusion and deep learning, provided in an embodiment of the present invention, is shown below. Figure 1 Each step is explained in detail.

[0073] Step S110: Obtain multimodal data of patients, perform data preprocessing, and construct training dataset;

[0074] Furthermore, by interfacing with hospital information systems (HIS), picture archiving and communication systems (PACS), laboratory information systems (LIS), and electronic medical records (EMR), multimodal data of RA patients at multiple consecutive follow-up time points are collected.

[0075] Furthermore, ensure that the collected data has a clear timestamp and perform time alignment, linking data from different sources to the same follow-up time point. For a single follow-up time point, the dataset includes:

[0076] (1) Imaging data: DICOM files of high-frequency ultrasound examinations of the patient's bimanual joints at each follow-up point were exported from the PACS system, including both B-mode (grayscale) images and power Doppler (PDI) images.

[0077] (2) Structured clinical indicator time series data: Laboratory test results on the same day as the imaging examination were exported from the LIS and EMR systems, including but not limited to C-reactive protein (CRP), erythrocyte sedimentation rate (ESR), rheumatoid factor (RF), and anti-cyclic citrullinated peptide antibody (ACPA); as well as clinical assessment scale scores completed by physicians, including 22 numerical indicators such as the 28 Joint Disease Activity Score (DAS28), Health Assessment Questionnaire (HAQ), and Simplified Disease Activity Index (SDAI).

[0078] (3) Export the unstructured text of the doctor’s outpatient record corresponding to each follow-up from the EMR system, including the patient’s chief complaint, physical examination findings (such as joint swelling, tenderness count) and diagnostic opinions.

[0079] Furthermore, the collected data undergoes preprocessing, specifically as follows:

[0080] (1) Label definition: “Disease relapse” is defined as a binary label. If a patient’s DAS28 score increases by more than 1.2 within 3 months after a follow-up visit compared to the previous follow-up visit, or if there is a clear record of “disease activity”, “relapse” or need for an upgraded treatment plan (such as adding biologics) in the medical record, then the label at that time point is recorded as 1 (relapse), otherwise it is 0 (stable).

[0081] (2) Image Preprocessing: For B-mode and PDI images, experienced ultrasound physicians first annotated the regions of interest (ROIs) of the metacarpophalangeal joints (MCPs) and wrist joints. Subsequently, a pre-trained UNet model was used to automatically segment the ROIs in all images. The segmented ROI regions were scaled to 224×224 pixels, and the B-mode and PDI images were stacked as two channels. Image pixel values ​​were normalized by subtracting the mean of the ImageNet dataset and dividing by its standard deviation.

[0082] (3) Clinical indicator preprocessing: For the 22 numerical indicators, Z-score standardization (subtracting the mean and dividing the standard deviation) was first used. For missing values ​​in the data, if the missing rate was less than 10%, linear interpolation based on adjacent time points was used to fill the missing values; if it was greater than 10%, the patient sample was discarded.

[0083] (4) Text preprocessing: For medical record texts, data cleaning is first performed to remove formatting tags and meaningless symbols. Subsequently, the text is segmented and tokenized using the vocabulary of the ClinicalBERT model pre-trained on a large clinical text corpus (such as MIMIC-III). To adapt to the model input, the text sequence is padded or truncated to a maximum length of 512 tokens.

[0084] Step S120: Input the data of each modality into the corresponding modality-specific encoder network and extract their respective feature information.

[0085] For details, see Figure 2 The diagram shown is a schematic of a multimodal data feature extraction network provided in an embodiment of the present invention. The preprocessed modal data is input into parallel encoder branches:

[0086] (1) Imaging Encoder (21): A ResNet-50 pre-trained on ImageNet is used as the backbone network. This choice is based on its strong feature extraction capability and good performance-efficiency balance verified in image recognition tasks. Two-channel 224×224 ultrasound images are input into the network, and the parameters of all convolutional layers are fine-tuned during training. The feature map output from the last residual block is extracted and passed through a global average pooling layer to finally generate an initial 2048-dimensional imaging feature vector f for each image. img .

[0087] (2) Clinical indicator encoder (22): A bidirectional gated recurrent unit (GRU) network consisting of two stacked layers is used. The bidirectional structure is chosen to allow the model to utilize both past and future information (within the sequence) when encoding indicators at any given time point, thereby more accurately capturing trends. A 22-dimensional indicator sequence of length 4 (representing 4 consecutive time points) is input into the network, with 128 hidden units per GRU layer. Finally, the forward and backward hidden states of the last time step are extracted and concatenated to generate a 256-dimensional initial clinical indicator feature vector f. clin .

[0088] (3) Medical Record Text Encoder (23): The ClinicalBERT-based model was adopted. This model was chosen instead of the standard BERT because it was pre-trained on massive amounts of clinical text and has a deeper understanding of medical terminology, abbreviations, and sentence structures. A token sequence of length 512 was input into the model, and the output of the last hidden layer corresponding to its special start symbol [CLS] was extracted to obtain an initial medical record text feature vector f of 768 dimensions. txt .

[0089] Step S130: The image features from two perspectives are fused through a cross-modal attention module to obtain an enhanced 3D structural representation;

[0090] For details, see Figure 3 The diagram shown is a schematic of the two-stage fusion core provided in an embodiment of the present invention. This step corresponds to the cross-modal attention module (31):

[0091] (1) Linear projection: f txt (768 dimensions), f img (2048 dimensions), f clin (256 dimensions) are mapped to a unified 768-dimensional space through different fully connected layers (linear projection layers). The stitched f img and f clin The projection of f is used as Key K and Value V. txt The projection of is used as Query Q.

[0092] (2) Scaling Dot Product Attention: The formula for calculating the attention score is as follows Where d k It is the dimension of the key vector (768 in this case), divided by sqrt(d). k This is to prevent the gradient from being too small.

[0093] (3) Multi-head mechanism: To capture diverse associations, this embodiment adopts a 4-head multi-head attention mechanism. That is, Q, K, V are linearly projected again into 4 different subspaces with a dimension of 192 (768 / 4), attention is calculated independently in each subspace, and then the outputs of the 4 heads are concatenated and passed through a linear layer again.

[0094] (4) Finally, the module outputs a 768-dimensional text feature vector f enhanced with image and clinical indicator information. attn .

[0095] Step S140: Input the fused features into the gated fusion module to generate a fused state vector;

[0096] Specifically, this step corresponds to Figure 3 The gating fusion module (32) in the middle is designed to intelligently determine the selection and weight of each information path:

[0097] (1) Feature concatenation: Combine the four feature vectors f txt (768 dimensions), f img (2048 dimensions), f clin (256 dimensions), f attn (768 dimensions) are concatenated along the dimensions to form a joint feature vector f with a total dimension of 3840. concat .

[0098] (2) Gating signal calculation: f concat The input is a gated network consisting of a fully connected layer (3840-dimensional input, 3840-dimensional output) and a sigmoid activation function. Its output, g, is a 3840-dimensional gate vector, where each element is between (0,1) and represents the pass rate of the corresponding feature dimension. Mathematically, g = σ(W g *f concat +b g ).

[0099] (3) Gating operation: Gating vector g and joint feature vector f concat Performing element-wise multiplication (Hadamard product), i.e., f gated =g⊙f concat This operation dynamically amplifies important features (corresponding to dimensions with g values ​​close to 1) and suppresses minor or noisy features (corresponding to dimensions with g values ​​close to 0).

[0100] (4) Reduce to output: reduce f gated By passing through the last fully connected layer, its dimension is reduced from 3840 to 1024, resulting in the final fused state vector v. sThis vector is the most comprehensive and refined digital representation of the patient at that point in time.

[0101] Step S150: Input the fused state vector sequence into the upper-layer time series prediction model for dynamic prediction;

[0102] For details, see Figure 4 The diagram shown is a schematic of the upper-layer time-series dynamic prediction module provided in an embodiment of the present invention.

[0103] (1) Sequence construction: Repeat steps S120-S140 for the four most recent follow-up time points (t-3, t-2, t-1, t) of the patients in the set to obtain four 1024-dimensional fusion state vectors v. s (t-3),…,v s (t). Arrange them in chronological order to form a state sequence of shape (4, 1024).

[0104] (2) Temporal modeling: The sequence is input into an upper-level temporal prediction model consisting of a two-layer stacked LSTM network (40). The number of hidden units in each LSTM layer is 512. A Dropout layer with a rate of 0.3 is added between the two LSTM layers to prevent overfitting.

[0105] (3) Multi-task output (50). The 512-dimensional output hidden state of the last time step of the LSTM is fed into two independent output heads:

[0106] Regression head: A linear layer that maps 512-dimensional input to a 3-dimensional output, representing a prediction of the DAS28 score for each of the next 3 months.

[0107] Classification Head: A linear layer that maps the 512-dimensional input to a 1-dimensional output (logit), and then uses a Sigmoid function to convert it into a probability value between (0,1), representing the risk of disease recurrence within the next 3 months.

[0108] Step S160: Train and optimize the network based on the composite loss function.

[0109] To optimize both prediction tasks simultaneously, a composite loss function L is used. total =λ reg *λ reg +λ cls *L cls .in,

[0110] L regSmooth L1 Loss is employed because it combines the robustness of L1 loss to outliers with the differentiability of L2 loss near zero, making it well-suited for handling DAS28 scores that may have measurement errors.

[0111] L cls We employ a weighted binary cross-entropy loss. Since relapsed samples are typically far fewer than stable samples, we assign higher loss weights to the minority class (relapsed) samples based on the ratio of the two classes in the training set, thus forcing the model to focus more on accurately predicting relapsed cases.

[0112] Weight λ setting: Through cross-validation, λ is set in this embodiment. reg =0.4, λ cls =0.6, giving a slightly higher weight to the more important clinical early warning task (relapse risk prediction).

[0113] Step S170: Train and apply the overall model for prediction.

[0114] For details, see Figure 5 The diagram shown is a user interface diagram of the result display module provided in an embodiment of the present invention.

[0115] (1) Training Details: The AdamW optimizer was used, whose decoupled weight decay method helps improve the model's generalization ability. The initial learning rate was set to 1e-4, and a cosine annealing learning rate scheduler with a 10-epoch warm-up was used. The model was trained on a server equipped with 4 NVIDIA A100 GPUs, with a batch size of 16, for a total of 200 epochs. During training, the model performance was evaluated on the validation set (20% of the total data) after each epoch, and the model with the highest F1 score on the recurrence prediction task on the validation set was saved as the final model.

[0116] (2) Application and Inference: The final model will be deployed as a clinical decision support system. Doctors will interact with the system through a front-end interface (e.g., ...). Figure 5 (As shown) After selecting a patient, the system performs inference in the backend. The inference process takes an average of 1.2 seconds on a single A100 GPU. The prediction results are presented in the form of a visual dashboard (401), a trajectory graph (402), and a list of key factors (403), providing fast, intuitive, and interpretable support for clinical decision-making.

[0117] Through the above implementation methods, we can quickly and accurately generate dynamic predictions of a patient's future disease status based on the patient's complex multimodal historical data.

[0118] Any aspects of this invention not described in detail are well-known to those skilled in the art.

[0119] The technical means disclosed in this invention are not limited to those disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications are also considered within the scope of protection of this invention.

Claims

1. A dynamic prediction method for rheumatoid arthritis patients based on multimodal data fusion and deep learning, characterized in that, Includes the following steps: Step S1: Obtain the patient's multimodal data, which includes at least imaging data, structured clinical indicator time series data, and unstructured medical record text data. Perform data preprocessing to construct a training dataset. Step S2: Input the data of each modality into the corresponding modality-specific encoder network to extract the initial imaging features, initial clinical indicator features and initial medical record text features respectively; Step S3: The multiple initial features are interactively enhanced through a cross-modal attention module to learn and generate enhanced fusion features that characterize the intermodal synergistic effect; Step S4: Input the enhanced fusion features and the initial features into a gated fusion unit to adaptively filter and integrate information to generate a fusion state vector that represents the patient's overall state at a specific time point; Step S5: The multiple fused state vectors arranged in chronological order are combined into a state sequence, and the sequence is input into an upper-level time series prediction model to capture the evolution pattern of the disease and output dynamic prediction results. Step S6: Perform end-to-end training and optimization of the overall network model involved in the method based on a composite loss function to improve the accuracy and robustness of prediction. Step S7: Train the overall network model constructed from steps S2, S3, S4, and S5 so that it can generate dynamic predictions of the patient's future disease status based on the input multimodal data.

2. The dynamic prediction method for rheumatoid arthritis patients based on multimodal data fusion and deep learning according to claim 1, characterized in that, In step S1, the specific steps are as follows: Step 1: Collect images, blood test results, rating scales, and physician diagnostic records for the same patient through the Hospital Information System (HIS), Picture Archiving and Communication System (PACS), Laboratory Information System (LIS), and Electronic Medical Record (EMR) system; Step 2: Time-align the collected data to ensure that all data points correspond to specific follow-up time points and remove samples with severe data loss. Step 3: Normalize the structured data, crop and unify the size of the region of interest (ROI) in the image data, and perform word segmentation and tokenization on the medical record text to construct a formatted training dataset.

3. The dynamic prediction method for rheumatoid arthritis patients based on multimodal data fusion and deep learning according to claim 1, characterized in that, Step S2 specifically includes: Step 1: Input the imaging data into a convolutional neural network (CNN) encoder to extract spatial features that reflect the lesion condition; Step 2: Input the time-series data of structured clinical indicators into a recurrent neural network (RNN) encoder to extract time features reflecting the dynamic changes of the indicators; Step 3: Input the unstructured medical record text data into a pre-trained language model encoder based on the Transformer architecture to extract deep semantic features that reflect the patient's state description.

4. The dynamic prediction method for rheumatoid arthritis patients based on multimodal data fusion and deep learning according to claim 1, characterized in that, In step S3, the specific steps are as follows: Step 1: Construct a cross-modal attention module, which uses features from one modality as queries and features from another or more modalities as keys and values. Step 2: Employ a multi-head attention mechanism to compute attention weights in different subspaces in parallel, in order to capture the complex relationships between modalities from different perspectives; Step 3: Apply the calculated attention weights to the value features to generate one or more fusion features enhanced by contextual information from other modalities.

5. The dynamic prediction method for rheumatoid arthritis patients based on multimodal data fusion and deep learning according to claim 1, characterized in that, In step S4, the specific steps are as follows: Step 1: Design a gated fusion unit that receives all the initial features generated in step S2 and the enhanced fusion features generated in step S3 as input; Step 2: Calculate a gating vector using a small fully connected network and a sigmoid activation function. Each element of this vector has a value between 0 and 1, representing the importance of the corresponding information channel. Step 3: Perform element-wise multiplication of the gate vector with all input features to achieve adaptive weighting and filtering of information, and finally output a highly refined fusion state vector.

6. The dynamic prediction method for rheumatoid arthritis patients based on multimodal data fusion and deep learning according to claim 1, characterized in that, In step S5, the specific steps are as follows: Step 1: Collect patient data at multiple consecutive time points (e.g., ... The fusion state vector generated in step S4 is arranged into a sequence in chronological order. Step 2: Input the state vector sequence into an upper-level time series prediction model composed of a Long Short-Term Memory (LSTM) network or its variant; Step 3: The model outputs a predicted value for the disease activity score (such as DAS28) at one or more future time points, or a probability value for the risk of disease recurrence within a specific future window period.

7. The dynamic prediction method for rheumatoid arthritis patients based on multimodal data fusion and deep learning according to claim 1, characterized in that, In step S6, the specific steps are as follows: Step 1: Construct a composite loss function for multi-task learning to simultaneously optimize different prediction objectives of the model; Step 2: The composite loss function includes a regression loss term, such as mean squared error (MSE), which is used to measure the difference between the predicted disease activity trajectory and the actual trajectory. Step 3: The composite loss function includes a classification loss term, such as binary cross-entropy, which measures the difference between the predicted relapse risk probability and the actual relapse situation. Step 4: Combine the regression loss term and the classification loss term using a weighted summation method, and update all parameters of the overall network model using the backpropagation algorithm.

8. The dynamic prediction method for rheumatoid arthritis patients based on multimodal data fusion and deep learning according to claim 1, characterized in that, After step S7 is completed, the specific application steps are as follows: Step 1: Input the multimodal raw data of the patient to be predicted at the current and past time points into the trained and optimized prediction model; Step 2: The model automatically performs feature extraction in step S2, interaction enhancement in step S3, and gated fusion in step S4, generating a fusion state vector for each time point. Step 3: The model automatically executes step S5, inputting the generated state vector sequence into the upper-level time series prediction model; Step 4: Output the dynamic prediction results of the patient's future disease trajectory in real time and present them to the clinician.

9. A dynamic prediction method for rheumatoid arthritis patients based on multimodal data fusion and deep learning according to any one of claims 1 to 7, characterized in that, The method is further deployed as a system comprising: Step 1: The model is deployed on the hospital's server or cloud platform and integrated with the hospital's existing information system for data exchange; Step 2: Provide a visual user interface to display the forecast results in the form of dashboards, trend charts, and risk warnings; Step 3: Integrate an interpretability analysis module that can highlight the key indicators or features that contribute the most to the current prediction results, helping doctors understand the basis of the model's decision-making. Step 4: Provide the function of generating and exporting prediction reports to facilitate case discussions and communication with patients.

Images