A method for intelligent assessment of rheumatoid arthritis activity based on double adaptive weighted multi-modal fusion

Abstract

The application discloses a kind of based on double self-adapting weighted multimodal fusion rheumatoid arthritis activity intelligent evaluation method, belong to medical information technology field.This method inputs clinical test, imageology, patient report and joint sign multimodal data, after abnormal value processing, missing value filling and standardization preprocessing, carries out modal internal feature purification enhancement;With modal reliability factor and bidirectional cross-modal attention weight, realize double self-adapting weighted depth fusion;Output disease activity grade and DAS28 score, and adopt global clinical correction factor to complete result calibration.The application integrates multi-dimensional data complementary value, realizes rheumatoid arthritis activity precision, objectification, forward-looking evaluation, provides reliable basis for clinical intervention.

Description

Technical Field

[0001] This invention belongs to the field of medical information technology, specifically relating to an intelligent assessment method for rheumatoid arthritis activity based on dual adaptive weighted multimodal fusion. Background Technology

[0002] Rheumatoid arthritis (RA) is a chronic, progressive autoimmune disease characterized by symmetrical polyarticular synovitis. Its core harm lies in the vicious cycle of synovial hyperplasia, inflammatory infiltration, and bone and cartilage destruction triggered by abnormal activation of the immune system. This process is significantly irreversible; patients can develop occult articular cartilage erosion within one year of onset. Without effective intervention, the disability rate can reach 30% within 5 years and exceed 50% within 10 years, ultimately leading to joint deformities, loss of function, and severely reduced quality of life. The core anchor point throughout the entire RA diagnosis and treatment process is the accurate assessment of "disease activity."

[0003] Disease activity is a core indicator for quantifying the degree of joint inflammation and the rate of tissue damage progression in rheumatoid arthritis (RA) patients. Essentially, it is a comprehensive representation of multi-dimensional information including "inflammatory marker levels, joint lesion morphology, patient subjective feelings, and degree of functional impairment." Its core significance lies in three aspects: First, it guides treatment plan development; clinicians need to adjust the intensity of intervention based on the activity level (e.g., maintaining basic treatment for low-activity patients, while strengthening antirheumatic therapy or combination therapy for moderate- and high-activity patients); second, it assesses treatment effectiveness; by dynamically monitoring changes in activity, it determines whether the current plan is effective (e.g., a significant decrease in activity after treatment suggests the plan is feasible); and third, it predicts prognostic risks; the longer the high activity persists, the higher the risk of joint bone destruction and disability, making it a key basis for prognostic assessment.

[0004] The pathological damage in rheumatoid arthritis (RA) is characterized by its irreversibility and progressive nature. If early, mild inflammation is not intervened in time, it can rapidly progress to synovial hyperplasia and bone erosion, significantly increasing the difficulty and effectiveness of subsequent treatment. Disease activity directly reflects the state of inflammatory progression. Accurately predicting a patient's activity level in the future (e.g., whether it will progress to high activity in one month) provides a "prospective intervention window" for clinicians, avoiding waiting until significant joint swelling and pain appear before adjusting treatment plans. This allows for "early detection, early intervention, and early control," slowing disease progression from its source. Furthermore, RA patients exhibit significant individual differences (e.g., different patients respond differently to medications and have varying rates of inflammatory progression). Accurate activity prediction enables "personalized diagnosis and treatment," avoiding drug waste or adverse reactions caused by indiscriminate medication use.

[0005] The current mainstream clinical method for assessing disease activity, represented by the DAS28 score (28 joint activity scores), suffers from three major flaws: ① The assessment dimensions are one-sided, relying only on two types of data: "number of joint swelling / tenderness + inflammatory markers (ESR / CRP)," ignoring key imaging pathological features such as synovial hyperplasia and bone erosion, as well as patients' subjective feelings such as pain and fatigue; ② Significant subjective bias: the number of joint swelling / tenderness depends on the physician's subjective judgment during physical examination, leading to significant differences in assessment results among different physicians, and patients' self-reported pain scores also exhibit individual subjective bias; ③ Insufficient timeliness and prospectivity: the DAS28 is a "retrospective" score (reflecting current activity) and cannot achieve "prospective prediction," and it has low sensitivity to early, mild inflammation, making it difficult to detect occult lesions. These flaws prevent accurate assessment and prediction of disease activity in clinical practice, thereby affecting treatment outcomes and increasing the risk of disability.

[0006] Accurate and objective prediction of disease activity can bring multiple clinical and social benefits: ① At the patient level, it significantly reduces the risk of disability, extends the lifespan of joints with normal function, and improves quality of life (e.g., early intervention in high-activity patients can reduce the incidence of bone erosion by more than 40%); ② At the clinical level, it improves the efficiency and accuracy of diagnosis and treatment, reduces ineffective treatments and adverse drug reactions, and optimizes the allocation of medical resources; ③ At the social level, it reduces the medical burden related to RA. Data shows that precise intervention can reduce the average annual medical expenditure of RA patients by 30%, while also reducing the loss of labor capacity due to disability.

[0007] Based on the aforementioned pain points and value, this invention proposes an intelligent assessment and prediction algorithm based on dual adaptive weighted multimodal data fusion (clinical laboratory, imaging, patient reports, and joint signs), which becomes the key to breaking through existing diagnostic and treatment bottlenecks. Summary of the Invention

[0008] The purpose of this invention is to propose an intelligent assessment method for rheumatoid arthritis activity based on dual adaptive weighted multimodal fusion. By integrating the complementary value of multidimensional data, it can achieve "precise, objective, and prospective" prediction of disease activity, providing a reliable basis for clinical intervention.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0010] First, this invention proposes an intelligent assessment method for rheumatoid arthritis activity based on dual adaptive weighted multimodal fusion, comprising the following steps:

[0011] Input patient multimodal data, including clinical laboratory data, imaging data, patient report data, and joint sign data;

[0012] Enter the patient's basic medical history, including the duration of the disease since the diagnosis of rheumatoid arthritis;

[0013] Preprocessing of patient multimodal data, including outlier handling, missing value imputation, and standardization;

[0014] Intramodal feature enhancement involves refining and enhancing the features of preprocessed data from each modality to obtain enhanced clinical features, imaging features, and patient-reported joint signs fusion features.

[0015] Cross-modal feature fusion is achieved by designing an attention-guided modal reliability-bidirectional attention dual adaptive weighted fusion module, introducing modal reliability factors and bidirectional cross-modal attention weights, to realize deep fusion of clinical features, imaging features, patient reports, and joint signs.

[0016] The system provides intelligent assessment and output of disease activity, including prediction results such as activity level and DAS28 score, and introduces a global clinical correction factor to correct the prediction results.

[0017] Secondly, this invention also proposes an intelligent assessment system for rheumatoid arthritis disease activity based on multimodal data fusion, comprising:

[0018] The patient multimodal data input module is used to input clinical laboratory data, imaging data, patient report data, and joint sign data.

[0019] The patient basic disease information input module is used to input data including the duration of the disease after the diagnosis of rheumatoid arthritis;

[0020] The preprocessing module for patient multimodal data is used for outlier handling, missing value imputation, and standardization of multimodal data.

[0021] The intramodal feature enhancement module is used to purify and enhance the features of the preprocessed data of each modality to obtain enhanced clinical features, imaging features, and patient-reported joint signs fusion features.

[0022] A cross-modal feature fusion module is used to design an attention-guided modal reliability-bidirectional attention dual adaptive weighted fusion module. It introduces modal reliability factors and bidirectional cross-modal attention weights to achieve deep fusion of clinical features, imaging features, patient reports, and joint signs.

[0023] The intelligent assessment and output module for disease activity is used to output the prediction results, namely the activity level and the DAS28 score, and introduces a global clinical correction factor to correct the prediction results.

[0024] In addition, this invention also proposes an intelligent assessment device for rheumatoid arthritis disease activity based on multimodal data fusion, comprising:

[0025] processor;

[0026] Memory used to store processor-executable instructions;

[0027] The processor is configured to invoke instructions stored in the memory to execute an intelligent assessment method for rheumatoid arthritis activity based on dual adaptive weighted multimodal fusion.

[0028] Finally, the present invention also proposes a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement an intelligent assessment method for rheumatoid arthritis activity based on dual adaptive weighted multimodal fusion.

[0029] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0030] 1. Introducing a modal reliability factor to construct an adaptive weighting mechanism: Unlike traditional fixed-weight fusion methods, this method calculates the modal reliability factor γ through single-mode prediction entropy. i This mechanism dynamically adjusts the fusion weights of multimodal data, including clinical data, imaging data, and patient reports. It can automatically identify and reduce the weight of distorted data (such as test errors and patient subjective biases), significantly improving the stability of evaluation results and addressing the industry-wide technical pain point of large overall evaluation deviations when a single modality fails in existing technologies.

[0031] 2. Design a cross-modal attention interaction module to uncover potential associations: Innovatively construct an attention-guided feature interaction structure, and calculate the bidirectional cross-modal attention weight β between modalities. ij This module captures the intrinsic correlations between heterogeneous data (such as the positive correlation between the inflammatory marker hs-CRP and ultrasound synovial thickness, and the correlation between bone erosion score and morning stiffness duration). Compared to fusion schemes that only perform simple feature stitching, this module can deeply mine the synergistic value of multi-dimensional data, resulting in a significant improvement in assessment accuracy compared to traditional fusion algorithms.

[0032] 3. Integration of supervised and unsupervised learning with dedicated clinical correction mechanism: On the one hand, it combines supervised learning (SVM classification / regression) to ensure assessment accuracy, and on the other hand, it adapts to some core data missing scenarios through unsupervised learning (GAN imputation, PCA feature screening); at the same time, it innovatively introduces a global clinical correction factor θ to dynamically fine-tune the results for the irreversible characteristics of RA course, especially improving the assessment accuracy of patients with long disease course, making the clinical adaptability far exceed that of general algorithms without specific correction. Attached Figure Description

[0033] Figure 1This is the overall architecture diagram of an intelligent assessment method for rheumatoid arthritis activity based on dual adaptive weighted multimodal fusion.

[0034] Figure 2 This is a flowchart of an intelligent assessment method for rheumatoid arthritis activity based on dual adaptive weighted multimodal fusion.

[0035] Figure 3 This is a multi-class ROC curve of an intelligent assessment method for rheumatoid arthritis activity based on dual adaptive weighted multimodal fusion. Detailed Implementation

[0036] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings.

[0037] Example 1

[0038] This invention proposes an intelligent assessment method for rheumatoid arthritis activity based on dual adaptive weighted multimodal fusion. This method is an intelligent assessment and prediction algorithm that integrates multimodal data (clinical laboratory tests, imaging, patient reports, and joint signs), combining the complementary value of multidimensional data to achieve "precise, objective, and prospective" prediction of disease activity, providing a reliable basis for clinical intervention. Please refer to [link / reference]. Figure 1 , 2 As shown below, the multimodal input data and the detailed intelligent evaluation method steps are described in detail.

[0039] I. Multimodal Input Data

[0040] The input multimodal data covers four core data sources, as follows:

[0041] 1. Modality 1 (clinical laboratory data, denoted as D1): includes inflammatory markers, autoantibodies, complete blood count indicators, metabolic indicators, and liver function indicators, specifically: erythrocyte sedimentation rate (ESR), high-sensitivity C-reactive protein (hs-CRP), rheumatoid factor (RF), white blood cell count (WBC), glucose (GLU), and alanine aminotransferase (ALT), for a total of 6 dimensions, i.e., D1 = [X 11 X 12 X 13 X 14 X 15 X 16 ] T .

[0042] The specific indicators and acquisition methods for each subcategory of clinical laboratory data are as follows:

[0043] ① Inflammatory markers: Erythrocyte sedimentation rate was measured using the Westergren method after venous blood collection, and high-sensitivity C-reactive protein was measured using immunoturbidimetric assay.

[0044] ② Autoantibodies: Rheumatoid factor was detected by enzyme-linked immunosorbent assay (ELISA) after venous blood collection.

[0045] ③ Blood routine indicators: After venous blood collection, EDTA anticoagulant tubes are injected, and white blood cell count is detected by a fully automated blood cell analyzer.

[0046] ④ Metabolic indicators: Serum was separated after venous blood collection and glucose was detected using a fully automated biochemical analyzer.

[0047] ⑤ Liver function indicators: After venous blood collection, serum was separated and alanine aminotransferase was detected by a fully automated biochemical analyzer.

[0048] 2. Modality 2 (Imaging data, denoted as D2): Includes joint ultrasound data and X-ray data, specifically: synovial thickness (ST), joint effusion grade (JE), synovial blood flow signal (PS), and bone erosion score (GE). After feature extraction, a four-dimensional feature vector is obtained, i.e., D2 = [X... 21 X 22 X 23 X 24 ]ᵀ.

[0049] The methods for acquiring imaging data are as follows:

[0050] ① Joint ultrasound data: The ultrasound physician operates a color Doppler ultrasound machine to scan the patient's wrists, metacarpophalangeal joints, proximal interphalangeal joints and other joints that are prone to RA. Two-dimensional grayscale images (used to measure synovial thickness ST and joint effusion grade JE) and color Doppler images (used to assess synovial blood flow signal PS) are acquired and the images are stored in DICOM format.

[0051] ② X-ray data: The radiologist takes X-ray images of the patient's hands and wrists in the anteroposterior position, with the tube voltage set to 40-60kV and the tube current to 50-100mA. The images are also stored in DICOM format.

[0052] First, the Region of Interest (ROI) was segmented using professional image processing software (MATLAB). For joint ultrasound images, the synovium, joint cavity, and blood flow signal areas were segmented; for X-ray images, the joint bone tissue and erosion foci were segmented. Then, multiple core features were extracted from the segmented ROIs: synovial thickness (ST), joint effusion grade (JE), synovial blood flow signal (PS), and bone erosion score (GE). Finally, all extracted features were concatenated to form a 4-dimensional feature vector (D2).

[0053] 3. Modal 3 (Patient Reported Data, denoted as D3): Includes Visual Analogue Scale (VAS), Health Assessment Questionnaire (HAQ), number of swollen joints (SJC28), and number of tender joints (TJC28), totaling 4 dimensions, i.e., D3 = [X 31X 32 X 33 X 34 ] T .

[0054] The patient report data is obtained in the following ways:

[0055] ① Subjective rating type (Visual Analog Scale (VAS), Health Assessment Questionnaire (HAQ): After medical staff explain the scoring rules to the patient in detail, the patient fills out the questionnaire independently during outpatient visits or hospitalization through a paper questionnaire or electronic questionnaire system (such as the hospital patient follow-up platform). The VAS score (pain, fatigue) is marked by the patient on a scale of 0-10 (0 points for no pain / fatigue, 10 points for the most severe). The HAQ calculates the total score through the ability assessment of 8 dimensions (dressing, washing, etc.).

[0056] ② Joint counting (SJC28, TJC28): Rheumatologists will conduct a physical examination of the patient's joints according to the 28-joint assessment criteria, count and record the number of swollen SJC28 joints and tender TJC28 joints.

[0057] 4. Modal 4 (Joint Signs Data, denoted as D4): Includes joint range of motion (ROM) and morning stiffness duration (MSD), totaling two dimensions, i.e., D4 = [X... 41 X 42 ] T .

[0058] The methods for obtaining joint vital signs data are as follows:

[0059] ① Range of motion (ROM): Medical staff use a goniometer to measure the flexion, extension, and rotation angles of key joints such as the wrist, elbow, and knee. Each joint is measured three times and the average value is recorded as the specific angle value (unit: degrees).

[0060] ② Morning stiffness duration MSD: The patient reports the onset time and complete relief time of joint stiffness after waking up in the morning. Medical staff calculate the difference between the two and record it (unit: minutes). If the stiffness is not completely relieved, it is recorded as "continued until the time of visit" and the specific duration is marked.

[0061] II. Patient's Basic Medical History

[0062] Patient baseline disease information, including the duration of rheumatoid arthritis after diagnosis, serves as a fundamental correlation feature throughout the entire process. It is used for screening similar samples to fill missing values ​​in clinical laboratory data and for calculating coefficients in subsequent clinical calibration stages. The method for obtaining patient baseline disease information is as follows:

[0063] ① Basic information collection: Review the patient's electronic medical records, outpatient and inpatient medical records to determine the date of the first diagnosis of rheumatoid arthritis.

[0064] ② Duration calculation and recording: Duration of disease = Date of this hospitalization / outpatient assessment - Date of the first diagnosis of rheumatoid arthritis, accurate to the month, and recorded in "years"; and according to the needs of subsequent clinical calibration, the disease course grade (≤1 year, 1-2 years, >2 years) is marked simultaneously to provide standardized data support for subsequent missing value filling and clinical outcome calibration.

[0065] III. Methods and Steps

[0066] Step 1: Multimodal data preprocessing

[0067] After acquiring patients' multimodal data, it is necessary to eliminate the dimensional differences, outlier interference, and missing value effects of heterogeneous data to lay the foundation for fusion.

[0068] 1.1 Outlier Handling

[0069] Adopting the improved 3 The criteria identify outliers and truncate values ​​that are outside the acceptable range:

[0070]

[0071] in, Let be the original value of the j-th index in the i-th mode. This is the mean of the indicator. The standard deviation of this indicator. This is the value after outlier handling.

[0072] 1.2 Missing value imputation

[0073] To address the missing characteristics of different modalities of data, a layered imputation strategy was adopted to adapt to the distribution characteristics of each modality of data, ensuring that the imputed data does not deviate from the original clinical and imaging features, as detailed below:

[0074] Clinical laboratory data D1: Imputation was performed using the mean of similar samples within the modality. The core logic is as follows: using the clinical laboratory indicators used in this invention as key matching features, indicators with strong correlation to the missing features (ESR, hs_CRP, RF, WBC, GLU, ALT) are prioritized to quantify the differences between samples. Within the D1 modality, the five other non-missing valid samples with the highest similarity to the missing sample features are selected, and imputation is completed using the arithmetic mean of their corresponding missing indicators. This method is suitable for the continuity and correlation characteristics of clinical laboratory data, can preserve the correlation of data within the modality to the greatest extent, avoid the interference of extreme values ​​and irrelevant samples on the imputation results, and ensure that the imputed values ​​truly reflect the patterns of clinical laboratory data, laying a solid foundation for subsequent feature enhancement within the modality.

[0075] Imaging data D2 / Patient report data D3 / Joint sign data D4: An imputation method based on generative adversarial network (GAN) is adopted. Through adversarial training between the generator and the discriminator, the generator learns and fits the true distribution of each modality of data, thereby generating missing values ​​consistent with the features of the original data, adapting to the characteristics of each modality of data to avoid feature distortion.

[0076] Patient baseline disease information: The median of baseline similar samples was used for imputation. If the patient's disease information was missing, non-missing samples with highly similar baseline characteristics were selected based on gender, age, and core inflammatory markers (erythrocyte sedimentation rate, high-sensitivity C-reactive protein, rheumatoid factor). The median of the disease course of this group of samples was used to complete the imputation, ensuring the clinical authenticity and distribution consistency of the disease course data and meeting the needs of subsequent similar sample screening and result calibration.

[0077] 1.3 Standardization Processing

[0078] Z-score normalization is used to map all preprocessed data to the same scale:

[0079]

[0080] in, The standardized index values ​​ensure that the data from each modality are comparable.

[0081] This step only performs Z-score standardization on the four modal data D1-D4. The patient's basic disease course is an independent clinical baseline attribute, and a hierarchical form is used for subsequent similar sample matching and result calibration. It does not participate in the dimensional unification and feature transformation of the modal data.

[0082] Step 2: Intramodal Feature Enhancement

[0083] Feature purification and enhancement are performed on the standardized data for each modality to improve the representation capability of single-modal data.

[0084] 2.1 Enhanced D1 features in clinical laboratory data

[0085] Principal component analysis (PCA) combined with mutual information was used to screen core features.

[0086] (1) Calculate the covariance matrix Σ1 of D1 and solve for the eigenvalues ​​λ. 11 ≥λ 12 ≥...≥λ 1m and the corresponding eigenvector ξ 11 ,ξ 12 ,...,ξ 1m .

[0087] (2) Select the first t principal components with a cumulative contribution rate ≥ 85% and construct the feature matrix P1 = [ξ 11 ,ξ 12 ,...,ξ 1t ].

[0088] (3) Calculate the enhanced clinical feature vector The dimension is t×1.

[0089] 2.2 D2 Feature Enhancement of Imaging Data

[0090] Convolutional Neural Networks (CNNs) are used to extract deep features, and attention mechanisms are combined to enhance features in key regions.

[0091] Let the output of the last fully connected layer of the CNN be the feature vector of the original image. The dimension is s×1, and bidirectional cross-modal attention weights are introduced. If the dimension is s×1, then the enhanced image feature vector is:

[0092]

[0093] Where ⊙ represents the element-wise product, and the bidirectional cross-modal attention weights. Calculated using the sigmoid function:

[0094]

[0095] Where W2 and b2 are learnable parameters, and σ is the sigmoid activation function.

[0096] 2.3 Patient-reported D3 and joint sign data showed enhanced features in D4.

[0097] Multilayer perceptron (MLP) is used to fuse two types of data and enhance feature representation:

[0098]

[0099] in, express and The vectors are concatenated column-wise with a concatenation dimension of (p+q)×1. W3 and b3 are the corresponding bias vectors, with W3 having a dimension of r×(p+q) and b3 having a dimension of r×1. Both are learnable parameters of the model. This is the activation function for the rectified linear unit, used to introduce nonlinear characteristic transformations; The enhanced patient-reported fusion features of joint signs were defined as r×1.

[0100] Step 3: Cross-modal feature fusion

[0101] The design of the "attention-guided modal reliability-bidirectional attention dual adaptive weighted fusion module" achieves deep fusion of F1 (clinical), F2 (imaging), and F3 (patient report combined with joint signs). The core is to introduce a "modal reliability factor" to dynamically allocate the weight of each modality.

[0102] 3.1 Calculation of Modal Reliability Factor

[0103] Innovative definition of modal reliability factor This measure assesses the reliability of each modality of data, quantifying its reliability and predictive stability. , A larger value indicates more reliable modal data and lower prediction uncertainty. Its core innovation lies in calculating the prediction probability distribution entropy based on single-modal features, rather than traditional subjective or simple objective credibility evaluation. The calculation formula is:

[0104]

[0105] in, Enhanced features based on the i-th mode The probability distribution entropy for predicting disease activity y. The smaller the entropy value, the lower the uncertainty of the single-modal prediction result and the higher the reliability of the modal data. Represents the enhanced features based on the i-th mode. The probability of predicting disease activity y is calculated using the softmax function:

[0106]

[0107] in, Let be the prediction weight matrix for the i-th mode, with dimensions (C×d). i (C represents the disease activity level, such as C=4 for clinical remission / low / moderate / high activity; d) i For the enhanced features of the i-th mode (dimensions), used to analyze features Perform a linear transformation.

[0108] in, Let be the prediction bias vector for the i-th mode, with dimension (C×1), used to correct the offset after linear transformation. The core function is to transform the result of a linear transformation ( ), which are mapped to numerical values ​​that conform to a probability distribution, with each probability value ∈ [0,1], and the sum of the probabilities of all categories is 1.

[0109] 3.2 Bidirectional Cross-modal Attention Weight Learning

[0110] To uncover the intrinsic correlations (rather than sample similarity) between features of different modalities, a bidirectional cross-modal attention mechanism is constructed, and the bidirectional cross-modal attention weights are calculated using the following formula. , representing the degree of dependence of the i-th modality on the j-th modality, is fundamentally different from traditional single-direction attention, as it can simultaneously capture the inter-modal dependencies:

[0111]

[0112] Where d is the mean of the feature dimensions, used to eliminate the interference of differences in feature dimensions across different modalities on the calculation of bidirectional cross-modal attention weights; to avoid the influence of dimensional differences. is the inner product of the feature vectors of modality i and modality j, specifically used to quantify the intrinsic correlation between the features of the two modalities (rather than the characteristics of individual patient data). The larger the inner product value, the closer the feature association between the two modalities and the higher the degree of dependence. k is the modality index, with values ​​of 1, 2, and 3, corresponding to the clinical laboratory data modality, the imaging data modality, and the patient report and joint sign fusion modality, respectively, and is used to achieve weight normalization through summation of all modalities.

[0113] It should be added that, with Correspondingly, (The subscripts i and j are reversed) represent the dependence of the j-th mode on the i-th mode. The reversed subscript order and different physical meanings of the two subscripts together constitute a cross-modal bidirectional attention association, providing support for the subsequent final weight calculation; similarly, This indicates the degree of dependence of the j-th mode on the k-th mode. It is only used to replace the subscript symbol in subsequent summation calculations. Its calculation logic is always based on the inner product of modal features, rather than sample similarity or individual data features.

[0114] 3.3 Final fusion feature calculation

[0115] Innovative combination of modal reliability factor γ i (This application is original, differing from the credibility scores in existing technologies) and bidirectional cross-modal attention weights. (See ② Bidirectional cross-modal attention weight learning for definition, which represents the dependence of the j-th modality on the i-th modality.) Construct a dual adaptive weighted fusion mechanism to calculate the final weights of each modality. The core is to achieve a dual consideration of "modal credibility + global modal correlation", which differs from the traditional single-dimensional weight calculation method:

[0116]

[0117] in, This is the sum of bidirectional cross-modal attention weights from all modalities to the i-th modality, used to characterize the degree of attention the i-th modality receives in global association (this is the core application of bidirectional attention, distinct from the weight calculation of traditional unidirectional attention); in the formula... and All are modal reliability factors (independently designed in this application to quantify the confidence of modal self-prediction), and k is a dummy variable used for summation and iteration. and All are bidirectional cross-modal attention weights calculated based on the inner product of modal features; the only difference is the replacement of the subscript signs in the summation calculation. The denominator is a global normalization term, ensuring that the final weights satisfy... ,in ;

[0118] Final fused feature vector F 融合 for:

[0119] .

[0120] This fusion method, through dual adaptive weights (reliability factor + bidirectional attention), can automatically suppress the weight ratio of distorted modes and strengthen the contribution of key modes. Its core innovation lies in combining the reliability evaluation of the mode itself with the bidirectional dependency association across modes to construct a dual adaptive weighting mechanism, which is different from the traditional fusion method of single attention or single credibility weight.

[0121] Step 4: Intelligent assessment and output of disease activity

[0122] Based on fusion feature F 融合 An improved support vector machine (SVM) is used to classify or regress disease activity.

[0123] 4.1 Classification Task: Output Activity Level

[0124] Objective: To classify patients into four categories: clinical remission (R), low activity (L), moderate activity (M), and high activity (H). The improved SVM decision function is as follows:

[0125]

[0126] Where N is the number of training samples, For Lagrange multipliers, {1,2,3,4} corresponds to R, L, M, H. The subscript z is the training sample index, which is only used to traverse the training samples; This represents the fused feature vector obtained after multimodal fusion of the k-th training sample, and the fused feature vector of the current sample to be predicted. Correspondingly, K(·,·) is the radial basis kernel function, K(a,b)=exp(-||ab|| 2 / 2σ 2 ), where b is the bias term.

[0127] 4.2 Regression Task: Output DAS28 score

[0128] Objective: To directly output the specific DAS28 score (continuous value), using an SVM regression model.

[0129]

[0130] in, , Let be the regression coefficient, satisfying .

[0131] 4.3 Result Calibration

[0132] In response to the irreversible clinical characteristics of rheumatoid arthritis, an innovative disease course stratified targeted calibration mechanism was designed. Pre-trained clinical calibration rules were introduced to perform targeted linear calibration on the original predicted DAS28 scores. This mechanism is specifically designed to correct systematic prediction biases in different disease course subgroups, improving clinical adaptability. The calibration formula and execution rules are as follows:

[0133] (1) Calibration formula

[0134]

[0135] (2) Definition of fixed parameters

[0136] θ is a global clinical correction factor obtained through training with clinical data. This factor is used to uniformly regulate the calibration intensity. It is a fixed parameter during the model training phase and does not change with subsequent samples.

[0137] (3) Rules for the composition of correction items

[0138]

[0139] The baseline coefficient for disease duration is determined based on the length of the patient's disease: 0.1 for ≤1 year, 0.2 for 1-2 years, and 0.3 for >2 years.

[0140] Determination of sign coefficient

[0141] During the model training phase, the training set was divided into three subgroups according to the above-mentioned disease course classification criteria, and the average deviation between the original predicted DAS28 score and the true DAS28 score in each subgroup was calculated:

[0142] If the average deviation of a certain subgroup is greater than 0, it indicates that the model has systematically overestimated the group. The sign coefficient of the group is fixed at -1 (to correct the overestimation deviation in reverse).

[0143] If the average deviation of a subgroup is less than 0, it indicates that the model has systematically underestimated the subgroup. The sign coefficient of the subgroup is fixed at +1 (to positively correct the underestimation deviation).

[0144] If the mean bias of a subgroup is 0, it indicates that the model has no systematic bias towards that subgroup and no calibration is required; the sign coefficient for that subgroup is fixed at 0.

[0145] The sign coefficients corresponding to the three disease course subgroups are saved together with θ as fixed calibration parameters for the model and will not be changed.

[0146] (4) Test set and clinical sample execution method

[0147] For the test set samples and subsequent newly added clinical samples, it is only necessary to determine their disease course group. All calibration rules and parameters use the fixed results from the training phase, directly calling the sign coefficient, baseline disease course coefficient, and other parameters already determined during the training phase for that group. Substitute the values ​​into the calibration formula above to complete the calibration.

[0148] Example 2

[0149] To verify the accuracy, robustness, and clinical suitability of this method, a systematic test was conducted based on multi-center clinical data. The test process strictly followed the principle of "data stratification → multi-index evaluation → clinical calibration and verification → comparative experimental verification". The specific test results are presented below.

[0150] 1. Test Dataset Description

[0151] The test data came from RA patients admitted to the First Affiliated Hospital of Anhui University of Traditional Chinese Medicine (Anhui Provincial Hospital of Traditional Chinese Medicine) between 2023 and 2025. The patients met the ACR / EULAR 2010 RA classification criteria, and the disease activity grading strictly followed the grading criteria based on the DAS28 score in the European League Against Rheumatism (EULAR) 2019 Guidelines for the Management of Rheumatoid Arthritis and the Chinese Rheumatology Society 2024 Guidelines for the Diagnosis and Treatment of Rheumatoid Arthritis.

[0152] In the multimodal data, clinical laboratory data came from the hospital's Laboratory Information System (LIS), imaging data from the hospital's Picture Archiving and Communication System (PACS), and patient report data and joint sign data from the hospital's Electronic Medical Record System (EMR). The raw data from these multimodal sources were extracted for algorithm training. Additionally, disease progression data was also obtained from the hospital's EMR.

[0153] After excluding patients who did not meet the diagnostic information, had other related complications, or had incomplete multimodal data, the specific information is as follows:

[0154] Sample size: A total of 1200 patients were included, of which 350 were in clinical remission (DAS28 score < 2.6), 205 were low activity (2.6 ≤ DAS28 score ≤ 3.2), 295 were moderate activity (3.2 ≤ DAS28 score ≤ 5.1), and 350 were high activity (DAS28 score > 5.1), which met the constraint requirement of ≥200 samples for each activity category.

[0155] Data composition: All patients were able to provide complete data in all four modalities (clinical laboratory, imaging, patient reports, joint signs) and complete disease course data, with no missing core data.

[0156] Dataset partitioning: The dataset was randomly partitioned into a training set (840 cases), a validation set (120 cases), and a test set (240 cases) in a 7:1:2 ratio. During the partitioning process, the composition of disease activity levels and disease course distribution in each subset were kept consistent with the total set to avoid the impact of data distribution bias on model performance evaluation.

[0157] 2. Test Evaluation Metrics

[0158] Combining the algorithm's classification task (outputting activity level) and regression task (outputting DAS28 score), and taking into account relevant evaluation criteria and the clinical needs of rheumatoid arthritis diagnosis and treatment, the following evaluation index system is established:

[0159] 2.1 Evaluation Indicators for Classified Tasks

[0160] Common multi-classification metrics include: accuracy, weighted precision, weighted recall, and weighted F1 score. These metrics are weighted to adapt to multi-classification scenarios and to offset evaluation bias caused by the imbalance of sample sizes at different activity levels.

[0161] Key clinical endpoint: High-activity patient recall (Recall_H), specifically used to evaluate the model's ability to identify high-risk cases with high disease activity and reduce the probability of missing diagnoses of high-risk patients.

[0162] Overall discrimination index: Area under the multi-class AUC-ROC curve, used to quantify the model's overall ability to distinguish and differentiate the four groups of disease activity levels.

[0163] 2.2 Regression Task Evaluation Indicators

[0164] Error metrics: Mean Absolute Error (MAE), Mean Squared Error (MSE), and Root Mean Squared Error (RMSE), used to quantify the deviation between the predicted value and the actual DAS28 score.

[0165] Fit index: Coefficient of determination (R²) 2 R0 is used to measure the linear correlation between predicted and actual values. 2 The closer to 1, the better the fit.

[0166] Clinical error index: Percentage of samples with absolute error ≤ 0.3 (P 0.3 When the DAS28 score error is ≤0.3, there is no significant difference in clinical intervention programs, and this indicator directly reflects the clinical practical value of the algorithm.

[0167] 3. Test Results

[0168] 3.1 Classification task test results (activity level prediction)

[0169] The classification performance of this method on the test set is shown in Table 1. Each indicator can objectively reflect the classification accuracy, recognition comprehensiveness and clinical suitability of the model.

[0170] Table 1

[0171] Evaluation indicators numerical values Clinical significance explanation Accuracy 91.25% Nearly 92% of the samples were correctly classified, indicating high overall classification reliability. Weighted Precision 90.83% The classification results are highly accurate, with a low probability of misclassifying non-target levels as target levels. Weighted Recall 91.25% The system has a high degree of comprehensiveness in identifying patients with various activity levels, resulting in fewer missed diagnoses. Weighted F1 score 91.04% The combined performance of precision and recall is excellent, with no obvious bias. Recall rate of highly active patients (Recall_H) 93.14% The rate of missed diagnosis for high-risk patients was only 6.86%, which meets the needs of clinical risk control. Area under the AUC-ROC curve (multi-class AUC) 0.956 The algorithm has an extremely strong ability to distinguish between different activity levels.

[0172] See the corresponding multi-class ROC curve for the model. Figure 3 The curve corresponds to a multi-class AUC value of 0.956, further verifying that the model has a stable and reliable ability to distinguish between the four categories of clinical remission, low activity, moderate activity, and high activity.

[0173] Additional explanation: The confusion matrix statistics based on the classification results of the test set samples show that misclassified cases are mainly concentrated in the boundary area between low activity and medium activity levels, with a misclassification rate of 4.2%; the misclassification rate at the boundary between medium activity and high activity levels is only 2.1%, indicating that the model has higher accuracy in distinguishing between high-risk and non-high-risk cases, which matches the priority of clinical disease intervention.

[0174] 3.2 Regression Task Test Results (DAS28 Score Prediction)

[0175] This method performs numerical regression prediction of DAS28 scores on an independent test set. The various errors and fitting indices are shown in Table 2. The prediction bias is within the error tolerance range of clinical diagnosis and treatment, and the numerical fitting degree and prediction stability meet the actual use requirements.

[0176] Table 2

[0177] Evaluation indicators numerical values Clinical significance explanation Mean Absolute Error (MAE) 0.186 The average deviation between the predicted and actual scores was 0.186 points, which is below the clinical critical decision threshold. Mean Squared Error (MSE) 0.054 The prediction results show no significant extreme errors, and the model's numerical prediction stability is good. Root mean square error (RMSE) 0.232 The overall error index is at a low level, which can support the needs of precise clinical assessment. coefficient of determination (R 2 ) 0.893 The predicted scores and the actual scores have a strong linear correlation, and the overall fit is good. <![CDATA[Percentage of samples with absolute error ≤ 0.3 (P 0.3 ).]]> 94.58% The prediction errors in the vast majority of samples will not lead to changes in disease activity grading or clinical intervention protocols.

[0178] 3.3 Results after clinical calibration

[0179] Based on the disease course grouping rules, sign coefficients, and global clinical correction factor θ (optimized to θ=0.12 in this embodiment), determined during the training phase, targeted clinical adaptability calibration was performed on the original DAS28 score prediction results: the overall mean absolute error (MAE) of the regression task was further reduced from 0.186 to 0.162, and the percentage of samples with an absolute error ≤0.3, P... 0.3 The recall rate for high-activity classification tasks increased from 94.58% to 96.2%; the prediction bias of patients in different disease course subgroups was effectively suppressed and reduced, and the overall clinical fit and assessment robustness were significantly enhanced.

[0180] 4. Results

[0181] To adapt to different use cases (algorithm development, clinical diagnosis and treatment), the output results of this method are presented in a multi-dimensional and structured format, thus balancing professionalism and readability:

[0182] 4.1 Standardized Output of Results

[0183] Classification results: The output is standardized in JSON format, including the patient's unique ID, activity level (R / L / M / H, where R corresponds to clinical remission, L to low activity, M to moderate activity, and H to high activity), predicted probability for each level, and confidence score (0-1 range, with a confidence score ≥0.8 indicating high confidence). Taking patient ID: 400427006 as an example, after inputting the relevant multimodal data for this patient and processing it using this method, the output classification results are as follows:

[0184] {"patient_id": "400427006", "activity_level": "R", "probabilities":{"R": 0.86, "L": 0.07, "M": 0.05, "H": 0.02}, "confidence": 0.89}};

[0185] Regression results: Output in JSON format, including patient ID, predicted DAS28 score, corrected DAS28 score, and disease course correction description. Using this patient as an example, the regression results are as follows:

[0186] {"patient_id": "400427006", "predicted_DAS28": 2.28, "calibrated_DAS28": 2.26, "calibration_note": "Disease duration 1.0 year, correction factor 0.12×-0.1=-0.012"}}.

[0187] 4.2 Clinical report format (for doctors' reference during diagnosis and treatment)

[0188] Based on the output of classification and regression, this method transforms the results into structured text when presenting them to doctors, providing them with clear and intuitive data on the patient's rheumatoid arthritis disease activity. Taking patient ID 400427006 as an example, the following report is generated based on the classification and regression output:

[0189] Clinical report:

[0190] Key findings: Rheumatoid arthritis disease activity assessment results: clinical remission (DAS28 corrected score 2.26), with an assessment confidence level of 89%.

[0191] Intervention recommendations: It is recommended to maintain the current treatment plan, and to have ESR, CRP and joint ultrasound checked every 3 months to continuously monitor changes in disease activity and avoid relapse after discontinuation of medication.

[0192] In summary, this invention integrates the complementary value of multi-dimensional data to achieve "precise, objective, and prospective" prediction of disease activity, providing a reliable basis for clinical intervention and ultimately breaking the vicious cycle of RA "inflammation progression-tissue destruction-disability." Its clinical significance is mainly reflected in:

[0193] In terms of performance: the overall accuracy of multi-class classification exceeds 90%, and the average absolute error of DAS28 score prediction is less than 0.2. Its comprehensive performance is better than single-modal data models and traditional fusion solutions, and it has the performance foundation for clinical application.

[0194] In terms of clinical risk control: The recall rate of high-activity patients is >93%, which can be further improved after clinical calibration, effectively reducing the risk of missed diagnosis of high-risk patients and providing technical support for early intensive treatment and prevention of joint damage.

[0195] In terms of practical application: the prediction error of more than 94% of the samples is ≤0.3, and the prediction results are highly compatible with the clinical intervention decision threshold; at the same time, the method is compatible with some application scenarios where non-core data is missing, reducing the data collection threshold for primary healthcare institutions and meeting the conditions for large-scale clinical promotion.

Claims

1. A method for intelligent assessment of rheumatoid arthritis activity based on dual adaptive weighted multimodal fusion, characterized in that, Includes the following steps: Input patient multimodal data, including clinical laboratory data, imaging data, patient report data, and joint sign data; Enter the patient's basic medical history, including the duration of the disease since the diagnosis of rheumatoid arthritis; Preprocessing of patient multimodal data, including outlier handling, missing value imputation, and standardization; Intramodal feature enhancement involves refining and enhancing the features of preprocessed data from each modality to obtain enhanced clinical features, imaging features, and patient-reported joint signs fusion features. Cross-modal feature fusion is achieved by designing an attention-guided modal reliability-bidirectional attention dual adaptive weighted fusion module, introducing modal reliability factors and bidirectional cross-modal attention weights, to realize deep fusion of clinical features, imaging features, patient reports, and joint signs. The system provides intelligent assessment and output of disease activity, including prediction results such as activity level and DAS28 score, and introduces a global clinical correction factor to correct the prediction results.

2. The intelligent assessment method for rheumatoid arthritis activity based on dual adaptive weighted multimodal fusion as described in claim 1, characterized in that, In patient multimodal data: The clinical laboratory data D1 includes: erythrocyte sedimentation rate (ESR), high-sensitivity C-reactive protein (hs-CRP), rheumatoid factor (RF), white blood cell count (WBC), glucose (GLU), and alanine aminotransferase (ALT). The imaging data D2 includes: synovial thickness (ST), joint effusion grade (JE), synovial blood flow signal (PS), and bone erosion score (GE). The patient report data D3 includes: Visual Analogue Scale (VAS), Health Assessment Questionnaire (HAQ), number of joint swellings (SJC28), and number of joint tendernesses (TJC28). The joint vital signs data D4 include: range of motion (ROM) and duration of morning stiffness (MSD).

3. The intelligent assessment method for rheumatoid arthritis activity based on dual adaptive weighted multimodal fusion as described in claim 1 or 2, characterized in that, The preprocessing steps for patient multimodal data are as follows: ① Outlier handling Adopting the improved 3 The criteria identify outliers and truncate values ​​that are outside the acceptable range: in, Let be the original value of the j-th index in the i-th mode. This is the mean of the indicator. The standard deviation of this indicator. This is the value after outlier handling; ② Missing value imputation To address the missing characteristics of different modalities of data, a layered imputation strategy was adopted to adapt to the distribution characteristics of each modality of data, ensuring that the imputed data does not deviate from the original clinical and imaging features, as detailed below: Clinical laboratory data D1: Intramodal imputation was performed using the mean of similar samples. Clinical laboratory indicators were used as key matching features. Indicators with strong correlation to missing features, such as ESR, hs_CRP, RF, WBC, GLU, and ALT, were selected to quantify the differences between samples. Within the D1 modality, the five other non-missing valid samples with the highest similarity to the missing sample features were selected, and imputation was completed using the arithmetic mean of their corresponding missing indicators. Imaging data D2 / Patient report data D3 / Joint sign data D4: An imputation method based on generative adversarial network (GAN) is adopted. Through adversarial training between the generator and the discriminator, the generator learns and fits the true distribution of each modality of data, thereby generating missing values ​​consistent with the features of the original data, adapting to the characteristics of each modality of data to avoid feature distortion. Patient baseline disease information: Median of baseline similar samples was used for imputation. If patient disease information was missing, non-missing samples with highly similar baseline characteristics were selected based on gender, age, and core inflammatory markers. The median of the disease course of this group of samples was used to imputate the data, ensuring the clinical authenticity and distribution consistency of the disease data and meeting the needs of subsequent similar sample screening and result calibration. Core inflammatory markers include erythrocyte sedimentation rate (ESR), high-sensitivity C-reactive protein (HSR), and rheumatoid factor. ③ Standardized processing Z-score standardization was used to map all preprocessed clinical laboratory data D1, imaging data D2, patient report data D3, and joint sign data D4 to the same scale. in, The standardized index values ​​ensure that the data from each modality are comparable.

4. The intelligent assessment method for rheumatoid arthritis activity based on dual adaptive weighted multimodal fusion as described in claim 3, characterized in that, The intra-modal feature enhancement steps are as follows: ① Enhanced D1 feature in clinical laboratory data Principal component analysis (PCA) combined with mutual information was used to screen core features. (1) Calculate the covariance matrix Σ1 of D1 and solve for the eigenvalues ​​λ. 11 ≥λ 12 ≥...≥λ 1m and the corresponding eigenvector ξ 11 ,ξ 12 ,...,ξ 1m ; (2) Select the first t principal components with a cumulative contribution rate ≥ 85% and construct the feature matrix P1 = [ξ 11 ,ξ 12 ,...,ξ 1t ]; (3) Calculate the enhanced clinical feature vector The dimension is t×1; ② Enhancement of D2 features in imaging data Convolutional Neural Networks (CNNs) are used to extract deep features, and attention mechanisms are combined to enhance features in key regions. Let the output of the last fully connected layer of the CNN be the feature vector of the original image. The dimension is s×1, and bidirectional cross-modal attention weights are introduced. If the dimension is s×1, then the enhanced image feature vector is: Where ⊙ represents the element-wise product, and the bidirectional cross-modal attention weights. Calculated using the sigmoid function: Where W2 and b2 are learnable parameters, and σ is the sigmoid activation function; ③ Patient reports showed enhanced features in D3 and joint signs data in D4. Multilayer perceptron (MLP) is used to fuse two types of data and enhance feature representation: in, express and The vectors are concatenated column-wise with a concatenation dimension of (p+q)×1. W3 and b3 are the corresponding bias vectors, with W3 having a dimension of r×(p+q) and b3 having a dimension of r×1. Both are learnable parameters of the model. This is the activation function for the rectified linear unit, used to introduce nonlinear characteristic transformations; The enhanced patient-reported fusion features of joint signs were defined as r×1.

5. The intelligent assessment method for rheumatoid arthritis activity based on dual adaptive weighted multimodal fusion as described in claim 4, characterized in that, The steps for cross-modal feature fusion are as follows: ① Modal reliability factor calculation Define modal reliability factor This measure assesses the reliability of each modality of data, quantifying its reliability and predictive stability. , A larger value indicates more reliable modal data and lower prediction uncertainty. The calculation formula is: in, Enhanced features based on the i-th mode The probability distribution entropy for predicting disease activity y. The smaller the entropy value, the lower the uncertainty of the single-modal prediction result and the higher the reliability of the modal data. Represents the enhanced features based on the i-th mode. The probability of predicting disease activity y is calculated using the softmax function to ensure the reasonableness of the probability distribution. in, Let be the prediction weight matrix for the i-th mode, with dimensions (C×d). i ), used for features Perform a linear transformation; where C is the disease activity level number, C=4; d i For the enhanced features of the i-th mode The dimension; in, Let be the prediction bias vector for the i-th mode, with dimension (C×1), used to correct the offset after linear transformation; The core function is to transform the result of a linear transformation ( ), which are mapped to numerical values ​​that conform to a probability distribution, where each probability value ∈ [0,1] and the sum of the probabilities of all categories is 1; ② Bidirectional cross-modal attention weight learning To uncover the intrinsic correlations between features of different modalities, a bidirectional cross-modal attention mechanism is constructed, and the bidirectional cross-modal attention weights are calculated using the following formula. This represents the degree of dependence of the i-th mode on the j-th mode, and also captures the inter-modal dependencies: Where d is the mean of the feature dimensions, used to eliminate the interference of differences in feature dimensions across different modalities on the calculation of bidirectional cross-modal attention weights; to avoid the influence of dimensional differences. is the inner product of the feature vectors of modality i and modality j, specifically used to quantify the intrinsic correlation between the features of the two modalities. The larger the inner product value, the closer the feature association between the two modalities and the higher the degree of dependence. k is the modality index, with values ​​of 1, 2, and 3, corresponding to the clinical laboratory data modality, the imaging data modality, and the patient report and joint sign fusion modality, respectively, and is used to achieve weight normalization through full modality summation. ③ Final fusion feature calculation Combining modal reliability factor γ i and bidirectional cross-modal attention weights A dual adaptive weighted fusion mechanism is constructed to calculate the final weights of each modality. : in The sum of bidirectional cross-modal attention weights of all modes for the i-th mode is used to characterize the degree of attention the i-th mode receives in global association; in the formula... and These are all modal reliability factors, used to quantify the confidence of the mode's own predictions, and k is a dummy variable used for summation and iteration. and All are bidirectional cross-modal attention weights calculated based on the inner product of modal features; the only difference is the replacement of the subscript signs in the summation calculation. The denominator is a global normalization term, ensuring that the final weights satisfy... ,in ; Final fused feature vector F 融合 for: 。 6. The intelligent assessment method for rheumatoid arthritis activity based on dual adaptive weighted multimodal fusion as described in claim 5, characterized in that, Based on fusion feature F 融合 An improved support vector machine (SVM) is used to classify or regress disease activity prediction. The steps are as follows: ① Classification task: Output activity level Objective: To classify patients into four categories: clinical remission (R), low activity (L), moderate activity (M), and high activity (H); and to improve the decision function of the SVM as follows: Where N is the number of training samples, For Lagrange multipliers, {1,2,3,4} corresponds to R, L, M, H. The subscript z is the training sample index, which is only used to traverse the training samples; This represents the fused feature vector obtained after multimodal fusion of the k-th training sample, and the fused feature vector of the current sample to be predicted. Correspondingly, K(·,·) is the radial basis kernel function, K(a,b)=exp(-||ab|| 2 / 2σ 2 ), where b is the bias term; ② Regression task: Output DAS28 score Objective: Directly output the specific DAS28 score, using an SVM regression model. in, , Let be the regression coefficient, satisfying ; ③ Result calibration A pre-trained clinical calibration rule is introduced to perform targeted linear calibration on the original predicted DAS28 scores. This is specifically designed to correct systematic prediction biases in different disease course subgroups and improve clinical fit. The calibration formula and execution rules are as follows: (1) Calibration formula (2) Definition of fixed parameters θ is a global clinical correction factor obtained through training with clinical data. This factor is used to uniformly regulate the calibration intensity. It is a fixed parameter during the model training phase and does not change with subsequent samples. (3) Rules for the composition of correction items The baseline coefficient for disease duration is determined based on the length of the patient's disease: 0.1 for ≤1 year, 0.2 for 1-2 years, and 0.3 for >2 years. Determination of sign coefficient During the model training phase, the training set was divided into three subgroups according to the above-mentioned disease course classification criteria, and the average deviation between the original predicted DAS28 score and the true DAS28 score in each subgroup was calculated: If the average deviation of a certain subgroup is greater than 0, it indicates that the model has systematically overestimated the group. The sign coefficient of the group is fixed at -1 and used to correct the overestimation deviation in reverse. If the average deviation of a certain subgroup is <0, it indicates that the model has systematically underestimated the group. The sign coefficient of the group is fixed at +1 and used to positively correct the underestimation deviation. If the mean bias of a subgroup is 0, it indicates that the model has no systematic bias towards that subgroup and no calibration is required; the sign coefficient for that subgroup is fixed at 0. The sign coefficients corresponding to the three disease course subgroups are saved together with θ as fixed calibration parameters for the model and will not be changed. (4) Test set and clinical sample execution method For the test set samples and subsequent newly added clinical samples, it is only necessary to determine their disease course group. All calibration rules and parameters use the fixed results from the training phase, directly calling the sign coefficient, baseline disease course coefficient, and other parameters already determined during the training phase for that group. Substitute the values ​​into the calibration formula above to complete the calibration.

7. A smart assessment system for rheumatoid arthritis disease activity based on multimodal data fusion, characterized in that, include: The patient multimodal data input module is used to input clinical laboratory data, imaging data, patient report data, and joint sign data. The patient basic disease information input module is used to input data including the duration of the disease after the diagnosis of rheumatoid arthritis; The preprocessing module for patient multimodal data is used for outlier handling, missing value imputation, and standardization of multimodal data. The intramodal feature enhancement module is used to purify and enhance the features of the preprocessed data of each modality to obtain enhanced clinical features, imaging features, and patient-reported joint signs fusion features. A cross-modal feature fusion module is used to design an attention-guided modal reliability-bidirectional attention dual adaptive weighted fusion module. It introduces modal reliability factors and bidirectional cross-modal attention weights to achieve deep fusion of clinical features, imaging features, patient reports, and joint signs. The intelligent assessment and output module for disease activity is used to output the prediction results, namely the activity level and the DAS28 score, and introduces a global clinical correction factor to correct the prediction results.

8. A smart assessment device for rheumatoid arthritis disease activity based on multimodal data fusion, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method described in any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the method of any one of claims 1 to 6.

Images