Rheumatoid arthritis early-stage intelligent diagnosis method based on multi-modal medical image and deep learning

Abstract

The invention provides an early-stage intelligent diagnosis method for rheumatoid arthritis based on a multi-modal medical image and deep learning, and the method comprises the steps: carrying out the image cleaning and standardization, automatic bone joint positioning and segmentation, and quality control of a multi-modal original data set, and obtaining a joint slice library; performing intra-modal self-supervised pre-training and cross-modal alignment representation, and then performing focus level detection and quantification to obtain a lesion feature vector of each joint; performing joint diagram construction and quality perception multi-modal fusion to obtain fusion feature representation, and performing weak supervision multi-instance learning and multi-task loss calculation based on the fusion feature representation to obtain an uncalibrated model; and carrying out model calibration and explainable output to obtain an intelligent diagnosis model. According to the method, the diagnosis time point can be advanced to the reversible inflammation stage, interpretable evidence of patient-level decision and joint-level quantification is provided, the engineering feasibility of small samples and multi-center generalization is considered, and the method has high clinical transformation potential.

Description

Technical Field

[0001] This invention relates to the field of intelligent diagnosis technology for rheumatoid arthritis, and in particular to an early intelligent diagnosis method for rheumatoid arthritis based on multimodal medical imaging and deep learning. Background Technology

[0002] Rheumatoid arthritis (RA) is a chronic autoimmune disease. The body's immune system mistakenly attacks its own tissues, primarily targeting the synovium within the joints. This leads to persistent inflammation, joint pain, swelling, and morning stiffness, eventually causing cartilage and bone destruction, and may ultimately result in joint deformities and loss of function. RA is also a systemic disease, affecting not only the joints but also organs such as the heart, lungs, blood vessels, eyes, and skin.

[0003] Early diagnosis of rheumatoid arthritis (RA) can prevent joint damage, improve prognosis, increase remission rates, reduce complications, and improve quality of life. However, existing early diagnostic methods have the following drawbacks: 1) Strong data dependence: High-quality, consistently labeled large-scale medical data is difficult to obtain, and sharing is restricted by privacy and compliance constraints; labeling costs are high and require expert consensus. 2) Insufficient interpretability: Deep models are often considered "black boxes," making it difficult to clearly explain the decision-making basis to clinicians and patients. 3) Limited generalization ability: Performance degrades across hospitals, devices, ethnic groups, or different disease spectrum populations, and is susceptible to data bias. 4) Limited feature selection: Key biomarkers and temporal information may be overlooked, making it difficult to capture early atypical manifestations. 5) Easily confused with other diseases: Early RA symptoms overlap with osteoarthritis, gout, systemic lupus erythematosus, etc., posing a risk of misdiagnosis and missed diagnosis. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, the purpose of this invention is to provide an intelligent early diagnosis method for rheumatoid arthritis based on multimodal medical imaging and deep learning. This method solves the key challenges in multimodal fusion, data heterogeneity, generalization, and reliable output in early RA diagnosis. It can shift the diagnosis time point to the reversible inflammatory phase, provide patient-level decision-making and joint-level quantitative interpretable evidence, and take into account the engineering feasibility of small sample size and multi-center generalization, thus having high potential for clinical translation.

[0005] To achieve the above objectives, the present invention provides the following solution: an early intelligent diagnosis method for rheumatoid arthritis based on multimodal medical imaging and deep learning, comprising: Images and metadata are acquired to obtain a multimodal raw dataset. The multimodal raw dataset is then subjected to image cleaning and standardization, automatic bone and joint localization and segmentation, and quality control to obtain a joint slice library. Based on the joint slice library, intramodal self-supervised pre-training and cross-modal aligned representation are performed, followed by lesion-level detection and quantization to obtain the lesion feature vector of each joint; Based on the lesion feature vector, joint graph construction and quality-aware multimodal fusion are performed to obtain a fused feature representation. Based on the fused feature representation, weakly supervised multi-instance learning and multi-task loss calculation are performed to output a patient-level classifier and a joint-level regressor to obtain an uncalibrated model. Based on the uncalibrated model, model calibration and interpretable output are performed to obtain an intelligent diagnostic model.

[0006] Optionally, images and metadata are acquired to obtain a multimodal raw dataset. The multimodal raw dataset is then subjected to image cleaning and standardization, automatic bone and joint localization and segmentation, and quality control to obtain a joint slice library, including: Configure a DICOM router to collect raw medical images and basic information of ultrasound, MRI and X-ray in batches according to a unified process rule, and perform integrity and consistency checks on the raw medical images to obtain a multimodal raw dataset including raw ultrasound data, raw MRI data and raw X-ray data. For the raw ultrasound data, anti-aliasing, beam and depth normalization, PD signal standardization and spatial calibration operations are performed in sequence to obtain ultrasound standardized data packets. For the raw magnetic resonance data, N4 bias field correction, denoising, intensity normalization, resampling and coarse registration operations are performed sequentially to obtain a magnetic resonance normalized data packet; The raw X-ray data is subjected to the following operations in sequence: demeshing and scattering, histogram equalization, bone structure protrusion, scale unification and geometric correction, to obtain a standardized X-ray data package. Based on standardized data packets, a deep learning model is used to perform structural localization and segmentation of images, identify and mark key anatomical structures to obtain clear images, then extract key regions from the clear images to obtain a candidate slice set, and finally evaluate the quality of the candidate slice set to obtain a joint slice library.

[0007] Optionally, based on standardized data packets, a deep learning model is used to perform structural localization and segmentation on the images, identify and label key anatomical structures to obtain sharpened images, then key regions are extracted from the sharpened images to obtain a candidate slice set, and then the quality of the candidate slice set is evaluated to obtain a joint slice library, including: For the aforementioned standardized ultrasound data package, Keypoint-Detector is used to detect the articular surface and tendon sheath orientation, and Attention U-Net is used to segment the synovial thickness region and PD region to complete ultrasound localization and segmentation; For the aforementioned MRI standardized data package, the nnU-Net 3D segmentation model is used to simultaneously segment the carpal bones, distal metacarpals, proximal phalanges, joint cavity, and synovial thickening area. The three-dimensional region of each joint is cut out in the image to generate a 3D ROI block, thus completing the MRI segmentation. For the X-ray standardized data packet, the HRNet-W32 model is used to predict the metacarpal heads and phalangeal bases for key point localization. Adjacent key points are connected by lines, and a rectangular box is drawn around the connecting lines to obtain the joint cavity candidate region. Then, the U-Net++ model is used to segment the bone tissue of the joint cavity candidate region to complete the X-ray localization segmentation. The results of ultrasound localization and segmentation, magnetic resonance segmentation and X-ray localization and segmentation are integrated to obtain a clear image. Then, key regions are extracted from the clear image to obtain a candidate slice set. Based on the candidate slice set, quality dimensions and quality labels for different types of images are defined, and quality scores for different quality dimensions are output using an AI model. Then, Huber loss and RankLoss loss are combined to constrain the model and obtain slice quality scores. By comparing the quality score of the slices with a preset quality pass threshold, problem areas are marked and low-quality slices are screened out to obtain a joint slice library.

[0008] Optionally, based on the joint slice library, intra-modal self-supervised pre-training and cross-modal aligned representation are performed, followed by lesion-level detection and quantization to obtain the lesion feature vector for each joint, including: Based on the joint slice library, a masked autoencoder is used to learn 3D volumetric context, a contrastive learning model is used to enhance invariant features, and temporal consistency is used to learn the association between adjacent frames, thus completing intramodal self-supervised pre-training. Based on the joint slice library, positive and negative sample pairs are added for comparative learning, and a distance loss function is set to bring images of different modalities of the same joint closer together, resulting in an aligned cross-modal representation encoder. The Attention U-Net model is used for ultrasound lesion quantification, the 3D nnU-Net model is used for magnetic resonance lesion quantification, and the target detection model and distance transformation are used for X-ray lesion quantification. The uncertainty of lesion quantification is estimated by deep ensemble or MC Dropout. Then, for each joint, all modal features and lesion quantification results are concatenated to obtain the lesion feature vector.

[0009] Optionally, based on the lesion feature vector, joint map construction and quality-aware multimodal fusion are performed to obtain a fused feature representation. Based on the fused feature representation, weakly supervised multi-instance learning and multi-task loss are performed to output a patient-level classifier and a joint-level regressor, resulting in an uncalibrated model, including: Based on the lesion feature vector, the discrete joint vectors are organized into a graph structure including anatomical relationships, symmetry relationships, and propagation relationships. Based on the graph structure, the graph nodes are subjected to gating weighting and uncertainty-aware weighting to obtain the set of correction node inputs. Based on the set of inputs to the correction nodes, a graph neural network is used to enable mutual learning between nodes, the multi-head self-attention mechanism of the Transformer is used to enable interaction between nodes, and gating weights and prior knowledge are introduced to train and fine-tune the model to obtain joint-level fusion feature representations. Based on the fused feature representation, a supervision signal is defined and task modeling is performed to obtain a task head structure and a label dictionary. Based on the task head structure, multi-instance learning is used to converge the joint-level representation to the patient level in an interpretable attention manner, and quality weighting and uncertainty weighting are combined to automatically filter out evidence joints, thereby obtaining patient embeddings and joint-level attention weights. Define a multi-task loss, perform task optimization, and construct a model training strategy and validation metrics for training optimization to obtain a patient-level classifier head and a joint-level regression head. Then, encapsulate and version the patient-level classifier head and the joint-level regression head to output a patient-level classifier and a joint-level regressor, resulting in an uncalibrated model.

[0010] Optionally, the graph nodes of the graph structure are set as joint nodes, and the edges of the graph structure include adjacent edges representing adjacent joints of the same hand, symmetrical edges representing left and right joints with the same name, hierarchical edges representing the relationship between the distal and proximal ends, and optional cross-modal similar edges representing similarity connections.

[0011] Optionally, the task head structure includes patient-level tasks consisting of RA classification tasks and differential diagnosis tasks, and joint-level tasks consisting of RAMRIS regression tasks and lesion detection tasks. The label dictionary includes patient-level weak labels, joint-level strong labels, and joint-level pseudo-labels.

[0012] Optionally, the multi-task loss includes classification loss for optimizing patient-level classification tasks, regression loss for optimizing joint-level RAMRIS regression tasks, cross-modal consistency loss for maintaining consistency in predictions of the same joint from different modalities, symmetry prior loss for maintaining consistency in predictions of left-right symmetrical joints, domain adversarial loss for adapting to different data sources, and uncertainty regularization loss for controlling sample loss contributions.

[0013] Optionally, the calculation expression for the classification loss is: ; in, For cross-entropy, For patient-level classification heads, For patient-level fusion embedding, Patient-level truth category labels; The formula for calculating the regression loss is: ; in, Index for joint subsets. For segmented quadratic loss, For the return of the head, This represents joint-level fusion. For the corresponding truth value score, Joint-level reliable weights; The expression for calculating the cross-modal consistency loss is as follows: ; in, For joint indexing, For the set of modal pairs to be compared, The contribution of different modes to different joints, For the contrast loss with temperature, For modality For projection head, For modality For projection head, For the first Each joint in modality The encoding, For the first Each joint in modality The encoding; The expression for calculating the symmetric prior loss is as follows: ; in, This is a set of paired left and right joints with the same name. For symmetric constraint weights, This represents joint-level fusion. This is a symmetrical joint-level fusion representation; The expression for calculating the domain adversarial loss is as follows: ; in, For sample index, For domain discriminator, For the corresponding level features, For real domain tags; The expression for calculating the uncertainty regularization loss is as follows: ; in, All are sub-item weights. The mean uncertainty within the patient population. To measure the deviation between confidence level and success rate.

[0014] Optionally, based on the uncalibrated model, model calibration and interpretable output are performed to obtain an intelligent diagnostic model, including: Construct a calibration set including different disease stages and complete labels, use the calibration set to test the uncalibrated model, and calculate the uncalibrated baseline index and error distribution; Based on the uncalibrated baseline index and the error distribution, patient-level probabilistic calibration is performed through temperature scaling, Dirichlet calibration, and group calibration, and joint-level regression calibration is performed through piecewise equal probability regression and residual modeling to obtain calibration parameters for model calibration. A typical case feature library associated with known cases is constructed. Based on the typical case feature library, visualization or textual explanations are generated from four levels: case, region, joint, and lesion, to complete the interpretable output of the model and obtain the intelligent diagnostic model.

[0015] This invention discloses the following technical effects by providing an early intelligent diagnosis method for rheumatoid arthritis based on multimodal medical imaging and deep learning: 1. High-quality joint slice library: Through unified multimodal data acquisition and standardized processes, customized standardized pipelines for each modality, end-to-end automatic anatomical structure localization and segmentation, and an AI-driven quality control system, this achieves the following: 1) Solving the engineering challenges of "heterogeneous + noise + artifacts" in multimodal data from the source, significantly improving the usability and robustness of subsequent feature learning. 2) Automatic structure localization / segmentation is more stable in quality and more accurate in ROI under the synergistic constraints of the three modalities, significantly reducing manual intervention and error propagation. 3) Quality control introduces ranking and robust loss, improving the overall consistency and reproducibility of the "slice library" and reducing the interference of low-quality samples on model training.

[0016] 2. Lesion-level quantization enhances generalization ability: 1) Self-supervised learning maximizes the use of unlabeled images, alleviating the scarcity and high cost of medical data labeling and improving feature generalization ability. 2) Cross-modal alignment allows ultrasound, MRI, and X-ray to "come closer" to each other in the same joint dimension, making it easier to discover complementary relationships during subsequent fusion. 3) Lesion-level quantization provides structured and interpretable lesion evidence and reduces the negative impact of noise prediction on the system by using uncertainty weighting.

[0017] 3. Provide a consistent basis for judgment and prevent confusion with other diseases: 1) Graph structure fusion and prior constraints make the model more sensitive to "symmetry, proximity propagation, and hierarchical association," which is consistent with the clinical pattern of RA lesions in multiple joints of the hand being "symmetrical and propagable." 2) Quality / uncertainty perception avoids "bad data" dominating learning and improves robustness; MIL attention selects evidence joints, which are naturally interpretable, facilitating physician trust and clinical implementation. 3) Multi-task joint optimization incorporates patient-level diagnosis, joint-level quantification, cross-modal / symmetry consistency, and domain adaptation into the same framework, significantly improving generalization ability and transfer adaptability.

[0018] 4. Accurate Diagnosis and Interpretable Output: 1) Model probability and regression outputs achieve "consistent credibility," reducing overconfidence and underconfidence, and facilitating clinical risk communication and threshold setting. 2) Multi-level interpretive outputs connect "evidence-pattern-conclusion," improving transparency and auditability, and promoting clinical adoption.

[0019] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the method flow provided in an embodiment of the present invention; Figure 2 A flowchart for obtaining a joint slice library provided in an embodiment of the present invention; Figure 3 This is a flowchart illustrating the construction process of an uncalibrated model provided in an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] like Figure 1As shown, this invention provides an early intelligent diagnosis method for rheumatoid arthritis based on multimodal medical imaging and deep learning, comprising: Step 1, as follows Figure 2 As shown, images and metadata are collected to obtain a multimodal raw dataset. The multimodal raw dataset is then subjected to image cleaning and standardization, automatic bone and joint localization and segmentation, and quality control to obtain a joint slice library.

[0025] Step 1 includes: 1.1 Configure the DICOM router to collect raw medical images and basic information of ultrasound, MRI and X-ray in batches according to the unified process rules, and perform integrity and consistency checks on the raw medical images to obtain a multimodal raw dataset including raw ultrasound data, raw MRI data and raw X-ray data.

[0026] 1.2 For the raw ultrasound data, anti-aliasing, beam and depth normalization, PD signal standardization and spatial calibration operations are performed sequentially to obtain a standardized ultrasound data package.

[0027] 1) Anti-aliasing / Spot Removal (SRAD or NLM): Ultrasound images often contain speckle noise. Anti-aliasing makes the image edges smoother. SRAD (Anisotropic Diffusion Filter) or NLM (Non-Local Mean Filter) can specifically remove speckles while preserving tissue edges (such as the boundaries of tendons and blood vessels).

[0028] 2) Beam and depth normalization: Different ultrasound devices may have varying overall brightness and display tissue depth in their images, and text may even be superimposed on the images. This step standardizes the gain and depth scale, for example, ensuring all images display tissue at a depth of 5cm, and removes text through OCR recognition or template matching to avoid interference.

[0029] 3) PD signal standardization (Doppler blood flow signal processing): In ultrasound, PD (Power Doppler) refers to the color Doppler signal. Processing includes: extracting the color channels, converting to the HSV color space, and using a threshold to separate red blood flow pixels from the background; automatically adjusting the criteria for distinguishing blood flow from non-blood flow, filling in small gaps in blood flow areas, removing small noise, and generating a binary PD image and a histogram of blood flow intensity.

[0030] 4) Spatial calibration: Retain the millimeter / pixel scale, for example, 1 pixel corresponds to 0.1mm, to ensure that the actual size can be calculated from the image; if there is a probe tracking device, save this posture information simultaneously for easy subsequent alignment.

[0031] 1.3 For the original magnetic resonance data, N4 bias field correction, denoising, intensity normalization, resampling and coarse registration operations are performed sequentially to obtain a magnetic resonance normalized data package.

[0032] 1) N4 bias field correction: MRI images often show inconsistent brightness within the same tissue due to uneven magnetic fields from the equipment; for example, muscles on the left side may appear brighter than those on the right. This step uses algorithms to eliminate this low-frequency brightness unevenness, resulting in more consistent brightness within the same tissue.

[0033] 2) Noise reduction: MRI images may contain noise, similar to the static on an old television, which affects the observation of details. Rician-adaptive NLM is an intelligent denoising algorithm that can remove noise without blurring image details.

[0034] 3) Intensity normalization, z-score or Nyúl histogram standardization: The overall brightness range of MRI images can vary greatly between different patients (for example, image A's image is generally brighter, while image B's is generally darker). This step unifies the brightness sequentially.

[0035] z-score: Converts brightness values ​​into the degree of deviation from the average value, for example, setting the average brightness of all images to 0 and the standard deviation to 1.

[0036] Nyúl's method: to make the brightness distribution of different images as consistent as possible, ensuring that the same tissue is in a similar brightness range in different images.

[0037] 4) Resampling: A voxel is a pixel in a 3D image. Different MRI devices may have different voxel sizes, resulting in inconsistent proportions of the same area in the image. Resampling uses algorithms to adjust the voxel size across all MRI images to be uniformly between 0.5-0.7 mm. 3 Ensure that the three-dimensional dimensions are consistent between them.

[0038] 5) Coarse registration: Two MRI scans of the same patient may show discrepancies in the position / angle of the wrist due to different arm positioning angles. This step identifies the cortical bone edge and uses a rigid registration algorithm, similar to "rotating and translating images," without stretching or deforming, to align all MRI images to the same local coordinates of the patient's wrist, facilitating comparison of structures at the same location.

[0039] 1.4 For the raw X-ray data, perform the following operations in sequence: remove mesh artifacts and scattering, histogram equalization, bone structure protrusion, scale unification and geometric correction to obtain a standardized X-ray data package.

[0040] 1) Removing mesh artifacts and scattering: X-ray images may exhibit grid-like interference or scattered light due to equipment or shooting environment. These irrelevant interferences can be removed through frequency domain filtering and circular halo elimination.

[0041] 2) Histogram Equilibrium: Some X-ray images have low contrast, blurring the boundary between bone and soft tissue. CLAHE is a local contrast enhancement algorithm that can make the boundary between bone and soft tissue clearer while limiting the magnitude of contrast enhancement to avoid over-enhancing soft tissue noise and causing image clutter.

[0042] 3) Prominent bone structure: Further enhance bone contours. Frangi-like filtering excels at identifying blood vessels / tubular structures, while LoG filtering excels at identifying edges. Combining the two makes the bone outline clearer, facilitating subsequent localization, such as locating the axes of metacarpals and radius.

[0043] 4) Standardize the scale and resample to 0.15-0.2 mm / px: px stands for pixels in a 2D image. Different X-ray machines may have different actual millimeter values ​​per pixel, causing the same bone to appear to have different lengths in the image. Resampling unifies the pixel-to-millimeter correspondence across all X-ray images, ensuring consistent actual dimensions.

[0044] 5) Geometric correction: During the X-ray, the patient's hand may be tilted, causing the bones to appear crooked. By identifying the axis of the metacarpals and radius, that is, the line connecting the centers of the bones, the image can be rotated or stretched to align the bone axis and avoid structural deviations caused by the shooting angle.

[0045] 1.5 Based on standardized data packets, a deep learning model is used to perform structural localization and segmentation of images, identify and label key anatomical structures to obtain sharpened images, then key regions are extracted from the sharpened images to obtain a candidate slice set, and finally the quality of the candidate slice set is evaluated to obtain a joint slice library. Specifically, this includes: 1.51 For the ultrasound standardized data package, Keypoint-Detector is used to detect the direction of the articular surface and tendon sheath, and Attention U-Net is used to segment the synovial thickness region and PD region to complete ultrasound localization and segmentation.

[0046] 1.52 For the aforementioned MRI standardized data package, the nnU-Net 3D segmentation model is used to simultaneously segment the carpal bones, distal metacarpals, proximal phalanges, joint cavity, and synovial thickening area. The three-dimensional region of each joint is cut out in the image to generate a 3D ROI block, thus completing the MRI segmentation.

[0047] 1.53 For the X-ray standardized data packet, the HRNet-W32 model is used to predict the metacarpal heads and phalangeal bases for key point localization. Adjacent key points are connected by lines, and a rectangular box is drawn around the connecting lines to obtain the joint cavity candidate region. Then, the U-Net++ model is used to segment the bone tissue of the joint cavity candidate region to complete the X-ray localization segmentation.

[0048] 1.54 The results of ultrasound localization and segmentation, magnetic resonance segmentation and X-ray localization and segmentation are integrated to obtain a clear image. Then, key regions are extracted from the clear image to obtain a candidate slice set.

[0049] 1.55 Based on the candidate slice set, define the quality dimensions and quality labels of different types of images, and use the AI ​​model to output the quality scores of different quality dimensions. Then, combine Huber loss and RankLoss loss to constrain the model and obtain the slice quality score.

[0050] 1.56 Compare the slice quality score with the preset quality pass threshold to mark problem areas and screen out low-quality slices to obtain a joint slice library.

[0051] By using deep learning models to locate and segment images, the transformation from raw images to structured information has been achieved. Anatomical structures that previously required manual labeling by doctors can now be automatically identified and precisely outlined. This not only improves efficiency but also ensures consistency, avoiding subjective differences in labeling by different doctors. This lays the foundation for subsequent quantitative analysis, such as the calculation of indicators like joint space width, synovial volume, and blood flow area, as well as disease diagnosis and efficacy evaluation.

[0052] Step 2: Based on the joint slice library, perform intramodal self-supervised pre-training and cross-modal aligned representation, then perform lesion-level detection and quantization to obtain the lesion feature vector of each joint.

[0053] Step 2 includes: 2.1 Based on the joint slice library, a masked autoencoder is used to learn the 3D volumetric context, a contrastive learning model is used to enhance invariant features, and temporal consistency is used to learn the association between adjacent frames, thus completing the intramodal self-supervised pre-training.

[0054] 2.2 Based on the joint slice library, positive and negative sample pairs are added for comparative learning, and a distance loss function is set to bring images of different modalities of the same joint closer together, resulting in an aligned cross-modal representation encoder.

[0055] 2.3 The Attention U-Net model is used for ultrasound lesion quantification, the 3D nnU-Net model is used for magnetic resonance lesion quantification, the target detection model and distance transformation are used for X-ray lesion quantification, and the uncertainty of lesion quantification is estimated by deep ensemble or MCDropout. Then, for each joint, all modal features and lesion quantification results are concatenated to obtain the lesion feature vector.

[0056] Step 3, as follows Figure 3 As shown, based on the lesion feature vector, joint graph construction and quality-aware multimodal fusion are performed to obtain a fused feature representation. Based on the fused feature representation, weakly supervised multi-instance learning and multi-task loss calculation are performed to output a patient-level classifier and a joint-level regressor, resulting in an uncalibrated model.

[0057] Step 3 includes: 3.1 Based on the lesion feature vector, the discrete joint vectors are organized into a graph structure including anatomical relationships, symmetry relationships and propagation relationships. Based on the graph structure, the graph nodes are gating weighted and uncertainty-aware weighted to obtain the set of correction node inputs.

[0058] The graph nodes of the graph structure are set as joint nodes, and the edges of the graph structure include adjacent edges representing adjacent joints of the same hand, symmetrical edges representing left and right joints with the same name, hierarchical edges representing the relationship between the distal and proximal ends, and optional cross-modal similar edges representing similarity connections.

[0059] Node (V): Each joint is a node, such as "left hand third metacarpophalangeal joint" and "right hand second metacarpophalangeal joint". The attributes of each node are the joint feature vector, which includes disease quantification indicators, cross-modal features, quality score, uncertainty, etc.

[0060] Edge (E): Connects joint nodes with a "relationship". The attributes of the edge indicate the type or strength of the relationship, for example: Anatomical relationships: Adjacent joints, such as the "metacarpophalangeal joint" and the "proximal interphalangeal joint", are connected by edges, and the edge attribute records their anatomical distance.

[0061] Symmetrical relationship: corresponding joints on the left and right (such as L3MCP and R3MCP) are connected by edges, and the edge attribute is marked "symmetric".

[0062] Propagation association: Joints that often occur simultaneously in clinical practice, such as rheumatoid arthritis, are easily affected by multiple metacarpophalangeal joints. These are connected by edges, and the edge attributes record the probability of comorbidity.

[0063] 3.2 Based on the set of inputs of the correction nodes, a graph neural network is used to enable mutual learning between nodes, the multi-head self-attention mechanism of the Transformer is used to enable interaction between nodes, and gating weights and prior knowledge are introduced to train and fine-tune the model to obtain the joint-level fusion feature representation.

[0064] Transformer encoding: Transformer is a model that excels at handling sequence relationships. The model allows each joint node to focus on nodes related to it. For example, the pathological features of a joint in the left hand will be referenced to see if there are similar lesions in its symmetrical right hand joints; the metacarpophalangeal joints will be referenced to see if there are any associated lesions in the adjacent wrist joint. Through this attention mechanism, the features of each node are integrated with information from related nodes, and are no longer isolated.

[0065] Injecting prior knowledge: Incorporating prior knowledge of human anatomy into the auxiliary model: Symmetry prior: The left and right joints are usually symmetrical, and the model will reinforce this symmetry. If the difference in features between the left and right joints is too large, it may indicate an anomaly.

[0066] Anatomical priors: The position of the joints is fixed, such as the metacarpophalangeal joint at the base of the finger. The model will utilize this positional relationship, such as joints that are close to each other are more likely to influence each other.

[0067] 3.3 Based on the fused feature representation, a supervision signal is defined and task modeling is performed to obtain a task head structure and a label dictionary. Based on the task head structure, multi-instance learning is used to converge the joint-level representation to the patient level in an interpretable attention manner. In addition, quality weighting and uncertainty weighting are combined to automatically filter out evidence joints and obtain patient embeddings and joint-level attention weights.

[0068] The task head structure includes patient-level tasks consisting of RA classification and differential diagnosis tasks, and joint-level tasks consisting of RAMRIS regression and lesion detection tasks. The label dictionary includes patient-level weak labels, joint-level strong labels, and joint-level pseudo-labels (coarse labels or labels automatically generated by the model).

[0069] 3.4 Define multi-task loss, perform task optimization, and construct model training strategies and validation metrics for training optimization to obtain patient-level classification heads and joint-level regression heads. Then, encapsulate and version the patient-level classification heads and joint-level regression heads to output patient-level classifiers and joint-level regressors, resulting in an uncalibrated model.

[0070] The multi-task loss includes classification loss for optimizing patient-level classification tasks, regression loss for optimizing joint-level RAMRIS regression tasks, cross-modal consistency loss for maintaining consistency in predictions of the same joint from different modalities, symmetry prior loss for maintaining consistency in predictions of left-right symmetrical joints, domain adversarial loss for adapting to different data sources, and uncertainty regularization loss for controlling sample loss contributions.

[0071] The formula for calculating the classification loss is: ; in, For cross-entropy, For patient-level classification heads, For patient-level fusion embedding, Patient-level truth category labels; The formula for calculating the regression loss is: ; in, Index for joint subsets. For segmented quadratic loss, For the return of the head, This represents joint-level fusion. For the corresponding truth value score, Joint-level reliable weights; The expression for calculating the cross-modal consistency loss is as follows: ; in, For joint indexing, For the set of modal pairs to be compared, The contribution of different modes to different joints, For the contrast loss with temperature, For modality For projection head, For modality For projection head, For the first Each joint in modality The encoding, For the first Each joint in modality The encoding; The expression for calculating the symmetric prior loss is as follows: ; in, This is a set of paired left and right joints with the same name. For symmetric constraint weights, This represents joint-level fusion. This is a symmetrical joint-level fusion representation; The expression for calculating the domain adversarial loss is as follows: ; in, For sample index, For domain discriminator, For the corresponding level features, For real domain tags; The expression for calculating the uncertainty regularization loss is as follows: ; in, All are sub-item weights. The mean uncertainty within the patient population. To measure the deviation between confidence level and success rate.

[0072] Step 4: Based on the uncalibrated model, perform model calibration and interpretable output to obtain the intelligent diagnostic model.

[0073] Step 4 includes: 4.1 Construct a calibration set including different disease stages and complete labels, use the calibration set to test the uncalibrated model, and calculate the uncalibrated baseline index and error distribution.

[0074] Data requirements for constructing the calibration set: The data comes from different centers (such as hospitals A / B for training and hospitals C / D for calibration), covering different disease stages, such as early RA, late RA, healthy individuals, and different equipment models, ensuring the representativeness of the data and reflecting real clinical scenarios.

[0075] Includes complete annotations: patient-level diagnostic markers, joint-level RAMRIS gold standard scores, and corresponding imaging and feature data.

[0076] Stratified design: Sampling is stratified by center, disease type, and imaging modality to avoid excessive data of a certain type, such as only containing early RA, which could lead to calibration bias.

[0077] Uncalibrated baseline metrics: Expected calibration error (ECE) and maximum calibration error (MCE). The smaller the ECE, the closer the predicted probability is to the actual prevalence.

[0078] Error distribution: mean absolute error, root mean square error, and distribution deviation to see if the predicted value and the actual value distribution are consistent.

[0079] 4.2 Based on the uncalibrated baseline index and the error distribution, patient-level probabilistic calibration is performed through temperature scaling, Dirichlet calibration, and group calibration, and joint-level regression calibration is performed through piecewise equal probability regression and residual modeling to obtain calibration parameters for model calibration.

[0080] Temperature scaling: The simplest and most commonly used method is to divide the model's predicted logit value by a temperature parameter T to adjust the probability distribution.

[0081] Dirichlet calibration: A more flexible approach that learns a Dirichlet distribution parameter for each category, such as RA positive / negative, and can handle multi-class or uneven probability distributions.

[0082] Group calibration: The predicted probability is divided into multiple intervals, and each interval is adjusted individually. This is suitable for situations where the calibration error varies with the probability interval.

[0083] 4.3 Construct a typical case feature library associated with known cases. Based on the typical case feature library, generate visual or textual explanations from four levels: case, region, joint, and lesion, and complete the interpretable output of the model to obtain the intelligent diagnostic model.

[0084] Typical case feature database, such as: RA positive prototype: contains typical feature vectors of polymetacarpophalangeal synovitis and bone erosion; RA negative prototype, such as osteoarthritis OA: contains typical feature vectors of distal interphalangeal joint osteophytes and no synovitis; Healthy human prototype: contains typical feature vectors of normal joint structure and no lesions.

[0085] Each prototype is accompanied by a clinical description, such as Prototype 1: Active RA phase, involving 3 metacarpophalangeal joints and original imaging fragments, for doctors' reference.

[0086] Therefore, this invention provides an intelligent early diagnosis method for rheumatoid arthritis based on multimodal medical imaging and deep learning, which solves the key challenges of multimodal fusion, data heterogeneity, generalization and reliable output in early RA diagnosis. It can shift the diagnosis time point to the reversible inflammatory phase, provide patient-level decision-making and joint-level quantitative interpretable evidence, and take into account the engineering feasibility of small sample and multi-center generalization, thus having high clinical translation potential.

[0087] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0088] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for early intelligent diagnosis of rheumatoid arthritis based on multimodal medical imaging and deep learning, characterized in that, include: Images and metadata are acquired to obtain a multimodal raw dataset. The multimodal raw dataset is then subjected to image cleaning and standardization, automatic bone and joint localization and segmentation, and quality control to obtain a joint slice library. Based on the joint slice library, intramodal self-supervised pre-training and cross-modal aligned representation are performed, followed by lesion-level detection and quantization to obtain the lesion feature vector of each joint; Based on the lesion feature vector, joint graph construction and quality-aware multimodal fusion are performed to obtain a fused feature representation. Based on the fused feature representation, weakly supervised multi-instance learning and multi-task loss calculation are performed to output a patient-level classifier and a joint-level regressor to obtain an uncalibrated model. Based on the uncalibrated model, model calibration and interpretable output are performed to obtain an intelligent diagnostic model.

2. The method for early intelligent diagnosis of rheumatoid arthritis based on multimodal medical imaging and deep learning according to claim 1, characterized in that, Images and metadata are acquired to obtain a multimodal raw dataset. The multimodal raw dataset undergoes image cleaning and standardization, automatic bone and joint localization and segmentation, and quality control to obtain a joint slice library, including: Configure a DICOM router to collect raw medical images and basic information of ultrasound, MRI and X-ray in batches according to a unified process rule, and perform integrity and consistency checks on the raw medical images to obtain a multimodal raw dataset including raw ultrasound data, raw MRI data and raw X-ray data. For the raw ultrasound data, anti-aliasing, beam and depth normalization, PD signal standardization and spatial calibration operations are performed in sequence to obtain ultrasound standardized data packets. For the raw magnetic resonance data, N4 bias field correction, denoising, intensity normalization, resampling and coarse registration operations are performed sequentially to obtain a magnetic resonance normalized data packet; The raw X-ray data is subjected to the following operations in sequence: demeshing and scattering, histogram equalization, bone structure protrusion, scale unification and geometric correction, to obtain a standardized X-ray data package. Based on standardized data packets, a deep learning model is used to perform structural localization and segmentation of images, identify and mark key anatomical structures to obtain clear images, then extract key regions from the clear images to obtain a candidate slice set, and finally evaluate the quality of the candidate slice set to obtain a joint slice library.

3. The method for early intelligent diagnosis of rheumatoid arthritis based on multimodal medical imaging and deep learning according to claim 2, characterized in that, Based on standardized data packets, a deep learning model is used to perform structural localization and segmentation on images, identify and label key anatomical structures to obtain sharpened images, then key regions are extracted from the sharpened images to obtain a candidate slice set, and finally the quality of the candidate slice set is evaluated to obtain a joint slice library, including: For the aforementioned standardized ultrasound data package, Keypoint-Detector is used to detect the articular surface and tendon sheath orientation, and Attention U-Net is used to segment the synovial thickness region and PD region to complete ultrasound localization and segmentation; For the aforementioned MRI standardized data package, the nnU-Net 3D segmentation model is used to simultaneously segment the carpal bones, distal metacarpals, proximal phalanges, joint cavity, and synovial thickening area. The three-dimensional region of each joint is cut out in the image to generate a 3D ROI block, thus completing the MRI segmentation. For the X-ray standardized data packet, the HRNet-W32 model is used to predict the metacarpal heads and phalangeal bases for key point localization. Adjacent key points are connected by lines, and a rectangular box is drawn around the connecting lines to obtain the joint cavity candidate region. Then, the U-Net++ model is used to segment the bone tissue of the joint cavity candidate region to complete the X-ray localization segmentation. The results of ultrasound localization and segmentation, magnetic resonance segmentation and X-ray localization and segmentation are integrated to obtain a clear image. Then, key regions are extracted from the clear image to obtain a candidate slice set. Based on the candidate slice set, quality dimensions and quality labels for different types of images are defined, and quality scores for different quality dimensions are output using an AI model. Then, Huber loss and RankLoss loss are combined to constrain the model and obtain slice quality scores. By comparing the quality score of the slices with a preset quality pass threshold, problem areas are marked and low-quality slices are screened out to obtain a joint slice library.

4. The method for early intelligent diagnosis of rheumatoid arthritis based on multimodal medical imaging and deep learning according to claim 3, characterized in that, Based on the joint slice library, intramodal self-supervised pre-training and cross-modal aligned representation are performed, followed by lesion-level detection and quantization to obtain the lesion feature vector for each joint, including: Based on the joint slice library, a masked autoencoder is used to learn 3D volumetric context, a contrastive learning model is used to enhance invariant features, and temporal consistency is used to learn the association between adjacent frames, thus completing intramodal self-supervised pre-training. Based on the joint slice library, positive and negative sample pairs are added for comparative learning, and a distance loss function is set to bring images of different modalities of the same joint closer together, resulting in an aligned cross-modal representation encoder. The Attention U-Net model is used for ultrasound lesion quantification, the 3D nnU-Net model is used for magnetic resonance lesion quantification, and the target detection model and distance transformation are used for X-ray lesion quantification. The uncertainty of lesion quantification is estimated by deep ensemble or MC Dropout. Then, for each joint, all modal features and lesion quantification results are concatenated to obtain the lesion feature vector.

5. The method for early intelligent diagnosis of rheumatoid arthritis based on multimodal medical imaging and deep learning according to claim 4, characterized in that, Based on the lesion feature vector, joint map construction and quality-aware multimodal fusion are performed to obtain a fused feature representation. Based on the fused feature representation, weakly supervised multi-instance learning and multi-task loss are performed to output a patient-level classifier and a joint-level regressor, resulting in an uncalibrated model, including: Based on the lesion feature vector, the discrete joint vectors are organized into a graph structure including anatomical relationships, symmetry relationships, and propagation relationships. Based on the graph structure, the graph nodes are subjected to gating weighting and uncertainty-aware weighting to obtain the set of correction node inputs. Based on the set of inputs to the correction nodes, a graph neural network is used to enable mutual learning between nodes, the multi-head self-attention mechanism of the Transformer is used to enable interaction between nodes, and gating weights and prior knowledge are introduced to train and fine-tune the model to obtain joint-level fusion feature representations. Based on the fused feature representation, a supervision signal is defined and task modeling is performed to obtain a task head structure and a label dictionary. Based on the task head structure, multi-instance learning is used to converge the joint-level representation to the patient level in an interpretable attention manner, and quality weighting and uncertainty weighting are combined to automatically filter out evidence joints, thereby obtaining patient embeddings and joint-level attention weights. Define a multi-task loss, perform task optimization, and construct a model training strategy and validation metrics for training optimization to obtain a patient-level classifier head and a joint-level regression head. Then, encapsulate and version the patient-level classifier head and the joint-level regression head to output a patient-level classifier and a joint-level regressor, resulting in an uncalibrated model.

6. The method for early intelligent diagnosis of rheumatoid arthritis based on multimodal medical imaging and deep learning according to claim 5, characterized in that, The graph nodes of the graph structure are set as joint nodes, and the edges of the graph structure include adjacent edges representing adjacent joints of the same hand, symmetrical edges representing left and right joints with the same name, hierarchical edges representing the relationship between the distal and proximal ends, and optional cross-modal similar edges representing similarity connections.

7. The method for early intelligent diagnosis of rheumatoid arthritis based on multimodal medical imaging and deep learning according to claim 6, characterized in that, The task head structure includes patient-level tasks consisting of RA classification and differential diagnosis tasks, and joint-level tasks consisting of RAMRIS regression and lesion detection tasks. The label dictionary includes patient-level weak labels, joint-level strong labels, and joint-level pseudo-labels.

8. The method for early intelligent diagnosis of rheumatoid arthritis based on multimodal medical imaging and deep learning according to claim 7, characterized in that, The multi-task loss includes classification loss for optimizing patient-level classification tasks, regression loss for optimizing joint-level RAMRIS regression tasks, cross-modal consistency loss for maintaining consistency in predictions of the same joint from different modalities, symmetry prior loss for maintaining consistency in predictions of left-right symmetrical joints, domain adversarial loss for adapting to different data sources, and uncertainty regularization loss for controlling sample loss contributions.

9. The method for early intelligent diagnosis of rheumatoid arthritis based on multimodal medical imaging and deep learning according to claim 8, characterized in that: The formula for calculating the classification loss is: ； in, For cross-entropy, For patient-level classification heads, For patient-level fusion embedding, Patient-level truth category labels; The formula for calculating the regression loss is: ； in, Index for joint subsets. For segmented quadratic loss, For the return of the head, This represents joint-level fusion. For the corresponding truth value score, Joint-level reliable weights; The expression for calculating the cross-modal consistency loss is as follows: ； in, For joint indexing, For the set of modal pairs to be compared, The contribution of different modes to different joints, For the contrast loss with temperature, For modality For projection head, For modality For projection head, For the first Each joint in modality The encoding, For the first Each joint in modality The encoding; The expression for calculating the symmetric prior loss is as follows: ； in, This is a set of paired left and right joints with the same name. For symmetric constraint weights, This represents joint-level fusion. This is a symmetrical joint-level fusion representation; The expression for calculating the domain adversarial loss is as follows: ； in, For sample index, For domain discriminator, For the corresponding level features, For real domain tags; The expression for calculating the uncertainty regularization loss is as follows: ； in, All are sub-item weights. The mean uncertainty within the patient population. To measure the deviation between confidence level and success rate.

10. A method for early intelligent diagnosis of rheumatoid arthritis based on multimodal medical imaging and deep learning according to claim 9, characterized in that, Based on the uncalibrated model, model calibration and interpretable output are performed to obtain an intelligent diagnostic model, including: Construct a calibration set including different disease stages and complete labels, use the calibration set to test the uncalibrated model, and calculate the uncalibrated baseline index and error distribution; Based on the uncalibrated baseline index and the error distribution, patient-level probabilistic calibration is performed through temperature scaling, Dirichlet calibration, and group calibration, and joint-level regression calibration is performed through piecewise equal probability regression and residual modeling to obtain calibration parameters for model calibration. A typical case feature library associated with known cases is constructed. Based on the typical case feature library, visualization or textual explanations are generated from four levels: case, region, joint, and lesion, to complete the interpretable output of the model and obtain the intelligent diagnostic model.

Images