Rheumatoid arthritis activity grading device based on multimodal data

Abstract

The present invention discloses a rheumatoid arthritis activity grading device based on multimodal data, which mainly includes: an image preprocessing module for converting multiple joint images of a case into image features; an extraction module for extracting all joint image features using TransformerD; an optimized image representation module for obtaining lesion classification predictions of corresponding images and disease grading predictions of the entire image, and calculating a first cross-entropy loss with the corresponding lesion label; an other modality feature extraction module for extracting and enhancing other modality features using MLP; a feature fusion module for interacting and fusing the above-mentioned modality features using TransformerP to obtain disease grading predictions of multimodal data, and calculating a second cross-entropy loss with the true grading label; a loss function optimization module for training a disease activity grading prediction network; a disease grading prediction module for outputting the disease activity of the case from the grading prediction network given the joint images and other modality data of the case.

Description

Technical Field

[0001] The present application relates to the technical field of multimodal medical analysis, and in particular to a rheumatoid arthritis activity grading device based on multimodal data. Background Art

[0002] Rheumatoid arthritis (RA) is a common chronic inflammatory disease characterized by erosive arthritis. RA develops slowly and insidiously, typically beginning in the small joints of the hands and feet, gradually progressing to symmetrical multi-joint involvement, ultimately leading to typical joint inflammatory lesions such as joint pain, morning stiffness, and swelling, accompanied by fatigue and decreased quality of life. Therefore, early identification and intervention for RA are crucial. Existing clinical diagnostic methods require both professional evaluation by physicians and self-assessment by patients. This, while time-consuming and labor-intensive, requires significant resources from specialized physicians. Furthermore, self-assessment by non-expert patients can bias diagnostic results. However, with the advancement of deep learning technology, the ability to efficiently and accurately predict RA disease activity grading using deep learning can reduce the cost of evaluation and diagnosis for specialized physicians, help physicians seize the therapeutic window, select appropriate drug regimens for patients, and thus improve treatment compliance rates. This holds significant clinical significance.

[0003] In general, the existing RA disease activity grading prediction methods can be divided into clinical data-based methods and imaging data-based methods. The former aims to use clinical data regression to fit the disease activity. For example, Lee et al. [1] proposed a linear regression model using variables such as the number of damaged joints presented by bone imaging (BS-positive), erythrocyte sedimentation rate (ESR) and patient global assessment (PGA) to predict the rheumatoid arthritis disease activity index score. The latter aims to use deep neural networks to mine imaging data representations and then predict the disease activity grade. For example, Chinese invention patent CN114140393A, published on March 4, 2022, discloses a deep learning-based rheumatoid arthritis X-ray scoring method, which uses the ResNet-Dwise50 network model to extract features from X-ray images of rheumatoid arthritis patients and then predict the corresponding disease activity grade.

[0004] Methods based on clinical data use typical machine learning techniques to regress and fit disease activity. However, these methods have the following drawbacks: Firstly, traditional machine learning models require manual feature extraction, resulting in simple model structures and poor representational capabilities, making it difficult to accurately fit disease activity. Secondly, some clinical data is obtained by doctors examining each patient's joints individually, which is labor-intensive, inefficient, and costly, resulting in significant human and material resources consumption and potentially delaying diagnosis and treatment.

[0005] The disadvantages of the disease grading method based on imaging data are: on the one hand, only a single modality of medical imaging data is used, the information is single, it is difficult to obtain accurate and comprehensive representation, and the prediction accuracy is low; on the other hand, using separate joint images to predict disease activity does not fully explore the relationship between different joint images, resulting in poor prediction results. Summary of the Invention

[0006] The purpose of the embodiment of the present application is to provide a rheumatoid arthritis activity grading device based on multimodal data to solve the technical problems existing in the related art, such as poor model representation ability, single modal information and insufficient exploration of the connection between different joint images.

[0007] According to a first aspect of an embodiment of the present application, a device for grading rheumatoid arthritis activity based on multimodal data is provided, comprising:

[0008] An image preprocessing module is used to perform normalized coding on multiple joint images of a case to obtain a coding matrix, and convert the image coding matrix into joint image features;

[0009] The extraction module is used to extract all joint image features using Transformer D to obtain the features and classification features corresponding to each joint image;

[0010] An optimized image representation module is used to input the features and classification features corresponding to each joint image into a first feedforward neural network (FFN) to obtain a lesion classification prediction for the corresponding image and a disease activity grading prediction for the entire image, and calculate a first cross entropy loss with the actual image lesion category and disease activity grading label;

[0011] The other modality feature extraction module is used to extract other modality features of all joints using MLP, obtain the features of basic information and laboratory examination data respectively, and then use the cross-attention mechanism to enhance the two to obtain enhanced features;

[0012] The feature fusion module is used to input the features corresponding to each joint image and the enhanced features into the feature fusion TransformerP for feature interaction and fusion to obtain fused features, and then input the fused features into the second feedforward neural network FFN to obtain the disease activity grade prediction of multimodal data, and calculate the second cross entropy loss between the disease activity grade prediction and the actual disease activity grade;

[0013] a loss function optimization module for training a disease activity grading prediction network using a total loss as a linear combination of a first cross entropy loss and a second cross entropy loss, and iteratively updating parameters of the prediction network until the total loss function reaches a preset convergence condition, wherein the disease activity grading prediction network comprises an extraction module, an optimized image representation module, and a feature fusion module;

[0014] The disease grading prediction module is used to input multiple joint images, basic information and laboratory test data of a given case into a trained disease activity grading prediction network, and output the disease activity level to which the case belongs.

[0015] Optionally, the image preprocessing module includes:

[0016] Unified image size unit, used to use bilinear interpolation algorithm to convert all images They are all scaled to a fixed size so that the input image meets the input specifications of the image feature extraction network;

[0017] A normalization coding unit is used to divide all pixel values ​​in the image g of the same size by 255, and then normalize the image value (val) to obtain the image normalization code;

[0018] Linearization representation unit, used to flatten each encoding matrix into a one-dimensional vector s1,s2,...,s n , mapped to u1, u2, ..., u through the linear mapping layer n ;

[0019] Image feature output unit, used for u1, u2, ..., u n Add classification vector and positional encoding The image feature h0 is obtained as shown in formula (1);

[0020]

[0021] Optionally, the extraction module includes:

[0022] A self-attention unit is used to extract image features using Transformer D, where Transformer D consists of L stacked self-attention layers, where:

[0023] Transformer D takes h0 as input and uses L self-attention layers in Transformer D to enhance the image feature h0. The output h of the Lth self-attention layer is L This is the output of Transformer D, as shown in formula (5):

[0024]

[0025] Optional, optimized image characterization module, including:

[0026] An image lesion classification prediction unit is used to obtain a lesion classification prediction of the corresponding image using the first feedforward neural network layer (FFN) and overall disease activity grading prediction based on imaging

[0027] Image feature optimization unit for using classification loss Optimize image features, including classification loss As shown in formula (6):

[0028]

[0029]

[0030] Where i represents the joint image index, u represents the image lesion category, represents the probability that the i-th joint image belongs to the u-th lesion category, c1 represents the total number of disease categories, represents the probability that the joint image as a whole belongs to grade v, and c2 represents the total number of disease grades.

[0031] Optional, other modal feature extraction modules include:

[0032] Other modal feature extraction units are used to extract basic information of all joints and laboratory examination data features using MLP to obtain feature u M and u S ;

[0033] Cross attention unit, used to M and u S Calculate the cross attention and weight the features of the two according to the similarity to obtain the weighted feature f M and f S , the cross attention mechanism will u S Transformed into key vector K and value vector V,u M Transformed into query vector Q for cross attention calculation. e represents other modal features after the cross attention layer, as shown in formula (8):

[0034] h e =[f M , f S ]#(8).

[0035] Optional feature fusion module, including:

[0036] Feature fusion unit is used to combine the features corresponding to each joint image And the enhanced feature f M and f S Input into Transformer P for feature interaction and fusion to obtain the fused feature f P , where the structure of Transformer P is the same as that of Transformer D in the extraction module;

[0037] Hierarchical loss calculation unit, used to combine the fusion feature f of the feature fusion module P Input into the second feedforward network FFN, select GELU(·) function as the activation function, and obtain the disease activity grade prediction of multimodal data And use the cross entropy loss shown in formula (9) to calculate the classification loss

[0038]

[0039] Where c represents the total number of disease activity grades, p u represents the probability that the input case belongs to grade u.

[0040] According to the second aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the rheumatoid arthritis activity grading device based on multimodal data as described above.

[0041] According to a third aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the apparatus for grading rheumatoid arthritis activity based on multimodal data as described above is implemented.

[0042] The technical solutions provided by the embodiments of the present application may have the following beneficial effects:

[0043] It can be seen from the above embodiments that the present application adopts deep learning technology to perform characterization learning on the three modalities of basic information, laboratory test data and joint imaging data respectively, and interacts and fuses the features of the three modalities of data, and uses the three modalities of data to jointly evaluate the activity of rheumatoid arthritis, thereby avoiding the shortcoming that it is difficult to obtain accurate characterization of single modality data. In addition, the present invention uses the Transformer network to extract multiple joint imaging features, and doctors do not need to conduct additional examinations, which greatly saves manpower, material resources and medical resources and improves the efficiency of technical implementation. Finally, the present invention uses high-confidence data such as gender, age, BMI, medical history, joint imaging, etc. to effectively solve the data that is prone to errors such as patient self-assessment and doctor's assessment of disease activity or overall health status, thereby improving the accuracy of the model.

[0044] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0046] Figure 1 This is a block diagram of a device for grading rheumatoid arthritis activity based on multimodal data according to an exemplary embodiment.

[0047] Figure 2 4 is a structural diagram of a device for grading rheumatoid arthritis activity based on multimodal data according to an exemplary embodiment. DETAILED DESCRIPTION

[0048] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0049] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0050] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0051] Figure 1 is a block diagram of a device for grading rheumatoid arthritis activity based on multimodal data according to an exemplary embodiment. Figure 1 As shown, the device includes:

[0052] Image preprocessing module 1, used for normalizing and encoding multiple joint images of a case to obtain a coding matrix, and converting the image coding matrix into joint image features;

[0053] Extraction module 2 is used to extract all joint image features using Transformer D to obtain the features and classification features corresponding to each joint image;

[0054] Optimizing the image representation module 3, for inputting the features and classification features corresponding to each joint image into the first feedforward neural network FFN to obtain the lesion classification prediction of the corresponding image and the disease activity grade prediction of the entire image, and calculating the first cross entropy loss with the actual image lesion category and disease activity grade label;

[0055] Other modal feature extraction module 4 is used to extract other modal features of all joints using MLP, obtain features of basic information and laboratory examination data respectively, and then enhance the two using the cross attention mechanism to obtain enhanced features;

[0056] Feature fusion module 5 is used to input the features corresponding to each joint image and the enhanced features into the feature fusion Transformer P for feature interaction and fusion to obtain fused features, and input the fused features into the second feedforward neural network FFN to obtain the disease activity grade prediction of multimodal data, and calculate the second cross entropy loss between the disease activity grade prediction and the actual disease activity grade;

[0057] a loss function optimization module 6 for training a disease activity grading prediction network with a total loss as a linear combination of a first cross entropy loss and a second cross entropy loss, and iteratively updating parameters of the prediction network until the total loss function reaches a preset convergence condition, wherein the disease activity grading prediction network comprises an extraction module, an optimized image representation module, and a feature fusion module;

[0058] The disease grading prediction module 7 is used to input multiple joint images, basic information and laboratory test data of a given case into a trained disease activity grading prediction network, and output the disease activity level to which the case belongs.

[0059] It can be seen from the above embodiments that the present application adopts deep learning technology to perform characterization learning on the three modalities of basic information, laboratory test data and joint imaging data respectively, and interacts and fuses the features of the three modalities of data, and uses the three modalities of data to jointly evaluate the activity of rheumatoid arthritis, thereby avoiding the shortcoming that it is difficult to obtain accurate characterization of single modality data. In addition, the present invention uses the Transformer network to extract multiple joint imaging features, and doctors do not need to conduct additional examinations, which greatly saves manpower, material resources and medical resources and improves the efficiency of technical implementation. Finally, the present invention uses high-confidence data such as gender, age, BMI, medical history, joint imaging, etc. to effectively solve the data that is prone to errors such as patient self-assessment and doctor's assessment of disease activity or overall health status, thereby improving the accuracy of the model.

[0060] The image preprocessing module 1 includes:

[0061] Unified image size unit, used to use bilinear interpolation algorithm to convert all images They are all scaled to a fixed size so that the input image meets the input specifications of the image feature extraction network;

[0062] A normalization coding unit is used to divide all pixel values ​​in the image g of the same size by 255, and then normalize the image value (val) to obtain the image normalization code;

[0063] Linearization representation unit, used to flatten each encoding matrix into a one-dimensional vector s1, s2, ..., s n , mapped to u1, u2, ..., u through the linear mapping layer n , to obtain a representation that meets the input specifications of the image feature extraction module, as shown in formula (1):

[0064] u i =W D s i +b D (1)

[0065] Among them, W D Indicates that b D represents the bias and i represents the vector index.

[0066] Image feature output unit, used for u1, u2, ..., u n Add classification vector and positional encoding The image feature h0 is obtained as shown in formula (2):

[0067]

[0068] The extraction module 2 includes:

[0069] The self-attention unit is used to extract image features using Transformer D, where Transformer D consists of L stacked self-attention layers to capture the correlation between different parts of the joint image and explore the connection between different joint images. Transformer D takes the image feature representation h0 obtained by the image preprocessing module 1 as input and the prediction output h of the Lth layer as output. L is the final output. Among them, the self-attention layer is composed of layer normalization function (LN), self-attention function (MSA) and multi-layer perceptron (MLP). Specifically:

[0070] The first self-attention layer takes h0 as input, and the lth self-attention layer (l=2, 3, ..., L) takes the output h of the l-1th self-attention layer l-1 As input, output enhanced image feature h l For the lth self-attention layer, input it into h l-1 , which is transformed from the transformation matrix into the corresponding query vector Q l , key vector K l and the value vector V l , as shown in formula (3):

[0071]

[0072]

[0073]

[0074] Then the self-attention function can be expressed by formula (4):

[0075]

[0076] Among them, d K is the key vector K l Dimensions, K l The transpose of .

[0077] The specific structure of the self-attention layer can be expressed as follows by formula (5):

[0078] z l =LN(MSA(Q l , K l , V l)+h l-1 ),

[0079] h l =LN(MLP(z l )+z l )#(5)

[0080] where z l represents the output of the attention mechanism in layer l, h l represents the image features after the lth self-attention layer. LN(·) is the layer normalization function, which is used to solve the problem that the network value is too large due to the depth of the network, thus affecting the gradient descent speed, as shown in formula (6):

[0081]

[0082] Among them, γ and β are hyperparameters, mean represents the mean of the layer, and std represents the standard deviation of the layer.

[0083] MLP stands for Multi-Layer Perceptron, which consists of multiple fully connected layers and activation functions, as shown in formula (7):

[0084] MLP(x)=g(W2(W1x+b1)+b2)#(7)

[0085] Where W1 and W2 represent weights, b1 and b2 represent biases, and g(·) represents the activation function. g(·) can be selected from sigmoid, RELU, tanh, and other functions.

[0086] The output h of the Lth self-attention layer L This is the output of Transformer D, as shown in formula (8):

[0087]

[0088] The optimized image representation module 3 includes:

[0089] An image lesion classification prediction unit is used to obtain a lesion classification prediction of the corresponding image using the first feedforward neural network layer (FFN) and overall disease activity grading prediction based on imaging Specifically:

[0090] The model structure of the feedforward neural network layer (FFN) can be expressed by formula (9):

[0091]

[0092] in, is the weight transformation matrix, is the bias coefficient, GELU(·) is the Gaussian error linear activation function, as shown in formula (10):

[0093]

[0094] Image feature optimization unit for using classification loss Optimize image features, including classification loss As shown in formula (11):

[0095]

[0096]

[0097] Where i represents the joint image index, u represents the image lesion category, represents the probability that the i-th joint image belongs to the u-th lesion category, c1 represents the total number of disease categories, represents the probability that the joint image as a whole belongs to grade v, and c2 represents the total number of disease grades.

[0098] The other modal feature extraction module 4 includes:

[0099] Other modal feature extraction units are used to extract basic information of all joints and laboratory examination data features using MLP to obtain feature u M and u S ;

[0100] Cross attention unit, used to M and u S Calculate cross attention to model the cross-modal association between basic information and laboratory test data. Weight the features of both according to the similarity to obtain the weighted feature f M and f S , the cross attention mechanism will u S Transformed into key vector K and value vector V,u M Transform into query vector Q for cross attention calculation.

[0101] The cross attention layer can be expressed as formula (12):

[0102]

[0103] z e =LN(MHA(Q, K, V)+u M ),

[0104] h e =LN(MLP(z e )+z e )#(13)

[0105] Where MHA(Q, K, V) represents the cross attention function, d K represents the dimension of the key vector K, z e Represents the output of the cross-attention mechanism.

[0106] h e represents other modal features after the cross attention layer, as shown in formula (13):

[0107] h e =[f M , f S ]#(13).

[0108] The feature fusion module 5 includes:

[0109] Feature fusion unit is used to combine the features corresponding to each joint image And the enhanced feature f M and f S Input into Transformer P for feature interaction and fusion to obtain accurate and comprehensive multimodal representation and obtain fusion feature f P , where the structure of Transformer P is the same as that of Transformer D in the extraction module;

[0110] Hierarchical loss calculation unit, used to combine the fusion feature f of the feature fusion module P Input into the second feedforward network FFN, select GELU(·) function as the activation function, and obtain the disease activity grade prediction of multimodal data And use the cross entropy loss shown in formula (9) to calculate the classification loss

[0111]

[0112] Where c represents the total number of disease activity grades, p u represents the probability that the input case belongs to grade u.

[0113] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the rheumatoid arthritis activity grading device based on multimodal data as described above.

[0114] Accordingly, the present application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the above-mentioned rheumatoid arthritis activity grading device based on multimodal data.

[0115] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present application are indicated by the claims.

[0116] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A rheumatoid arthritis activity grading device based on multimodal data, characterized in that: include: An image preprocessing module is used to perform normalized coding on multiple joint images of a case to obtain a coding matrix, and convert the coding matrix into joint image features; The extraction module is used to extract all joint image features using Transformer D to obtain the features and classification features corresponding to each joint image; An optimized image representation module is used to input the features and classification features corresponding to each joint image into a first feedforward neural network (FFN) to obtain a lesion classification prediction for the corresponding image and a disease activity grading prediction for the entire image, and calculate a first cross entropy loss with the actual image lesion category and disease activity grading label; The other modality feature extraction module is used to extract other modality features of all joints using MLP, obtain the features of basic information and laboratory examination data respectively, and then use the cross-attention mechanism to enhance the two to obtain enhanced features; The feature fusion module is used to input the features corresponding to each joint image and the enhanced features into the feature fusion Transformer P for feature interaction and fusion to obtain fused features, and then input the fused features into the second feedforward neural network FFN to obtain the disease activity grade prediction of multimodal data, and calculate the second cross entropy loss between the disease activity grade prediction and the actual disease activity grade; a loss function optimization module for training a disease activity grading prediction network using a total loss as a linear combination of a first cross entropy loss and a second cross entropy loss, and iteratively updating parameters of the prediction network until the total loss function reaches a preset convergence condition, wherein the disease activity grading prediction network comprises an extraction module, an optimized image representation module, and a feature fusion module; The disease grading prediction module is used to input multiple joint images, basic information and laboratory test data of a given case into a trained disease activity grading prediction network, and output the disease activity level to which the case belongs.

2. The device according to claim 1, characterized in that The image preprocessing module includes: Unified image size unit, used to use bilinear interpolation algorithm to convert all images They are all scaled to a fixed size so that the input image meets the input specifications of the image feature extraction network; Normalized coding unit, used to convert images of the same size All pixel values ​​in the image are divided by 255, and then the values ​​of the image are Perform normalization processing to obtain image normalization coding; Linearization representation unit, used to flatten each encoding matrix into a one-dimensional vector , which is mapped to ; An image feature output unit is used to output the image feature to the Add classification vector and positional encoding , get the image features , as shown in formula (1); 3. The device according to claim 1, characterized in that The extraction module comprises: A self-attention unit is used to extract image features using Transformer D, where Transformer D consists of L stacked self-attention layers, where: Transformer D As input, the L self-attention layers in Transformer D are used to analyze the image features Enhanced; the output of the Lth self-attention layer This is the output of Transformer D, as shown in formula (5):

4. The device according to claim 1, characterized in that Optimize image characterization modules, including: Image lesion classification prediction unit, used to obtain lesion classification prediction of the corresponding image using the first feedforward neural network layer (FFN) and overall disease activity grading prediction based on imaging ; Image feature optimization unit for using classification loss Optimize image features, including classification loss As shown in formula (6): ； ； in represents the joint image index, Indicates the type of imaging lesion, Indicates the The joint image belongs to The probability of each lesion category, c1 represents the total number of disease categories, represents the probability that the joint image as a whole belongs to grade v, and c2 represents the total number of disease grades.

5. The device according to claim 1, characterized in that Other modal feature extraction modules include: Other modal feature extraction units are used to extract basic information of all joints and laboratory examination data features using MLP to obtain features ; Cross attention unit, used to and Calculate cross attention and weight the features of both according to the similarity to obtain weighted features and , the cross attention mechanism will Transformed into a key vector Sum value vector , Transformed into query vector Perform cross-attention calculations; represents other modal features after the cross attention layer, as shown in formula (8):

6. The device according to claim 1, characterized in that Feature fusion module, including: Feature fusion unit is used to combine the features corresponding to each joint image And the enhanced features and Input into Transformer P for feature interaction and fusion to obtain fused features , where the structure of Transformer P is the same as that of Transformer D in the extraction module; Hierarchical loss calculation unit, used to combine the fusion features of the feature fusion module Input to the second feedforward network FFN, select Function as activation function to obtain disease activity grade prediction of multimodal data , and use the cross entropy loss shown in formula (9) to calculate the classification loss ; ； Where c represents the total number of disease activity grades, represents the probability that the input case belongs to grade u.

7. An electronic device comprising: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the rheumatoid arthritis activity grading device based on multimodal data as described in any one of claims 1 to 6.

8. A computer-readable storage medium having computer instructions stored thereon, wherein when the instructions are executed by a processor, the device for grading rheumatoid arthritis activity based on multimodal data as described in any one of claims 1 to 6 is implemented.

Images