Axis type spondyloarthritis auxiliary analysis system based on multi-modal data fusion

Abstract

The invention discloses a multi-modal data fusion-based auxiliary analysis system for central axis type spondyloarthritis, which integrates medical images and clinical data, and realizes automatic processing and comprehensive analysis of data through preprocessing, coding, fusion and prediction modules. The system extracts image features by using a deep residual network, and fuses clinical data through a cross-modal attention mechanism, so that the diagnosis accuracy is improved, the automatic process reduces manual intervention, and an analysis result is provided for a doctor as a reference.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data fusion, and in particular to an auxiliary analysis system for axial spondyloarthritis based on multimodal data fusion. Background Art

[0002] Axial spondyloarthritis (axSpA) is a chronic inflammatory joint disease that primarily affects the sacroiliac joints and spine. The treatment of axSpA is mainly based on medication to inhibit the progression of the disease, however, the efficacy of medication decreases as the disease progresses. Therefore, it is very important to diagnose patients in a timely manner.

[0003] Medical imaging is one of the important diagnostic criteria for axSpA. Magnetic resonance imaging (MRI) has the advantages of being non-invasive, radiation-free, and having high resolution. It has images of multiple different modalities, such as T1 and T2. The information represented by images of different modalities has a certain complementary relationship. Joint learning using multimodal MRI images can obtain comprehensive and accurate feature representation information of the described object and improve the accuracy of classification results. However, the early imaging features of axSpA often lack specificity, and the image interpretation process is limited by the experience of clinicians, resulting in unreliable disease information obtained only from images.

[0004] Clinical data refers to information related to the patient's health status collected during medical activities, including medical history, physical examination, laboratory tests, imaging examinations, pathological examinations, genomic examinations and other results. These data are important bases for doctors to diagnose diseases, formulate treatment plans, and monitor the condition. Clinical data related to the diagnosis of axSpA, such as HLA-B27 positivity, can assist in diagnosis to a certain extent. However, although HLA-B27 positivity is closely related to axSpA, it is not a specific indicator for the diagnosis of axSpA, and some non-axSpA patients may also be positive. Therefore, the diagnosis of axSpA requires a comprehensive judgment based on the characteristics of multiple modalities.

[0005] The multimodal data fusion method combines data from different sources and different types, can learn effective feature representation from the original data, is suitable for processing structured and unstructured data, and has strong robustness and scalability. These advantages make it widely used in the diagnosis of axSpA, effectively make up for the lack of single modality information, significantly improve the accuracy of diagnosis, and provide important support for the early screening and treatment of axSpA.

[0006] Therefore, a system that can fuse MRI images and clinical information to assist in the diagnosis of axSpA has important application value. The system can use deep learning models to fully mine implicit label information in heterogeneous data such as imaging and clinical data, and fuse and complement the extracted information to obtain data-driven auxiliary diagnostic opinions, which can be compared with clinical priors to assist doctors in diagnosing the disease. Summary of the invention

[0007] In view of the shortcomings of the prior art, the purpose of the present invention is to provide an auxiliary analysis system for axial spondyloarthritis based on multimodal data fusion, which utilizes deep learning technology to extract hidden features from MRI images and clinical data, and fuses and complements them, and finally the classifier outputs the analysis results for doctors to assist in reference.

[0008] To achieve the above object, the present invention provides the following technical solution: an auxiliary analysis system for axial spondyloarthritis based on multimodal data fusion, comprising:

[0009] Case data acquisition module, which extracts medical images and clinical data related to patients from the server;

[0010] Data preprocessing module, which preprocesses the read medical images and clinical data;

[0011] The image data encoding module inputs the images of T1 and T2 into the encoder respectively and splices them to obtain image representation;

[0012] A clinical data coding module, which inputs clinical form data into a coder for coding;

[0013] The data fusion module fuses the preprocessed clinical data with the image representation based on multimodal data to obtain a combined representation;

[0014] The prediction module inputs the combined representation into the classifier to obtain the analysis result.

[0015] As a further improvement of the present invention, the medical image includes MRI data of T1 and T2 modalities.

[0016] As a further improvement of the present invention, the preprocessing includes standardization of clinical data, cropping and denoising of image data.

[0017] As a further improvement of the present invention, the image data encoding module extracts the image features of the two modalities T1 and T2 through ResNet to obtain the corresponding encoding vector x T1_enc and x T2_enc , concatenate and expand the dimension to get x fusion .

[0018] As a further improvement of the present invention, the clinical data encoding module converts clinical data into a high-dimensional vector representation x by one-hot encoding and MLP mapping. final_table .

[0019] As a further improvement of the present invention, the data fusion module converts x fusion With x final_table Perform cross-modal attention fusion to obtain the final merged representation vector x final .

[0020] As a further improvement of the present invention, the prediction module converts x final The data is input into the classifier, the axSpA judgment result is analyzed, and the result is displayed back to the professional as an auxiliary reference.

[0021] Beneficial effects of the present invention:

[0022] 1) In the present invention, multimodal data is used for diagnosis and prediction of axSpA. The high-dimensional feature vectors in the image and clinical data are automatically extracted by the encoder, and fusion operations are performed in the high-dimensional space to achieve information complementarity between the modalities.

[0023] 2) In the present invention, the diagnostic efficiency can be improved by using automated computer-aided diagnostic tools.

[0024] 3) In the present invention, a multimodal data-driven axSpA diagnostic model is constructed to make up for the shortcomings of single-modality clinical data or medical images in disease diagnosis and provide clinicians with more reliable auxiliary diagnostic opinions as a reference. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a structural diagram of an embodiment of an axial spondyloarthritis auxiliary analysis system based on multimodal data fusion according to the present invention;

[0026] Figure 2 It is a schematic diagram of the structure of the encoder of the present invention;

[0027] Figure 3 It is a schematic diagram of the structure of cross-modal data fusion of the present invention;

[0028] Figure 4 This is a schematic diagram of the encoder structure of the present invention translated from English to Chinese;

[0029] Figure 5 Schematic diagram of the English-Chinese translation of the structure of cross-modal data fusion of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described below in conjunction with the embodiments given in the accompanying drawings.

[0031] Reference Figure 1-5 As shown, an auxiliary analysis system for axial spondyloarthritis based on multimodal data fusion includes:

[0032] The case data acquisition module extracts medical images and clinical data related to the patient in the server. The medical images include MRI data in T1 and T2 modes. This step ensures the integrity of the data and provides a basis for subsequent analysis.

[0033] The data preprocessing module preprocesses the read medical images and clinical data. The preprocessing includes standardization of clinical data, cropping and denoising of image data. Through the preprocessing steps, the consistency and comparability of the data are improved, laying the foundation for subsequent feature extraction and fusion.

[0034] The image data encoding module inputs the images of T1 and T2 into the encoder respectively and splices them to obtain image representation. The image data encoding module extracts the image features of the two modalities of T1 and T2 through a deep residual network (ResNet) to obtain the corresponding encoding vector x T1_enc and x T2_enc , concatenate and expand the dimension to get x fusion ,This step utilizes the powerful feature extraction capability of the deep residual network (ResNet) to capture key information from multimodal images and provide rich feature representation for subsequent fusion and prediction.

[0035] The clinical data encoding module inputs the clinical table data into the encoder for encoding. The clinical data encoding module converts the clinical data into a high-dimensional vector representation xf through one-hot encoding and MLP mapping. inal_table This step converts unstructured clinical data into a format that can be processed by machine learning models through encoding transformation, while retaining the original information and relevance of the data.

[0036] The data fusion module fuses the preprocessed clinical data with the image representation based on multimodal data to obtain a combined representation. The data fusion module fusion With x final_table Perform cross-modal attention fusion to obtain the final merged representation vector x final ,This step adopts the cross-modal attention mechanism to automatically learn the weight distribution between ,data of different modalities, achieve effective fusion and complementarity of information, and improve ,the accuracy of diagnosis.

[0037] The prediction module will combine the representations into a classifier to obtain the analysis results. The prediction module will finalThe data is input into the classifier, the axSpA judgment result is analyzed, and the result is displayed to the professionals as an auxiliary reference. This step utilizes the powerful classification ability of the classifier to conduct a comprehensive analysis of the fused multimodal data to provide auxiliary reference.

[0038] Specifically, the process of the data preprocessing module is as follows:

[0039] Read all clinical data x; set a discrimination threshold, which is a positive integer. When the number of values ​​of the variable in x is less than the threshold, it is considered a categorical variable, otherwise it is a continuous variable; for continuous variables, standardize them so that their distribution becomes a distribution with a mean of 0 and a variance of 1;

[0040] Read all MRI-T1 and MRI-T2 image data; scale all MRI slices to a resolution of 224*224 to meet the input image size required by the deep residual network (ResNet); use up and down sampling to ensure that the number of slices in each sequence is fixed at 20.

[0041] Specifically, the process of the image data encoding module is as follows:

[0042] Read T1 and T2 image data x T1 and x T2 ; Use deep residual network (ResNet) and MLP as image encoder; x T1 and x T2 The two image modalities share all the parameters of the deep residual network (ResNet), so that different inputs can complement and correct the parameters in real time, and the encoding vectors x of the two modalities are obtained. T1_enc and x T2_enc ;

[0043] x T1_enc and x T2_enc Splice to x img_hidden , and then input into MLP to get the same as x final_table The feature vector x of the same dimension fusion .

[0044] Specifically, the process of the clinical data coding module is as follows:

[0045] For the categorical variables, perform one-hot embedding operations on them to obtain the encoding x of the categorical variables emb ;

[0046] For continuous variables, each variable is input into an MLP model, and high-dimensional mapping is performed in advance to obtain the continuous variable encoding x con_MLP After the above operation, first concatenate xemb and x con_MLP For x table_hidden , and then input into MLP to get the high-dimensional vector x final_table .

[0047] Specifically, the process of the data fusion module is as follows:

[0048] x img_hidden As Q, x table_hidden As K, V, input into the multi-head attention layer, we get the attention x of clinical data on medical images. table_img_attention ;

[0049] x table_hidden As Q, x img_hidden As K, V, input into the multi-head attention layer, we get the attention x of medical images on clinical data. img_table_attention ;

[0050] Splicing x final_table With x img_table_attention For x table , splicing x fusion With x table_img_attention For x img ;

[0051] Specifically, the process of the prediction module is as follows:

[0052] x table Input into the classifier of clinical data to obtain the logits vector x of clinical data table_prob , x img Input into the medical image classifier to obtain the logits vector x of the medical image img_prob ;

[0053] x table_prob and x img_prob Add them up, and then input the result into softmax to get the normalized probability distribution;

[0054] Use argmax to obtain the binary classification ID of the disease diagnosis, 0 means no disease, and 1 means disease.

[0055] The present invention will be further described below in conjunction with the accompanying drawings:

[0056] Clinical data and MRI-T1 and MRI-T2 medical images were collected, and attribute alignment was performed using the case number as the unique identifier after desensitization;

[0057] The clinical data x is divided into categorical variables and continuous variables and encoded separately. The specific steps are as follows:

[0058] Set a discrimination threshold, which is a positive integer. When the number of values ​​of the variable in x is less than the threshold, it is identified as a categorical variable x. cls , otherwise it is a continuous variable x con ;

[0059] Perform one-hot embedding operation on the categorical variable to obtain the encoding x of the categorical variable emb :

[0060] x emb =f emb (x cls θ emb )

[0061] where θ emb Parameters of the Embedding layer;

[0062] The continuous variable is Z-score standardized and scaled to a distribution with a mean of 0 and a variance of 1:

[0063] Z=(x con -μ) / σ

[0064] Where μ is the mean of the data set and σ is the standard deviation of the data set;

[0065] Input Z into the MLP model for high-dimensional mapping to obtain the continuous variable encoding x con_MLP :

[0066] x con_MLP =f MLP (x con θ MLP )

[0067] where θ MLP are the MLP model parameters;

[0068] x emb and x con_MLP Splice to x table_hidden , this vector serves as the attention vector for clinical data:

[0069] x table_hidden =Concat(x emb , x con_MLP )

[0070] x table_hidden Input into MLP to get a high-dimensional vector x final_table :

[0071] x final_table =f MLP (x table_hidden θ MLP )

[0072] where θ MLP are the parameters of the MLP model.

[0073] The medical imaging data x T1 and x T2 Fusion, the specific steps are as follows:

[0074] The 3D ResNet-101 model is used to encode MRI image data, accepting T1 and T2 data inputs in sequence to ensure that the two modalities share all model parameters:

[0075] x T1_enc =ResNet(x T1 θ encoder )

[0076] x T2 _e nc =ResNet(x T2 θ encoder )

[0077] where e encoder are the parameters of ResNet;

[0078] x T1_enc and x T2_enc Perform a simple merge to form eigenvectors x that are twice as large as each other img_hidden , as the attention vector of medical images:

[0079] x img_hidden =Concat(x T1_enc， x T2_enc )

[0080] x img_hidden The input decoding layer obtains a feature vector x with the same dimension as the clinical vector representation fusion :

[0081] x fusion =f decoder (x img_hidden ;O decoder )

[0082] O decoder are the parameters of the decoder, and the decoding layer is MLP.

[0083] To perform cross-modal attention fusion on the data, the specific steps are as follows:

[0084] x img_hidden As Q, x table_hidden As K, V, input into the multi-head attention layer, we get the attention x of clinical data on medical images. table_img_attention :

[0085] xtable_img_attention =attention(Q img , K table , V table )

[0086] x table_hidden As Q, x img_hidden As K, V, input into the multi-head attention layer, we get the attention x of medical images on clinical data. img_table_attention :

[0087] x img_table_attention =attention(Q t.ble , K img , V img )

[0088] Splicing x final_table With x img_table_attention For x table , splicing x fusion With x table_img_attention For x img , combining two different fusion angles to make the final prediction.

[0089] x table =Concat(x final_table , x tmg_table_attention )

[0090] x img =Concat(x fusion , x table_img_attention )

[0091] The final representation vector is input into the decoder to obtain the prediction result of axSpA. The specific steps are as follows:

[0092] x table Input into the classifier of clinical data to obtain the logits vector x of clinical data table_prob :

[0093] x table_prob = TableClassifier(x table )

[0094] Where TableClassifier is MLP;

[0095] x img Input into the medical image classifier to obtain the logits vector x of the medical image img_prob :

[0096] x img_prob =ImgClassifier(x img )

[0097] Among them, ImgClassifier is the MLP of different TableClassifiers;

[0098] x table_prob and x img_prob Add them up and then input the result into Softmax to get the normalized probability distribution:

[0099]

[0100] in It means element-wise addition;

[0101] Use argmax to obtain the binary classification ID of the disease diagnosis, 0 means no disease, 1 means disease:

[0102] predict=argmax(x logits )

[0103] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. An auxiliary analysis system for axial spondyloarthritis based on multimodal data fusion, characterized in that: include: Case data acquisition module, which extracts medical images and clinical data related to patients from the server; Data preprocessing module, which preprocesses the read medical images and clinical data; The image data encoding module inputs the images of T1 and T2 into the encoder respectively and splices them to obtain image representation; A clinical data coding module, which inputs clinical form data into a coder for coding; The data fusion module fuses the preprocessed clinical data with the image representation based on multimodal data to obtain a combined representation; The prediction module inputs the combined representation into the classifier to obtain the analysis result.

2. The axial spondyloarthritis auxiliary analysis system based on multimodal data fusion according to claim 1, characterized in that: The medical images include MRI data in T1 and T2 modalities.

3. The axial spondyloarthritis auxiliary analysis system based on multimodal data fusion according to claim 2 is characterized in that: The preprocessing includes standardization of clinical data, cropping and denoising of image data.

4. The axial spondyloarthritis auxiliary analysis system based on multimodal data fusion according to claim 3 is characterized in that: The image data encoding module extracts the image features of the two modalities T1 and T2 through ResNet to obtain the corresponding encoding vector x T1_enc and x T2_enc , concatenate and expand the dimension to get x fusion .

5. The axial spondyloarthritis auxiliary analysis system based on multimodal data fusion according to claim 4 is characterized in that: The clinical data encoding module converts clinical data into a high-dimensional vector representation x through one-hot encoding and MLP mapping. final_table .

6. The axial spondyloarthritis auxiliary analysis system based on multimodal data fusion according to claim 5, characterized in that: The data fusion module converts x fusion With x final_table Perform cross-modal attention fusion to obtain the final merged representation vector x final .

7. The axial spondyloarthritis auxiliary analysis system based on multimodal data fusion according to claim 6, characterized in that: The prediction module will final The data is input into the classifier, the axSpA judgment result is analyzed, and the result is displayed back to the professional as an auxiliary reference.

Images