3DRes-ViT knee osteoarthritis classification method based on multi-modal fusion

Abstract

The invention relates to a knee osteoarthritis (KOA) multi-classification method based on multi-modal fusion, and the method fuses the characteristics of three modals of two different magnetic resonance imaging (MRI) sequences and X-ray films to carry out four classifications on knee osteoarthritis grades, namely normal, mild, moderate and severe. According to the method, a three-dimensional convolutional neural network (3D CNN) is designed to respectively extract three-dimensional shallow layer features of two different MRI sequences and fuse the features, then a dependency relationship between feature channels after fusion is captured through an efficient channel attention (ECA) module, and the dependency relationship is input into a Vision Transformer (ViT) encoder, so that the MRI sequence of the two different MRI sequences can be identified, and the MRI sequence of the two different MRI sequences can be identified. In combination with the advantages of the 3D CNN and the ViT, the local features and the global features of the two MRI sequences are efficiently aggregated, and the output of the ViT is spliced with the X-ray image features extracted by the two-dimensional convolutional neural network, so that the classification performance is further improved. According to the method, through fusion of three pieces of modal information, the model can capture different physiological and anatomical information, and more reliable support is provided for diagnosis of knee osteoarthritis.

Description

Technical Field

[0001] The present invention relates to a knee osteoarthritis classification method, in particular to a knee osteoarthritis multi-classification method based on deep learning. The method integrates medical imaging information of three different modalities to classify knee osteoarthritis. By designing 3DRes CNN (Convolutional Neural Networks), three-dimensional shallow features of two different MRI sequences are extracted and fused. The fused features are firstly passed through an Efficient Channel Attention (ECA) module to effectively capture the dependency between channels, and then input into a Vision Transformer (ViT) encoder to learn the global correlation information between features. The output of ViT is then integrated with X-ray image features extracted by 2D CNN to classify the severity of knee osteoarthritis into four categories. The method enables the model to capture different physiological and anatomical information, and provides more reliable support for the medical diagnosis and treatment of knee osteoarthritis. Background Art

[0002] Knee osteoarthritis (KOA) is the most common form of arthritis and a degenerative disease of the knee joint that causes pain and limited mobility, affecting the quality of life of millions of people. In the late stage of knee arthritis, the abnormalities shown on imaging are usually irreversible, and late-stage knee arthritis (OA) can only be treated with total knee replacement. Therefore, early diagnosis of knee OA is crucial for pathology, clinical decision-making, prediction of disease progression, and treatment. Traditional OA diagnosis requires well-trained or experienced radiologists to complete. Computer-aided diagnosis of KOA can help doctors improve diagnostic efficiency.

[0003] X-rays are a commonly used examination method for diagnosing knee osteoarthritis, and the classification method of knee OA also focuses on processing X-ray images with 2D CNN. However, X-ray images are not sensitive to early OA, and are only highly sensitive to the diagnosis of moderate and severe OA. MRI can accurately display the subtle structure and morphology of various tissues in the joint, which is of immeasurable importance for the early identification of damage and lesions of tissues such as articular cartilage, synovium and ligaments. Therefore, the present invention can achieve complementary advantages of the two imaging technologies by combining the use of these two imaging technologies, thereby further improving the classification accuracy of KOA and providing more reliable support for the diagnosis of knee OA.

[0004] With the rapid development of computer hardware, in the medical field, computer-assisted medical diagnosis methods have become an important field in medical imaging, diagnostic radiology, and computer science. Among them, deep learning, which is more advanced than traditional algorithms, has achieved many excellent results after entering the medical field. X-ray images are not sensitive to early OA, and MRI can better present soft tissues such as cartilage, bone marrow lesions and effusions. Therefore, radiologists in hospitals now often use more sensitive MRI for early detection of OA. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention proposes a 3DRes-ViT knee osteoarthritis classification network based on multimodal fusion, which fuses two different MRI sequences (including sagittal three-dimensional dual echo steady-state echo (SAG 3D-DESS) sequence and coronal medium weighted fast spin echo (COR IW-TSE) sequence) and features extracted from X-ray images for classification. The study found that the features of the two different MRI sequences, SAG 3D-DESS and COR IW-TSE, are complementary. The local features of the two different MRI sequences are extracted by 3D CNN respectively, and then ViT is used to learn the global correlation information between these features. Finally, the X-ray image features extracted by the 2D CNN network are fused, and based on the Kellgren and Lawrence (KL) grading system, the fused information is sent to the classification layer to classify the knee osteoarthritis grade into four categories, namely normal (KL=0), mild OA (KL=2), moderate OA (KL=3), and severe OA (KL=4).

[0006] In order to achieve the above object, the technical scheme of the present invention is as follows: a 3DRes-ViT knee osteoarthritis classification method based on multimodal fusion, comprising the following steps: Step 1: Preprocess the data used, which includes the following sub-steps: (a) Locate the ROI (Region of Interest) of the left and right knees in the X-ray image and convert it into PNG format; (b) Convert two different MRI sequences into Nifti’s standard format, perform center cropping to obtain the ROI sub-region of the MRI sequence image, and remove the start and end slices in each MRI sequence; (c) Perform minimum-maximum normalization and enhanced training on the image data of X-ray and MRI sequences respectively Set samples; Step 2: Use 3D CNN to extract shallow features of SAG 3D-DESS and COR IW-TSE respectively. The shallow feature extraction network consists of 3D Conv block and 3D Res Blocks, and feature extraction is performed through 3D convolution and residual connection; Step 3: The shallow features extracted by the two 3DCNN networks are fused and sent to the channel attention module 3DECA, which specifically includes the following sub-steps: (a) Concatenate the shallow features of the two MRI sequences obtained in step 2; (b) The concatenated 3D shallow features are fed into the 3D ECA module to adaptively learn the weight of each feature channel; Step 4: Feed the features output by the 3D ECA module into the encoder of the Vision Transformer to extract the global interaction information between each feature, which includes the following sub-steps: (a) Map each 3D feature map output by the 3D ECA module into a vector and perform position embedding; (b) Input the position-embedded features into the three-layer ViT encoder; Step 5: Design a 2D CNN network to extract X-ray image features, merge them with the features obtained in step 4, and finally send them to the MLP (Multilayer Perceptron) layer for classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a flow chart of the steps of the method of the present invention; Figure 2 It is the overall architecture diagram of the network model of the present invention; Figure 3 3D Conv block structure diagram of the present invention; Figure 4 This is a structural diagram of the 3D Res block of the present invention. DETAILED DESCRIPTION

[0008] For those skilled in the art, it is understandable that some well-known structures and their descriptions may be omitted in the drawings. The technical solution of the present invention is further described below in conjunction with the drawings and embodiments.

[0009] like Figure 1 As shown, a 3DRes-ViT knee osteoarthritis classification method based on multimodal fusion includes the following steps: Step 1: Preprocess the data used; Step 1.1: Extract the region of interest (ROI) of the image. Annotate the bounding boxes of all X-ray images. Perform center cropping based on the center point of the annotated box to extract an image of size 560×560 and convert it to PNG format. For MRI sequences, first convert the DICOM format to the standard format of the Neuroimaging Informatics Technology Initiative (Nifti), then perform center cropping to obtain the ROI sub-region of the MRI sequence image. The sagittal three-dimensional dual-echo steady-state echo (SAG 3D-DESS) slice size is cropped from 384x384 to 160x160, and the coronal medium-weighted fast spin echo (COR IW-TSE) slice size is cropped from 384x384 to 256x256 and scaled to 160×160. Step 1.2: Remove the slices at the beginning and end of each MRI sequence. The removed slices do not contain bone or cartilage information and also contain too much noise; Step 1.3: Perform minimum-maximum normalization on the image data of X-ray and MRI sequences respectively, and enhance the training set samples by random rotation and gamma correction; Step 2: Construct the MRI sequence feature extraction network 3D Res CNN to extract the shallow features of two different MRI sequences, SAG 3D-DESS and COR IW-TSE, and then splice and fuse them; Step 2.1: Figure 2 As shown in the figure, the shallow feature extraction network of the two MRI sequences is composed of 3D Conv block and 3DRes Blocks. Feature extraction is performed through 3D convolution and residual connection. In order to extract the difference information of SAG 3D-DESS and CORIW-TSE, a 3DCNN network is constructed and trained for the two respectively. They have the same network structure. Step 2.1.1: Figure 3 As shown in the figure, the 3D Conv block includes a 3D convolution layer with a convolution kernel size of 5×5×3 and a stride of (2, 2, 1), a BN layer, and a ReLu activation function; Step 2.1.2: If Figure 4 As shown in the figure, the 3D Res block performs jump connections on the features obtained by 3D convolution. This module uses two convolutional layers. Each block has a convolutional layer of size 3×3×3 and a stride of (1, 1, 1). The same padding operation is performed on all blocks, and batch normalization and ReLU activation function are performed after each convolutional layer. Step 2.2: The final feature dimensions of the two modalities are equal by controlling the stride of the first convolutional layer in each 3D Res block. The depth stride of SAG 3D-DESS is (1, 2, 2, 2), and the depth stride of COR IW-TSE is (1, 1, 1, 1). Step 2.3: Obtain the features of the two MRI sequences, SAG 3D-DESS and COR IW-TSE, by alternately stacking 3D Res blocks, and then perform feature splicing and fusion; Step 3: Figure 2 As shown, the fused features are sent to the channel attention module 3D ECA to adaptively learn the weight of each feature channel to achieve the complementary advantages of the two imaging methods; Step 4: Figure 2 As shown, the features output by the ECA module are fed into the encoder of the three-layer Vision Transformer to extract the global interaction information between various features; Step 4.1: The feature map dimension obtained by the 3D ECA module is (1024, 10, 10, 10). Each 3D feature map (10*10*10) is directly mapped into a vector and positionally embedded as the input of the ViT encoder. Step 4.2: The encoder module of ViT is used to learn the global correlation information between inputs. The present invention uses a three-layer ViT encoder to extract global features; Step 5: Use 2D CNN to extract features of X-ray images; Step 5.1: Figure 2 As shown in the figure, an X-ray feature extraction network is built on the basis of the VGG-19 architecture. The network contains five convolution blocks in total. By stacking 3×3 small-scale convolutional layers with a stride of 1 and a padding of 1 instead of large convolutional layers, the network parameters are reduced without reducing the receptive field. Step 5.2: Different from the VGG-19 network, a BN layer is added after the convolution layer of each module to enhance the generalization ability, and a ReLu activation function is used after the BN layer; Step 5.3: Use a maximum pooling layer of size 2×2 and stride 2 to reduce the dimension after the first four convolutional blocks, and perform global average pooling on the X-ray features after the fifth convolutional block; Step 6: Figure 2 As shown in the figure, the MRI modality global interaction features obtained by the Vision Transformer encoder are concatenated and fused with the features extracted from X-rays, and finally sent to the MLP layer for classification.

Claims

1. A 3DRes-ViT knee osteoarthritis classification method based on multimodal fusion, characterized in that: This method combines medical imaging information from three different modalities to classify knee osteoarthritis. First, a new 3D convolutional neural network (CNN) architecture 3D Res CNN is designed to extract local features of two different MRI sequences and fuse them. The fused features are first passed through the Efficient channel Attention (ECA) module to effectively capture the dependencies between channels, and then enter the encoder of the Vision Transformer (ViT) to learn the global correlation information between features. Finally, the output of ViT is fused with the X-ray image features extracted by 2D CNN and based on the Kellgren and Lawrence (KL) grading standard, knee osteoarthritis is divided into four levels: normal (KL=0), mild OA (KL=2), moderate OA (KL=3), and severe OA (KL=4). This method can capture different physiological and anatomical information through the fusion of three modal information, so it can improve the accuracy of CAD (Computer Aided Diagnosis) diagnosis. The specific steps include: Step 1: Preprocess the image data used; Step 1.1: Extract the region of interest (ROI) of the image. Annotate the bounding boxes of all X-ray images. Perform center cropping based on the center point of the annotated box to extract an image of size 560×560 and convert it to PNG format. For MRI sequences, first convert the DICOM format to the standard format of the Neuroimaging Informatics Technology Initiative (Nifti), then perform center cropping to obtain the ROI sub-region of the image. The sagittal three-dimensional dual-echo steady-state echo (SAG 3D-DESS) slice size is cropped from 384x384 to 160x160, and the coronal medium-weighted fast spin echo (COR IW-TSE) slice size is cropped from 384x384 to 256x256 and scaled to 160×160. Step 1.2: Remove the slices at the beginning and end of each MRI sequence. The removed slices do not contain bone or cartilage information and also contain too much noise; Step 1.3: Perform minimum-maximum normalization on the image data of X-ray and MRI sequences respectively, and enhance the training set samples by random rotation and gamma correction; Step 2: Construct an MRI sequence feature extraction network 3D Res CNN to extract shallow features of two different MRI sequences, SAG 3D-DESS and COR IW-TSE, and then perform splicing and fusion; Step 2.1: The shallow feature extraction network of the two MRI sequences is composed of 3D Conv block and 3D Res Blocks. Feature extraction is performed through 3D convolution and residual connection. In order to extract the difference information of SAG 3D-DESS and COR IW-TSE, a 3DCNN network is constructed and trained for both of them. They have the same network structure. Step 2.1.1: The 3D Conv block includes a 3D convolution layer with a kernel size of 5×5×3, a stride of (2, 2, 1), a BN layer, and a ReLu activation function; Step 2.1.2: 3D Res block performs skip connections on the features obtained by 3D convolution. This module uses two convolutional layers. Each block has a convolutional layer of size 3×3×3 and stride (1, 1, 1), and the same padding operation is performed. Batch Normalization and ReLU activation function are performed after each convolutional layer. Step 2.2: The final feature dimensions of the two modalities are equal by controlling the stride of the first convolutional layer in each 3D Res block. The depth stride of SAG 3D-DESS is (1, 2, 2, 2), and the depth stride of COR IW-TSE is (1, 1, 1, 1). Step 2.3: Obtain the features of the two MRI sequences, SAG 3D-DESS and COR IW-TSE, by alternately stacking 3D Res blocks, and then perform feature splicing and fusion; Step 3: Send the fused features to the channel attention module 3D ECA to adaptively learn the weight of each feature channel to achieve the complementary advantages of the two imaging methods; Step 4: Feed the features output by the ECA module into the encoder of the three-layer Vision Transformer to extract the global interaction information between various features; Step 4.1: The feature map dimension obtained by the 3D ECA module is (1024, 10, 10, 10). Each 3D feature map (10*10*10) is directly mapped into a vector and positionally embedded as the input of the ViT encoder. Step 4.2: The encoder module of ViT is used to learn the global correlation information between inputs. The present invention uses a three-layer ViT encoder to extract global features; Step 5: Use 2D CNN to extract features of X-ray images; Step 5.1: Based on the VGG-19 architecture, an X-ray feature extraction network was built. The network consists of five convolutional blocks. The large convolutional layers are replaced by stacking 3×3 small-scale convolutional layers with a stride of 1 and a padding of 1, which reduces the network parameters without reducing the receptive field. Step 5.2: Different from the VGG-19 network, a BN layer is added after the convolution layer of each module to enhance the generalization ability, and a ReLu activation function is used after the BN layer; Step 5.3: Use a maximum pooling layer of size 2×2 and stride 2 to reduce the dimension after the first four convolutional blocks, and perform global average pooling on the X-ray features after the fifth convolutional block; Step 6: The MRI modality global interaction features obtained by the Vision Transformer encoder are concatenated and fused with the features extracted from X-rays, and finally sent to the MLP layer (Multilayer Perceptron) for classification.

Images