A Deep Learning-Based Automatic Classification Method for MRI Images of Synovial Diseases

Abstract

This invention discloses an automatic classification method for MRI images of synovial diseases based on deep learning, belonging to the field of medical image processing technology. The invention uses a segmentation network to segment the lesion regions of each image in the input MRI image sequence, generating a lesion mask map, which is then combined with the original MRI images to form a joint input. In the synovial disease classification network, features are extracted by combining 2.5D and 3D network architectures to classify synovial diseases. This method optimizes segmentation performance by introducing position-aware loss in the segmentation network and improves classification accuracy by combining multi-dimensional features in the classification network, demonstrating good clinical application value and promising prospects for wider application.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to an automatic classification method for MRI (Magnetic Resonance Imaging) images of synovial diseases based on deep learning. Background Technology

[0002] Synovitis is a common condition that can cause joint pain and dysfunction, significantly impacting patients' quality of life. The largest synovial lining layer in the human body is located in the knee joint, making it one of the most susceptible joints to synovitis. Therefore, accurate identification of knee synovitis is a crucial task in clinical practice. However, accurate diagnosis presents various challenges. One challenge is differentiating knee synovitis from other types of arthritis, as their treatments may differ. A typical scenario is differentiating pigmented nodular synovitis (PVNS) from gouty arthritis (GA). Lesions such as lumps caused by PVNS can sometimes have similar imaging features to gouty tophi, leading to misdiagnosis and delayed treatment.

[0003] In recent years, deep learning methods for MRI recognition have become an important technology in clinical medicine, significantly improving the accuracy and efficiency of diagnosis. Related methods have been well-developed for identifying various diseases, such as pulmonary nodules and cancer. While research on other joints or other medical data has been conducted on synovitis, research on knee MRI images remains in the exploratory stage. Recent research has proposed a method for classifying several knee joint diseases, including PVNS and GA, representing a significant breakthrough. They segmented the suprapatellar bursa and infrapatellar fat pad and employed several traditional machine learning models for classification, achieving competitive performance. However, two problems remain in the current classification of PVNS and GA. The first is how to enhance the model's focus on lesions caused by PVNS and GA. Lesions such as synovial membranes, nodular masses, edema, and tophi may be distributed over a large area on MRI, exhibiting various shapes and sizes, making it difficult for models to accurately perceive their location. Classification models, especially 3D convolutional neural networks, may not be able to fully extract the critical features of such non-fixed regions. Therefore, segmentation-based schemes are beneficial for recognition. The second is how to achieve effective model training with limited data. On the one hand, PVNS is a relatively rare disease, making the collection of large amounts of data challenging. On the other hand, differentiating PVNS from GA based on MRI is not a straightforward task, and data annotation by clinicians is both laborious and time-consuming. Therefore, deep learning-based solutions can fully realize their potential. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes an automatic classification method for MRI images of synovial diseases based on deep learning. This method combines segmentation and classification to enhance the model's ability to focus on lesion areas, while also improving classification performance through feature fusion of different dimensions. The aim is to achieve accurate and efficient automatic classification of synovial diseases.

[0005] The technical solution adopted in this invention is as follows:

[0006] A deep learning-based automatic classification method for MRI images of synovial diseases, comprising the following steps:

[0007] A segmentation network is used to segment the synovial lesion region of the input knee joint MRI image sequence (original MRI image sequence), generating a segmentation mask map (i.e., prediction mask map) of the synovial lesion region in each frame of the image sequence; this step aims to allow the model to first learn and locate the spatial location of the lesion.

[0008] The input knee MRI image sequence includes coronal and sagittal sequences; the predicted mask map includes: spatial location information of the lesion area and boundary contour information of the lesion area.

[0009] The segmentation mask image is merged with the corresponding original MRI image in the channel dimension to form a joint classification input sequence, which is an MRI slice sequence for synovial disease classification.

[0010] The joint classification input sequence is fed into the synovial disease classification network for classification processing to generate synovial disease classification results.

[0011] The synovial disease classification network includes a 2.5D feature extraction branch and a 3D spatial feature extraction branch, a fusion module based on an adaptive weight mechanism, and a classification output module. The inputs to both the 2.5D and 3D spatial feature extraction branches are joint classification input sequences. The 2.5D feature extraction branch comprises several parallel 2D backbone networks, each used to extract 2D features from one frame of the classification input sequence. Then, based on the 3D backbone network, a secondary feature extraction is performed on all extracted 2D features to output a 3D feature map. The 3D spatial feature extraction branch extracts 3D features from the classification input sequence based on the 3D backbone network and outputs a 3D feature map.

[0012] The fusion module introduces a learnable parameter λ to fuse the 3D feature maps output from the 2.5D feature extraction branch and the 3D spatial feature extraction branch. Then, the classification output module classifies the fused features to generate synovial disease classification results. Here, the parameter λ is the fusion weight of the 3D feature map output from the 3D spatial feature extraction branch.

[0013] In this invention, the 2.5D feature extraction branch of the synovial disease classification network (hereinafter referred to as the classification network) is fully trained using a large amount of two-dimensional slice information, while the 3D spatial feature extraction branch focuses on capturing three-dimensional spatial context information. When weighted fusion of the features extracted by the two branches, a learnable parameter is used to dynamically balance the contributions of two-dimensional and three-dimensional information, so as to reduce the negative impact from the 3D branch when the 3D branch cannot be trained well.

[0014] Furthermore, the synovial disease classification network generates two categories of synovial diseases: pigmented villonodular synovitis (PVNS) and gouty arthritis (GA).

[0015] Furthermore, the segmentation network employs a lightweight Ultralight VM-UNet network.

[0016] Furthermore, the segmentation loss function of the segmentation network during training is set to... ;

[0017] in, This represents the binary cross-entropy loss. This indicates Dice's loss. Indicates position loss;

[0018] The position loss is used to constrain the consistency of the projections of the predicted mask and the real mask in different directions (i.e., the sum of pixels in each row or column), rather than strict pixel-to-pixel matching.

[0019] Furthermore, position loss The expression is:

[0020]

[0021] in, This indicates the set of rotation angles, and the set of rotation angles... The multiple rotation angles in the equation are rotation angles that are equally spaced. Represents the set of rotation angles Any rotation angle in, , According to the rotation angle Pixel values ​​of the rotated real mask and the segmented mask.

[0022] Furthermore, rotation angle set The mid-angle interval is set to 45°. (e.g., rotation angle set) The rotation angles are arranged in order of magnitude, with each adjacent rotation angle spaced 45° apart.

[0023] Furthermore, the 2D backbone network of the 2.5D feature extraction branch adopts a 2D backbone network based on residual networks, such as the ResNet-18 residual network backbone network.

[0024] Furthermore, the 3D backbone network is a 3D backbone network based on residual networks.

[0025] Furthermore, the synovial disease classification network uses the classification cross-entropy loss function as the loss function during training.

[0026] Furthermore, during training, the learnable parameter λ of the fusion module in the synovial disease classification network is initially set to 1.

[0027] The technical solution provided by this invention brings at least the following beneficial effects:

[0028] This invention realizes MRI classification of pigmented villonodular synovitis and gouty arthritis based on deep learning, which can be used to assist in medical diagnosis;

[0029] This invention employs a "segmentation-then-classification" strategy and a novel location loss method, enabling the model to focus more precisely on lesion areas and effectively distinguish morphologically similar diseases. Its classification accuracy is superior to that of existing baseline models.

[0030] This invention employs an innovative hybrid 2.5D and 3D network architecture with learnable fusion weights, enabling efficient model training even with limited 3D scan data. It fully utilizes a large amount of 2D slice information while selectively incorporating 3D spatial information, avoiding the problem of training pure 3D models due to insufficient data, and effectively solving the challenge of limited data.

[0031] This invention provides a feasible and automated solution for differential diagnosis of morphologically similar joint diseases in data-constrained clinical settings, significantly improving the efficiency of auxiliary diagnosis while achieving comparable accuracy to clinical data. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a schematic diagram of the lesion masking for PVNS and GA patients in an embodiment of the present invention.

[0034] Figure 2This is a schematic diagram illustrating the processing steps of an automatic classification method for MRI images of synovial diseases based on deep learning, as proposed in an embodiment of the present invention.

[0035] Figure 3 This is a schematic diagram illustrating the calculation principle of location loss in an embodiment of the present invention.

[0036] Figure 4 This is a schematic diagram of the architecture of the hybrid 2.5D and 3D classification network in an embodiment of the present invention.

[0037] Figure 5 This is a schematic diagram of the structure of the 2.5D feature extraction branch in an embodiment of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be described in detail and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Generally, the components of the embodiments of the present invention described and shown in the accompanying drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present invention.

[0039] This invention provides an automatic classification method for MRI images of synovial diseases based on deep learning. It adopts a strategy of lesion region segmentation and multi-dimensional feature fusion classification, combined with 2.5D and 3D network structures, to achieve accurate identification of synovial diseases in knee MRI.

[0040] In one embodiment, see Figure 1 The automatic classification method for MRI images of synovial diseases based on deep learning provided in this embodiment includes the following steps:

[0041] Step S1: Input the original MRI images of the knee joint, including coronal and sagittal sequence images;

[0042] Step S2: Use a lightweight segmentation network (such as a UNet-structured segmentation network) to automatically segment the synovial lesion region and obtain a synovial lesion mask image, including: spatial location information of the lesion region and boundary contour information of the lesion region;

[0043] In this embodiment, the segmentation network adopts the Ultralight VM-UNet structure, and a position-aware loss is introduced into the loss function to improve the lesion localization accuracy;

[0044] Step S3: The original MRI image and the corresponding mask image are concatenated at the channel level to form a joint classification input sequence for classification.

[0045] Step S4: The joint classification sequence obtained in step S3 is input into the synovial disease classification network. This network includes a 2.5D feature extraction branch and a 3D spatial feature extraction branch, a fusion module based on an adaptive weight mechanism, and a classification output module. The inputs to the 2.5D and 3D spatial feature extraction branches are both joint classification inputs. Feature information is extracted through these branches and fed into the fusion module, which performs feature fusion based on an adaptive weight mechanism. The classification output module outputs the synovial disease classification results from PVNS or GA. In this embodiment, the classification output module is a fully connected layer with a softmax function.

[0046] In one embodiment, see Figure 2 The segmentation network uses the Ultralight VM-UNet architecture. This network is an improved design based on U-Net, introducing a lightweight module to reduce the number of parameters. Since this model is not specifically designed for PVNS and GA, it exhibits low sensitivity in initial experiments. To address this issue, this invention designs a new loss function that focuses on the approximate positional constraints of foreground pixels, replacing pixel-by-pixel calculation by calculating the overall prediction probability of each column in the ground truth label and the predicted mask. Specifically, the introduced loss function is as follows:

[0047] 1) Dice loss L dice :

[0048]

[0049] in, , The actual labels (real mask) and predicted masks are located at the pixel positions, respectively. The pixel value at that location. Dice loss is used to enhance the training of the foreground region and suppress background dominance, and is suitable for scenarios with small synovial lesion areas and inter-class imbalance.

[0050] 2) Position loss L loc :

[0051]

[0052] In this invention, position loss is performed based on the real label and the predicted mask image at several equally spaced rotation angles. In the calculation, four equally spaced rotation angles were set in the implementation, among which... Represents any rotation angle, , According to the rotation angle The pixel values ​​of the rotated ground truth label and the predicted mask image. The rotation angle settings (number of rotation angles and angle values) can be configured based on actual processing requirements. In this embodiment, the four rotation angles are set to 0°, 45°, 90°, and 135°. This invention optimizes the network's ability to learn spatial location information by performing consistency calculations on the cylindrical projections of the lesion region in multiple rotation angles (0°, 45°, 90°, 135°).

[0053] 3) The joint loss expression is:

[0054]

[0055] Among them, L bce It is the binary cross-entropy.

[0056] In one embodiment, see Figure 4 The synovial disease classification network employs an innovative parallel fusion architecture of 2.5D and 3D networks. This architecture is specifically optimized for applications with limited medical imaging data to address the issues of insufficient training data for 3D networks and information loss in 2.5D networks. Its structural features include:

[0057] (1) 2.5D feature extraction branch:

[0058] See Figure 5 This branch can use ResNet-18 as its backbone network, with input consisting of MRI slice sequences (image sequences concatenated from original MRI images and corresponding mask images). This branch fully utilizes a large amount of slice data to address the problem of insufficient samples. 2D convolutions and residual blocks extract features from the MRI slices, while 3D operations enhance the model's perception of the fundamental correlations between slices. With sufficient 2D images, the model can undergo comprehensive training. The 3D operations connect the extracted 2D features in a new dimension, forming 2.5D features.

[0059] In one example, the network structure of the 2.5D feature extraction branch in this embodiment is as follows: using the ResNet-18 backbone network as a residual network-based 2D backbone network, multiple parallel 2D backbone networks are set up to perform 2D feature extraction on the input MRI slice sequence, with each 2D backbone network processing one frame of the input MRI slice sequence; that is... Figure 5The value of n is consistent with the number of frames in the input MRI slice sequence. The extracted n-channel 2D features are then fed into a 3D backbone network based on a residual network (optionally, the 2D convolutional layers in the selected 2D backbone network can be replaced with 3D convolutional layers, and the pooling layers can also be replaced with 3D pooling layers). The 3D backbone network first performs dimensional concatenation on the n-channel 2D features to obtain a 3D feature map, and then performs feature extraction based on the 3D convolutional residual structure, outputting a 3D output feature map of the 2.5D feature extraction branch.

[0060] (2) 3D spatial feature extraction branch:

[0061] Employing a lightweight 3D convolutional neural network structure, it extracts spatial voxel features through cascaded stacked 3D convolutional layers, focusing on capturing spatial information.

[0062] In this embodiment, the 3D spatial feature extraction branch uses a 3D backbone network based on residual network to extract 3D spatial features from the input MRI slice sequence, and obtains the 3D output feature map of the 3D spatial feature extraction branch.

[0063] (3) Feature fusion module:

[0064] This module dynamically fuses the 3D output feature maps of the two branches using a learnable channel weighting method to obtain fused features, which are expressed as follows:

[0065]

[0066] A learnable parameter λ is introduced to participate in gradient calculation and optimization, with an initial value of 1. λ will become smaller when the 3D branch cannot be trained well to reduce the negative impact from the 3D branch.

[0067] (3) Classification output module:

[0068] The fused features are input into a fully connected layer network, combining features extracted from both sagittal and coronal views along the channel dimension. The model outputs a vector containing predicted probabilities for each class, including PVNS and GA.

[0069] It should be noted that in this invention, the segmentation network and the synovial disease classification network are trained separately, wherein the loss function used during the training of the segmentation network is: Classification networks can be trained using the cross-entropy loss method directly during training.

[0070] The method proposed in this embodiment of the invention is compared with other existing 3D MRI recognition models. Table 1 shows the performance comparison between different existing classification algorithms and the classification algorithm provided in this embodiment of the invention.

[0071] Table 1

[0072]

[0073] As shown in Table 1, since the listed existing methods are not specifically designed for knee MRI classification, their performance in PVNS and GA classification may not be ideal. Considering the lack of existing solutions with a similar framework to this invention, its competitive performance demonstrates the effectiveness of this invention.

[0074] This invention was experimentally evaluated on an MRI dataset provided by a university hospital. The model performed excellently in classification tasks for PVNS and GA diseases. The segmentation model achieved 97.05% pixel-level accuracy, 72.7% sensitivity, 97.9% specificity, and 45.5% mIoU. The recognition model achieved 93.8% accuracy, 95.8% sensitivity, 91.7% specificity, and 94.4% AUROC. Its performance surpasses that of several existing methods.

[0075] The deep learning-based automatic classification method for MRI images of synovial diseases provided in this invention can assist in the diagnosis of synovial diseases. By combining 2.5D and 3D technologies, the classification performance of synovial diseases can be significantly improved, achieving an accuracy comparable to that of human experts, and greatly increasing the diagnostic speed, thus enabling the auxiliary diagnosis of synovial diseases.

[0076] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0077] Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include at least one of those features.

[0078] Any process or method description described in this specification can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order according to the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0079] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0080] Note that, unless otherwise explicitly stated, all features disclosed in this specification can be replaced by alternative features for achieving the same, equivalent, or similar purpose. Therefore, unless explicitly stated otherwise, each disclosed feature is merely one example of a set of equivalent or similar features. Where used, "further," "preferably," "even further," and "more preferably" are simply starting points for describing another embodiment based on the foregoing embodiments, the combination of which with the foregoing embodiments constitutes the complete configuration of another embodiment. Any combination of several "further," "preferably," "even further," or "more preferably" settings following the same embodiment constitutes yet another embodiment.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0082] The above descriptions are merely some embodiments of the present invention. Those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A deep learning-based automatic classification method for synovial membrane disease MRI images, characterized by, The method comprises the following steps: The synovial lesion region of the input knee MRI image sequence is segmented by using a segmentation network to generate a segmentation mask image of the synovial lesion region of each frame of the image sequence; wherein the input knee MRI image sequence comprises a coronal sequence image and a sagittal sequence image; the predicted mask image comprises spatial position information of the lesion region and boundary contour information of the lesion region; The segmentation mask image and the corresponding knee MRI image are merged in the channel dimension to form a joint classification input sequence; The joint classification input sequence is sent to the synovial disease classification network for classification processing to generate a synovial disease classification result; The synovial disease classification network comprises a 2.5D feature extraction branch and a 3D spatial feature extraction branch, a fusion module based on an adaptive weight mechanism, and a classification output module; wherein the input of the 2.5D feature extraction branch and the 3D spatial feature extraction branch is the joint classification input sequence, the 2.5D feature extraction branch comprises a plurality of parallel 2D backbone networks, each of which is used for 2D feature extraction on one frame of the classification input sequence, and then based on a 3D backbone network, the extracted 2D features are extracted again to output a 3D feature map; the 3D spatial feature extraction branch extracts 3D features based on a 3D backbone network on the classification input sequence to output a 3D feature map; The fusion module introduces a learnable parameter λ to fuse the 3D feature maps output by the 2.5D feature extraction branch and the 3D spatial feature extraction branch, and then the classification output module classifies the fused features to generate a synovial disease classification result; wherein the parameter λ is the fusion weight of the 3D feature map output by the 3D spatial feature extraction branch.

2. The method of claim 1, wherein the step of determining the location of the mobile device is performed by the mobile device. The synovial disease classification result generated by the synovial disease classification network comprises pigmented villonodular synovitis and gouty arthritis.

3. The method of claim 1, wherein the step of determining the location of the mobile device is performed by a location server. The segmentation network adopts a lightweight Ultralight VM-UNet network.

4. The method of claim 1, wherein the step of determining the location of the mobile device is performed by a location server. The segmentation loss function of the segmentation network during training is set as: ； wherein, denotes binary cross-entropy loss, denotes Dice loss, denotes position loss; The position loss is used to constrain the consistency of the projections of the predicted mask image and the real mask image in different directions.

5. The method as described in claim 4, characterized in that, Position loss The expression for position loss is: ； wherein, denotes a set of rotation angles set, and the set of rotation angles the plurality of rotation angles in the set of rotation angles denotes any one rotation angle in the set of rotation angles , , are pixel values of the real mask map and the segmentation mask map respectively after rotation according to the rotation angle .

6. The method of claim 5, wherein the step of determining the location of the mobile device is performed by the mobile device. Set of rotation angles The middle angle interval is set to 45°.

7. The method of claim 1, wherein the step of determining the location of the mobile device is performed by a location server. The 2D backbone network of the 2.5D feature extraction branch adopts a 2D backbone network based on a residual network.

8. The method of claim 1, wherein the step of determining the location of the mobile device is performed by a location server. The 3D backbone network is a 3D backbone network based on a residual network.

9. The method as described in claim 1, characterized in that, The loss function of the synovial disease classification network during training adopts a cross-entropy loss function of classification.

10. The method of claim 1, characterized in that, During training of the synovial disease classification network, the initial value of the learnable parameter λ of the fusion module is set to 1.

Images