Multi-model fusion musculoskeletal anomaly diagnosis method based on improved lightweight network

Abstract

The invention provides a multi-model fusion musculoskeletal anomaly diagnosis method based on an improved lightweight network. The method based on the convolutional neural network can automatically extract features and solve the complex musculoskeletal anomaly diagnosis problem, but the high accuracy of the convolutional neural network often depends on a complex network architecture, which means that larger calculation amount and parameter amount are needed. In addition, an existing musculoskeletal anomaly diagnosis method usually uses a single network, and due to the fact that the performance of the single network is limited, a certain influence can be caused on a diagnosis result. Therefore, in order to solve the problems, the invention provides an improved OfficientNet-B2 model, and the improved OfficientNet-B2 model and the MobileNetV2 are subjected to model fusion based on the improved model, namely Fusion Net. The method has the advantages that firstly, compared with the OfficientNet-B2, the performance of the improved OfficientNet-B2 is improved, and the parameter quantity and the size of the occupied storage space are reduced by 17%; and secondly, on the basis of the improved OfficientNet-B2, the performance of the FusionNet is further improved, the limitation of the performance of a single network is overcome, meanwhile, the method has the characteristics of a plurality of networks, and the balance between the performance and the complexity is achieved.

Description

Technical Field

[0001] The present invention relates to a method for diagnosing musculoskeletal abnormalities based on deep learning. Specifically, the present invention proposes a multi-model fusion musculoskeletal abnormality diagnosis technology based on an improved lightweight network. The technology can be used as an auxiliary tool for radiologists to diagnose musculoskeletal abnormalities and has good application prospects. Background Art

[0002] The diagnosis of abnormalities is very important for the further treatment of musculoskeletal diseases. Therefore, it is necessary to study an accurate and efficient automatic musculoskeletal abnormality diagnosis method.

[0003] Traditional methods for diagnosing musculoskeletal diseases are usually done by manually extracting specific features. For example, Zhang et al. proposed a computer-aided image classification method for diagnosing osteoarthritis in finger joints. Based on image processing and machine learning, this method can accurately identify signs of osteoarthritis through analysis and feature extraction of optical images, providing doctors with a powerful auxiliary diagnostic tool (Computer-aided classification of optical images for diagnosis of osteoarthritis in the finger joints. J Xray Sci Technol. 2011; 19 (4): 531-44.). Al-Ayyoub et al. proposed a machine learning-based system for automatically detecting the type of long bone fractures using X-ray images (Determining the type of long bone fractures in x-ray images [J]. WSEAS Transactions on Information Science and Applications, 2013, 10 (8): 261-270.). Mahendran et al. proposed a fusion classification technique to detect long bone fractures (An enhanced tibia fracture detection tool using imageprocessing and classification fusion techniques in X-ray images[J].GlobalJournal of Computer Science and Technology,2011,11(14):22-28.). Chai et al. proposed a method to evaluate the efficacy of femoral long bone fractures using Gray-Level Co-occurrence Matrix (GLCM) (Gray-level co-occurrence matrix bone fracture detection[J].WSEAS Transactions on Systems,2011,10(1):7-16.). The feature extraction of the above algorithms mainly relies on the experience of researchers, which may lead to a certain degree of information loss.

[0004] In contrast, automatic learning methods such as deep learning can overcome the above shortcomings to some extent because they can automatically learn features from data and can adapt to different tasks and data distributions.

[0005] In recent years, with the rapid development of deep learning technology represented by convolutional neural networks (CNN), the application of deep learning models in the field of medical image analysis has gradually attracted attention.

[0006] At present, some scholars have conducted some research on musculoskeletal X-ray images using methods based on deep learning neural networks. For example, Cohen et al. used an artificial intelligence algorithm based on deep learning to analyze and diagnose wrist X-rays and compared them with the performance of radiologists in wrist fracture X-ray detection. The study confirmed that artificial intelligence performed better than non-professional radiologists in wrist fracture detection on X-rays (Artificial intelligence vs. radiologist: accuracy of wrist fracture detection on radiographs. Eur Radiol. 2023 Jun; 33 (6): 3974-3983.). Nam et al. proposed a method for automatic diagnosis of nasal bone fractures based on EfficientNet-B7. By learning a large number of X-ray images of nasal bone fractures and non-fractures, the study was able to quickly and accurately classify and judge new X-ray images (Diagnosis of nasal bone fractures on plain radiographs via convolutional neural networks. Sci Rep 12, 21510 (2022).). Oka et al. proposed a novel network architecture based on VGG-16 to diagnose distal radius fractures. This method can automatically analyze and diagnose biplane X-ray images by learning fracture features and patterns (Artificial intelligence to diagnose distal radius fracture using biplane plain X-rays. J Orthop Surg Res. 2021 Nov25; 16(1): 694.). He et al. proposed a method for abnormality detection in musculoskeletal X-ray images using a calibrated deep learning ensemble method. By training a deep learning model and using an ensemble learning method, the accuracy and efficiency of abnormality detection can be improved (A calibrated deep learning ensemble for abnormality detection in musculoskeletal radiographs. Sci Rep. 2021 Apr 27; 11(1): 9097.).Singh et al. proposed a new hybrid architecture ComDNet-512 based on deep neural network to effectively detect bone abnormalities in patients' musculoskeletal X-rays (Hybrid Deep Learning Approach for Automatic Detection in Musculoskeletal Radiographs. Biology (Basel). 2022 Apr 26; 11(5): 665.). Yi et al. used pre-trained ResNet for transfer learning to quickly and accurately classify children's bone X-ray images (Automated semantic labeling of pediatric musculoskeletal radiographs using deeplearning. Pediatr Radiol. 2019 Jul; 49(8): 1066-1070.). Yao et al. proposed a two-stage bone classification and abnormality detection deep learning framework based on X-rays to improve the diagnosis speed of orthopedic diseases (Enhanced deep residual network for bone classification and abnormality detection. MedPhys. 2022 Nov; 49(11): 6914-6929.). Cheng et al. used DenseNet-121 to detect and localize hip fractures on pelvic radiographs (PXR) (Application of a deep learning algorithm for detection and visualization of hip fractures on plain pelvic radiographs. Eur Radiol. 2019 Oct; 29(10): 5469-5477.). Choi et al. proposed a dual-input deep learning model based on ResNet, which automatically detects supracondylar fractures in children in conventional radiographs using anteroposterior and lateral elbow radiographs (Using a Dual-Input Convolutional Neural Network for Automated Detection of Pediatric Supracondylar Fracture on Conventional Radiography. Invest Radiol. 2020 Feb; 55(2): 101-110.).Cheng et al. proposed a multi-scale deep learning algorithm called PelviXNet to detect pelvic and hip fractures on PXR. The study showed that PelviXNet showed comparable performance to radiologists and orthopedic surgeons (A scalable physician-level deep learning algorithm detects previous trauma on pelvic radiographs. Nat Commun. 2021 Feb 16; 12(1): 1066.). Thian et al. used Inception-ResNet and Faster R-CNN for fracture detection and localization on wrist X-rays. The study demonstrated that object detection CNNs have high sensitivity and specificity in detecting and localizing radial and ulnar fractures on wrist X-rays (Convolutional Neural Networks for Automated Fracture Detection and Localization on Wrist Radiographs. Radiol Artif Intell. 2019 Jan 30; 1(1): e180001.). Ye et al. built a deep learning network model based on the DenseNet-169 architecture to distinguish acetabular fractures on pelvic anteroposterior radiographs. The experimental results showed that the model achieved comparable or even better diagnostic performance than clinicians (Development and validation of a deep learning-based model to distinguish acetabular fractures on pelvic anteroposterior radiographs. Front Physiol. 2023Apr 28; 14: 1146910.). Wang et al. proposed a U-Net-based rib fracture detection method that extracts rib fracture features at the pixel level for rapid and accurate detection of rib fractures (Composite Attention ResidualU-Net for Rib Fracture Detection. Entropy (Basel). 2023Mar 7; 25(3): 466.).Jin et al. proposed a deep learning model called FracNet for detecting and segmenting rib fractures (Deep-learning-assisted detection and segmentation of rib fractures from CT scans: Development and validation of FracNet. EBioMedicine. 2020 Dec; 62: 103106.). Ghosh et al. used a patch-based image analysis method and used ResNet for transfer learning to develop a deep learning model for detecting rib fractures in frontal radiographs of children under 2 years old (A Patch-Based Deep Learning Approach for Detecting Rib Fractures on Frontal Radiographs in Young Children. J DigitImaging. 2023 Aug; 36(4): 1302-1313.).

[0007] Although the above-mentioned methods based on convolutional neural networks can automatically extract features and solve complex musculoskeletal abnormality diagnosis problems, the high accuracy of convolutional neural networks often relies on complex network architectures, which means that greater computational effort and parameter volume are required. By increasing one or more dimensions of the convolutional neural network, such as depth and width, the performance of the network is usually improved, but the number of parameters of the network will also increase, making the network too complex to be deployed on edge devices. This is the main disadvantage of the above-mentioned methods based on convolutional neural networks. In addition, these abnormality diagnosis methods usually use a single network, which may have a certain impact on the diagnostic results due to the limited performance of a single network.

[0008] In view of these problems and considering the lightweight and high performance of EfficientNet-B2 and MobileNetV2, the present invention proposes an improved EfficientNet-B2 network model. In order to obtain more accurate musculoskeletal abnormality detection results, a classification method based on the fusion of improved EfficientNet-B2 and MobileNetV2 models is proposed. The present invention verifies the above method using the public dataset MURA.

[0009] Some terminology:

[0010] Machine learning: Machine learning is a branch of artificial intelligence that aims to enable computers to learn and extract patterns from data by building and training models, thereby achieving prediction and decision-making on unknown data.

[0011] Deep learning: Deep learning is a branch of machine learning that simulates the working principles of human neural networks and achieves learning and analysis of data by building deep neural network models.

[0012] Convolutional Neural Network: Convolutional Neural Network is a neural network model specifically designed to process grid-structured data (such as images). It has important applications in image processing and computer vision.

[0013] Lightweight Network: Lightweight networks refer to network architectures with smaller size and lower complexity. They are often used to handle resource-limited environments such as embedded devices, mobile devices, and IoT devices.

[0014] Multi-model fusion: Multi-model fusion refers to the integration and fusion of data from different models to obtain more accurate and comprehensive results or decisions.

[0015] Feature extraction: Feature extraction refers to extracting representative and distinguishable features from raw data to describe and represent important information of the data.

[0016] The computational amount of convolutional neural networks: The computational amount of convolutional neural networks refers to the total amount of computational operations between each layer in the network structure. The measurement indicator of computational amount is usually the number of floating-point multiplication operations (FLOPs).

[0017] Parameters of convolutional neural networks: The number of parameters of convolutional neural networks is an important indicator to measure the scale and complexity of the network. A larger number of parameters usually means a more complex network structure and stronger expressive power.

[0018] Depth of convolutional neural networks: The depth of a convolutional neural network refers to the number of layers in the network. Depth is an important attribute of a convolutional neural network, which has an important impact on the network's expressiveness and learning capabilities.

[0019] Width of Convolutional Neural Network: The width of a convolutional neural network refers to the number of channels or nodes in each layer of the network. Width is an important property of a convolutional neural network, which affects the capacity and complexity of the network.

[0020] EfficientNet-B2: EfficientNet-B2 is a convolutional neural network model in the EfficientNet series. The EfficientNet series aims to provide efficient and high-performance convolutional neural network models to balance computing resources and model performance.

[0021] MobileNetV2: MobileNetV2 is a lightweight convolutional neural network model designed to achieve efficient image recognition and classification tasks with limited computing resources.

[0022] Coordinate Attention: Coordinate Attention (CA) is an attention mechanism used in computer vision tasks, which can select and focus on specific areas in an image instead of simply processing the entire image. This can reduce the amount of computation and improve the efficiency and performance of the model.

[0023] Squeeze and Excitation: Squeeze and Excitation (SE) is an attention mechanism used in computer vision tasks. The SE module enables the network to adaptively adjust the importance of channels by learning the weights between channels, thereby improving the expressiveness of the model.

[0024] Efficient Channel Attention: Efficient Channel Attention (ECA) is an attention mechanism used to enhance the performance of neural networks. Unlike the SE module, the ECA module uses a one-dimensional convolution operation instead of a fully connected layer to reduce computational complexity.

[0025] MBConv: The main component of EfficientNet-B2, used to extract features from input images. Summary of the invention

[0026] The purpose of the present invention is to provide a multi-model fusion musculoskeletal abnormality diagnosis method based on an improved lightweight network, which utilizes the lightweight and high-performance characteristics of the improved EfficientNet-B2 and MobileNetV2 to realize the diagnosis of musculoskeletal abnormalities. The advantages of the present invention are: (1) Compared with EfficientNet-B2, the improved EfficientNet-B2 has fewer parameters and occupies less storage space, and has better model performance; (2) FusionNet overcomes the limitations of a single network performance and has the characteristics of multiple networks, achieving a balance between performance and complexity. (3) Both the improved EfficientNet-B2 and MobileNetV2 can quickly and accurately judge musculoskeletal abnormalities.

[0027] In order to achieve the above object, the present invention adopts the following technical solution:

[0028] (1) Divide the samples to be input into the model into a training set and a validation set in a ratio of 8:2;

[0029] (2) For the training set, firstly, random cropping is performed to crop the image to 260×260. Then, horizontal flipping is performed to enhance the diversity of the data set. Then, data type conversion is performed to convert the image to a tensor. Finally, normalization is performed to make the pixel values ​​of each channel of the image have similar range and distribution. For the validation set, first, the image size is adjusted to 260×260. Then, center cropping is performed to crop the image from the center to 260×260. Then, data type conversion is performed to convert the image to a tensor. Finally, normalization is performed to make the pixel values ​​of each channel of the image have similar range and distribution.

[0030] (3) Take the processed image as input and the corresponding label as output, use FusionNet and improved EfficientNet-B2 for training, monitor and save indicators such as Precision, AUC and Accuracy on the validation set, and select the weight with the best performance on these indicators as the final weight to obtain the optimized model;

[0031] (4) Select a test sample and resize the image to 260×260. Then perform center cropping on the image, cropping the image from the center to 260×260 to retain the central part of the image. Then perform data type conversion to convert the image into a tensor. Finally, perform standardization so that the pixel values ​​of each channel of the image have a similar range and distribution;

[0032] (5) Use the processed image as the input of the optimization model and determine the image label based on the model output. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 The improved MBConv module in the present invention;

[0034] Figure 2 It is the improved EfficientNet-B2 in the present invention;

[0035] Figure 3 It is the principle diagram of FusionNet in the present invention;

[0036] Algorithm 1 is a classification algorithm based on the improved EfficientNet-B2 and MobileNetV2 multi-model fusion in the present invention;

[0037] Figure 4 This is a performance comparison chart of different models in the present invention on the MURA dataset;

[0038] Figure 5The figure is a comparison chart of the parameter quantity, storage space occupied and floating-point operation number of different models in the present invention.

[0039] Specific implementation methods and model effect evaluation

[0040] The present invention is implemented on a PC provided by the Intelligent Information Processing Laboratory of Hunan University of Technology. The hardware environment for network model training is Intel Xeon Gold 5320, Nvidia RTX A4000, 32GB memory and Linux operating system, and the deep learning framework used is Pytorch. In order to objectively compare the performance of each network model and avoid the influence of the training mechanism on the experiment, the parameters designed in the experiment are processed in the same way. Each network model is trained for 150 rounds on the same data set, and the sample batch size is 16.

[0041] The present invention is generally divided into the following contents:

[0042] (1) Improved MBConv module: This module can provide more powerful feature extraction and expression capabilities while maintaining low computational complexity and parameter count;

[0043] (2) Improved EfficientNet-B2: It extracts features from input images while maintaining a good balance between performance and computing resources.

[0044] (3) FusionNet: This network model overcomes the limitations of a single network performance and has the characteristics of multiple networks. The extracted features are more effective, achieving a balance between performance and complexity.

[0045] Model performance evaluation: In this invention, in order to accurately verify the performance of FusionNet and improved EfficientNet-B2, we compared the abnormality diagnosis performance of FusionNet with improved EfficientNet-B2, EfficientNet-B2, MobileNetV2 and ResNet-50 on seven data sets in MURA, and selected Precision, AUC and Accuracy as performance evaluation indicators. Precision is used to measure the accuracy of the model in positive example prediction, AUC is used to measure the classification ability and discrimination of the model, and Accuracy is used to measure the accuracy of the overall classification model. The specific experimental results are as follows: Figure 4As shown, (a)-(g) are the experimental results of each model on the shoulder, elbow, finger, forearm, hand, humerus, and wrist datasets. The experimental results show that the performance of FusionNet on the seven datasets (a)-(g) is significantly better than that of other models, especially on the elbow dataset, where Precision, AUC, and Accuracy reach 92.93%, 93.89%, and 87.10%, respectively. The performance of the improved EfficientNet-B2 on the seven datasets (a)-(g) is lower than that of FusionNet but substantially higher than that of other models, which means that FusionNet overcomes the limitation of the performance of a single network, has the features of multiple networks at the same time, and the extracted features are more effective, which ultimately improves the overall performance. The performance of ResNet-50 on the seven datasets (a)-(g) is substantially better than that of EfficientNet-B2 and MobileNetV2. Therefore, FusionNet and the improved EfficientNet-B2 have excellent performance in the diagnosis of musculoskeletal abnormalities.

[0046] Model lightweight evaluation: In this invention, in order to accurately verify the lightweight of FusionNet and improved EfficientNet-B2, we compared FusionNet with improved EfficientNet-B2, EfficientNet-B2, MobileNetV2 and ResNet-50, and selected Parameters, Model size and Flops as lightweight evaluation indicators. Parameters are usually used to measure the complexity of the model, Model size is usually used to measure the storage requirements and computing resource consumption, and Flops is usually used to measure the computational complexity and speed of the model. The lightweight degree of different models is compared as follows: Figure 5As shown. From the results, it can be seen that the parameters and model size of the improved EfficientNet-B2 are reduced by 1.33M and 5.04MB respectively compared with Efficientnet-B2. Compared with ResNet-50, the parameters, model size and Flops are only 27%, 27% and 17% respectively. This means that the complexity of the improved EfficientNet-B2 is extremely low. Although the parameters, model size and Flops of FusionNet have increased by 1.81M, 6.9MB and 0.94G respectively compared with Efficientnet-B2, the parameters, model size and Flops of ResNet-50 are still 2.5 times, 2.5 times and 4.0 times. This means that FusionNet has achieved a good balance between performance and lightweight, avoiding the extreme situation of low model complexity and low performance.

Claims

1. A method for diagnosing musculoskeletal abnormalities based on a multi-model fusion of an improved lightweight network, characterized in that: The EfficientNet-B2 network model was improved: first, EfficientNet-B2 was combined with CoordinateAttention (CA) to better allocate weights when extracting features; secondly, in order to reduce the number of parameters of EfficientNet-B2, the Mobile Inverted Residual Bottleneck Convolution (MBConv) in EfficientNet-B2 was improved; in addition, in order to obtain more accurate diagnostic results for musculoskeletal abnormalities, the improved EfficientNet-B2 and MobileNetV2 were used for multi-model fusion.

2. According to the method for diagnosing musculoskeletal abnormalities based on a multi-model fusion improved lightweight network according to claim 1, the EfficientNet-B2 network model is improved, characterized in that: ① Squeeze and Excitation (SE) in the MBConv module is a channel attention mechanism that can improve the performance of the network by adaptively adjusting the channel weights of the feature map. However, it only processes information globally within a single channel, ignoring the information interaction in space and not making full use of its information. The CA module combines the channel and spatial attention mechanisms, embedding the position information into the channel attention, which can avoid the loss of position information in two-dimensional global pooling. The CA module is flexible and lightweight enough, so we combine it with EfficientNet-B2 to better allocate feature weights and improve the performance of the model. ②The SE module needs to introduce two fully connected layers to calculate the channel attention weights, which increases the number of parameters in the MBConv module. EfficientNet-B2 is a network built by stacking MBConv modules, which increases the complexity of EfficientNet-B2. Efficient Channel Attention (ECA) does not contain a fully connected layer. It calculates the channel attention weights through a learnable one-dimensional convolution operation to achieve efficient calculation. The ECA module not only captures local channel dependencies by applying a one-dimensional convolution operation on the channel dimension, but also involves only a small number of parameters. This modeling of local relationships can more effectively extract important channel features. Therefore, in order to improve the performance of the model while reducing model parameters, we replace the SE module in the MBConv module with the ECA module. ③ In order to overcome the limitation of single network performance and have the characteristics of multiple networks at the same time, we will fuse the improved EfficientNet-B2 and MobileNetV2 for multi-model. When the image is input into the two basic networks of improved EfficientNet-B2 and MobileNetV2, the input image is respectively passed through the feature extractors of improved EfficientNet-B2 and MobileNetV2 to obtain the extracted features. Then the two features are fused together and sent to the fully connected layer to obtain the final classification result.

3. According to the method for diagnosing musculoskeletal abnormalities based on multi-model fusion of improved lightweight networks in claim 1, the technical solution for diagnosing abnormalities in musculoskeletal X-ray images is as follows: ① Divide the samples that need to be input into the model for training into a training set and a validation set in a ratio of 8:2; ②For the training set, first perform random cropping to crop the image to 260×260. Then perform horizontal flipping to enhance the diversity of the data set. Then perform data type conversion to convert the image to a tensor. Finally, perform standardization so that the pixel values ​​of each channel of the image have a similar range and distribution; for the validation set, first resize the image to 260×260. Then perform a center cropping operation to crop the image from the center to 260×260. Then perform data type conversion to convert the image to a tensor. Finally, perform standardization so that the pixel values ​​of each channel of the image have a similar range and distribution; ③ Take the processed image as input and the corresponding label as output, use FusionNet and improved EfficientNet-B2 for training, monitor and save indicators such as Precision, AUC and Accuracy on the validation set, and select the weight with the best performance on these indicators as the final weight to obtain the optimized model; ④ Select a test sample and resize the image to 260×260. Then perform center cropping on the image, cropping the image from the center to 260×260 to retain the central part of the image. Then perform data type conversion to convert the image into a tensor. Finally, perform standardization so that the pixel values ​​of each channel of the image have a similar range and distribution; ⑤Use the processed image as the input of the optimization model and determine the image label based on the model output.

Images