A medical image recognition and classification method based on a multi-view integrated network

Abstract

The present application belongs to the field of medical image detection, and particularly relates to a medical image recognition and classification method based on a multi-view integrated network, which is used for intelligent recognition and classification of a target region of interest. The method comprises the following steps: collecting medical image data and performing labeling; performing data enhancement on different views of the data; utilizing the feature expression capability of a C-DSC module for the data; adopting an integrated learning strategy to perform model training, and constructing a multi-view integrated classification model based on the region of interest. The present application effectively improves the detection efficiency and accuracy of the region of interest in the medical image, assists doctors in recognition and classification through an automatic method, reduces personnel detection, and provides efficient and reliable technical support for medicine.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of medical image detection, and particularly relates to a medical image recognition and classification method based on a multi-view integrated network. BACKGROUND

[0002] With the rapid development of medical imaging technology, medical image detection has become an important means of modern medical diagnosis. As a common endocrine disease, thyroid disease often relies on the auxiliary analysis of medical images such as ultrasound and CT. However, the traditional manual film reading method is easily affected by the experience level and subjective judgment of doctors, which may lead to uncertainty in the diagnosis results. Therefore, developing an efficient and accurate automated classification method is of great significance to improve the efficiency and accuracy of diagnosis.

[0003] Traditional machine learning methods mainly rely on manually designed feature extraction and classification models in medical image recognition. The typical process includes pre-processing the images (such as denoising, segmentation, etc.), extracting features based on texture, shape or statistics, and then completing classification through support vector machines (SVM), random forests (RF) or k-nearest neighbors (k-NN) classifiers. However, this method is highly dependent on features, and the quality of features directly affects the performance of the model. At the same time, feature design often requires the knowledge of domain experts, which has great subjectivity and limitations.

[0004] With the rapid development of deep learning technology, medical image analysis has entered a new stage of automation and intelligence. By combining multiple advanced algorithms, deep learning methods can autonomously learn and extract key features, greatly improving the efficiency and accuracy of recognition and classification. SUMMARY

[0005] The application fully excavates the image features of the region of interest, uses a multi-view integration strategy to fuse information from different perspectives and levels, and constructs a classification model with strong robustness and excellent generalization ability. This technology can significantly improve the efficiency and accuracy of medical image classification, providing scientific and reliable auxiliary decision support for clinical practice, and has broad application prospects.

[0006] The technical solution adopted by the application to achieve the above-mentioned purposes is:

[0007] A medical image recognition and classification method based on a multi-view integrated network, comprising the following steps:

[0008] Collecting CT medical images and pre-processing them to construct a CT image dataset for training;

[0009] Construct a medical image classification model using a multi-view ensemble network, including a C-DSC module based on Curvelet transform and depthwise separable convolution, followed by a multi-expert ensemble network. Supervised training of the medical image recognition and classification model is performed using CT image datasets, and the model with the smallest classification error is selected as the ideal classification model.

[0010] The unknown CT image is passed into the ideal classification model to obtain the recognition and classification results of the current image.

[0011] The pretreatment comprises the following steps:

[0012] Modify the image data to a uniform size;

[0013] Manually mark the region of interest on the collected CT image as the image label R;

[0014] Perform data augmentation on the annotated regions of interest to expand the dataset and obtain multiple views C of N regions of interest of different sizes;

[0015] The original grayscale view O is superimposed, and N+1 multi-views are finally obtained.

[0016] The data enhancement method is:

[0017] The ROI area is cropped according to the marked box. Through the expansion operation, the center of the marked box is used as the reference, first expanding along the long side direction, then expanding along the wide side direction, each time expanding by 1 / 4 times, and each expansion is based on the previous step to obtain a new view.

[0018] In the C-DSC module, the data is subjected to Curvelet transform to obtain two curvelet components (H and L), which are stacked with the original image to form a three-channel image. The spatial features of each channel are extracted through depthwise separable convolution, and then the grayscale, high-frequency and low-frequency features are effectively integrated.

[0019] The Curvelet transform is:

[0020] Convert it from the spatial domain to the frequency domain through Fast Fourier Transform FFT;

[0021] The frequency domain data is rotated and resampled by setting random directions and angles to capture more diverse edge and directional features;

[0022] The frequency domain information is converted into the spatial domain through the inverse fast Fourier transform IFFT to obtain two curvelet components H and L; the high-frequency component H, the low-frequency component L and the original grayscale D image form a three-channel input X.

[0023] The depthwise separable convolution is:

[0024] The size of the feature map obtained by inputting X to a 3*3 convolution is the same as X ′ ; and then the size is unchanged through a ReLU activation function;

[0025] After information fusion through a 1*1 convolution, a feature map X1' is obtained, and the X1' contains weight information of each channel; and then the size of the feature map is unchanged through a ReLU activation function, and a feature map D is obtained.

[0026] The multi-expert integrated network comprises a main network and a slave network, and is any one of DenseNet, GoogleNet, ResNet, ShuffleNet, Efficientnet, AlexNet network pre-trained on ImageNet.

[0027] The multi-expert integrated network comprises:

[0028] The multi-feature map D is input into the first N+1 different slave networks Expert Net with frozen parameters, and the image of the region of interest with the largest expansion scale is input into the Expert Net 6;

[0029] In the slave network, five Expert Net networks are used for training the views of N+1 (C and O) channels to be processed, and the classification results output by the five Expert Net networks are x1, x2, x3, x4 and x5 respectively; the results are spliced into a feature map f1, and a Cat operation is performed to change the feature map from f1 to f2;

[0030] The result x6 output by the main network Expert Net 6 and the f2 are spliced, a Cat operation is performed, and the result is changed into f3, which is output as the final recognition result.

[0031] The main network adopts an Efficientnet network without frozen parameters, and the slave network adopts an AlexNet, DenseNet, GoogleNet, ResNet or ShuffleNet network with frozen parameter operation; and the last fully connected layer of each network in the integrated network is set to 2 units.

[0032] A medical image recognition and classification system based on a multi-view integrated network comprises a memory and a processor; the memory is used for storing a computer program; and the processor is used for loading and executing the computer program to realize the medical image recognition and classification method based on the multi-view integrated network.

[0033] The present application has the following advantages and benefits:

[0034] 1. The present application is based on ensemble learning, using C-DSC to enhance the feature expression ability of medical images. Through the combination of Curvelet transform and depth separable convolution, the data is converted between the frequency domain and the spatial domain, and the model extracts key information from the frequency domain and the spatial domain respectively, performs feature fusion, enriches the feature extraction process, and improves the expression ability of the model.

[0035] 2. At the same time, the present application combines AlexNet, DenseNet, GoogleNet, ResNet, ShuffleNet and Efficientnet networks pre-trained on the ImageNet dataset to build an ensemble network for training and classification of CT image datasets. It effectively solves the problem of difficulty in capturing complex multi-scale and multi-directional features of medical images, significantly improves the sensitivity of the model to texture and edge information, and effectively improves the robustness and generalization ability of the model by integrating multiple pre-trained models and adopting the division strategy of the primary specialist network and the auxiliary specialist network. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 The method flowchart of the present application is shown in the figure;

[0037] Figure 2 The structure diagram of the thyroid classification model based on multi-view integration is shown in the figure;

[0038] Figure 3 The structure diagram of the C-DSC module is shown in the figure.

[0039] Figure 4 The experimental results of the integrated network are shown in the figure DETAILED DESCRIPTION

[0040] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation method of the present application will be described in detail below with reference to the accompanying drawings. In the following description, many specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the spirit of the present application, therefore the present application is not limited by the specific implementation disclosed below.

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing the specific embodiments and are not intended to limit the present application. The present application will be further described in detail below with reference to the accompanying drawings and examples.

[0042] In this example, thyroid medical CT images are used as the recognition and classification object, and the classification and recognition are performed according to the following method.

[0043] As Figure 1 shown, a medical image recognition classification method based on a multi-view integrated network comprises the following steps:

[0044] Step 1: Collect thyroid CT image data set, pretreatment, manual annotation.

[0045] Step 2: Crop the region of interest to obtain image R by pre-processing the data and annotating the data, and perform data enhancement (at least one of flipping, expanding, and scaling) centered on the region of interest.

[0046] Step 3: First, enhance feature expression through the C-DSC module.

[0047] Step 4: Then pass through the pre-trained (AlexNet, DenseNet, GoogleNet, ResNet, ShuffleNet, Efficientnet) integrated classification network, and finally output the feature map f2.

[0048] Step 5: Input the processed data into the integrated network to obtain the final thyroid benign and malignant classification result.

[0049] As Figure 2 shown, the specific process of the multi-view integrated thyroid classification model is as follows:

[0050] The collected thyroid CT image data set in step 1 is scaled to a uniform size of 224*224 pixels.

[0051] In step 2, the image data is annotated, specifically, the obtained image is manually annotated, and the region of interest is framed to obtain image R as the candidate label of the thyroid CT image.

[0052] The expansion is performed with the region of interest as the center, and the expansion is performed in a counterclockwise direction, with the expansion ratio being 1 / 4 of the length and width of the original region of interest each time, generating N different expansion rates of the region of interest (C) in the region of interest. Figure 3 In this example, N is 4.

[0053] At this point, the original gray region of interest O is superimposed, and there are N+1 (C and O) views of channels to be processed.

[0054] In step 3, the Curvelet transform is a random direction and random angle transform for the input image, which obtains high-frequency and low-frequency feature maps of the image, and superimposes the original gray image as a 3-channel feature map.

[0055] The integrated network (AlexNet, DenseNet, GoogleNet, ResNet, ShuffleNet, Efficientnet) maps the output layer of each network to 2 dimensions.

[0056] The C-DSC module described in step 3 is composed of Curvelet and depthwise separable convolution.

[0057] As shown in Figure 3 The specific workflow of the C-DSC is as follows:

[0058] Step 3.1: For the same batch of 5 views of the region of interest, perform Curvelet transformation according to steps 3.1a-3.1c, and obtain the feature map X.

[0059] Step 3.1a: Convert it from the spatial domain to the frequency domain representation by Fast Fourier Transform (FFT).

[0060] Step 3.1b: Set random directions and angles to rotate and resample the frequency domain data to capture more diverse edge and direction features. Finally, convert the frequency domain information to the spatial domain by Inverse Fast Fourier Transform (IFFT).

[0061] Step 3.1c: Form the three-channel input X with the obtained high-frequency component H, low-frequency component L, and original grayscale D image.

[0062] Step 3.2: Input X into the depthwise separable convolution module and perform depthwise separable convolution operation according to steps 3.2a-3.2b to obtain the feature map X1'.

[0063] Step 3.2a: Input X into the depthwise separable convolution, first pass through 3*3 convolution to obtain the feature map X' with the same size as X, and pass through the ReLU activation function without changing the size.

[0064] Step 3.2b: After 1*1 convolution and information fusion, obtain the feature map X1', pass through a ReLU activation function without changing the size of the feature map, and finally obtain the feature map D; where H is the height of the image pixels, W is the width of the image pixels, and C is the number of convolution channels.

[0065] In step 4, the integrated classification network is composed of a certain number of expert sub-networks Expert Net. The Expert Net is any AlexNet, DenseNet, GoogleNet, ResNet, ShuffleNet, or Efficientnet network structure. The number of Expert Nets is related to the number of channels N+1 of the view to be processed, and is N+2. The first N+1 Expert Nets are used to process each channel view after the pre-trained model is frozen and output the candidate recognition result f2. The last Expert Net is used to process the original view after the non-frozen parameters and output the candidate recognition result x6. The results f2 and x6 are further fused to obtain f3 as the final classification result output. During the training process, the network parameters that perform best on the test set are used as the network parameters of the ideal thyroid classification model.

[0066] The specific process is as follows:

[0067] Step 4.1: Input the feature map D into the first five different Expert Nets with frozen parameters, and Expert Net 6 is not frozen and the input is the image of the region of interest with the largest expanded scale.

[0068] Step 4.2: The first five Expert Nets are used to train the N+1 (C and O) channel views to be processed, and the output classification results are x1, x2, x3, x4, and x5, respectively. The results are spliced into a feature map f1, and the Cat operation is performed to change the feature map from f1 to f2. In this example, the first five Expert Nets are AlexNet, DenseNet, GoogleNet, ResNet, and ShuffleNet from networks with frozen parameters.

[0069] Step 4.3: The results x6 and f2 output by Expert Net 6 are spliced and subjected to Cat operation to change to f3, which is the final recognition result output. In this example, Expert Net 6 uses an Efficientnet backbone network.

[0070] Among them, the Efficientnet backbone network and the AlexNet, DenseNet, GoogleNet, ResNet, and ShuffleNet from networks with frozen parameters, the last fully connected layer of each network is set to 2 units.

[0071] Firstly, layer_fc1 processing is adopted: the first connection layer processing is performed on the network with frozen parameters: DenseNet, GoogleNet, ResNet, ShuffleNet and Efficientnet, the final feature vectors of the five networks are spliced, and after the Fc full connection layer processing, the number of feature maps is expanded to 128; then after BatchNorm1d, Dropout and the activation function LeakyReLU, the number of feature maps is reduced to 64 through an Fc full connection layer operation; then after the activation function LeakyReLU, the number of feature maps is changed to 10 through the next Fc full connection layer operation; and then after the last LeakyReLU activation operation.

[0072] Secondly, layer_fc2 processing is adopted: the output layer of the backbone network Efficientnet is combined with the output of layer_fc1, and after the first Fc full connection layer processing, the number of feature maps is expanded to 128; then after BatchNorm1d, Dropout and the activation function LeakyReLU, the number of feature maps is reduced to 64 through an Fc full connection layer operation; then after the activation function LeakyReLU, the number of feature maps is changed to the score of each dimension corresponding to a class through the next Fc full connection layer operation.

[0073] In step 5, the medical image to be recognized is preprocessed, data is enhanced, the feature expression is enhanced through the C-DSC module, and then input into the thyroid ideal classification model for classification and recognition, and the feature map is output.

[0074] Embodiment:

[0075] The modeling step of the application is:

[0076] Step 1: data collection. The data of the application is derived from the thyroid CT image data collected by the First Affiliated Hospital of Shenyang Medical University.

[0077] Step 2: data preprocessing, the collected data is manually labeled.

[0078] Step 3: the labeled region is expanded in a counterclockwise direction, and each expansion ratio is 1 / 4 of the length and width of the original region of interest, thereby generating four regions of interest with different expansion rates.

[0079] Step 4: the data set after preprocessing is scaled to a uniform size of 224*224 pixels.

[0080] Step 5: the data with uniform size is subjected to Curvelet transformation and depth separable processing.

[0081] Step 6: Build an ensemble network using the pre-trained ImageNet models: DenseNet, GoogleNet, ResNet, ShuffleNet, EfficientNet, and AlexNet. Convert the final fully connected layers of these networks to 2D maps. Leave the EfficientNet backbone network unfrozen, while freezing the remaining branches. Train on the thyroid CT image dataset and use the network parameters that perform best on the test set as the thyroid classification model.

[0082] Steps: Resize the image or data set that needs to be classified sent by the user, then pass it into the thyroid classification network, and finally output the thyroid classification and recognition results based on the image.

[0083] like Figure 4 As shown in FIG, the schematic diagram of the integrated network experimental results, the specific workflow is as follows:

[0084] Step 1: Compare the proposed network with ensemble networks of different numbers.

[0085] Step 2: This paper uses EfficientNet as the backbone network. The best results are achieved with 5 auxiliary networks and 6 ensemble networks, with the highest accuracy reaching 86.22 and the F1 index reaching 84.00.

[0086] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural change made to the above embodiment based on the technical essence of the present invention shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A medical image recognition and classification method based on a multi-view integrated network, characterized in that, The method comprises the following steps: Collecting CT medical images and preprocessing them to construct a CT image dataset for training; Constructing a medical image recognition classification model based on a multi-view integrated network, comprising a C-DSC module constructed based on Curvelet transformation and deep separable convolution, and a multi-expert integrated network connected thereto; using the CT image dataset to supervise the training of the medical image recognition classification model, and selecting the model with the smallest classification error as an ideal classification model; the multi-expert integrated network comprises: Inputting the feature map D into the first N+1 different frozen parameter slave networks Expert Net, and inputting the region of interest image with the largest expansion scale into the Expert Net 6 which is not frozen; The five Expert Net networks in the slave network are used to train N+1 channels of views to be processed, and the classification results output by the five Expert Net networks are x1, x2, x3, x4 and x5 respectively; the results are spliced into a feature map f1, and a Cat operation is performed to change the feature map from f1 to f2; The result x6 output by the main network Expert Net 6 and f2 are spliced, a Cat operation is performed, and f3 is obtained as the final recognition result output; The unknown CT image is input into the ideal classification model to obtain the recognition classification result of the current image. 2.The medical image recognition and classification method based on multi-view ensemble network according to claim 1, characterized in that, The preprocessing comprises the following steps: Modifying the image data to a uniform size; Manually labeling the region of interest of the collected CT image as a label R of the image; Performing data enhancement processing on the labeled region of interest to expand the dataset to obtain N multiple views of regions of interest with different sizes; Stacking the original gray view O to finally obtain N+1 multiple views. 3.The medical image recognition and classification method based on multi-view ensemble network according to claim 2, characterized in that, The data enhancement processing comprises the following steps: According to the labeled box, the ROI region is cropped, and through expansion operation, the center of the labeled box is taken as the reference, the expansion is first performed along the long side direction, and then along the wide side direction, each time the expansion is 1 / 4 times, and each time the expansion is increased on the basis of the previous step to obtain a new view. 4.The medical image recognition and classification method based on multi-view ensemble network of claim 1, wherein, In the C-DSC module, the data is subjected to Curvelet transformation to obtain two curvelet components, and the original image is stacked to form a three-channel image, the spatial features of each channel are extracted through deep separable convolution, and then the gray, high-frequency and low-frequency features are effectively integrated.

5. The medical image recognition classification method based on a multi-view integrated network according to claim 4, wherein the Curvelet transformation comprises: Converting the data from the spatial domain to the frequency domain through fast Fourier transform (FFT); Rotating and resampling the frequency domain data by setting random directions and angles to capture more diverse edge and direction features; Converting the frequency domain information to the spatial domain through inverse fast Fourier transform (IFFT) to obtain two curvelet components; and forming three channels of inputs X by the high-frequency component, the low-frequency component and the original gray image. 6.The medical image recognition and classification method based on multi-view ensemble network according to claim 5, characterized in that, The deep separable convolution comprises: X input to 3*3 convolution gets feature map of the same size as X ′ ; then through ReLU activation function, size remains unchanged; After information fusion through 1*1 convolution, the feature map X1' is obtained, which contains the weight information of each channel; and the size of the feature map is unchanged through a ReLU activation function to obtain the feature map D. 7.The medical image recognition and classification method based on multi-view ensemble network of claim 1, wherein, The multi-expert integrated network comprises a backbone network and a slave network, and is any one of DenseNet, GoogleNet, ResNet, ShuffleNet, Efficientnet, and AlexNet networks pre-trained on ImageNet. 8.The medical image recognition and classification method based on multi-view ensemble network of claim 1, wherein, The backbone network adopts an Efficientnet network without frozen parameters, and the slave network adopts AlexNet, DenseNet, GoogleNet, ResNet, and ShuffleNet networks with frozen parameter operation; and the last fully connected layer of each network in the integrated network is set to two units. 9.A medical image recognition and classification system based on a multi-view ensemble network, characterized in that, The method comprises a memory and a processor; the memory is used for storing a computer program; and the processor is used for loading and executing the computer program to realize the medical image recognition and classification method based on the multi-view integrated network according to any one of claims 1-8.

Images