Medical image classification method based on multi-modal feature fusion

Abstract

The invention relates to medical images, in particular to a medical image classification method based on multi-modal feature fusion, and the method comprises the steps: obtaining medical image data, and constructing a training set, a verification set and a test set; extracting fusion features of the medical images in the training set, the verification set and the test set; performing model training on the inspection item classification model by using the training set, and performing parameter adjustment and performance evaluation on the trained inspection item classification model by using the verification set and the test set; inputting a to-be-classified medical image into the trained examination item classification model to obtain an examination item category label to which the medical image belongs; according to the technical scheme provided by the invention, the defect that examination items of medical images cannot be classified in the prior art can be effectively overcome.

Description

Technical Field

[0001] The present invention relates to medical images, and in particular to a medical image classification method based on multimodal feature fusion. Background Art

[0002] Medical imaging plays an irreplaceable and important role in clinical diagnosis and treatment. It can provide doctors with key information to support accurate diagnosis and treatment plans. The importance of classifying medical imaging examination items is becoming increasingly prominent. With the rapid development of artificial intelligence technology, establishing high-quality medical imaging datasets and effectively classifying them has become a key step in training and optimizing artificial intelligence models. However, in practice, the classification of medical images mainly relies on the experience and knowledge of doctors, which is not only time-consuming and labor-intensive, but also has problems such as subjectivity and consistency.

[0003] At present, some automated medical image classification methods have been proposed, most of which are based on traditional machine learning or deep learning methods. However, these methods often only consider the medical image itself and ignore the rich attribute information in DICOM images, such as examination type, patient information, etc. Therefore, there is a need to combine image data with attribute information for comprehensive analysis to improve the accuracy and efficiency of medical image classification.

[0004] In response to the above challenges, the proposal of a medical image classification method based on multimodal feature fusion is of great significance. This method can not only comprehensively utilize the image data and attribute information in DICOM images for accurate classification, but also lay the foundation for subsequent quality control classification training and artificial intelligence model training. By using a multimodal fusion network to extract rich fusion features, the accuracy and consistency of medical image classification can be effectively improved, laying a solid foundation for establishing a high-quality medical image dataset.

[0005] In the existing technology, it is still at the primary level of using deep neural networks to classify DICOM images into two-dimensional medical images / three-dimensional medical images. For some more detailed medical image classifications, such as the examination item categories of MRI (magnetic resonance imaging) examination mode, there are 22 categories:

[0006] {

[0007] 'MR head plain scan-transverse T1WI',

[0008] 'MR head plain scan-transverse T2WI',

[0009] 'MR head plain scan - transverse T2WI FLAIR',

[0010] 'MR head plain scan-sagittal T1WI',

[0011] 'MR head plain scan-sagittal T2WI',

[0012] 'MR head plain scan - transverse DWI-b1000',

[0013] 'MR lumbar spine (lumbar plexus) plain scan - sagittal T1WI',

[0014] 'MR lumbar spine (lumbar plexus) plain scan - sagittal T2WI',

[0015] 'MR lumbar spine (lumbar plexus) plain scan - sagittal T2WI fat suppression',

[0016] 'MR lumbar spine (lumbar plexus) plain scan - transverse axial T2WI',

[0017] 'MR upper abdomen plain scan - transverse axial T2WI',

[0018] 'MR upper abdomen plain scan-coronal T2WI',

[0019] 'MR upper abdomen plain scan - transverse T2WI fat suppression',

[0020] 'MR upper abdomen plain scan - transverse DWI-b800',

[0021] 'MR head plain scan-transverse DWI-ADC',

[0022] 'MR head plain scan-transverse DWI-b0',

[0023] 'MR head plain scan - transverse T2WI water and fat suppression',

[0024] 'MR upper abdomen plain scan - transverse axial DWI-ADC',

[0025] 'MR upper abdomen plain scan - transverse axial T1WI same phase',

[0026] 'MR upper abdomen plain scan - transverse axial T1WI reverse phase',

[0027] 'MR upper abdomen plain scan - coronal T2WI fat suppression',

[0028] 'MR upper abdomen plain scan - transverse DWI-b50'

[0029] };

[0030] For the more detailed medical image classification mentioned above, such as the examination item category of "MR upper abdomen plain scan-transverse axial T1WI anti-phase", the existing artificial intelligence model cannot identify and classify it well, and manual intervention is required for manual classification. In addition, the existing artificial intelligence model has a low utilization rate of the attribute information of DICOM images, which is limited to the classification of image data and cannot fully mine the useful information contained in the attribute information. Summary of the invention

[0031] 1. Technical issues to be resolved

[0032] In view of the above-mentioned shortcomings of the prior art, the present invention provides a medical image classification method based on multimodal feature fusion, which can effectively overcome the defect of the prior art that medical images cannot be classified into examination items.

[0033] (II) Technical solution

[0034] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0035] A medical image classification method based on multimodal feature fusion includes the following steps:

[0036] S1. Obtain medical imaging data and construct training sets, validation sets, and test sets;

[0037] S2, extracting the fusion features of each medical image in the training set, validation set and test set;

[0038] S3. Use the training set to train the inspection item classification model, and use the validation set and test set to adjust the parameters and perform performance evaluation on the trained inspection item classification model;

[0039] S4. Input the medical image to be classified into the trained examination item classification model to obtain the examination item category label to which the medical image belongs.

[0040] Preferably, obtaining medical imaging data in S1 and constructing a training set, a validation set and a test set include:

[0041] S11. Obtain medical image data from original medical image DICOM;

[0042] S12, selecting a representative frame image from the medical image data, and extracting image data and attribute information of the representative frame image;

[0043] S13. Construct a training set, a validation set, and a test set based on representative frame images.

[0044] Preferably, obtaining medical image data from the original medical image DICOM in S11 includes:

[0045] Read multiple DICOM images from the original medical image DICOM and obtain the image data contained in the DICOM images;

[0046] For DICOM images belonging to three-dimensional medical images, the read DICOM images are sorted according to the slice position and the scanning order to obtain an initial DICOM sequence;

[0047] Among them, the multiple DICOM images read from the original medical image DICOM each time belong to the same examination part;

[0048] The image data includes the category labels of the standardized inspection items to which they belong. The category labels of the standardized inspection items are manually annotated and are saved using one-hot 01 coding;

[0049] The slice position and scanning order are derived from the tag values ​​stored in the medical image DICOM.

[0050] Preferably, in S12, selecting a representative frame image from the medical image data and extracting image data and attribute information of the representative frame image includes:

[0051] For DICOM images belonging to three-dimensional medical images, the center, that is, the Int(L / 2)th DICOM image in the initial DICOM sequence is selected as the representative frame image;

[0052] For a DICOM image belonging to a two-dimensional medical image, the DICOM image is used as a representative frame image;

[0053] Extracting image data and attribute information of each representative frame image, and performing data processing on the image data and attribute information;

[0054] Wherein, L is the sequence length of the initial DICOM sequence, the image data includes pixel values ​​and image size, and the attribute information includes scanning mode and sequence information.

[0055] Preferably, the data processing of the image data and attribute information includes:

[0056] Standardize and resample image data to ensure data quality and consistency;

[0057] Combine each attribute information into a text containing multiple parameters, and combine the texts corresponding to multiple attribute information into a long text;

[0058] Each attribute information contains four parameters: tag, attribute name, attribute encoding type and attribute value.

[0059] Preferably, constructing a training set, a validation set and a test set according to the representative frame images in S13 includes:

[0060] All representative frame images are divided into training set, validation set and test set according to preset ratios.

[0061] Preferably, S2 extracts the fusion features of each medical image in the training set, the validation set, and the test set, including:

[0062] S21, resampling all representative frame images and converting them into three-channel Tensor type vectors, that is, obtaining image vectors of each representative frame image;

[0063] S22, scaling the long text composed of the attribute information of each representative frame image using a summary algorithm based on the Longformer model, and encoding it using the BERT model to obtain a text encoding vector of each representative frame image;

[0064] S23, input the image vector and text encoding vector of each representative frame image into the MultimodalBert model respectively to obtain the fusion features of each representative frame image.

[0065] Preferably, in S3, the inspection item classification model is trained using the training set, including:

[0066] S31. Considering the data imbalance problem of classification tasks, the weighted loss function CrossEntropyLoss containing the Softmax function is used as the loss function of the inspection item classification model:

[0067]

[0068] Among them, ω i is the positive weight of the i-th representative frame image. The calculation method of the positive weight is: 1-the total positive rate of the training set sample labels, y i is the inspection item category label value of the i-th representative frame image, y i ∈{0,1},p i is the predicted probability value of the i-th representative frame image, N is the number of representative frame images in the training set, and σ(·) is the Softmax function;

[0069] S32, inputting the fusion features of each representative frame image in the training set into the inspection item classification model for model training;

[0070] S33, calculating the loss value based on the weighted loss function CrossEntropyLoss, and the optimizer updates the model parameters according to the loss value and the network gradient information;

[0071] S34. Compare the loss value with the preset threshold. If the loss value is less than the preset threshold, the model training is completed, and the current inspection item classification model is the trained inspection item classification model. Otherwise, return to S32 to continue model training.

[0072] Preferably, in S4, the medical image to be classified is input into a trained examination item classification model to obtain the examination item category label to which the medical image belongs, including:

[0073] S41, extracting fusion features of the medical image to be classified, and inputting the fusion features into the trained examination item classification model to obtain the predicted probability value of the medical image under each examination item category label;

[0074] S42. Compare the predicted probability values, and use the examination item category corresponding to the examination item category label with the highest predicted probability value as the classification result of the medical image.

[0075] (III) Beneficial effects

[0076] Compared with the prior art, the medical image classification method based on multimodal feature fusion provided by the present invention has the following beneficial effects:

[0077] 1) It can effectively process more detailed classification of medical imaging examination items, and through the method of multimodal feature fusion, it can realize the automatic classification of various subcategories under specific examination modes;

[0078] 2) Make full use of the attribute information of DICOM images and effectively fuse the image data and attribute information to mine more comprehensive fusion features for medical image classification, effectively improving the accuracy and efficiency of classification;

[0079] 3) Improve AI classification performance, especially in identifying more detailed categories with higher accuracy and reliability, thereby reducing human intervention;

[0080] 4) Reduce the subjectivity and limitations of medical image classification, realize the automated classification process, reduce the workload of medical staff, and improve work efficiency;

[0081] 5) For three-dimensional medical images obtained under MRI, CT and other examination modes, only one DICOM image in the center is selected from the initial DICOM sequence as the representative frame image for fusion feature extraction, which effectively saves the amount of calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0083] Figure 1 It is a schematic diagram of the process of the present invention;

[0084] Figure 2 It is a schematic diagram of the process of extracting fusion features of each medical image in the training set, validation set and test set in the present invention. DETAILED DESCRIPTION

[0085] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0086] A medical image classification method based on multimodal feature fusion, such as Figure 1 and Figure 2 As shown, S1. Obtain medical imaging data and construct training sets, validation sets, and test sets, including:

[0087] S11. Obtain medical image data from original medical image DICOM;

[0088] S12, selecting a representative frame image from the medical image data, and extracting image data and attribute information of the representative frame image;

[0089] S13. Construct a training set, a validation set, and a test set based on representative frame images.

[0090] 1) S11 obtains medical image data from the original medical image DICOM, including:

[0091] Read multiple DICOM images from the original medical image DICOM and obtain the image data contained in the DICOM images;

[0092] For DICOM images belonging to three-dimensional medical images (three-dimensional medical images obtained under examination modes such as MRI and CT), the read DICOM images are sorted according to the slice position and the scanning order to obtain an initial DICOM sequence;

[0093] Among them, the multiple DICOM images read from the original medical image DICOM each time belong to the same examination part;

[0094] The image data includes the category labels of the standardized inspection items to which they belong. The category labels of the standardized inspection items are manually annotated and are saved using one-hot 01 coding;

[0095] The slice position and scanning order are derived from the tag values ​​stored in the medical image DICOM.

[0096] 2) In S12, a representative frame image is selected from the medical image data, and image data and attribute information of the representative frame image are extracted, including:

[0097] For DICOM images belonging to three-dimensional medical images, the center, that is, the Int(L / 2)th DICOM image in the initial DICOM sequence is selected as the representative frame image;

[0098] For a DICOM image belonging to a two-dimensional medical image, the DICOM image is used as a representative frame image;

[0099] Extracting image data and attribute information of each representative frame image, and performing data processing on the image data and attribute information;

[0100] Wherein, L is the sequence length of the initial DICOM sequence, the image data includes pixel values ​​and image size, and the attribute information includes scanning mode and sequence information.

[0101] Specifically, the image data and attribute information are processed, including:

[0102] Standardize and resample image data to ensure data quality and consistency;

[0103] Combine each attribute information into a text containing multiple parameters, and combine the texts corresponding to multiple attribute information into a long text;

[0104] Each attribute information contains four parameters: tag, attribute name, attribute encoding type and attribute value.

[0105] 3) In S13, a training set, a validation set, and a test set are constructed based on the representative frame images, including:

[0106] All representative frame images are divided into training set, validation set and test set according to the preset ratio (7:1:2).

[0107] In the above technical solution, each attribute information is combined into a text form containing multiple parameters as "tag attribute name attribute encoding type: 'attribute value';", as shown below:

[0108] tag: (0008,0060);

[0109] Attribute name: Modality;

[0110] Attribute encoding type: CS;

[0111] Attribute value: MR;

[0112] The text composed of the above four parameters is: "(0008,0060)Modality CS:'MR';"

[0113] The texts corresponding to the multiple pieces of attribute information are combined into a long text (variable long text): "(0008,0060) ModalityCS:'MR'; (0008,0070) ManufacturerLO:'Philips Medical Systems'; (0008,0080) Institution NameLO:”; (0008,0081) Institution Address ST:”; (0008,0090) Referring Physician's NamePN:”; (0008,0100) Code ValueSH:'hnsdermyy'; (0008,1010) Station NameSH:'PHILIPS-B3557D4';".

[0114] Since DICOM images obtained under different examination modes have different attribute information, the technical solution of the present application processes the attribute information by combining the texts corresponding to multiple attribute information into a long text, thereby avoiding the problem of generating null values ​​due to selecting fixed attribute information.

[0115] S2. Extract the fusion features of each medical image in the training set, validation set, and test set, including:

[0116] S21, resampling all representative frame images (resampled to a size of 224*224), and converting them into three-channel Tensor type vectors, that is, obtaining image vectors of each representative frame image;

[0117] S22, scaling the long text composed of the attribute information of each representative frame image using a summary algorithm based on the Longformer model, and encoding it using the BERT model to obtain a text encoding vector of each representative frame image;

[0118] S23, input the image vector and text encoding vector of each representative frame image into the MultimodalBert model respectively to obtain the fusion features of each representative frame image.

[0119] S3. Use the training set to train the inspection item classification model, and use the validation set and test set to adjust the parameters and perform performance evaluation on the trained inspection item classification model.

[0120] Specifically, the training set is used to train the inspection item classification model, including:

[0121] S31. Considering the data imbalance problem of classification tasks, the weighted loss function CrossEntropyLoss containing the Softmax function is used as the loss function of the inspection item classification model:

[0122]

[0123] Among them, ω i is the positive weight of the i-th representative frame image. The calculation method of the positive weight is: 1-the total positive rate of the training set sample labels, y i is the inspection item category label value of the i-th representative frame image, y i ∈{0,1},p i is the predicted probability value of the i-th representative frame image, N is the number of representative frame images in the training set, and σ(·) is the Softmax function;

[0124] S32, inputting the fusion features of each representative frame image in the training set into the inspection item classification model for model training;

[0125] S33, calculating the loss value based on the weighted loss function CrossEntropyLoss, and the optimizer updates the model parameters according to the loss value and the network gradient information;

[0126] S34. Compare the loss value with the preset threshold. If the loss value is less than the preset threshold, the model training is completed, and the current inspection item classification model is the trained inspection item classification model. Otherwise, return to S32 to continue model training.

[0127] S4. Input the medical image to be classified into the trained examination item classification model to obtain the examination item category label to which the medical image belongs, specifically including:

[0128] S41, extracting fusion features of the medical image to be classified, and inputting the fusion features into the trained examination item classification model to obtain the predicted probability value of the medical image under each examination item category label;

[0129] S42. Compare the predicted probability values, and use the examination item category corresponding to the examination item category label with the highest predicted probability value as the classification result of the medical image.

[0130] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A medical image classification method based on multimodal feature fusion, characterized by: The following steps are involved: S1. Obtain medical imaging data and construct training sets, validation sets, and test sets; S2, extracting the fusion features of each medical image in the training set, validation set and test set; S3. Use the training set to train the inspection item classification model, and use the validation set and test set to adjust the parameters and perform performance evaluation on the trained inspection item classification model; S4. Input the medical image to be classified into the trained examination item classification model to obtain the examination item category label to which the medical image belongs.

2. The medical image classification method based on multimodal feature fusion according to claim 1, characterized in that: Obtain medical imaging data from S1 and construct training sets, validation sets, and test sets, including: S11. Obtain medical image data from original medical image DICOM; S12, selecting a representative frame image from the medical image data, and extracting image data and attribute information of the representative frame image; S13. Construct a training set, a validation set, and a test set based on representative frame images.

3. The medical image classification method based on multimodal feature fusion according to claim 2, characterized in that: S11 obtains medical image data from the original medical image DICOM, including: Read multiple DICOM images from the original medical image DICOM and obtain the image data contained in the DICOM images; For DICOM images belonging to three-dimensional medical images, the read DICOM images are sorted according to the slice position and the scanning order to obtain an initial DICOM sequence; Among them, the multiple DICOM images read from the original medical image DICOM each time belong to the same examination part; The image data includes the category labels of the standardized inspection items to which they belong. The category labels of the standardized inspection items are manually annotated and are saved using one-hot 01 coding; The slice position and scanning order are derived from the tag values ​​stored in the medical image DICOM.

4. The medical image classification method based on multimodal feature fusion according to claim 3, characterized in that: In S12, a representative frame image is selected from the medical image data, and image data and attribute information of the representative frame image are extracted, including: For DICOM images belonging to three-dimensional medical images, the center, that is, the Int(L / 2)th DICOM image in the initial DICOM sequence is selected as the representative frame image; For a DICOM image belonging to a two-dimensional medical image, the DICOM image is used as a representative frame image; Extracting image data and attribute information of each representative frame image, and performing data processing on the image data and attribute information; Wherein, L is the sequence length of the initial DICOM sequence, the image data includes pixel values ​​and image size, and the attribute information includes scanning mode and sequence information.

5. The medical image classification method based on multimodal feature fusion according to claim 4, characterized in that: The data processing of the image data and attribute information includes: Standardize and resample image data to ensure data quality and consistency; Combine each attribute information into a text containing multiple parameters, and combine the texts corresponding to multiple attribute information into a long text; Each attribute information contains four parameters: tag, attribute name, attribute encoding type and attribute value.

6. The medical image classification method based on multimodal feature fusion according to claim 4, characterized in that: In S13, a training set, a validation set, and a test set are constructed based on representative frame images, including: All representative frame images are divided into training set, validation set and test set according to preset ratios.

7. The medical image classification method based on multimodal feature fusion according to claim 2, characterized in that: S2 extracts the fusion features of each medical image in the training set, validation set, and test set, including: S21, resampling all representative frame images and converting them into three-channel Tensor type vectors, that is, obtaining image vectors of each representative frame image; S22, scaling the long text composed of the attribute information of each representative frame image using a summary algorithm based on the Longformer model, and encoding it using the BERT model to obtain a text encoding vector of each representative frame image; S23, input the image vector and text encoding vector of each representative frame image into the MultimodalBert model respectively to obtain the fusion features of each representative frame image.

8. The medical image classification method based on multimodal feature fusion according to claim 7, characterized in that: In S3, the training set is used to train the inspection item classification model, including: S31. Considering the data imbalance problem of classification tasks, the weighted loss function CrossEntropyLoss containing the Softmax function is used as the loss function of the inspection item classification model: Among them, ω i is the positive weight of the i-th representative frame image. The calculation method of the positive weight is: 1-the total positive rate of the training set sample labels, y i is the inspection item category label value of the i-th representative frame image, y i ∈{0,1},p i is the predicted probability value of the i-th representative frame image, N is the number of representative frame images in the training set, and σ(·) is the Softmax function; S32, inputting the fusion features of each representative frame image in the training set into the inspection item classification model for model training; S33, calculating the loss value based on the weighted loss function CrossEntropyLoss, and the optimizer updates the model parameters according to the loss value and the network gradient information; S34. Compare the loss value with the preset threshold. If the loss value is less than the preset threshold, the model training is completed, and the current inspection item classification model is the trained inspection item classification model. Otherwise, return to S32 to continue model training.

9. The medical image classification method based on multimodal feature fusion according to claim 8, characterized in that: In S4, the medical image to be classified is input into the trained examination item classification model to obtain the examination item category label to which the medical image belongs, including: S41, extracting representative frame images and fusion features of the medical image to be classified, and inputting the fusion features into the trained examination item classification model to obtain the predicted probability value of the medical image under each examination item category label; S42. Compare the predicted probability values, and use the examination item category corresponding to the examination item category label with the highest predicted probability value as the classification result of the medical image.

Images