A training method and a classification method of a medical image classification model and related products

Abstract

The application discloses a medical image classification model training method, a classification method and related products. A plurality of feature extraction paths are designed for the model. On one hand, the category of an image is predicted in one of the feature extraction paths to obtain a category prediction result. On the other hand, consistency feature comparison is performed on the same image in the remaining two feature extraction paths in combination with a consistency regularization rule to obtain a consistency comparison result, and differential feature comparison is performed on different images in combination with a similarity calculation technology to obtain a similarity comparison result. In this way, the global and local features of the image are extracted to realize fine-grained feature extraction of the image, so that the model extraction capability can be better trained. Compared with manual image classification by medical staff, the medical image classification model trained by the application can capture global high-level features and local fine-grained features in the image, thereby improving the accuracy of medical image classification and the efficiency of medical image classification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, in particular to a medical image classification model training method, a medical image classification method and related products. BACKGROUND

[0002] At present, in the technical field of medical image processing, medical staff mainly manually classifies medical images. However, medical images do not have obvious foreground and background differences like natural images, and it is difficult to complete recognition only by using high-level features of images. In addition, the identification of whether an organ in a medical image is diseased mainly depends on subtle changes in texture, and the identification task of medical staff needs to pay more attention to the details of medical images. Since the image classification by medical staff has a certain subjectivity, it will affect the accuracy of image classification to some extent. In addition, paying attention to the details of medical images will also make medical staff spend a lot of energy on image classification, resulting in low image classification efficiency. Therefore, how to improve the accuracy and efficiency of medical image classification is a key problem for those skilled in the art. SUMMARY

[0003] Based on the above problems, the present application provides a medical image classification model training method, a medical image classification method and related products to improve the accuracy and efficiency of medical image classification.

[0004] The embodiments of the present application disclose the following technical solutions:

[0005] The first aspect of the present application provides a medical image classification model training method. The training method comprises:

[0006] The to-be-trained model respectively extracts features of the first image, the second image and the third image to obtain a first feature extraction result of the first image, a second feature extraction result of the second image and a third feature extraction result of the third image. The first image and the second image are two different images obtained by enhancing a first sample medical image. The third image is an image obtained by enhancing a second sample medical image, and the first image and the third image are enhanced in the same way.

[0007] Based on the first feature extraction result, the class of the first sample medical image is predicted to obtain a class prediction result.

[0008] Based on the first feature extraction result, the second feature extraction result and the third feature extraction result, a feature extraction consistency comparison result of the to-be-trained model for the first image and the second image is obtained, and a feature extraction similarity comparison result of the to-be-trained model for the first image and the third image is obtained.

[0009] based on the gap between the class prediction result and the labeled class of the first sample medical image, the feature extraction consistency comparison result, and the feature extraction similarity comparison result, adjusting parameters of the to-be-trained model until the to-be-trained model meets a training stop condition, and stopping training to obtain the medical image classification model.

[0010] The second aspect of the present application provides a medical image classification method. The medical image classification method comprises:

[0011] obtaining a medical image to be classified;

[0012] taking the medical image to be classified as an input of a medical image classification model, performing feature extraction and class prediction on the medical image to be classified by the medical image classification model, and obtaining an image class output by the medical image classification model; the medical image classification model is a model trained according to the training method of the medical image classification model of the first aspect.

[0013] The third aspect of the present application provides a training device of a medical image classification model. The training device comprises:

[0014] an image feature extraction unit configured to perform feature extraction on a first image, a second image, and a third image respectively by a to-be-trained model, to obtain a first feature extraction result of the first image, a second feature extraction result of the second image, and a third feature extraction result of the third image; the first image and the second image are two different images obtained after enhancement processing of a first sample medical image; the third image is an image obtained after enhancement processing of a second sample medical image, and the first image and the third image are processed in the same manner;

[0015] a prediction result obtaining unit configured to predict a class of the first sample medical image based on the first feature extraction result, to obtain a class prediction result;

[0016] a comparison result obtaining unit configured to obtain a feature extraction consistency comparison result of the to-be-trained model for the first image and the second image, and obtain a feature extraction similarity comparison result of the to-be-trained model for the first image and the third image based on the first feature extraction result, the second feature extraction result, and the third feature extraction result;

[0017] a model parameter adjusting unit configured to adjust parameters of the to-be-trained model based on the gap between the class prediction result and the labeled class of the first sample medical image, the feature extraction consistency comparison result, and the feature extraction similarity comparison result, until the to-be-trained model meets a training stop condition, and stop training to obtain the medical image classification model.

[0018] The fourth aspect of the present application provides a medical image classification device. The medical image classification device comprises:

[0019] a medical image acquisition unit configured to acquire a medical image to be classified;

[0020] an image category obtaining unit configured to take the medical image to be classified as an input of a medical image classification model, perform feature extraction and category prediction on the medical image to be classified by the medical image classification model, and obtain an image category output by the medical image classification model; the medical image classification model is a model trained according to the training method of the medical image classification model of the first aspect.

[0021] The fifth aspect of the present application provides a computer device. The recognition device comprises:

[0022] a memory having a computer program stored thereon;

[0023] a processor configured to execute the computer program in the memory to implement the steps of the training method of the medical image classification model provided in the first aspect, or the steps of the medical image classification method provided in the second aspect.

[0024] The sixth aspect of the present application provides a computer readable storage medium having a computer program stored thereon. The program, when executed by a processor, implements the steps of the training method of the medical image classification model provided in the first aspect, or the steps of the medical image classification method provided in the second aspect.

[0025] Compared with the prior art, the present application has the following beneficial effects:

[0026] In the present application, firstly, the first image, the second image and the third image are respectively subjected to feature extraction by the to-be-trained model to obtain a first feature extraction result of the first image, a second feature extraction result of the second image and a third feature extraction result of the third image; on this basis, the category of the first sample medical image is predicted according to the first feature extraction result to obtain a category prediction result; then, the feature extraction consistency comparison result of the to-be-trained model for the first image and the second image is obtained, and the feature extraction similarity comparison result of the to-be-trained model for the first image and the third image is obtained according to the first feature extraction result, the second feature extraction result and the third feature extraction result; finally, the parameters of the to-be-trained model are adjusted according to the gap between the category prediction result and the labeled category of the first sample medical image, the feature extraction consistency comparison result and the feature extraction similarity comparison result, until the to-be-trained model meets the training stopping condition, and the training is stopped to obtain the medical image classification model. It can be seen that in the present application, three feature extraction paths are designed for the model. On the one hand, the category of the image is predicted in one of the feature extraction paths to obtain the category prediction result, and on the other hand, the consistency comparison result is obtained by combining the consistency regularization rule to compare the consistent features of the same image in the remaining two feature extraction paths, and the similarity comparison result is obtained by combining the similarity calculation technology to compare the differentiated features of different images, so that the global and local features of the image are extracted to realize the fine-grained feature extraction of the image, so that the model extraction capability can be better trained. Compared with the manual image classification by medical staff, the medical image classification model trained by the present application can capture global high-level features and local fine-grained features in the image, thereby improving the accuracy of medical image classification and the efficiency of medical image classification. BRIEF DESCRIPTION OF DRAWINGS DETAILED DESCRIPTION OF THE INVENTION BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0028] Figure 1 A flowchart of a medical image classification model training method provided by an embodiment of the present application;

[0029] Figure 2 An enhanced processing schematic diagram of a medical image classification model training method provided by an embodiment of the present application;

[0030] Figure 3 An image preprocessing schematic diagram of a medical image classification model training method provided by an embodiment of the present application;

[0031] Figure 4 A model structure diagram of a medical image classification model training method provided by an embodiment of the present application;

[0032] Figure 5 A flowchart of another medical image classification model training method provided by an embodiment of the present application;

[0033] Figure 6 A model structure diagram of another medical image classification model training method provided by an embodiment of the present application;

[0034] Figure 7a A flowchart of another medical image classification model training method provided by an embodiment of the present application;

[0035] Figure 7b A model structure diagram of another medical image classification model training method provided by an embodiment of the present application;

[0036] Figure 8 A flowchart of a medical image classification method provided by an embodiment of the present application;

[0037] Figure 9 A structure diagram of a medical image classification model training device provided by an embodiment of the present application;

[0038] Figure 10 A structure diagram of a medical image classification device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0039] As described above, at present, in the field of medical image processing technology, medical staff mainly rely on manual image classification of medical images, for example, classifying images into abnormal images and normal images. Here, abnormal and normal can refer to the same organ. However, medical images do not have obvious foreground and background differences like natural images, and only relying on image high-level features cannot well complete recognition, and the behavior of identifying whether an organ in a medical image is diseased mainly depends on the subtle changes of texture, and the identification task of medical staff needs to pay more attention to the details of medical images. Since image classification by manual work has a certain subjectivity, it will affect the accuracy of image classification to some extent. In addition, paying attention to the details of medical images will also make medical staff spend a lot of effort on image classification of medical images, resulting in low image classification efficiency. Therefore, how to improve the accuracy and efficiency of medical image classification is the focus of the technical personnel in the field.

[0040] In view of the above problems, a solution is provided in the embodiments of the present application, and a training method and a classification method of a medical image classification model and related products are proposed, aiming to improve the accuracy of medical image classification and the efficiency of medical image classification. In the technical solution of the present application, the first feature extraction result can be obtained by performing feature extraction on the first image by the to-be-trained model, the second feature extraction result can be obtained by performing feature extraction on the second image, and the third feature extraction result can be obtained by performing feature extraction on the third image. Then, the category of the first sample medical image can be predicted according to the first feature extraction result to obtain a category prediction result. The feature extraction consistency comparison result of the to-be-trained model for the first image and the second image can be obtained according to the first feature extraction result, the second feature extraction result and the third feature extraction result, and the feature extraction similarity comparison result of the to-be-trained model for the first image and the third image can be obtained. Finally, the parameters of the to-be-trained model can be adjusted according to the gap between the category prediction result and the labeled category of the first sample medical image, the feature extraction consistency comparison result and the feature extraction similarity comparison result, until the to-be-trained model meets the training stopping condition, and the medical image classification model is obtained by stopping training.

[0041] It can be seen that three feature extraction paths are designed for the model in the present application. For one aspect, the category of the image is predicted in one of the feature extraction paths to obtain a category prediction result, and for another aspect, consistency feature comparison is performed on the same image in the remaining two feature extraction paths to obtain a consistency comparison result, and difference feature comparison is performed on different images to obtain a similarity comparison result. In this way, the global features of the image and the local features of the image are extracted to realize fine-grained feature extraction of the image, so that the model extraction capability can be better trained. Compared with manual image classification by medical staff, the medical image classification model trained by the present application can capture global high-level features and local fine-grained features in the image, thereby improving the accuracy of medical image classification and the efficiency of medical image classification. In this way, whether from the accuracy aspect or the efficiency aspect, the classified medical image can improve the auxiliary role of the medical staff in the medical scene, and further improve the utilization rate of the medical image. In addition, the generalization ability of the small data training model can also be improved.

[0042] In order to enable personnel in the technical field to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0043] Referring to Figure 1 , the figure is a flowchart of a medical image classification model training method provided by an embodiment of the present application. As shown in Figure 1 , the medical image classification model training method comprises:

[0044] S101: performing feature extraction on the first image, the second image and the third image by the to-be-trained model to obtain a first feature extraction result of the first image, a second feature extraction result of the second image and a third feature extraction result of the third image.

[0045] In this step, the first image and the second image are two different images obtained after the first sample medical image is enhanced, and the third image is an image obtained after the second sample medical image is enhanced, and the first image and the third image are enhanced in the same way.

[0046] As an example, for the operation of obtaining the first image after the first sample medical image is enhanced, the operation can be specifically performed as follows: performing image flipping and / or translation on the first sample medical image to obtain the first image. For the operation of obtaining the third image after the second sample medical image is enhanced, the operation can be specifically performed as follows: performing image flipping and / or translation on the second sample medical image to obtain the third image. It can be understood that in the present application, the operation of image flipping and / or translation can be regarded as weak enhancement. In this way, the first sample medical image and the second sample medical image can be weakly enhanced to obtain the first image and the third image.

[0047] For the operation of obtaining the second image after the first sample medical image is enhanced, the operation can be specifically performed as follows: using one or a combination of multiple of the preset enhancement processing manners on the first sample medical image to obtain the second image; the preset enhancement processing manners include brightness adjustment, contrast adjustment, histogram equalization, random region occlusion, scaling, rotation, sharpening or blurring. It can be understood that in the present application, the operation of performing preset enhancement processing on the image can be regarded as strong enhancement. In this way, the first sample medical image can be strongly enhanced to obtain the second image. For example, the first sample medical image can be enhanced by combining brightness adjustment, contrast adjustment and histogram equalization to obtain the second image. Figure 2 An enhancement processing schematic diagram of the medical image classification model training method provided by an embodiment of the present application is shown in Figure 2 , wherein Figure 2 four second images obtained by using four different preset enhancement processing manners on the first sample medical image are shown in

[0048] In addition, since the sizes of medical images generated by different imaging devices are different, and the model generally requires input of medical images of the same size, the medical images can be unified to a fixed size, such as 224mm*224mm, through image preprocessing methods such as cropping or scaling. Figure 3 An image preprocessing schematic diagram of a medical image classification model training method provided by an embodiment of the present application is shown in Figure 3 . Figure 3 The (1) diagram in (1) represents a normal medical image, and the (2) diagram represents a pulmonary tuberculosis medical image. The (1) diagram and the (2) diagram are both subjected to image preprocessing operations to adjust the image size, so as to obtain medical images conforming to the input specifications of the model.

[0049] Next, the composition structure of the to-be-trained model is introduced. Referring to Figure 4 , Figure 4 A model structure schematic diagram of a medical image classification model training method provided by an embodiment of the present application is shown in Figure 4 .

[0050] The to-be-trained model includes a convolutional neural network, the convolutional neural network includes a feature extraction network and a fully connected classifier, the fully connected classifier includes a first fully connected layer and a classification layer, and the to-be-trained model further includes a second fully connected layer and a third fully connected layer. The convolutional neural network has a first output end, a second output end and a third output end, the first output end is connected to the input end of the first fully connected layer, the second output end is connected to the input end of the second fully connected layer, and the third output end is connected to the input end of the third fully connected layer.

[0051] It can be understood that, in order to better train the feature extraction capability of the model, three feature extraction paths are set for the model of the present application. The first feature extraction path is that the feature extraction network is connected to the first fully connected layer, the first fully connected layer is connected to the classification layer, and the first feature extraction path is used to predict the category of the image, so as to finally obtain the difference between the predicted category and the labeled category of the image. The second feature extraction path is that the feature extraction network is connected to the second fully connected layer, and is used to obtain a consistency comparison result compared with the image features in the first feature extraction path. The third feature extraction path is that the feature extraction network is connected to the third fully connected layer, and is used to obtain a similarity comparison result compared with the image features in the first feature extraction path. Among them, the feature extraction networks in the three feature extraction paths are shared, which facilitates the adjustment of the parameters of the model at one time according to the loss functions generated by the three paths in the subsequent process.

[0052] It can be further understood that the first image, the second image and the third image are input into the three feature extraction paths respectively. The first sample medical image is subjected to different enhancement processing to obtain the first image and the second image, the first image is input into the first feature extraction path, the second image is input into the second feature extraction path, and the second sample medical image is subjected to enhancement processing to obtain the third image, and the third image is input into the third feature extraction path.

[0053] Further, the first image, the second image and the third image are all subjected to feature extraction by the feature extraction network, the first feature extraction result of the first image is input into the first full connection layer through the first output end, the second feature extraction result of the second image is input into the second full connection layer through the second output end, and the third feature extraction result of the third image is input into the third full connection layer through the third output end. In this way, the global and local features of the image are extracted through different feature extraction paths to realize fine-grained feature extraction of the image, so that the model extraction capability can be better trained.

[0054] In addition, the specific composition of the feature extraction network and the full connection classifier is described in combination with Table 1. It should be noted that the feature extraction network can be a Resnet network structure or a self-defined network structure, which is not specifically limited here. The feature extraction network mentioned in Table 1 is a self-defined network structure, which includes module 1, module 2, module 3, module 4 and module 5. Each of module 1, module 2, module 3, module 4 and module 5 includes three convolutional layers and one max-pooling layer. For example, module 1 includes convolutional layer 1, convolutional layer 2, convolutional layer 3 and max-pooling layer 1, wherein convolutional layer 1, convolutional layer 2, convolutional layer 3 and max-pooling layer 1 are connected in sequence, the input column of convolutional layer 1 represents the number of channels, H represents the vertical size, W represents the horizontal size, 1x1 represents the size of the convolution kernel, and the output column of max-pooling layer 1 represents the number of channels, 1 / 2H represents reducing the vertical size by 1 / 2, and 1 / 2W represents reducing the horizontal size by 1 / 2. Similarly, the remaining convolutional layers in module 1 and the convolutional layers and max-pooling layers in the remaining modules are similar to the above representation. The full connection classification layer includes FC-1024 (the first full connection layer) and FC-2 (the classification layer), and it can be understood that FC-1024 outputs 1024 neurons, and then FC-2 outputs two neurons, i.e. the predicted image classification situation. It should be noted that the model of the present application is also applicable to a multi-classification scenario.

[0055] Table 1

[0056]

[0057] S102: predicting a class of the first sample medical image based on the first feature extraction result, to obtain a class prediction result.

[0058] In this step, the class prediction result can be a class prediction probability. For example, the classification of an image has two categories: abnormal image and normal image. Therefore, the class prediction result can be the probability of the image being an abnormal image and the probability of the image being a normal image. The sum of the prediction probabilities of different categories can be 1.

[0059] S103: obtaining a feature extraction consistency comparison result of the to-be-trained model for the first image and the second image based on the first feature extraction result, the second feature extraction result and the third feature extraction result, and obtaining a feature extraction similarity comparison result of the to-be-trained model for the first image and the third image.

[0060] The following steps S1031-S1033 are described. Figure 5 Another flowchart of the training method of the medical image classification model provided by the embodiment of the present application is shown in Figure 5 .

[0061] S1031: obtaining a first feature vector output by the first full connection layer after processing the first feature extraction result, obtaining a second feature vector output by the second full connection layer after processing the second feature extraction result, and obtaining a third feature vector output by the third full connection layer after processing the third feature extraction result.

[0062] S1032: obtaining a feature extraction consistency comparison result of the to-be-trained model for the first image and the second image based on the first feature vector and the second feature vector.

[0063] As shown in Figure 6 , Figure 6 Another model structure diagram of the training method of the medical image classification model provided by the embodiment of the present application is shown in the figure. In the three feature extraction paths of the present application, a normalization layer is connected after the first full connection layer, the second full connection layer and the third full connection layer respectively (it should be noted that the normalization layer connected after the full connection layer is not embodied in Figure 6 ). In order to normalize the first feature vector, the second feature vector and the third feature vector output by the first full connection layer, the second full connection layer and the third full connection layer respectively. Finally, the comparison results obtained by training for each single sample image pair (the first sample medical image and the second sample medical image) are output in the three feature extraction paths respectively. The normalization layer includes a softmax activation function.

[0064] Specifically, the first feature vector is normalized by a normalization layer connected to the output end of the first fully connected layer to obtain a first normalized vector, and the second feature vector is normalized by a normalization layer connected to the output end of the second fully connected layer to obtain a second normalized vector. Then, based on the first normalized vector and the second normalized vector, the feature extraction consistency comparison result l can be calculated using the following formula cr (that is, the feature extraction consistency comparison result l of the image on the first feature extraction path and the second feature extraction path is calculated cr ):

[0065]

[0066] wherein the first fully connected layer and the second fully connected layer each have n neurons, and the first normalized vector and the second normalized vector are each n-dimensional vectors; Q i represents the value of the i-th dimension in the first normalized vector, and P i represents the value of the i-th dimension in the second normalized vector. It can be understood that the purpose of training the model is to make the consistency of the image on the first feature extraction path and the second feature extraction path as high as possible, and the value of the corresponding consistency comparison result l cr is as small as possible.

[0067] S1033: Based on the first feature vector and the third feature vector, obtain a feature extraction similarity comparison result of the to-be-trained model for the first image and the third image.

[0068] Specifically, the first feature vector is normalized by a normalization layer connected to the output end of the first fully connected layer to obtain a first normalized vector, and the third feature vector is normalized by a normalization layer connected to the output end of the third fully connected layer to obtain a third normalized vector. Then, based on the first normalized vector and the third normalized vector, the feature extraction similarity comparison result l can be calculated using the following formula cos (that is, the feature extraction similarity comparison result l of the image on the first feature extraction path and the third feature extraction path is calculated cos ):

[0069]

[0070] wherein the first fully connected layer and the third fully connected layer each have n neurons, and the first normalized vector and the third normalized vector are each n-dimensional vectors; Q i represents the value of the i-th dimension in the first normalized vector, and R i represents the value of the i-th dimension in the third normalized vector. It can be understood that the purpose of training the model is to make the difference of the image on the first feature extraction path and the third feature extraction path as large as possible, and the value of the corresponding similarity comparison result l cosThe smaller the better.

[0071] It should be noted that the above steps S1032 and S1033 are not in order, and the two steps can be performed simultaneously.

[0072] In addition, it can be understood that in the first feature extraction path, the classification layer can be used to classify the first normalized vector to obtain a classification result, and then the normalized layer connected to the output end of the classification layer is used to normalize the classification result to obtain a class prediction result. For example, the classification result can include a feature representation of the first sample medical image belonging to an image of infected tuberculosis and a feature representation of the first sample medical image belonging to an image of non-infected tuberculosis. As an example, the class prediction result can be represented by the following formula

[0073]

[0074] wherein, represents the probability that the class prediction result is the i-th class, x i represents the value corresponding to the position of the i-th class in the classification result; x p represents the value corresponding to the position of the p-th class in the classification result, k represents the number of classes, p takes a value in 1 to k, k is an integer, and p is also an integer.

[0075] Further, the labeled class of the first sample medical image can be obtained, wherein the labeled class of the first sample medical image is the class labeled by a professional medical worker on the medical image, and then the gap l between the class prediction result and the labeled class of the first sample medical image can be represented by the following formula ce (that is, the gap l between the class prediction result on the first feature extraction path and the labeled class of the first sample medical image ce ):

[0076]

[0077] wherein, represents the gap between the class prediction result of the j-th sample medical image (specifically, the image as the first sample medical image) and its corresponding labeled class, k represents the number of classes, represents the class of the j-th sample medical image is p, if the class of the j-th sample medical image is p, is 1, otherwise 0; represents the probability that the j-th sample medical image is predicted as the p-th class.

[0078] S104: adjusting the parameters of the to-be-trained model based on the gap between the category prediction result and the labeled category of the first sample medical image, the feature extraction consistency comparison result, and the feature extraction similarity comparison result, until the to-be-trained model meets a training stop condition, and stopping training to obtain a medical image classification model.

[0079] It should be noted that the to-be-trained model has a corresponding training data set, and the sample medical images in the training data set are divided into multiple training batches, and the sample medical images in each training batch form multiple sample medical image pairs, and each sample medical image pair includes a first sample medical image and a second sample medical image used for mutual comparison in the model training process.

[0080] It can be understood that "first" and "second" in the first sample medical image and the second sample medical image are only concepts, and "second" is relative to "first", and "second" is a random one taken from "first". The first sample medical image can be the same image in different sample medical image pairs, and the second sample medical image can be different images in different sample medical image pairs, so as to form comparison. For example, the AB sample medical image pair includes A sample medical image and B sample medical image, and the AC sample medical image pair includes A sample medical image and C sample medical image.

[0081] Further, Figure 7a Another flowchart of a medical image classification model training method provided by an embodiment of the present application is shown in FIG. 10. Step S104 (for the training process of the target training batch, adjusting the parameters of the to-be-trained model based on the gap between the category prediction result and the labeled category of the first sample medical image, the feature extraction consistency comparison result, and the feature extraction similarity comparison result) includes steps S1041-S1045: Figure 7a And,

[0082] Another model structure diagram of a medical image classification model training method provided by an embodiment of the present application is shown in FIG. 11. In the model training process, after obtaining the gap between the category prediction result and the labeled category, the feature extraction consistency comparison result, and the feature extraction similarity comparison result corresponding to each sample medical image pair, the category prediction loss, the consistency loss, and the similarity loss of the to-be-trained model are also calculated. S1041: calculating the category prediction loss of the to-be-trained model based on the gap between the category prediction result and the labeled category corresponding to each sample medical image pair in the target training batch. Figure 7b Figure 7b

[0083] ​​Since the first feature extraction path in the above process has been trained for a single sample image pair (the first sample medical image and the second sample medical image) and the gap between the class prediction result and the labeled class of the single sample image pair has been obtained, the following formula can be used to calculate the class prediction loss L of each sample medical image pair in the to-be-trained model ce (that is, the class prediction loss L of each sample medical image in the first feature extraction path in the to-be-trained model ce ):

[0084]

[0085] where m is the total number of sample medical image pairs. Assuming that there are m sample medical images in the target training batch, each sample medical image can be taken as the first sample medical image, and a different sample medical image can be randomly selected as the second sample medical image to construct m sample medical image pairs.

[0086] S1042: Based on the feature extraction consistency comparison result corresponding to each sample medical image pair in the target training batch, the consistency loss of the to-be-trained model is calculated.

[0087] Since the second feature extraction path in the above process has been trained for a single sample image pair and the feature extraction consistency comparison result of the single sample image pair has been obtained, the following formula can be used to calculate the consistency loss L of each sample medical image with respect to the to-be-trained model cr (that is, the consistency loss L of the second feature extraction path in the to-be-trained model cr ):

[0088]

[0089] where m is the total number of sample medical image pairs.

[0090] S1043: Based on the feature extraction similarity comparison result corresponding to each sample medical image pair in the target training batch, the similarity loss of the to-be-trained model is calculated.

[0091] Since the third feature extraction path in the above process has been trained for a single sample image pair and the feature extraction similarity comparison result of the single sample image pair has been obtained, the following formula can be used to calculate the similarity loss L of each sample medical image with respect to the to-be-trained model cos (that is, the similarity loss L of the third feature extraction path in the to-be-trained model cos ):

[0092]

[0093] Wherein, m is the total number of sample medical image pairs.

[0094] S1044: According to the category prediction loss, the consistency loss and the similarity loss, the total loss of the to-be-trained model in the target training batch is calculated.

[0095] Specifically, the total loss of the to-be-trained model in the target training batch can be calculated by the following formula:

[0096] L = a1 x L ce + a2 x L cr + a3 x L cos

[0097] Wherein, L represents the total loss, L ce represents the category prediction loss, L cr represents the consistency loss, L cos represents the similarity loss, a1 represents the weight of the category prediction loss in the total loss, a2 represents the weight of the consistency loss in the total loss, and a3 represents the weight of the similarity loss in the total loss. It should be noted that a1, a2 and a3 are adjustable loss function weights, and the adjustment of the weights mainly depends on the training data set. Different values can be set according to different training data sets, such as setting a1, a2 and a3 to 10, 5 and 1 respectively.

[0098] S1045: Adjust the parameters of the to-be-trained model according to the total loss.

[0099] In this way, by designing the feature extraction path of the model, the ability of the model to extract global and local features of the image can be better trained, the fine-grained feature extraction of the image can be realized, and the parameters of the to-be-trained model are adjusted by integrating the category prediction loss, the consistency loss and the similarity loss to obtain the total loss, so as to complete the construction of the medical image classification model with high classification accuracy and efficiency.

[0100] In the above process, the training process of the medical image classification model has been described in detail. Next, the application process of the medical image classification model is introduced. Figure 8 The flowchart of a medical image classification method provided by the embodiment of the present application is shown in Figure 8 The medical image classification method comprises:

[0101] S801: Obtain a medical image to be classified.

[0102] Wherein, the medical image to be classified can be an X-ray image to be predicted whether infected with pulmonary tuberculosis. The X-ray image can be obtained from a medical system. Then the X-ray image can be processed to obtain an image meeting the input model specification size. The processing process is similar to the above image processing process, and will not be described here.

[0103] S802: input the medical image to be classified into a medical image classification model, perform feature extraction and category prediction on the medical image to be classified by the medical image classification model, and obtain an image category output by the medical image classification model.

[0104] In this step, in the foregoing model training process, the consistency regularization rule and the similarity calculation technology are combined to train the model to extract the global and local features of the image, so as to realize the fine-grained feature extraction of the image. In this way, in the model application process, the medical image classification model can capture the global high-level features and the local fine-grained features in the image, perform feature extraction and category prediction on the medical image to be classified, determine whether the medical image to be classified belongs to the medical image of infected tuberculosis or the medical image of uninfected tuberculosis, and output the image category, which further improves the accuracy and efficiency of medical image classification. The medical image classification model is a model trained according to the training method of the medical image classification model.

[0105] It should be noted that different medical image classification models can be trained for medical image classification processing of different bodies. It only needs to input the X-ray image sample pair corresponding to the body in the initial stage of training the model, so that the medical image classification model corresponding to the body can be generated after the model is trained.

[0106] Based on the training method of the medical image classification model provided in the foregoing embodiments, the present application also correspondingly provides a training device of a medical image classification model. Figure 9 A structural schematic diagram of a training device of a medical image classification model provided in an embodiment of the present application is shown in FIG. 9. Figure 9 The training device includes:

[0107] An image feature extraction unit 901 is configured to perform feature extraction on a first image, a second image and a third image by a to-be-trained model respectively, to obtain a first feature extraction result of the first image, a second feature extraction result of the second image and a third feature extraction result of the third image. The first image and the second image are two different images obtained after enhancement processing of a first sample medical image. The third image is an image obtained after enhancement processing of a second sample medical image, and the first image and the third image are processed in the same manner.

[0108] A prediction result obtaining unit 902 is configured to predict the category of the first sample medical image based on the first feature extraction result, to obtain a category prediction result.

[0109] The comparison result obtaining unit 903 is configured to obtain a feature extraction consistency comparison result of the to-be-trained model for the first image and the second image based on the first feature extraction result, the second feature extraction result, and the third feature extraction result, and obtain a feature extraction similarity comparison result of the to-be-trained model for the first image and the third image.

[0110] The model parameter adjusting unit 904 is configured to adjust parameters of the to-be-trained model based on a gap between the category prediction result and a labeled category of the first sample medical image, the feature extraction consistency comparison result, and the feature extraction similarity comparison result, until the to-be-trained model meets a training stop condition, and stop training to obtain a medical image classification model.

[0111] Optionally, the to-be-trained model comprises a convolutional neural network, the convolutional neural network comprises a feature extraction network and a fully connected classifier, the fully connected classifier comprises a first fully connected layer and a classification layer, and the to-be-trained model further comprises a second fully connected layer and a third fully connected layer; the convolutional neural network has a first output end, a second output end, and a third output end, the first output end is connected to an input end of the first fully connected layer, the second output end is connected to an input end of the second fully connected layer, and the third output end is connected to an input end of the third fully connected layer.

[0112] The extraction result output unit is configured to perform feature extraction on the first image, the second image, and the third image by the feature extraction network, and the first feature extraction result, the second feature extraction result, and the third feature extraction result are input into the first fully connected layer, the second fully connected layer, and the third fully connected layer via the first output end, the second output end, and the third output end, respectively.

[0113] Optionally, the comparison result obtaining unit 903 comprises:

[0114] The feature vector obtaining unit is configured to obtain a first feature vector output by the first fully connected layer after processing the first feature extraction result, obtain a second feature vector output by the second fully connected layer after processing the second feature extraction result, and obtain a third feature vector output by the third fully connected layer after processing the third feature extraction result.

[0115] The consistency comparison result obtaining unit is configured to obtain the feature extraction consistency comparison result of the to-be-trained model for the first image and the second image based on the first feature vector and the second feature vector.

[0116] The similarity comparison result obtaining unit is configured to obtain a feature extraction similarity comparison result of the to-be-trained model for the first image and the third image based on the first feature vector and the third feature vector.

[0117] Optionally, the to-be-trained model is configured to have a corresponding training data set, sample medical images in the training data set are divided into a plurality of training batches, and sample medical images in each training batch constitute a plurality of sample medical image pairs, one sample medical image pair including one first sample medical image and one second sample medical image used for mutual comparison in a model training process.

[0118] The model parameter adjustment unit 904 includes:

[0119] The prediction loss calculation unit is configured to calculate a category prediction loss of the to-be-trained model based on a difference between a category prediction result corresponding to each sample medical image pair in the target training batch and a labeled category.

[0120] The consistency loss calculation unit is configured to calculate a consistency loss of the to-be-trained model based on a feature extraction consistency comparison result corresponding to each sample medical image pair in the target training batch.

[0121] The similarity loss calculation unit is configured to calculate a similarity loss of the to-be-trained model based on a feature extraction similarity comparison result corresponding to each sample medical image pair in the target training batch.

[0122] The total loss calculation unit is configured to calculate a total loss of the to-be-trained model in the target training batch according to the category prediction loss, the consistency loss, and the similarity loss.

[0123] The total loss adjustment parameter unit is configured to adjust parameters of the to-be-trained model according to the total loss.

[0124] Optionally, the consistency comparison result obtaining unit is specifically configured to:

[0125] The first feature vector is normalized to obtain a first normalized vector, and the second feature vector is normalized to obtain a second normalized vector.

[0126] The feature extraction consistency comparison result lcr is calculated based on the first normalized vector and the second normalized vector by using the following formula:

[0127]

[0128] The first fully connected layer and the second fully connected layer each have n neurons, and the first normalized vector and the second normalized vector are each an n-dimensional vector. idenotes the value of the i-th dimension in the first normalized vector, P i denotes the value of the i-th dimension in the second normalized vector.

[0129] Optionally, the similarity comparison result obtaining unit is specifically configured to:

[0130] The first feature vector is normalized to obtain a first normalized vector, and the third feature vector is normalized to obtain a third normalized vector.

[0131] Based on the first normalized vector and the third normalized vector, a feature extraction similarity comparison result lcos is calculated using the following formula:

[0132]

[0133] The first full connection layer and the third full connection layer each have n neurons, and the first normalized vector and the third normalized vector are each n-dimensional vectors; Q i denotes the value of the i-th dimension in the first normalized vector, R i denotes the value of the i-th dimension in the third normalized vector.

[0134] Optionally, the calculation formula unit is configured to calculate the total loss of the target training batch of the to-be-trained model using the following formula:

[0135] L = α1×L ce + α2×L cr + α3×L cos

[0136] In the formula, L represents the total loss, L ce represents the category prediction loss, L cr represents the consistency loss, L cos represents the similarity loss, α1 represents the weight of the category prediction loss in the total loss, α2 represents the weight of the consistency loss in the total loss, and α3 represents the weight of the similarity loss in the total loss.

[0137] Optionally, an image enhancement processing unit is configured to

[0138] The first image is obtained after the first sample medical image is enhanced, and the first image includes:

[0139] The first image is obtained by performing image flipping and / or translation on the first sample medical image.

[0140] The second image is obtained after the first sample medical image is enhanced, and the second image includes:

[0141] adopting one or more combinations of preset enhancement processing manners on the first sample medical image to obtain the second image; the preset enhancement processing manners include brightness adjustment, contrast adjustment, histogram equalization, random region occlusion, scaling, rotation, sharpening or blurring;

[0142] obtaining a third image after enhancing the second sample medical image, including:

[0143] performing image flipping and / or translation on the second sample medical image to obtain the third image.

[0144] Based on the medical image classification method provided in the foregoing embodiments, the application also correspondingly provides a medical image classification device. Figure 10 A structural schematic diagram of a medical image classification device provided in the embodiments of the application is shown in FIG. 1. Figure 10 The training device includes:

[0145] a medical image acquisition unit 1001 configured to acquire a medical image to be classified;

[0146] an image category obtaining unit 1002 configured to take the medical image to be classified as an input of a medical image classification model, perform feature extraction and category prediction on the medical image to be classified by using the medical image classification model, and obtain an image category output by the medical image classification model; the medical image classification model is a model trained according to the medical image classification model training method of the first aspect.

[0147] The application also provides a computer device. The recognition device includes:

[0148] a memory having a computer program stored thereon.

[0149] a processor configured to execute the computer program in the memory to implement part or all of the steps of the medical image classification model training method introduced in the foregoing embodiments, or implement part or all of the steps of the medical image classification method introduced in the foregoing embodiments.

[0150] The application also provides a computer readable storage medium having a computer program stored thereon. The program is executed by a processor to implement part or all of the steps of the medical image classification model training method introduced in the foregoing embodiments, or implement part or all of the steps of the medical image classification method introduced in the foregoing embodiments.

[0151] It should be noted that each of the embodiments of the present specification is described in a progressive manner, and the same or similar parts between each embodiment can be understood by referring to each other, and each embodiment focuses on the difference from other embodiments. In particular, for the device and equipment embodiments, since they are basically similar to the method embodiments, they are described more simply, and the relevant parts can be understood by referring to the part of the method embodiment. The above-described device and equipment embodiments are only illustrative, and the units described as separate components can or can not be physically separated, and the components indicated as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiments of the present application according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0152] The above is only one specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any skilled person in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for training a medical image classification model, characterized in that, Comprise: The first image, the second image and the third image are respectively extracted by the to-be-trained model, and the first feature extraction result of the first image, the second feature extraction result of the second image and the third feature extraction result of the third image are obtained; The first image and the second image are two different images obtained after the first sample medical image is enhanced; The third image is an image obtained after the second sample medical image is enhanced, and the first image and the third image are enhanced in the same way; The category of the first sample medical image is predicted based on the first feature extraction result, and a category prediction result is obtained; Based on the first feature extraction result, the second feature extraction result and the third feature extraction result, the feature extraction consistency comparison result of the to-be-trained model for the first image and the second image is obtained, and the feature extraction similarity comparison result of the to-be-trained model for the first image and the third image is obtained; Based on the gap between the category prediction result and the labeled category of the first sample medical image, the feature extraction consistency comparison result and the feature extraction similarity comparison result, the parameters of the to-be-trained model are adjusted until the to-be-trained model meets the training stop condition, and the medical image classification model is obtained.

2. The training method of claim 1, wherein, The to-be-trained model comprises a convolutional neural network, the convolutional neural network comprises a feature extraction network and a fully connected classifier, the fully connected classifier comprises a first fully connected layer and a classification layer, and the to-be-trained model further comprises a second fully connected layer and a third fully connected layer; The convolutional neural network has a first output end, a second output end and a third output end, the first output end is connected to the input end of the first fully connected layer, the second output end is connected to the input end of the second fully connected layer, and the third output end is connected to the input end of the third fully connected layer; The first image, the second image and the third image are all extracted by the feature extraction network, and the first feature extraction result, the second feature extraction result and the third feature extraction result are respectively input into the first fully connected layer, the second fully connected layer and the third fully connected layer through the first output end, the second output end and the third output end.

3. The training method of claim 2, wherein, The first feature vector output by the first fully connected layer after processing the first feature extraction result, the second feature vector output by the second fully connected layer after processing the second feature extraction result, and the third feature vector output by the third fully connected layer after processing the third feature extraction result are obtained; ​ obtaining a feature extraction consistency comparison result of the to-be-trained model for the first image and the second image based on the first feature vector and the second feature vector; obtaining a feature extraction similarity comparison result of the to-be-trained model for the first image and the third image based on the first feature vector and the third feature vector.

4. The training method of claim 3, wherein, The to-be-trained model has a corresponding training data set, and sample medical images in the training data set are divided into multiple training batches, and sample medical images in each training batch form multiple sample medical image pairs, and a sample medical image pair includes a first sample medical image and a second sample medical image used for mutual comparison in a model training process; For the training process of the target training batch, the parameters of the to-be-trained model are adjusted based on the gap between the category prediction result and the labeled category of the first sample medical image, the feature extraction consistency comparison result, and the feature extraction similarity comparison result, including: calculating a category prediction loss of the to-be-trained model based on the gap between the category prediction result and the labeled category of each sample medical image pair in the target training batch; calculating a consistency loss of the to-be-trained model based on the feature extraction consistency comparison result of each sample medical image pair in the target training batch; calculating a similarity loss of the to-be-trained model based on the feature extraction similarity comparison result of each sample medical image pair in the target training batch; calculating a total loss of the to-be-trained model in the target training batch according to the category prediction loss, the consistency loss, and the similarity loss; adjusting the parameters of the to-be-trained model according to the total loss.

5. The training method according to claim 3 or 4, characterized in that, The obtaining of the feature extraction consistency comparison result of the to-be-trained model for the first image and the second image based on the first feature vector and the second feature vector includes: normalizing the first feature vector to obtain a first normalized vector, and normalizing the second feature vector to obtain a second normalized vector; calculating a feature extraction consistency comparison result lcr based on the first normalized vector and the second normalized vector using the following formula: The first fully connected layer and the second fully connected layer each have n neurons, the first normalization vector and the second normalization vector are each n-dimensional vectors; Q i represents a value of an i-th dimension in the first normalization vector, P i represents a value of an i-th dimension in the second normalization vector.

6. The training method according to claim 3 or 4, characterized in that, The obtaining of the feature extraction similarity comparison result of the to-be-trained model for the first image and the third image based on the first feature vector and the third feature vector includes: normalizing the first feature vector to obtain a first normalized vector, and normalizing the third feature vector to obtain a third normalized vector; calculating a feature extraction similarity comparison result lcos based on the first normalized vector and the third normalized vector using the following formula: The first fully connected layer and the third fully connected layer each have n neurons, the first normalization vector and the third normalization vector are each n-dimensional vectors; Q i represents a value of an i-th dimension in the first normalization vector, R i represents a value of an i-th dimension in the third normalization vector.

7. The training method of claim 4, wherein, The total loss of the to-be-trained model in the target training batch is calculated by the following formula: L = a1 x L ce + a2 x L cr + a3 x L cos In the formula, L represents the total loss, L ce represents the category prediction loss, L cr represents the consistency loss, L cos represents the similarity loss, a1 represents the weight of the category prediction loss in the total loss, a2 represents the weight of the consistency loss in the total loss, and a3 represents the weight of the similarity loss in the total loss.

8. The training method of any one of claims 1-4, wherein the first image obtained by enhancing the first sample medical image includes: performing image flipping and / or translation on the first sample medical image to obtain the first image; the second image obtained by enhancing the first sample medical image includes: The first sample medical image is subjected to one or more combinations of preset enhancement processing modes to obtain the second image. The preset enhancement processing modes include brightness adjustment, contrast adjustment, histogram equalization, random region occlusion, scaling, rotation, sharpening, or blurring. The second sample medical image is subjected to enhancement processing to obtain a third image, including: The second sample medical image is subjected to image flipping and / or translation to obtain the third image.

9. A medical image classification method characterized by, Including: Obtaining a medical image to be classified; Taking the medical image to be classified as input of a medical image classification model, performing feature extraction and category prediction on the medical image to be classified by the medical image classification model, and obtaining an image category output by the medical image classification model; the medical image classification model is a model trained by the medical image classification model training method according to any one of claims 1-8. 10.A device for training a medical image classification model, comprising: Including: An image feature extraction unit is configured to perform feature extraction on a first image, a second image, and a third image by a to-be-trained model, respectively, to obtain a first feature extraction result of the first image, a second feature extraction result of the second image, and a third feature extraction result of the third image; the first image and the second image are two different images obtained by enhancing a first sample medical image; the third image is an image obtained by enhancing a second sample medical image, and the first image and the third image are enhanced in the same manner; A prediction result obtaining unit is configured to predict a category of the first sample medical image based on the first feature extraction result to obtain a category prediction result; A comparison result obtaining unit is configured to obtain a feature extraction consistency comparison result of the to-be-trained model for the first image and the second image based on the first feature extraction result, the second feature extraction result, and the third feature extraction result, and obtain a feature extraction similarity comparison result of the to-be-trained model for the first image and the third image; A model parameter adjustment unit is configured to adjust parameters of the to-be-trained model based on a gap between the category prediction result and a labeled category of the first sample medical image, the feature extraction consistency comparison result, and the feature extraction similarity comparison result, until the to-be-trained model meets a training stop condition, and a medical image classification model is obtained by stopping training.

11. A computer device, characterized by Including: A memory having a computer program stored thereon; A processor configured to execute the computer program in the memory to implement the steps of the medical image classification model training method according to any one of claims 1-8, or implement the steps of the medical image classification method according to claim 9.

12. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the medical image classification model training method according to any one of claims 1-8, or implement the steps of the medical image classification method according to claim 9.

Images