Medical image classification model training method and medical image classification method

Abstract

The application provides a medical image classification model training method, a medical image classification method, an electronic device and a storage medium. The training method comprises: obtaining a plurality of first training samples, each first training sample comprising a first randomly transformed image and a second randomly transformed image; taking the first randomly transformed image as the input of a first neural network in a self-supervised pre-training model and taking the second randomly transformed image as the input of a second neural network in the self-supervised pre-training model to train the self-supervised pre-training model; and migrating the model parameters of the trained self-supervised pre-training model to a medical image classification model and training the medical image classification model using a plurality of second training samples. The model parameters of the trained self-supervised pre-training model are migrated to the medical image classification model, so that the medical image classification model can also learn and converge well under a small sample condition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a medical image classification model training method, a medical image classification method, an electronic device and a storage medium. BACKGROUND

[0002] In recent years, the deep learning technology has developed rapidly. The technology has penetrated into the medical image field, and the deep learning technology is applied to automatic analysis of diseases in the medical image field, thereby assisting doctors in diagnosing patient conditions, such as identifying the nature of lung nodules, breast lesions and the like. However, the premise for deep learning to play a role is to need a large amount of labeled data. Compared with natural images, the acquisition and labeling of medical images are more difficult, and therefore how to effectively learn the deep network under the condition of small samples is a technical problem for the application of deep learning in the medical field.

[0003] Transfer learning is a method commonly used in the industry to solve the problem of small sample learning. This method replaces the randomly initialized network model parameters with the pre-trained model parameters of imagenet, so that the network can better converge and learn. However, the transferred model parameters are trained using natural images, and the learned features are more suitable for natural images, and the training effect on medical image models is small.

[0004] It should be noted that the information disclosed in the background section of the present application is only intended to deepen the understanding of the general background of the present application, and should not be regarded as acknowledging or implying in any form that the information constitutes prior art known to those skilled in the art. SUMMARY

[0005] The purpose of the present application is to provide a medical image classification model training method, a medical image segmentation method, an electronic device and a storage medium, which can solve the problem that the training effect of the medical image classification model cannot be improved by transferring the pre-trained model parameters of imagenet in the prior art.

[0006] To achieve the above-mentioned purpose, the present application provides a medical image classification model training method, which comprises:

[0007] Obtaining a plurality of first training samples, each of the first training samples comprising a first random transformation image and a second random transformation image obtained by randomly transforming a first medical training image in different ways;

[0008] inputting the first randomly transformed image in the first training sample as input of a first neural network in a self-supervised pre-training model and inputting the second randomly transformed image in the first training sample as input of a second neural network in the self-supervised pre-training model to train the self-supervised pre-training model until a first preset training end condition is met;

[0009] migrating model parameters of the trained self-supervised pre-training model to a medical image classification model and training the medical image classification model by using a plurality of second training samples until a second preset training end condition is met, each of the second training samples comprising a second medical training image and a class label corresponding to the second medical training image.

[0010] Optionally, the first training samples are obtained by the following steps:

[0011] detecting the obtained first initial medical training image to obtain position information of a target tissue region of interest;

[0012] cropping a corresponding region on the first initial medical training image according to the position information of the target tissue region of interest to obtain a first medical training image;

[0013] performing first random transformation and second random transformation on the first medical training image respectively to obtain a first randomly transformed image and a second randomly transformed image.

[0014] Optionally, the first random transformation and the second random transformation comprise any one or more of random cropping, random Gaussian blurring and random change of image properties.

[0015] Optionally, the first neural network and the second neural network each comprise a feature extraction subnetwork and a mapping subnetwork, the feature extraction subnetwork being configured to extract image features to obtain an image feature vector, and the mapping subnetwork being configured to reduce dimension of the image feature vector extracted by the corresponding feature extraction subnetwork to obtain a reduced dimension mapping image feature vector.

[0016] Optionally, the first preset training end condition is that:

[0017] a distance between the mapping image feature vector output by the first neural network and the mapping image feature vector output by the second neural network is less than or equal to a first preset threshold.

[0018] Optionally, the distance between the mapping image feature vector output by the first neural network and the mapping image feature vector output by the second neural network is calculated by the following formula:

[0019]

[0020] In the formula, y0 represents a mapping image feature vector output by the first neural network, y t represents a mapping image feature vector output by the second neural network.

[0021] Optionally, the method further comprises: taking the first randomly transformed image in the first training sample as input of the first neural network in the self-supervised pre-training model, taking the second randomly transformed image in the first training sample as input of the second neural network in the self-supervised pre-training model, and training the self-supervised pre-training model until a first preset training end condition is met, including:

[0022] taking the first randomly transformed image in the first training sample as input of the first neural network, taking the second randomly transformed image in the first training sample as input of the second neural network, updating model parameters of the first neural network by using a stochastic gradient descent method, and updating model parameters of the second neural network by using a sliding average method according to the updated model parameters of the first neural network until the first preset training end condition is met, wherein an initial value of the model parameters of the second neural network is the same as an initial value of the model parameters of the first neural network.

[0023] Optionally, the method further comprises:

[0024] updating the model parameters of the second neural network according to the following formula:

[0025] α←mα+(1-m)θ

[0026] In the formula, α represents the model parameters of the second neural network, m is a constant close to 1, and θ represents the model parameters of the first neural network.

[0027] Optionally, the method further comprises:

[0028] migrating the model parameters of the trained self-supervised pre-training model to the medical image classification model.

[0029] Optionally, the method further comprises:

[0030] taking the model parameters of the trained self-supervised pre-training model as initial values of model parameters of the medical image classification model.

[0031] Based on the initial values ​​of the model parameters of the medical image classification model and the second training sample, the medical image classification model is trained using the stochastic gradient descent method until the second preset training termination condition is met.

[0032] Optionally, the second training sample is obtained through the following steps:

[0033] The acquired second initial medical training image is detected to obtain the location information of the region of interest of the target tissue;

[0034] Based on the location information of the region of interest of the target tissue, the corresponding region is cropped out on the second initial medical training image to obtain the second medical training image.

[0035] To achieve the above objectives, the present invention also provides a medical image classification method, the classification method comprising:

[0036] Acquire medical images to be classified;

[0037] The medical image to be classified is classified using a trained medical image classification model to obtain the category corresponding to the medical image to be classified. The medical image classification model is trained using the training method described above.

[0038] Optionally, classifying the medical image to be classified using a trained medical image classification model includes:

[0039] The acquired medical images to be classified are detected to obtain the location information of the region of interest of the target tissue;

[0040] Based on the location information of the region of interest of the target tissue, the corresponding region is cropped out on the medical image to be classified to obtain the image of the region of interest of the target tissue.

[0041] A medical image segmentation model is used to classify the region of interest image of the target tissue.

[0042] Optionally, the step of detecting the acquired medical image to be classified to obtain the location information of the region of interest of the target tissue includes:

[0043] A trained YOLO detector is used to detect the acquired medical images to be classified in order to obtain the location information of the region of interest of the target tissue.

[0044] To achieve the above objectives, the present invention also provides an electronic device, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, it implements the medical image classification model training method or medical image classification method described above.

[0045] To achieve the above object, the application further provides a readable storage medium, wherein the readable storage medium stores a computer program, and the computer program is executed by a processor to implement the medical image classification model training method or the medical image classification method.

[0046] Compared with the prior art, the medical image classification model training method, the medical image segmentation method, the electronic device and the storage medium provided by the application have the following advantages:

[0047] The medical image classification model training method provided by the application comprises the following steps: first, a plurality of first training samples are obtained, each of the first training samples comprises a first random transformation image and a second random transformation image obtained by performing different random transformations on a first medical training image; second, the first random transformation image in the first training sample is taken as an input of a first neural network in a self-supervised pre-training model, and the second random transformation image in the first training sample is taken as an input of a second neural network in the self-supervised pre-training model, so as to train the self-supervised pre-training model until a first preset training end condition is met; and finally, the model parameters of the trained self-supervised pre-training model are migrated to a medical image classification model, and a plurality of second training samples are used to train the medical image classification model until a second preset training end condition is met, wherein the second training samples comprise a second medical training image and a category label corresponding to the second medical training image. Thus, based on the self-supervised learning method, the self-supervised pre-training model capable of directly learning the features of the medical image itself is trained, and the model parameters of the trained self-supervised pre-training model are migrated to the pre-created medical image classification model, so that the medical image classification model can also be well learned and converged under the condition of a small sample, and the training effect of the medical image classification model is effectively improved.

[0048] The medical image classification method provided by the application can accurately determine the category to which the to-be-classified medical image belongs (for example, whether the to-be-classified medical image is an image of a benign ovarian mass or an image of a malignant ovarian mass) by using the medical image classification model trained by the training method described above, so that the process of disease screening by doctors can be simplified, and the time required for diagnosis can be reduced.

[0049] The electronic device and the storage medium provided by the application belong to the same inventive concept as the medical image classification model training method or the medical image classification method described above, and therefore have all the advantages of the medical image classification model training method or the medical image classification method described above, which will not be described again. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 This is a flowchart illustrating a medical image classification model training method according to one embodiment of the present invention.

[0051] Figure 2 This is a schematic diagram of the specific process of obtaining the first training sample in one embodiment of the present invention;

[0052] Figure 3a This is a schematic diagram of a first initial medical training image in a specific example of the present invention;

[0053] Figure 3b For the sake of Figure 3a The diagram shows the bounding box of the region of interest of the target tissue obtained by detecting the first initial medical training image.

[0054] Figure 3c According to Figure 3b A schematic diagram of the first medical training image obtained by cropping the bounding box of the region of interest of the target tissue;

[0055] Figure 3d To Figure 3c A schematic diagram of the first randomly transformed image obtained by performing a first random transformation on the first medical training image shown;

[0056] Figure 3e To Figure 3c A schematic diagram of the second randomized image obtained by performing a second randomized transformation on the first medical training image shown;

[0057] Figure 4 This is a schematic diagram of the structure of a basic module in one embodiment of the present invention;

[0058] Figure 5 This is a schematic diagram of the residual submodule in one embodiment of the present invention;

[0059] Figure 6 This is a schematic diagram of the structure of the mapping subnetwork in one embodiment of the present invention;

[0060] Figure 7 This is a flowchart illustrating a medical image classification method according to one embodiment of the present invention;

[0061] Figure 8 This is a block diagram of an electronic device according to one embodiment of the present invention.

[0062] The reference numerals in the attached figures are as follows:

[0063] Processor-31; Communication interface-32; Memory-33; Communication bus-34. Detailed Implementation

[0064] The medical image classification model training method, the medical image classification method, the electronic device and the storage medium are further described in detail below in combination with the drawings and specific embodiments. The advantages and features of the present application will be clearer according to the following description. It should be noted that the drawings are greatly simplified and all use non-precise proportions, only to facilitate, clear and assist the purpose of describing the embodiments of the present application. In order to make the purpose, features and advantages of the present application more obvious and easy to understand, please refer to the accompanying drawings. It should be noted that the structure, proportion, size and the like shown in the drawings of the present application are only used to cooperate with the content disclosed in the specification, so that those skilled in the art can understand and read, and are not used to limit the conditions of the implementation of the present application. Any modification of structure, change of proportion relationship or adjustment of size, as long as it is the same or similar to the effect and purpose that can be achieved by the present application, should still fall within the scope of the technical content disclosed by the present application.

[0065] It should be noted that in this paper, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0066] In addition, in the description of the specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in combination with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of different embodiments or examples without contradiction.

[0067] The core idea of the present application is to provide a medical image classification model training method, a medical image segmentation method, an electronic device and a storage medium, which can solve the problem that the training effect of a medical image classification model cannot be improved by migrating pre-trained model parameters of imagenet in the prior art.

[0068] It should be noted that the medical image classification model training method of the present application can be applied to the medical image classification method of the present application, which can be configured on an electronic device, wherein the electronic device can be a personal computer, a mobile terminal, etc., and the mobile terminal can be a mobile phone, a tablet computer, etc. hardware devices with various operating systems. In addition, it should be noted that although the present application is described by training a medical image classification model for classifying benign and malignant ovarian mass images and classifying benign and malignant ovarian mass images using the trained medical image classification model, as can be understood by those skilled in the art, the present application can also train a medical image classification model for classifying other target tissue images, and classify other target tissue images using the trained medical image classification model, which is not limited by the present application.

[0069] To achieve the above idea, the present application provides a medical image classification model training method, please refer to Figure 1 which schematically shows the flowchart of the medical image classification model training method provided by an embodiment of the present application. As Figure 1 shown, the medical image classification model training method comprises the following steps:

[0070] Step S110, obtaining a plurality of first training samples, each of the first training samples comprising a first random transformation image and a second random transformation image obtained by performing different random transformations on a first medical training image.

[0071] Step S120, taking the first random transformation image in the first training sample as the input of the first neural network in the self-supervised pre-training model, and taking the second random transformation image in the first training sample as the input of the second neural network in the self-supervised pre-training model, to train the self-supervised pre-training model until the first preset training end condition is met.

[0072] Step S130, migrating the model parameters of the trained self-supervised pre-training model to a medical image classification model, and training the medical image classification model using a plurality of second training samples until a second preset training end condition is met, the second training sample comprising a second medical training image and a class label corresponding to the second medical training image.

[0073] Therefore, based on the self-supervised learning method, the self-supervised pre-training model capable of directly learning the features of the medical images is trained, and the model parameters of the trained self-supervised pre-training model are migrated to the pre-created medical image classification model, so that the medical image classification model can also be well learned and converged in the case of a small sample, and the training effect of the medical image classification model is effectively improved. It should be noted that, as can be understood by those skilled in the art, the model parameters of the neural network include two types: feature parameters and hyperparameters. The feature parameters are parameters used to learn image features. The feature parameters include weight parameters and bias parameters. The hyperparameters are parameters set by humans during training, and only by setting appropriate hyperparameters can the feature parameters be learned from the samples. The hyperparameters can include learning rate, number of hidden layers, convolution kernel size, number of training iterations, batch size for each iteration, and learning rate can be regarded as a step size. The present application mainly migrates the feature parameters in the trained self-supervised pre-training model to the medical image classification model.

[0074] It should be noted that, as can be understood by those skilled in the art, the image features extracted by the deep learning model are obtained by matrix multiplication calculation of the input of the model and the model parameters. Therefore, in the transfer learning, if the migration dataset (corresponding to the second training sample in this paper) is similar to the original dataset (corresponding to the first training sample in this paper), the edge, texture and other features (similar input, same model parameters) obtained by training (corresponding to the training of the self-supervised pre-training model in this paper) can be reused when migrating the model parameters; if they are not similar (different input, same model parameters), for example, from natural images to medical images, the features cannot be reused, and therefore the training effect of the medical image classification model is less improved. In the present application, since the original dataset and the migration dataset are similar (both are medical images, for example, both are ultrasound images, more specifically, both are ovarian mass ultrasound images), the migration method in the present application can reuse the edge, texture and other features, thereby improving the training effect of the medical image classification model. In addition, compared with general migration methods, the training samples (first training samples) used for pre-training (i.e., training the self-supervised pre-training model) in the present application do not require labels, so a large amount of unlabeled data can be used for self-supervised pre-training learning, while general transfer learning is supervised and requires labels, so the present application can effectively reduce the acquisition cost of the training samples required for pre-training.

[0075] In addition, it should be noted that, as can be understood by those skilled in the art, the number of the first training samples and the second training samples can be set according to specific conditions, and the number of the first training samples and the number of the second training samples can be the same or different, and the present application does not limit this.

[0076] Please continue to refer to Figure 2 , which schematically shows a specific flowchart of obtaining the first training sample provided by an embodiment of the present application. As shown in Figure 2 , in an exemplary embodiment, the first training sample is obtained by the following steps:

[0077] detecting the obtained first initial medical training image to obtain position information of a target tissue region of interest;

[0078] cropping a corresponding region on the first initial medical training image according to the position information of the target tissue region of interest to obtain a first medical training image;

[0079] respectively performing first random transformation and second random transformation on the first medical training image to obtain a first random transformation image and a second random transformation image.

[0080] Specifically, a pre-trained YOLO detector can be used to detect the obtained first initial medical training image to detect a region where the target tissue (such as an ovarian mass) may be located (i.e., a target tissue region of interest), and the position information of the target tissue region of interest (represented by the center point coordinates of the bounding box and the width and height of the bounding box) can be obtained according to the position information of the bounding box (such as a rectangular bounding box) of the target tissue region of interest; the first initial medical training image can be cropped according to the obtained position information of the target tissue region of interest, so that a corresponding target tissue region of interest can be cropped on the first initial medical training image, thereby obtaining a first medical training image; finally, by respectively performing first random transformation and second random transformation on the cropped first medical training image, a corresponding first random transformation image and a second random transformation image can be obtained, thereby obtaining a first training sample. Please refer to Figure 3a and Figure 3b , wherein Figure 3a is a schematic diagram of a first initial medical training image in a specific example of the present application; Figure 3b is a schematic diagram of a target tissue region of interest obtained by detecting the first initial medical training image shown in Figure 3a ; Figure 3c is a schematic diagram of a first medical training image obtained by cropping according to the bounding box of the target tissue region of interest shown in Figure 3b ; Figure 3d is a schematic diagram of a first random transformation image obtained by performing first random transformation on the first medical training image shown in Figure 3c ; Figure 3e is a schematic diagram of a second random transformation image obtained by performing second random transformation on the first medical training image shown in Figure 3c .

[0081] It should be noted that, as can be understood by those skilled in the art, the number of the acquired first initial medical training images is determined by the number of the required first training samples, and the corresponding number of the first training samples can be acquired by performing the above-mentioned operations on the first initial medical training images in the preset number.

[0082] In an exemplary embodiment, the first random transformation and the second random transformation include any one or more of random cropping, random Gaussian blur, and random change of image properties.

[0083] Specifically, when random cropping is performed, the size of the cropped image (the first random transformation image and the second random transformation image) is 0.2-1 times the size of the original image (the first medical training image), that is, the cropping ratio of the random cropping is 0.2-1. It should be noted that, as can be understood by those skilled in the art, the cropping ratios corresponding to the first random transformation image and the second random transformation image can be the same or different. When the cropping ratios corresponding to the first random transformation image and the second random transformation image are the same, different random Gaussian blurs and / or random changes of image properties can be performed on the two images of the same size cropped to obtain the first random transformation image and the second random transformation image obtained by different random transformations. When the cropping ratios corresponding to the first random transformation image and the second random transformation image are different, the sizes of the two images cropped can be first adjusted to the same size by interpolation, and then different random Gaussian blurs and / or random changes of image properties can be performed on the two images of the same size to obtain the first random transformation image and the second random transformation image obtained by different random transformations. Of course, as can be understood by those skilled in the art, after the sizes of the two images are adjusted to the same size, no other random changes are performed, and the two images are directly used as the first random transformation image and the second random transformation image, which is not limited by the present application. In addition, it should be noted that, as can be understood by those skilled in the art, the random change of image properties includes random change of brightness, contrast, saturation, and hue of the image, etc.

[0084] Since the neural network needs images of a uniform size as input, before training the self-supervised pre-training model, the method further includes adjusting the first random transformation image and the second random transformation image in all the first training samples to a first preset size to meet the input requirements of the first neural network and the second neural network.

[0085] In an exemplary embodiment, the first neural network and the second neural network have the same structure, both of which include a feature extraction subnetwork and a mapping subnetwork, the feature extraction subnetwork is configured to extract image features to obtain an image feature vector, and the mapping subnetwork is configured to reduce the dimension of the image feature vector extracted by the corresponding feature extraction subnetwork to obtain a reduced dimension mapping image feature vector. Specifically, assuming that the image feature vector is a 1 x k matrix (k is the total dimension of the features possessed by the image feature vector), and the mapping image feature vector is a 1 x m matrix (m is the total dimension of the features possessed by the image feature vector), then k > m. For example, assuming that the image feature vector is a 1 x 512 matrix, then the mapping image feature vector obtained after the dimension reduction operation can be a 1 x 128 matrix.

[0086] Specifically, the feature extraction subnetwork includes a plurality of cascaded basic modules. Please refer to Figure 4 , which schematically shows a structural schematic diagram of a basic module provided by an embodiment of the present application. As Figure 4 shown, the basic module includes a residual sub-module, an identity sub-module and a compression excitation sub-module, wherein the residual sub-module is configured to perform convolution processing on the input of the basic module; the compression excitation sub-module includes a compression layer and an excitation layer, the compression layer is configured to compress the input of the basic module to compress each two-dimensional feature channel into a real number, which has a global receptive field. The dimension of the compression layer output matches the number of feature channels of the input, which represents the corresponding global distribution on the feature channel, and enables the layers close to the input to obtain the global receptive field, and the excitation layer is configured to generate a weight for each feature channel, and the feature is recalibrated by multiplying the weight output by the compression excitation sub-module with the feature of each feature channel output by the residual module; the identity sub-module is configured to perform identity mapping on the input of the basic module, and the result of the identity mapping is added to the result of the feature recalibration to obtain the output of the basic module. Thus, the importance of each feature channel can be automatically obtained through the compression excitation sub-module, so that different weights can be set for different feature channels according to the importance of each feature channel, so that the features useful for medical image classification (including the edge features and texture features of the target organ tissue (such as ovarian mass, breast nodule, lung nodule, etc.)) can be enhanced and the features useless for medical image classification (including the edge features and texture features of the target organ tissue (such as ovarian mass, breast nodule, lung nodule, etc.)) can be suppressed. In addition, the feature extraction subnetwork in the present application can solve the degradation problem of deep network when the network is deepened.

[0087] In an exemplary embodiment, as Figure 4As shown, the compression excitation submodule specifically comprises a global average pooling layer, a first full connection layer, a second full connection layer and a first activation layer, wherein the global average pooling layer, the first full connection layer and the second full connection layer constitute the compression layer described above, the global average pooling layer is used for global pooling on the input (C HxW feature map) of the basic module to obtain a 1x1xC vector, the 1x1xC vector passes through the first full connection layer to obtain a 1x1xC / r vector, wherein r is a preset coefficient, the value of r can be set by a user according to experience, then the 1x1xC / r vector passes through the second full connection layer to obtain a 1x1xC vector. The obtained 1x1xC vector is processed by an activation function in the activation layer, and still a 1x1xC vector is obtained, wherein the activation function can be a Sigmoid function.

[0088] Please continue to refer to Figure 5 which schematically shows a structural schematic diagram of the residual submodule provided by an embodiment of the application. As shown in Figure 5 As shown, the residual submodule is a double-channel structure, specifically, the residual submodule comprises a first channel unit and a second channel unit, the first channel unit comprises a plurality of cascaded first convolution layers, the second channel unit comprises a plurality of cascaded second convolution layers, the second convolution layer corresponds to the first convolution layer one by one, wherein the output of the first convolution layer and the second convolution layer of the previous stage is added as the input of the first convolution layer and the second convolution layer of the next stage, and the input of the residual submodule is added to the output of the residual submodule as the output of the residual submodule. Since the residual submodule comprises a residual submodule, the residual submodule can extract image features (such as ovarian mass features) from different scales, further improving the accuracy of positioning of the application. In addition, since the first channel unit comprises a plurality of cascaded first convolution layers, and the second channel unit comprises a plurality of cascaded second convolution layers, deeper image feature information (such as ovarian mass feature information) can be extracted from different scales, which strengthens the extraction of local and global features (such as ovarian mass features) of the image, and is beneficial to improve the accuracy of classification.

[0089] As an example, as Figure 5As shown, the first channel unit includes 3 first convolutional layers in cascade, i.e., a first convolutional layer a, a first convolutional layer b and a first convolutional layer c, and the second channel unit includes 3 second convolutional layers in cascade, i.e., a second convolutional layer d, a second convolutional layer e and a second convolutional layer f. The size of the convolution kernel of the first convolutional layer is 3x3, and the activation function adopted by the first convolutional layer is Leaky ReLU. The size of the convolution kernel of the second convolutional layer is 5x5, and the activation function adopted by the second convolutional layer is also Leaky ReLU. Specifically, the outputs of the first convolutional layer a and the second convolutional layer d are added to serve as the inputs of the first convolutional layer b and the second convolutional layer e, the outputs of the first convolutional layer b and the second convolutional layer e are added to serve as the inputs of the first convolutional layer c and the second convolutional layer f, and the outputs of the first convolutional layer c and the second convolutional layer f are added to the input of the residual module to serve as the output of the residual module.

[0090] It should be noted that, as can be understood by those skilled in the art, Figure 5 In the residual module shown, the number of first convolutional layers and the number of second convolutional layers are both examples and should not be construed as limiting the embodiments of the present application. The number of first convolutional layers and the number of second convolutional layers can be set according to specific needs. In addition, it should be noted that, because the first convolutional layers and the second convolutional layers have a one-to-one correspondence, in the residual module provided by the embodiments of the present application, the number of first convolutional layers included in the first channel unit is equal to the number of second convolutional layers included in the second channel unit.

[0091] Please continue to refer to Figure 6 which schematically shows a structure diagram of the mapping sub-network provided by an embodiment of the present application. As shown in Figure 6 As shown, the mapping sub-network includes a third fully connected layer, a batch normalization layer, a second activation layer and a fourth fully connected layer, wherein the second activation layer adopts a ReLU function.

[0092] In an exemplary embodiment, the first preset training end condition is that the distance between the mapping image feature vector output by the first neural network and the mapping image feature vector output by the second neural network is less than or equal to a first preset threshold.

[0093] Since the first randomly transformed image and the second randomly transformed image in each first training sample are both obtained by performing different random transformations on a first medical training image, for the same first training sample, the mapping image feature vector output by the first neural network and the mapping image feature vector output by the second neural network should be relatively close. Therefore, the training purpose of the self-supervised pre-training model in the present application is to make the mapping image feature vector output by the first neural network and the feature vector obtained by the second neural network relatively close. Thus, the first preset training end condition can be set as that the distance between the mapping image feature vector output by the first neural network and the mapping image feature vector output by the second neural network is less than or equal to a first preset threshold.

[0094] Specifically, the distance between the mapping image feature vector output by the first neural network and the mapping image feature vector output by the second neural network is calculated by the following formula:

[0095]

[0096] In the formula, y0 represents the mapping image feature vector output by the first neural network, and y1 represents the mapping image feature vector output by the second neural network. t In the formula, y0 represents the mapping image feature vector output by the first neural network, and y1 represents the mapping image feature vector output by the second neural network.

[0097] It should be noted that, as understood by those skilled in the art, the smaller the distance between the mapping image feature vector output by the first neural network and the mapping image feature vector output by the second neural network, the higher the similarity between the mapping image feature vector output by the first neural network and the mapping image feature vector output by the second neural network.

[0098] In an exemplary embodiment, the first randomly transformed image in the first training sample is taken as the input of the first neural network in the pre-created self-supervised pre-training model, the second randomly transformed image in the first training sample is taken as the input of the second neural network in the self-supervised pre-training model, and the self-supervised pre-training model is trained until the first preset training end condition is met, comprising:

[0099] inputting the first randomly transformed image in the first training sample as input of the first neural network and inputting the second randomly transformed image in the first training sample as input of the second neural network; updating the model parameters of the first neural network by using a stochastic gradient descent method, and updating the model parameters of the second neural network by using a sliding average method according to the updated model parameters of the first neural network until a first preset training end condition is met, wherein the initial value of the model parameters of the second neural network is the same as the initial value of the model parameters of the first neural network.

[0100] In the present application, the gradient descent method is mainly used to update the weight parameters and bias parameters of the first neural network. In each round of training, the model parameters of the first neural network are first updated by the gradient descent method, and then the model parameters of the second neural network are updated by the sliding average method according to the updated model parameters of the first neural network.

[0101] In an exemplary embodiment, the updating the model parameters of the second neural network by using the sliding average method according to the updated model parameters of the first neural network comprises:

[0102] The model parameters of the second neural network are updated according to the following formula:

[0103] α←mα+(1-m)θ (2)

[0104] In the formula, α is the model parameters (including weight parameters and bias parameters) of the second neural network, m is a constant close to 1, and θ is the model parameters (including weight parameters and bias parameters) of the first neural network.

[0105] It should be noted that, as understood by those skilled in the art, the left α in the above formula (2) represents the updated model parameters of the second neural network, and the right α represents the model parameters of the second neural network before updating. In addition, it should be noted that the specific value of m can be set according to specific conditions, as long as m is close to 1, for example, m can be 0.999.

[0106] In an exemplary embodiment, the migrating the trained model parameters of the self-supervised pre-training model to the pre-created medical image classification model comprises:

[0107] Migrating the model parameters of the first neural network in the trained self-supervised pre-training model to the pre-created medical image classification model.

[0108] Since the model parameters of the first neural network are updated by employing the stochastic gradient descent method, by migrating the model parameters of the first neural network in the trained self-supervised pre-training model to the pre-created medical image classification model, it can be facilitated that the model parameters of the medical image classification model can also be updated by employing the stochastic gradient descent method subsequently.

[0109] In an exemplary embodiment, the training of the medical image classification model by employing the second training sample comprises:

[0110] The model parameters of the trained self-supervised training model (the model parameters of the first neural network) are taken as the initial values of the model parameters of the medical image classification model.

[0111] The medical image classification model is trained by employing the stochastic gradient descent method according to the initial values of the model parameters of the medical image classification model and the second training sample until the second preset training end condition is met.

[0112] Since the process of model training is actually a process of minimizing the loss function, and the derivative can quickly and simply achieve this goal, this method of derivative is the gradient descent method. Therefore, by training the medical image classification model by employing the stochastic gradient descent method, the training of the medical image classification model can be quickly and simply achieved. Specifically, the medical image classification model comprises a feature extraction part and a classification part, wherein the structure of the feature extraction part of the image classification model is the same as that of the feature extraction sub-network of the first neural network and the second neural network, and the classification part of the medical image classification model comprises at least one fully connected layer, which is used for nonlinear mapping regression of the image feature vector extracted by the feature extraction part to obtain the classification result of the medical image.

[0113] Since the training purpose of the medical image classification model is to make the predicted classification result of the medical image obtained by the model similar to the true classification result of the medical image, that is, the error of the two is reduced to a certain range, therefore, the second preset training end condition can be that the error value of the predicted classification result and the true classification result of the second medical training image converges to a preset error value. Specifically, a binary cross-entropy function can be employed as the loss function employed in the training process of the medical image classification model to measure the difference between the predicted classification result and the true classification result. Specifically, the binary cross-entropy function used in the present application is as follows:

[0114] L ce =-ylogy'-(1-y)log(1-y')

[0115] In the formula, y is a real class label, and y' is a predicted positive example probability value.

[0116] It should be noted that, as can be understood by those skilled in the art, when the present application is used to train an image classification model for classifying benign and malignant images, the benign can be set as a negative example, and the class label is 0, and the malignant is set as a positive example, and the class label is 1.

[0117] In addition, it should be noted that, as can be understood by those skilled in the art, the training process of the medical image classification model is a multiple loop iteration process, and therefore the number of iterations can be set to end the training, i.e., the second preset training end condition can also be that the number of iterations reaches a preset number of iterations.

[0118] In an exemplary embodiment, the second training sample is obtained by the following steps:

[0119] Detecting the obtained second initial medical training image to obtain position information of a target tissue region of interest;

[0120] According to the position information of the target tissue region of interest, a corresponding region is cropped on the second initial medical training image to obtain a second medical training image.

[0121] Specifically, a pre-trained YOLO detector can be used to detect the obtained second initial medical training image to detect a region where the target tissue (such as an ovarian mass) can be located (i.e., a target tissue region of interest), and the position information of the target tissue region of interest (represented by the center point coordinates of the bounding box and the width and height of the bounding box) can be obtained according to the position information of the bounding box (such as a rectangular bounding box) of the target tissue region of interest. According to the obtained position information of the target tissue region of interest, the second initial medical training image is cropped, and a corresponding target tissue region of interest is cropped on the second initial medical training image, thereby obtaining a second medical training image. It should be noted that, as can be understood by those skilled in the art, the second initial medical training image and the first medical training image can be the same medical image or different medical images, and the present application does not limit this.

[0122] Further, a first proportion of the second training samples can be determined as a training set and a second proportion of the second training samples can be determined as a validation set from the acquired preset number of second training samples, wherein the first proportion and the second proportion are 1, for example, 75% of the second training samples can be selected as the training set and 25% of the second training samples can be selected as the validation set, so that the medical image classification model can be trained by using the training set, and the output accuracy of the medical image classification model can be tested by using the validation set after each training round is completed, if it is found that the output accuracy of the medical image classification model no longer increases (i.e., the loss function value no longer decreases) or reaches the maximum iteration round or the loss function value is less than or equal to a second preset threshold after multiple rounds, the training is ended.

[0123] Based on the same inventive concept, the present application also provides a medical image classification method, please refer to Figure 7 which schematically shows a flowchart of a medical image classification method according to an embodiment of the present application. As shown in Figure 7 the medical image classification method comprises the following steps:

[0124] Step S210, acquiring a medical image to be classified.

[0125] Step S220, classifying the medical image to be classified by using a trained medical image classification model to obtain a category corresponding to the medical image to be classified.

[0126] The medical image classification model is trained by using the training method described above. Thus, the medical image classification method provided by the present application can accurately determine the category of the medical image to be classified (e.g., determine whether the medical image to be classified is a benign ovarian mass image or a malignant ovarian mass image) by using the medical image classification model trained by using the training method described above to classify the acquired medical image to be classified (e.g., an ovarian ultrasound image), thereby simplifying the process of disease screening by doctors and reducing the time required for diagnosis.

[0127] In an exemplary embodiment, the step of classifying the medical image to be classified by using the trained medical image classification model comprises:

[0128] detecting the acquired medical image to be classified to obtain position information of a target tissue region of interest;

[0129] cropping a corresponding region on the medical image to be classified according to the position information of the target tissue region of interest to obtain a target tissue region of interest image;

[0130] The target tissue region of interest image is classified by using a medical image segmentation model.

[0131] In an exemplary embodiment, the acquired medical image to be classified is detected to obtain the position information of the target tissue region of interest, including:

[0132] The acquired medical image to be classified is detected by using the trained YOLO detector to obtain the position information of the target tissue region of interest.

[0133] Based on the same inventive concept, the present application also provides an electronic device, please refer to Figure 8 which schematically shows the block structure of the electronic device provided by an embodiment of the present application. As shown in the figure, Figure 8 the electronic device includes a processor 101 and a memory 103, the memory 103 stores a computer program, and the computer program is executed by the processor 101 to realize the medical image classification model training method or the medical image classification method described above. Since the electronic device provided by the present application belongs to the same inventive concept as the medical image classification model training method or the medical image classification method provided by the present application, the electronic device provided by the present application has all the advantages of the medical image classification model training method or the medical image classification method described above, so it will not be described here.

[0134] As shown in the figure, Figure 8 the electronic device further includes a communication interface 102 and a communication bus 104, wherein the processor 101, the communication interface 102 and the memory 103 complete the communication among each other through the communication bus 104. The communication bus 104 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus 104 can be divided into address bus, data bus, control bus, etc. For the convenience of representation, only one thick line is shown in the figure, but it does not mean that there is only one bus or only one type of bus. The communication interface 102 is used for communication between the electronic device and other devices.

[0135] The processor 101 in the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The processor 101 is the control center of the electronic device, and is connected to various parts of the electronic device through various interfaces and lines.

[0136] The memory 103 can be used to store the computer program, and the processor 101 realizes various functions of the electronic device by running or executing the computer program stored in the memory 103 and calling the data stored in the memory 103.

[0137] The memory 103 can include non-volatile and / or volatile memory. The non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. The volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0138] The present application also provides a readable storage medium, wherein the readable storage medium stores a computer program, and the computer program is executed by a processor to realize the medical image classification model training method described above. Since the readable storage medium provided by the present application and the medical image classification model training method or the medical image classification method provided by the present application belong to the same inventive concept, the readable storage medium provided by the present application has all the advantages of the medical image classification model training method or the medical image classification method described above, and therefore will not be described here.

[0139] The readable storage medium of the embodiments of the present application can adopt any combination of one or more computer-readable media. The readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer hard disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this paper, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or apparatus.

[0140] The computer-readable signal medium can include a data signal propagating in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such a propagating data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit a program for use by or in combination with an instruction execution system, device or apparatus.

[0141] Computer program code for carrying out operations of the present application can be written in one or more programming languages or combinations of languages including object-oriented, such as Java, Smalltalk, C++, and conventional procedural, such as the "C" language or similar programming languages. Program code can execute entirely on a user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0142] In summary, compared with the prior art, the medical image classification model training method, medical image segmentation method, electronic device and storage medium provided by the present application have the following advantages:

[0143] The medical image classification model training method provided by the application comprises the following steps: first, a first training sample is obtained, wherein the first training sample comprises a first random transformation image and a second random transformation image obtained by performing different random transformations on a same first medical training image; second, a first neural network in a pre-created self-supervised pre-training model is taken as an input of the first random transformation image in the first training sample, and a second neural network in the self-supervised pre-training model is taken as an input of the second random transformation image in the first training sample, so as to train the self-supervised pre-training model until a first preset training end condition is met; finally, model parameters of the trained self-supervised pre-training model are migrated to a pre-created medical image classification model, and a second training sample is used to train the medical image classification model until a second preset training end condition is met, wherein the second training sample comprises a second medical training image and a category label corresponding to the second medical training image. Thus, the self-supervised pre-training model capable of directly learning the features of the medical image itself is trained based on the self-supervised learning method, and the model parameters of the trained self-supervised pre-training model are migrated to the pre-created medical image classification model, so that the medical image classification model can also be well learned and converged under the condition of a small sample, and the training effect of the medical image classification model is effectively improved.

[0144] The medical image classification method provided by the application can accurately determine the category to which the to-be-classified medical image belongs (for example, determine whether the to-be-classified medical image is an image of a benign ovarian mass or an image of a malignant ovarian mass), so that the process of disease screening by doctors can be simplified, and the time required for diagnosis can be reduced.

[0145] The electronic device and the storage medium provided by the application belong to the same inventive concept as the medical image classification model training method or the medical image classification method described above, and therefore have all the advantages of the medical image classification model training method or the medical image classification method described above, which will not be described again.

[0146] It should be noted that the apparatus and method disclosed in the embodiments herein can also be implemented in other ways. The apparatus embodiments described above are merely exemplary, and the flowcharts and block diagrams in the accompanying drawings show only one possible implementation of the apparatus, method and computer program product according to the embodiments herein. In this regard, each block in the flowcharts and block diagrams can represent a module, a procedure, or a part of a program, which comprises one or more executable instructions for implementing the specified logic function. It should also be noted that in some alternative implementations, the functions noted in the boxes can occur out of the order noted in the flowcharts. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or in the reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by dedicated hardware-based systems that perform the specified functions or acts, or can be implemented by a combination of dedicated hardware and computer instructions.

[0147] In addition, each functional module in the various embodiments herein can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0148] The above description is merely illustrative of the embodiments of the present application, and is not intended to limit the scope of the present application in any way. Any modifications, changes, and improvements made to the present application by any person skilled in the art, based on the above description, are intended to fall within the scope of the present application. It is apparent that those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and changes fall within the scope of the present application and equivalent technology thereof, they are intended to be included in the present application.

Claims

1. A method for training a medical image classification model, characterized in that, The method comprises: obtaining a plurality of first training samples, each of the first training samples comprising a first randomly transformed image and a second randomly transformed image obtained by performing different random transformations on a first medical training image; inputting the first randomly transformed image in the first training sample as input of a first neural network in a self-supervised pre-training model, inputting the second randomly transformed image in the first training sample as input of a second neural network in the self-supervised pre-training model, and training the self-supervised pre-training model until a first preset training end condition is met; migrating model parameters of the self-supervised pre-training model to a medical image classification model, and training the medical image classification model by using a plurality of second training samples until a second preset training end condition is met, each of the second training samples comprising a second medical training image and a category label corresponding to the second medical training image. 2.The method of claim 1, wherein, The first training samples are obtained by the following steps: detecting the obtained first initial medical training image to obtain position information of a target tissue region of interest; cropping a corresponding region on the first initial medical training image according to the position information of the target tissue region of interest to obtain a first medical training image; performing first random transformation and second random transformation on the first medical training image respectively to obtain a first randomly transformed image and a second randomly transformed image. 3.The method of claim 2, wherein, The first random transformation and the second random transformation comprise any one or more of random cropping, random Gaussian blurring, and random change of image attributes. 4.The method of claim 1, wherein, The first neural network and the second neural network each comprise a feature extraction subnetwork and a mapping subnetwork, the feature extraction subnetwork is configured to extract image features to obtain an image feature vector, and the mapping subnetwork is configured to reduce dimension of the image feature vector extracted by the corresponding feature extraction subnetwork to obtain a reduced dimension mapping image feature vector. 5.The method of claim 4, wherein, The first preset training end condition is that: a distance between a mapping image feature vector output by the first neural network and a mapping image feature vector output by the second neural network is less than or equal to a first preset threshold. 6.The method of claim 5, wherein, The distance between the mapping image feature vector output by the first neural network and the mapping image feature vector output by the second neural network is calculated by the following formula: In the formula, y0 represents a mapping image feature vector output by the first neural network, y t represents a mapping image feature vector output by the second neural network. 7.The method of claim 1, wherein, The inputting the first randomly transformed image in the first training sample as input of a first neural network in a self-supervised pre-training model, inputting the second randomly transformed image in the first training sample as input of a second neural network in the self-supervised pre-training model, and training the self-supervised pre-training model until a first preset training end condition is met, comprises: inputting the first randomly transformed image in the first training sample as input of the first neural network and inputting the second randomly transformed image in the first training sample as input of the second neural network; updating the model parameters of the first neural network by using a stochastic gradient descent method, and updating the model parameters of the second neural network by using a sliding average method according to the updated model parameters of the first neural network, until a first preset training end condition is met, wherein the initial value of the model parameters of the second neural network is the same as the initial value of the model parameters of the first neural network. 8.The method of claim 7, wherein, The updating of the model parameters of the second neural network according to the updated model parameters of the first neural network by using the sliding average method comprises: updating the model parameters of the second neural network according to the following formula: α←mα+(1-m)θ wherein, α is the model parameters of the second neural network, m is a constant close to 1, and θ is the model parameters of the first neural network. 9.The method of claim 7, wherein, The model parameters of the trained self-supervised pre-training model are migrated to a medical image classification model, comprising: The model parameters of the first neural network in the trained self-supervised pre-training model are migrated to the medical image classification model. 10.The method of claim 1, wherein, The training of the medical image classification model by using the second training samples comprises: The model parameters of the trained self-supervised pre-training model are used as the initial value of the model parameters of the medical image classification model. The medical image classification model is trained by using a stochastic gradient descent method according to the initial value of the model parameters of the medical image classification model and the second training samples, until a second preset training end condition is met. 11.The method of claim 1, wherein, The second training sample is obtained by the following steps: detecting the obtained second initial medical training image to obtain position information of a target tissue region of interest; cropping a corresponding region on the second initial medical training image according to the position information of the target tissue region of interest to obtain a second medical training image.

12. A medical image classification method characterized by, comprising: obtaining a medical image to be classified; classifying the medical image to be classified by using the trained medical image classification model to obtain a category corresponding to the medical image to be classified, wherein the medical image classification model is trained by using the training method in any one of claims 1 to 11.

13. The medical image classification method of claim 12, wherein, The classification of the medical image to be classified by using the trained medical image classification model comprises: detecting the obtained medical image to be classified to obtain position information of a target tissue region of interest; cropping a corresponding region on the medical image to be classified according to the position information of the target tissue region of interest to obtain a target tissue region of interest image; classifying the target tissue region of interest image by using a medical image segmentation model.

14. The medical image classification method of claim 13, wherein, The detection of the obtained medical image to be classified to obtain position information of a target tissue region of interest comprises: detecting the obtained medical image to be classified by using a trained YOLO detector to obtain position information of a target tissue region of interest.

15. An electronic device, comprising: A computer program product comprising a computer readable storage medium having computer program code embodied therewith, the computer program code configured such that, on execution by a computer, the computer is caused to perform the training method of any one of claims 1 to 11 or the classification method of any one of claims 12 to 14.

16. A readable storage medium, characterized by, A computer program product comprising a computer readable storage medium having computer program code embodied therewith, the computer program code configured such that, on execution by a computer, the computer is caused to perform the training method of any one of claims 1 to 11 or the classification method of any one of claims 12 to 14.

Images