A Breast Cancer Genotyping Prediction Method and Device Based on Deep Learning

Abstract

The present application provides a breast cancer gene typing prediction method and device based on deep learning. It includes: training an initial gene typing prediction model based on a sample medical image set and sample gene typing to obtain a trained gene typing prediction model; wherein, the initial gene typing prediction model includes an input layer, a branch network set, a fusion layer, and a fully connected block connected in sequence; the branch network set includes branch networks corresponding to each type of breast medical image; obtaining a target medical image set; using the trained gene typing prediction model to process the target medical image set to obtain a target gene typing. By setting multiple branch networks to extract features from different types of medical images, important features of each image type can be captured; by fusing information from multiple branches and comprehensively considering information in different modalities, it helps to improve the accuracy and generalization ability of model prediction.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, in particular to a breast cancer gene typing prediction method and device based on deep learning. Background Art

[0002] Breast cancer is one of the most common malignant tumors in women and is the number one killer of women's health. In 2000, Perou et al. first analyzed breast cancer gene typing, indicating the beginning of the research on the essence of tumors. It has been gradually found that there are obvious differences in disease expression, treatment, and survival outcomes due to different breast cancer susceptibility genes and carcinogenic sites. In 2003, StGallen et al. recommended gene analysis for hereditary breast cancer, and for patients with positive expression of breast cancer susceptibility genes (BRCA1 and BRCA2) with high risk factors, prophylactic contralateral mastectomy was selected to reduce the risk of breast cancer. Now, the medical community has recognized that the 21-gene test and 70-gene test for breast cancer are valuable for predicting the benefit of chemotherapy and have been approved for clinical application. However, under the current conditions, the high cost and difficult-to-achieve technical level limit the popularization and application of gene testing in clinics. The latest research has found that through the quantitative analysis of imaging features, the differences in tumor gene levels can be predicted to a certain extent, which indicates that the in-depth mining of medical images can discover the deep connection between images and genes and establish a direct bridge between breast cancer imaging examinations and precision medical treatments.

[0003] Currently, one of the hottest directions in the field of artificial intelligence is deep learning. The processing process of a deep learning network mimics the neural network of the human brain and is a process of continuous iteration and bottom-up abstraction. It can actively identify the natural world with powerful automatic feature extraction, complex model construction, and image processing capabilities, so it can automatically discover features without human preset prior knowledge. In 2012, Ciresan et al. in Switzerland applied deep learning to the automatic search for mitosis in breast cancer cell images, and the accuracy of this model far exceeded that of previous methods, winning the championship of the ICPR competition that year. However, the application of deep learning in radiomics is still in its infancy, and there are only 2 articles on Pubmed: Antropova N put 551 enhanced breast MR images into a pre-trained convolutional neural network, and the model extracted 4096 features and performed benign and malignant classification of breast masses. The results showed that the AUC value was 0.85, indicating that the model could better predict the benign and malignant of breast tumors. Huynh BQ et al. used "transfer learning" to solve the problem of small samples and achieved good results in the benign and malignant prediction of 219 breast lesions. However, these studies only stayed at judging the benign and malignant of lesions and did not further explore the association between radiomics and gene typing. Summary of the Invention

[0004] In view of the above-mentioned problems, the present application is proposed to provide a breast cancer gene typing prediction method and device based on deep learning that overcomes or at least partially solves the problems, including:

[0005] A breast cancer gene typing prediction method based on deep learning, including:

[0006] Obtain a sample medical image set and a sample gene typing of the sample cancerous breast; wherein, the sample medical image set includes at least three different types of breast medical images;

[0007] Train an initial gene typing prediction model based on the sample medical image set and the sample gene typing to obtain a trained gene typing prediction model; wherein, the initial gene typing prediction model includes an input layer, a branch network set, a fusion layer, and a fully connected block connected in sequence; the branch network set includes branch networks corresponding to each type of breast medical image;

[0008] Obtain a target medical image set of the target cancerous breast; wherein, the sample medical image set includes at least one type of breast medical image;

[0009] Process the target medical image set using the trained gene typing prediction model to obtain the target gene typing of the target cancerous breast.

[0010] Preferably, the step of training an initial gene typing prediction model based on the sample medical image set and the sample gene typing to obtain a trained gene typing prediction model includes:

[0011] Randomly mask the breast medical images in the sample medical image set to obtain a masked medical image set;

[0012] Train an initial gene typing prediction model based on the sample medical image set, the masked medical image set, and the sample gene typing to obtain a trained gene typing prediction model.

[0013] Preferably, the step of randomly masking the breast medical images in the sample medical image set to obtain a masked medical image set includes:

[0014] Generate a mean masked image corresponding to each type of breast medical image based on the sample medical image set;

[0015] Replace random regions in a random number of breast medical images in the sample medical image set with corresponding regions in the corresponding mean masked image to obtain a masked medical image set.

[0016] Preferably, the step of training the initial genotyping prediction model based on the sample medical image set, the masked medical image set, and the sample genotyping to obtain a trained genotyping prediction model includes:

[0017] Input the sample medical image set and the masked medical image set into the initial genotyping prediction model to obtain a predicted genotyping;

[0018] Calculate the loss value of the initial genotyping prediction model based on the predicted genotyping and the sample genotyping;

[0019] Adjust the parameters of the initial genotyping prediction model until the loss value is less than a preset value.

[0020] Preferably, the step of using the trained genotyping prediction model to process the target medical image set to obtain the target genotyping of the target cancerous breast includes:

[0021] When the number of breast medical images in the target medical image set is equal to the number of breast medical images in the sample medical image set, input the target medical image set into the trained genotyping prediction model to obtain the target genotyping of the target cancerous breast;

[0022] When the number of breast medical images in the target medical image set is less than the number of breast medical images in the sample medical image set, fill in the missing breast medical images in the target medical image set to obtain a filled medical image set;

[0023] Input the filled medical image set into the trained genotyping prediction model to obtain the target genotyping of the target cancerous breast.

[0024] Preferably, the step of filling in the missing breast medical images in the target medical image set to obtain a filled medical image set includes:

[0025] Generate a target filling image based on the existing breast medical images in the target medical image set;

[0026] Add the target filling image to the target medical image set to obtain a filled medical image set.

[0027] Preferably, the step of filling in the missing breast medical images in the target medical image set to obtain a filled medical image set includes:

[0028] Generate a target filling image based on the existing breast medical images in the sample medical image set and the target medical image set;

[0029] Add the target filled image to the target medical image set to obtain a filled medical image set.

[0030] A breast cancer gene typing prediction device based on deep learning, comprising:

[0031] A sample acquisition module for acquiring a sample medical image set and sample gene typing of a sample cancerous breast; wherein, the sample medical image set includes at least three different types of breast medical images;

[0032] A model training module for training an initial gene typing prediction model based on the sample medical image set and the sample gene typing to obtain a trained gene typing prediction model; wherein, the initial gene typing prediction model includes an input layer, a branch network set, a fusion layer, and a fully connected block connected in sequence; the branch network set includes branch networks corresponding to each type of breast medical image;

[0033] A target acquisition module for acquiring a target medical image set of a target cancerous breast; wherein, the sample medical image set includes at least one type of breast medical image;

[0034] A target prediction module for processing the target medical image set using the trained gene typing prediction model to obtain the target gene typing of the target cancerous breast.

[0035] A computer device, comprising a processor, a memory, and a computer program stored on the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the method described in any one of the above is implemented.

[0036] A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described in any one of the above is implemented.

[0037] This application has the following advantages:

[0038] In the embodiments of the present application, in view of the problem that the prior art cannot predict breast cancer gene typing through imaging examination results, the present application provides a solution for processing a target medical image set by using a gene typing prediction model with a multi-branch network to obtain its gene typing prediction result, specifically: "Obtain a sample medical image set and a sample gene typing of a sample cancerous breast; wherein, the sample medical image set includes at least three different types of breast medical images; train an initial gene typing prediction model based on the sample medical image set and the sample gene typing to obtain a trained gene typing prediction model; wherein, the initial gene typing prediction model includes an input layer, a branch network set, a fusion layer, and a fully connected block connected in sequence; the branch network set includes branch networks corresponding to each type of breast medical image; obtain a target medical image set of a target cancerous breast; wherein, the sample medical image set includes at least one type of breast medical image; use the trained gene typing prediction model to process the target medical image set to obtain the target gene typing of the target cancerous breast". By setting multiple branch networks to extract features from different types of medical images, each branch network can perform specific feature extraction according to the type of image it processes, allowing important features of each image type to be learned and captured in different ways, and each branch can be optimized separately for the type of image it processes, which helps to improve the accuracy and robustness of model prediction; by setting a fusion layer and a fully connected block to fuse the information from multiple branches and comprehensively consider the information of different modalities, it helps to improve the accuracy and generalization ability of model prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] To more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings required for the description of the present application. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0040] Figure 1 It is a flowchart of the steps of a method for predicting breast cancer gene typing based on deep learning provided by an embodiment of the present application;

[0041] Figure 2 It is a block diagram of the structure of a device for predicting breast cancer gene typing based on deep learning provided by an embodiment of the present application;

[0042] Figure 3 It is a schematic diagram of the structure of a computer device provided by an embodiment of the present application.

[0043] The reference numerals in the accompanying drawings of the specification are as follows:

[0044] 12. Computer device; 14. External device; 16. Processing unit; 18. Bus; 20. Network adapter; 22. I / O interface; 24. Display; 28. Memory; 30. Random access memory; 32. Cache memory; 34. Storage system; 40. Program / utilities; 42. Program module. Detailed implementation manners

[0045] To make the objectives, features, and advantages of the present application more obvious and understandable, the present application will be further described in detail below with reference to the accompanying drawings and specific implementation manners. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the scope of protection of the present application.

[0046] Refer to Figure 1 , which shows a breast cancer gene typing prediction method based on deep learning provided by an embodiment of the present application, including:

[0047] S110. Obtain a sample medical image set and a sample gene typing of a sample cancerous breast; wherein, the sample medical image set includes at least three different types of breast medical images;

[0048] S120. Train an initial gene typing prediction model according to the sample medical image set and the sample gene typing to obtain a trained gene typing prediction model; wherein, the initial gene typing prediction model includes an input layer, a branch network set, a fusion layer, and a fully connected block connected in sequence; the branch network set includes branch networks corresponding to each type of breast medical image;

[0049] S130. Obtain a target medical image set of a target cancerous breast; wherein, the sample medical image set includes at least one type of breast medical image;

[0050] S140. Process the target medical image set using the trained gene typing prediction model to obtain the target gene typing of the target cancerous breast.

[0051] In an embodiment of the present application, in view of the problem that the prior art cannot predict breast cancer gene typing through imaging examination results, the present application provides a solution for processing a target medical image set using a gene typing prediction model with a multi-branch network to obtain its gene typing prediction result, specifically: "Obtain a sample medical image set and a sample gene typing of a sample cancerous breast; wherein, the sample medical image set includes at least three different types of breast medical images; train an initial gene typing prediction model based on the sample medical image set and the sample gene typing to obtain a trained gene typing prediction model; wherein, the initial gene typing prediction model includes an input layer, a branch network set, a fusion layer, and a fully connected block connected in sequence; the branch network set includes branch networks corresponding to each type of breast medical image; obtain a target medical image set of a target cancerous breast; wherein, the sample medical image set includes at least one type of breast medical image; use the trained gene typing prediction model to process the target medical image set to obtain the target gene typing of the target cancerous breast". By setting multiple branch networks to extract features from different types of medical images, each branch network can perform specific feature extraction according to the type of image it processes, allowing it to learn and capture important features of each image type in different ways, and each branch can be optimized separately for the type of image it processes, which helps to improve the accuracy and robustness of model prediction; by setting a fusion layer and a fully connected block to fuse the information from multiple branches and comprehensively consider information in different modalities, it helps to improve the accuracy and generalization ability of model prediction.

[0052] Next, a method for predicting breast cancer gene typing based on deep learning in this exemplary embodiment will be further described.

[0053] As described in step S110, obtain a sample medical image set and a sample gene typing of a sample cancerous breast; wherein, the sample medical image set includes at least three different types of breast medical images.

[0054] Collect about 300 breast cancer cases diagnosed and treated in this hospital over the past decade; the inclusion criteria are: first visit, with complete clinical history and imaging, pathological, and genetic examination results, and the exclusion criteria are: already diagnosed with breast cancer and having undergone surgical resection or radiotherapy and chemotherapy; obtain the sample medical image set and the sample gene typing from the imaging and genetic examination results of the above cases.

[0055] Specifically, the sample medical image set includes the following types of images: X-ray images, MRI T1-weighted imaging, MRI T2-weighted imaging, MRI diffusion-weighted imaging, and MRI dynamic contrast-enhanced scan images; the sample gene typing is one of the following types: Luminal A type, Luminal B type, triple-negative type, and HER-2 type.

[0056] As described in step S120, the initial gene typing prediction model is trained based on the sample medical image set and the sample gene typing to obtain a trained gene typing prediction model; wherein, the initial gene typing prediction model includes an input layer, a branch network set, a fusion layer, and a fully connected block connected in sequence; the branch network set includes branch networks corresponding to each type of breast medical image.

[0057] Construct an initial gene typing prediction model, which includes an input layer, a branch network set, a fusion layer, and a fully connected block connected in sequence; the input layer is used to receive different types of breast medical images and input them to the next layer for processing; the branch network set includes branch networks corresponding to each type of breast medical image, and each branch network is an independent convolutional neural network, which is used to process specific type of image data and extract relevant features of specific type of images; the fusion layer is used to fuse the features from different branch networks to make full use of the information of different types of images. As an example, the fusion layer can achieve feature fusion through splicing, pooling, or other fusion strategies; the fully connected block is used to convert the features extracted from the previous layer into the final gene typing prediction result.

[0058] Specifically, the branch network includes a first convolutional layer, a first batch normalization layer, a second convolutional layer, a second batch normalization layer, a max pooling layer, and a spatial dropout layer connected in sequence. The first convolutional layer is used to extract features from the input breast medical images. It applies convolutional operations to detect edge, texture, and structural features. Each convolutional kernel slides over the input image and performs convolutional operations to generate feature maps. The first batch normalization layer is used to standardize the output of the first convolutional layer to accelerate the training process, improve the convergence speed of the model, and increase the robustness of the model. The second convolutional layer is used to continue extracting higher-level features from the output of the first convolutional layer. It further strengthens and abstracts the features of the previous convolutional layer through convolutional operations to obtain more representative feature representations. The second batch normalization layer is similar to the first batch normalization layer and is used to standardize the output of the second convolutional layer to accelerate the training process and improve the model performance. The max pooling layer is used to reduce the spatial size of the feature map while retaining the most significant features. It is achieved by selecting the maximum value within each pooling window. The max pooling layer helps reduce the computational burden, reduce the overfitting risk of the model, and improve the translational invariance of the model. The spatial dropout layer is used to randomly set the outputs of a part of the neurons to zero to reduce the overfitting risk. It helps improve the generalization ability of the model. It should be noted that the spatial dropout layer is only applied during training and removed during inference. By adopting the branch network in the above form, the features of breast medical images can be extracted and abstracted layer by layer from low-level edges and textures to high-level structures and patterns, while retaining the most representative features to improve the accuracy of genotype prediction.

[0059] The fully connected block includes a multi-head self-attention layer, a global average pooling layer, a first fully connected layer, a second fully connected layer, and an output layer connected in sequence. The multi-head self-attention layer is used to introduce the self-attention mechanism in the fully connected block to help the model understand the relationships between the input features. The self-attention mechanism allows the model to assign different weights to different features to better capture the dependencies between the features. The global average pooling layer is used to take the average of the values of each channel of the feature map to generate a global feature vector. The global average pooling layer helps reduce the feature dimension, lower the computational complexity, and capture the overall feature representation. The first fully connected layer is used to receive the feature vector after global average pooling and map it to a higher-dimensional feature space to learn more complex feature representations. The second fully connected layer further processes the feature vector and maps it to the feature representation finally used for genotype prediction. The output layer is used to generate the final genotype prediction result. As an example, the output layer uses an activation function to transform the output of the model into a probability distribution to represent the probabilities of different genotypes. By adopting the fully connected block in the above form, the features extracted from the convolutional and self-attention layers can be further integrated and abstracted to obtain the final genotype prediction result.

[0060] In a specific implementation, the input layer of the initial genotyping prediction model includes 5 neuron groups, and each neuron group includes 224 * 224 (number of pixels) * 3 (number of channels) = 150528 neurons, which are used to receive feature maps corresponding to five types of breast medical images, namely X-ray images, MRI T1-weighted imaging, MRI T2-weighted imaging, MRI diffusion-weighted imaging, and MRI dynamic contrast-enhanced scan images; the branch network set includes 5 corresponding branch networks; in each branch network, the number of convolutional kernels in the first convolutional layer is 32, the size of the convolutional kernel is 3x3, the stride is 1, the activation function is ReLU, and the total number of parameters is 32 * (3 * 3 + 1) = 320; the number of convolutional kernels in the second convolutional layer is 64, the size of the convolutional kernel is 3x3, the stride is 1, the activation function is ReLU, and the total number of parameters is 64 * (3 * 3 + 1) = 640; the size of the pooling window in the max pooling layer is 2x2, and the stride is 2; the dropout rate of the spatial dropout layer is 0.5, indicating that 50% of the neurons are discarded; the fusion layer uses weighted fusion, and the fused feature map = w1 * branch 1 feature map + w1 * branch 2 feature map + w1 * branch 3 feature map + w1 * branch 4 feature map + w1 * branch 5 feature map. By introducing weight parameters w1, w2, w3, w4, and w5, the importance of each branch in the fusion can be learned; in the fully connected block, the multi-head self-attention layer includes 4 heads of self-attention; the first fully connected layer includes 128 neurons, each neuron is connected to 64 feature map channels, the activation function is ReLU, and the total number of parameters is 128 * 64 + 128 = 8320; the second fully connected layer includes 64 neurons, each neuron is connected to 128 neurons, the activation function is ReLU, and the total number of parameters is 64 * 128 + 64 = 8256; the output layer uses the Softmax activation function and includes 4 neurons, which are used to represent the probability distributions of four genotyping types, namely Luminal A type, Luminal B type, triple-negative type, and HER-2 type, and the total number of parameters is 128 * 4 + 4 = 516.

[0061] Train the initial genotyping prediction model according to the sample medical image set and the sample genotyping to obtain a trained genotyping prediction model. During the training process, the parameters of the model will be gradually adjusted until the loss value is less than the preset value.

[0062] As described in step S130, obtain the target medical image set of the target cancerous breast; wherein, the sample medical image set includes at least one type of breast medical image.

[0063] Collect the imaging examination results of the target to be measured, and obtain a set of target medical images therefrom; specifically, the target medical images include at least one of the following types of images: X-ray images, MRI T1-weighted imaging, MRI T2-weighted imaging, MRI diffusion-weighted imaging, and MRI dynamic contrast-enhanced scan images.

[0064] As described in step S140, use the trained gene typing prediction model to process the set of target medical images to obtain the target gene typing of the target cancerous breast.

[0065] When the image types of the set of target medical images are complete, that is, the number of breast medical images in the set of target medical images is equal to the number of breast medical images in the set of sample medical images, input the set of target medical images into the trained gene typing prediction model for prediction to obtain the gene typing prediction result of the target cancerous breast;

[0066] When the image types of the set of target medical images are missing, that is, the number of breast medical images in the set of target medical images is less than the number of breast medical images in the set of sample medical images, fill in the missing target medical images to make the number of breast medical images in the set of target medical images meet the requirements, and input the filled set of target medical images into the trained gene typing prediction model to obtain the gene typing prediction result of the target cancerous breast.

[0067] In an embodiment of the present application, the specific process of the step of "training the initial gene typing prediction model according to the set of sample medical images and the sample gene typing to obtain a trained gene typing prediction model" can be further described in combination with the following description.

[0068] Randomly mask the breast medical images in the set of sample medical images to obtain a set of masked medical images. Specifically, the number of the set of masked medical images is a preset ratio (such as 5%-30%) of the number of the set of sample medical images.

[0069] Train the initial gene typing prediction model according to the set of sample medical images, the set of masked medical images, and the sample gene typing to obtain a trained gene typing prediction model. During the training process, use the set of sample medical images and their corresponding sample gene typing labels, as well as the set of masked medical images and their corresponding sample gene typing labels to train the model. After training, the model can be used to predict the gene typing of new cancerous breast medical images.

[0070] By randomly masking the breast medical images in the sample medical image set, more data can be generated for training, thereby improving the generalization performance of the model. It can also help the model learn to predict the genotyping of partially or wholly defective images, so as to better adapt to the defective images that may appear in the actual clinical scenario, thereby enhancing the robustness of the model.

[0071] In an embodiment of the present application, the specific process of the step of "randomly masking the breast medical images in the sample medical image set to obtain a masked medical image set" can be further described in combination with the following description.

[0072] Based on the sample medical image set, a mean masked image corresponding to each type of breast medical image is generated. As an example, based on all the sample medical images of the same type, a corresponding mean masked image is generated. The mean masked image is an image with the same dimension as the original image, and each pixel value therein is the mean of the pixel values at the same position in all the sample medical images of the corresponding type.

[0073] Replace the random regions in a random number of breast medical images in the sample medical image set with the corresponding regions in the corresponding mean masked images to obtain a masked medical image set. As an example, number all the sample medical image sets, use a random number generator to randomly generate the set numbers to be masked according to the preset ratio requirements to obtain the medical image sets to be masked; for each medical image set to be masked, number all the breast medical images therein, use a random number generator to first randomly generate the number of images to be masked, and then randomly generate the corresponding image numbers to obtain the breast medical images to be masked; for each breast medical image to be masked, use a random number generator to randomly generate the vertex coordinates of n (n≥3) regions to be masked, and replace the region enclosed by the n vertex coordinates with the corresponding region in the corresponding mean masked image to obtain a masked medical image.

[0074] By generating the mean masked images, the overall features of the sample medical images can be retained, and at the same time, the information loss caused by masking can be reduced to a certain extent. Since the masked regions are replaced with the corresponding regions in the mean masked images, the masked breast medical images still contain part of the information of the original images; by training the initial genotyping prediction model with the above-generated masked medical images, the accuracy of the model in predicting the genotyping of partially or wholly defective images can be further improved.

[0075] In one embodiment of the present application, the specific process of the step of "training the initial genotyping prediction model based on the sample medical image set, the masked medical image set, and the sample genotyping to obtain the trained genotyping prediction model" can be further described in combination with the following description.

[0076] Input the sample medical image set and the masked medical image set into the initial genotyping prediction model to obtain a predicted genotyping.

[0077] Calculate the loss value of the initial genotyping prediction model based on the predicted genotyping and the sample genotyping. As an example, a loss function (such as mean squared error, cross entropy, etc.) is used to calculate the difference between the predicted genotyping and the sample genotyping, and it is used as the loss value of the initial genotyping prediction model.

[0078] Adjust the parameters of the initial genotyping prediction model until the loss value is less than a preset value. As an example, an optimization algorithm (such as gradient descent method) is used to adjust the parameters of the initial genotyping prediction model to reduce the loss value. During this process, the model will automatically update the parameters according to the feedback signal of the loss function, making the prediction result closer to the sample genotyping.

[0079] In a specific implementation, the loss function of the model is:

[0080] ; (1)

[0081] Where, represents the loss value, N represents the number of samples, represents the eigenvalue corresponding to the sample genotyping, represents the eigenvalue corresponding to the predicted genotyping;

[0082] The parameter adjustment formula is:

[0083] ; (2)

[0084] Where, represents the model parameter at the t-th iteration, represents the learning rate (used to control the step size of parameter update), represents the gradient of the loss function with respect to the parameter.

[0085] In one embodiment of the present application, the specific process of the step of "processing the target medical image set using the trained genotyping prediction model to obtain the target genotyping of the target cancerous breast" can be further described in combination with the following description.

[0086] When the number of breast medical images in the target medical image set is equal to the number of breast medical images in the sample medical image set, input the target medical image set into the trained genotyping prediction model to obtain the target genotyping of the target cancerous breast.

[0087] When the number of breast medical images in the target medical image set is less than the number of breast medical images in the sample medical image set, fill in the missing breast medical images in the target medical image set to obtain a filled medical image set; input the filled medical image set into the trained genotyping prediction model to obtain the target genotyping of the target cancerous breast. As an example, use the existing breast medical images in the target medical image set to fill in the missing breast medical images. This method can preserve the overall characteristics of the original data and will not introduce additional data bias; as another example, combine a reference data set (such as the sample medical image set) to predict the missing breast medical images based on the existing breast medical images. In the case of high reference data diversity, this method can obtain relatively accurate prediction results.

[0088] By filling in the missing breast medical images in the target medical image set, a complete data set can be obtained to predict the genotyping of the target cancerous breast in the case where the imaging examination results of the target cancerous breast are missing.

[0089] In an embodiment of the present application, the specific process of the step of "filling in the missing breast medical images in the target medical image set to obtain a filled medical image set" can be further described in combination with the following description.

[0090] Generate a target filling image based on the existing breast medical images in the target medical image set. As an example, perform an averaging process on the pixels of the relevant existing breast medical images to obtain the target filling image.

[0091] Add the target filling image to the target medical image set to obtain a filled medical image set.

[0092] By using the average value of the existing breast medical images to fill in the missing breast medical images, the overall distribution trend of the original data can be preserved, no additional data bias will be introduced, and the computational amount is small and the operation cost is low. However, in practical applications, the correlation between the missing images and the existing images needs to be considered. For example, when the missing is MRIT1 weighted imaging, MRIT2 weighted imaging and MRI diffusion weighted imaging can be considered for filling.

[0093] In one embodiment of the present application, the specific process of the step of "filling the missing breast medical images in the target medical image set to obtain a filled medical image set" can be further described in combination with the following description.

[0094] Generate a target filling image based on the sample medical image set and the existing breast medical images in the target medical image set. As an example, based on the existing breast medical images, screen out a preset number (for example, 3-5) of approximate medical image sets from the sample medical image set; generate a target filling image based on the breast medical images of the same type as the missing breast medical image in the approximate medical image set.

[0095] In a specific implementation, for each existing image in the target set, calculate its image similarity with the corresponding type of sample image in each sample set according to the following formula:

[0096] ; (3)

[0097] where x and y respectively represent two images to be compared, and respectively represent the pixel means of x and y, and respectively represent the pixel standard deviations of x and y, represents the pixel covariance of x and y, and C1 and C2 are constants used to increase stability;

[0098] Sum the image similarities between each sample image in the sample set and the corresponding type of target image (when the corresponding target image is missing, the image similarity is recorded as 0) to obtain the overall similarity between each sample set and the target set; take the 3 sample sets with the highest overall similarity as the approximate medical image set; extract 3 breast medical images of the same type as the missing breast medical image included in the approximate medical image set, and perform mean processing on their pixels to obtain the target filling image.

[0099] Add the target filling image to the target medical image set to obtain a filled medical image set.

[0100] By screening out the closest images from the sample medical image set and generating the target filling image accordingly, it can better adapt to the characteristics and distribution of the target images and provide a relatively accurate filling result.

[0101] For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for the relevant parts, refer to the partial description of the method embodiment.

[0102] Refer to Figure 2, showing a breast cancer gene typing prediction device provided by an embodiment of the present application, including:

[0103] A sample acquisition module 210, configured to acquire a sample medical image set and a sample gene typing of a sample cancerous breast; wherein, the sample medical image set includes at least three different types of breast medical images;

[0104] A model training module 220, configured to train an initial gene typing prediction model based on the sample medical image set and the sample gene typing to obtain a trained gene typing prediction model; wherein, the initial gene typing prediction model includes an input layer, a branch network set, a fusion layer, and a fully connected block connected in sequence; the branch network set includes branch networks corresponding to each type of breast medical image;

[0105] A target acquisition module 230, configured to acquire a target medical image set of a target cancerous breast; wherein, the sample medical image set includes at least one type of breast medical image;

[0106] A target prediction module 240, configured to process the target medical image set using the trained gene typing prediction model to obtain the target gene typing of the target cancerous breast.

[0107] Refer to Figure 3 , showing a computer device of the present application, the computer device 12 is presented in the form of a general computing device; the computer device 12 includes: one or more processors or processing units 16, a memory 28, and a bus 18 connecting different system components (including the memory 28 and the processing unit 16).

[0108] The bus 18 can be one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any bus structure in a variety of bus structures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0109] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0110] Memory 28 may include computer system readable media in the form of volatile memory, such as random access memory 30 and / or cache memory 32. The computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used for reading from and writing to a non-removable, non-volatile magnetic medium (commonly referred to as a "hard disk drive"). Although Figure 3 not shown in FIG. Figure 3 , a disk drive for reading from and writing to a removable non-volatile disk (such as a "floppy disk"), and an optical disk drive for reading from and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical medium) may be provided. In these instances, each drive may be connected to the bus 18 by one or more data media interfaces. The memory may include at least one program product having a set (e.g., at least one) of program modules 42 that are configured to perform the functions of the embodiments of the present application.

[0111] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory. Such program modules 42 include an operating system, one or more application programs, other program modules 42, and program data, and an implementation of a network environment may be included in each or some combination of these examples. The program modules 42 generally perform the functions and / or methods in the embodiments described in the present application.

[0112] The computer device 12 may also communicate with one or more external devices 14 (such as a keyboard, a pointing device, a display 24, a camera, etc.), and may also communicate with one or more devices that enable an operator to interact with the computer device 12, and / or communicate with any device that enables the computer device 12 to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication may be through the I / O interface 22. Also, the computer device 12 may communicate with one or more networks (such as a local area network (LAN)), a wide area network (WAN), and / or a public network (such as the Internet) through the network adapter 20. As Figure 3 shown, the network adapter 20 communicates with other modules of the computer device 12 through the bus 18. It should be understood that although Figure 3 not shown in FIG. Figure 3 , other hardware and / or software modules may be used in conjunction with the computer device 12, including but not limited to: microcode, device drivers, redundant processing unit 16, external disk drive arrays, RAID systems, tape drives, and data backup storage systems 34, etc.

[0113] The processing unit 16 executes various functional applications and data processing by running the program stored in the memory 28, for example, implementing the breast cancer gene typing prediction method based on deep learning provided in any embodiment of the present application.

[0114] That is, when the above-mentioned processing unit 16 executes the above program, it can achieve: obtaining a sample medical image set and a sample gene typing of a sample cancerous breast; wherein, the sample medical image set includes at least three different types of breast medical images; training an initial gene typing prediction model according to the sample medical image set and the sample gene typing to obtain a trained gene typing prediction model; wherein, the initial gene typing prediction model includes an input layer, a branch network set, a fusion layer and a fully connected block connected in sequence; the branch network set includes branch networks corresponding to each type of breast medical image; obtaining a target medical image set of a target cancerous breast; wherein, the sample medical image set includes at least one type of breast medical image; using the trained gene typing prediction model to process the target medical image set to obtain the target gene typing of the target cancerous breast.

[0115] In an embodiment of the present application, there is also provided a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the breast cancer gene typing prediction method based on deep learning provided in any embodiment of the present application.

[0116] That is, when the program is executed by a processor, it can achieve: obtaining a sample medical image set and a sample gene typing of a sample cancerous breast; wherein, the sample medical image set includes at least three different types of breast medical images; training an initial gene typing prediction model according to the sample medical image set and the sample gene typing to obtain a trained gene typing prediction model; wherein, the initial gene typing prediction model includes an input layer, a branch network set, a fusion layer and a fully connected block connected in sequence; the branch network set includes branch networks corresponding to each type of breast medical image; obtaining a target medical image set of a target cancerous breast; wherein, the sample medical image set includes at least one type of breast medical image; using the trained gene typing prediction model to process the target medical image set to obtain the target gene typing of the target cancerous breast.

[0117] Any combination of one or more computer-readable media may be employed. The computer-readable media may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present document, the computer-readable storage medium may be any tangible medium that contains or stores a program which can be used by or in connection with an instruction execution system, apparatus, or device.

[0118] The computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which the computer-readable program code is carried. Such a propagated data signal may take many forms, including an electromagnetic signal, an optical signal, or any suitable combination of the foregoing. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.

[0119] The computer program code for performing the operations of the present application may be written in one or more programming languages or combinations thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code may execute entirely on the operator's computer, partly on the operator's computer, as a stand-alone software package, partly on the operator's computer and partly on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the operator's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider). Each embodiment in this specification is described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts among the various embodiments may be referred to each other.

[0120] Although the preferred embodiments of the embodiments of the present application have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0121] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or terminal device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or terminal device comprising the element.

[0122] The above has introduced in detail a breast cancer gene typing prediction method and device based on deep learning provided by the present application. Specific examples are used in this text to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A breast cancer gene typing prediction method based on deep learning, characterized in that, Including: Obtaining a sample medical image set and sample gene typing of a sample cancerous breast; wherein, the sample medical image set includes at least three different types of breast medical images; the types of the breast medical images include: X-ray images, MRI T1-weighted imaging, MRI T2-weighted imaging, MRI diffusion-weighted imaging, and MRI dynamic contrast-enhanced scan images; Training an initial gene typing prediction model based on the sample medical image set and the sample gene typing to obtain a trained gene typing prediction model; wherein, the initial gene typing prediction model includes an input layer, a branch network set, a fusion layer, and a fully connected block connected in sequence; the branch network set includes branch networks corresponding to each type of breast medical image; each branch network is an independent convolutional neural network for processing image data of a specific type and extracting relevant features of the specific type of image; the steps of training an initial gene typing prediction model based on the sample medical image set and the sample gene typing to obtain a trained gene typing prediction model include: inputting the sample medical image set and a masked medical image set into the initial gene typing prediction model to obtain a predicted gene typing; calculating a loss value of the initial gene typing prediction model based on the predicted gene typing and the sample gene typing; adjusting parameters of the initial gene typing prediction model until the loss value is less than a preset value; Obtaining a target medical image set of a target cancerous breast; wherein, the sample medical image set includes at least one type of breast medical image; Processing the target medical image set using the trained gene typing prediction model to obtain the target gene typing of the target cancerous breast.

2. The method according to claim 1, characterized in that, The steps of training an initial gene typing prediction model based on the sample medical image set and the sample gene typing to obtain a trained gene typing prediction model include: Randomly masking the breast medical images in the sample medical image set to obtain a masked medical image set; Training an initial gene typing prediction model based on the sample medical image set, the masked medical image set, and the sample gene typing to obtain a trained gene typing prediction model.

3. The method according to claim 2, wherein The steps of randomly masking the breast medical images in the sample medical image set to obtain a masked medical image set include: Generating a mean masked image corresponding to each type of breast medical image based on the sample medical image set; Replacing random regions in a random number of breast medical images in the sample medical image set with corresponding regions in the corresponding mean masked image to obtain a masked medical image set.

4. The method according to claim 1, characterized in that, The steps of processing the target medical image set using the trained gene typing prediction model to obtain the target gene typing of the target cancerous breast include: When the number of breast medical images in the target medical image set is equal to the number of breast medical images in the sample medical image set, input the target medical image set into the trained gene typing prediction model to obtain the target gene typing of the target cancerous breast; When the number of breast medical images in the target medical image set is less than the number of breast medical images in the sample medical image set, fill in the missing breast medical images in the target medical image set to obtain a filled medical image set; Input the filled medical image set into the trained gene typing prediction model to obtain the target gene typing of the target cancerous breast.

5. The method according to claim 4, wherein The step of filling in the missing breast medical images in the target medical image set to obtain a filled medical image set includes: Generating a target filling image based on the existing breast medical images in the target medical image set; Adding the target filling image to the target medical image set to obtain a filled medical image set.

6. The method according to claim 4, wherein The step of filling in the missing breast medical images in the target medical image set to obtain a filled medical image set includes: Generating a target filling image based on the existing breast medical images in the sample medical image set and the target medical image set; Adding the target filling image to the target medical image set to obtain a filled medical image set.

7. A breast cancer gene typing prediction device based on deep learning, characterized in that, Including: A sample acquisition module for acquiring a sample medical image set and a sample gene typing of a sample cancerous breast; wherein, the sample medical image set includes at least three different types of breast medical images; the types of the breast medical images include: X-ray images, MRI T1-weighted imaging, MRI T2-weighted imaging, MRI diffusion-weighted imaging, and MRI dynamic contrast-enhanced scan images; A model training module for training an initial gene typing prediction model based on the sample medical image set and the sample gene typing to obtain a trained gene typing prediction model; wherein, the initial gene typing prediction model includes an input layer, a branch network set, a fusion layer, and a fully connected block connected in sequence; the branch network set includes branch networks corresponding to each type of breast medical image; each branch network is an independent convolutional neural network for processing image data of a specific type and extracting relevant features of the image of the specific type; the step of training the initial gene typing prediction model based on the sample medical image set and the sample gene typing to obtain a trained gene typing prediction model includes: inputting the sample medical image set and a masked medical image set into the initial gene typing prediction model to obtain a predicted gene typing; calculating a loss value of the initial gene typing prediction model based on the predicted gene typing and the sample gene typing; adjusting parameters of the initial gene typing prediction model until the loss value is less than a preset value; A target acquisition module for acquiring a target medical image set of a target cancerous breast; wherein, the sample medical image set includes at least one type of breast medical image; A target prediction module, configured to process the target medical image set by using the trained genotyping prediction model, so as to obtain the target genotyping of the target cancerous breast.

8. A computer device, characterized in that, It includes a processor, a memory, and a computer program stored on the memory and capable of running on the processor. When the computer program is executed by the processor, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images