Breast lesion MRI image recognition method based on multi-branch CNN
The features of breast MRI images were extracted through the multi-branch CNN model, and the problem of poor feature extraction dependence on artificial and model generalization ability in breast cancer MRI image recognition was solved, and the four-class detection and high classification accuracy of breast MRI images were realized, which was adapted to small scale Dataset.
Patent Information
- Application Number
- CN202411101311.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-12-10
AI Technical Summary
The prior art has the ability to rely on artificial features in breast cancer MRI image recognition, poor generalization ability of the model, and it is difficult to effectively classify breast cancer subtypes, and the data set is small, resulting in poor classification effect.
Using the recognition method based on multi-branch CNN, the shallow and deep features of breast MRI images are extracted by constructing multi-branch modules, channel and spatial attention modules, and deep feature extraction modules, and the shallow and deep features of breast MRI images are classified through attention mechanism fusion features to adapt to small Scale datasets to avoid overfitting.
The four-classification detection of breast MRI images is realized, manpower is saved, and the accuracy of lesion type recognition is improved, ensuring the model's good generalization ability and high classification accuracy under limited data.
Smart Images
Figure CN119107488A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image recognition, and in particular to a breast lesion MRI image recognition method based on multi-branch CNN. Background Art
[0002] Breast cancer is the most common cancer in women and the second leading cause of death after cardiovascular disease. In 2020, more than 2.2 million new cases of breast cancer were diagnosed in women worldwide. It is reported that cancer would be detected if specific symptoms were present. However, studies have found that many women with cancer do not have any symptoms. Primary prevention of cancer is not realistic except for preventive mastectomy for those with BRCA1 or BRCA2 gene mutations known to cause cancer. Surgery, chemotherapy, radiotherapy and hormone therapy are the classic treatments for cancer patients. Therefore, screening is recommended as a mandatory step to detect cancer at an early stage. If cancer is detected at an early stage, then lives can be saved. Early detection of cancer helps in early diagnosis and treatment. Studies have shown that those who start treatment within 3 months of the onset of breast cancer symptoms have a higher chance of survival and can reduce the proliferation of malignant cells in the body.
[0003] Breast magnetic resonance imaging (MRI) is a good imaging technology. MRI can produce high-resolution images that can reveal the structure and function of the brain and other tissues in detail. This high-resolution imaging helps doctors diagnose lesions more accurately. In addition, compared with CT and X-rays, MRI does not use ionizing radiation, which means that patients can undergo repeated examinations without worrying about the potential risks of radiation accumulation. However, the increasing number of medical images has brought greater difficulties to radiologists, who may become fatigued, resulting in wrong diagnoses.
[0004] Many traditional machine learning algorithms have been applied to breast cancer analysis tasks. However, these traditional machine learning methods rely on manual feature extraction, have strong a priori, poor model generalization ability, low robustness, and are difficult to find discriminant features manually, making it difficult to solve the classification problem of breast cancer subtypes.
[0005] In recent years, with the development of deep learning technology, deep learning algorithms have been widely used in medical image processing, such as tumor detection and segmentation, benign and malignant differentiation, etc. A lot of work has been devoted to problems related to breast cancer subtype classification. They tried to extract discriminative features of breast cancer MRI by using deep learning models. Their experimental results illustrate the feasibility of predicting tumor categories based on MRI features of breast cancer. However, due to many limitations such as the rarity of the disease and the lack of medical expertise for proper labeling, the datasets for breast cancer subclassification are relatively small and scarce. At the same time, most of the existing studies are direct applications or slight adjustments of general deep learning architectures, which fail to fully optimize the specificity of breast cancer classification problems, resulting in room for improvement in classification results. Summary of the invention
[0006] The present invention provides a breast lesion MRI image recognition method based on multi-branch CNN to overcome the above technical problems.
[0007] A breast lesion MRI image recognition method based on multi-branch CNN, including
[0008] S1. Obtain a breast MRI image dataset, have a doctor outline the tumor contour in the breast MRI image, crop the breast MRI image based on the tumor contour to obtain a contour area image, perform data enhancement on the contour area image, and construct a training set based on the contour area image after data enhancement.
[0009] S2. Construct a multi-branch CNN network model, which includes a multi-branch module, a channel and spatial attention module, and a deep feature extraction module. The multi-branch module includes at least two types of branches, one type of branch is used to extract shallow features of the contour area image, and the other type of branch is used to extract deep features of the contour area image. The channel and spatial attention module includes a channel attention module and a spatial attention module. The channel attention module is used to learn the channel attention weights of the channels in the deep features, and filter the channels of the deep features according to the channel attention weights as channel features. The spatial attention module is used to learn the spatial attention weights of the spatial positions in the channel features, and filter the spatial positions of the channel features according to the spatial attention weights as channel spatial features. The deep feature extraction module is used to extract deep features from shallow features by increasing the number of channels, and fuse the channel spatial features with the deep features as classification features. The contour area image is classified according to the classification features.
[0010] S3, train the multi-branch CNN network model based on the training set, and obtain the trained multi-branch CNN network model,
[0011] S4. Obtain a breast MRI image to be classified, classify the breast MRI image to be classified based on the trained multi-branch CNN network model and obtain a classification result.
[0012] Preferably, the multi-branch module is a four-branch module.
[0013] Preferably, the four-branch module includes a first branch DP1, a second branch SP1, a third branch SP2 and a fourth branch SP3, the first branch DP1 is used to extract shallow features of the contour area image, and the second branch SP1, the third branch SP2 and the fourth branch SP3 are used to extract deep features of the contour area image.
[0014] Preferably, the processing flow of the first branch DP1 in the four-branch module is to obtain a first feature map of the contour area image, wherein the first feature map is obtained by inputting the contour area image after data enhancement into the convolution layer Conv1 for processing, the convolution kernel size of the convolution layer Conv1 is 3×3, the number of channels is 16, the step size is 1, and the padding is 1, and the size of the first feature map is 64×64×16. The first feature map is passed through the convolution layer Conv2 to obtain a second feature map, wherein the convolution kernel size of the convolution layer Conv2 is 1×1, the number of channels is 128, the step size is 1, and the padding is 0, and the dimension of the second feature map is 64×64×128. The second feature map with a size of 64×64×128 is used as the input of the deep convolution layer DW1, and the convolution kernel size of the deep convolution layer DW1 is 3×3, the number of channels is 128, step size is 1, padding is 1, the depth convolution layer DW1 outputs the third feature map of dimension 64×64×128; the third feature map of dimension 64×64×128 is used as the input of the squeeze and excitation block SE, and the squeeze and excitation block SE outputs the fourth feature map of dimension 64×64×128; the fourth feature map of dimension 64×64×128 is used as the input of the convolution layer Conv3, the convolution kernel size of the convolution layer Conv3 is 1×1, the number of channels is 16, the step size is 1, and the padding is 0, and the convolution layer Conv3 outputs the fifth feature map of dimension 64×64×16; the first feature map of dimension 64×64×16 is short-connected with the fifth feature map of dimension 64×64×16 to output the sixth feature map of dimension 64×64×16,
[0015] The processing flow of the second branch SP1 is to use the first feature map with a size of 64×64×16 as the input of the second branch SP1, first pass through the convolution layer Conv4, the convolution kernel size of the convolution layer Conv4 is 1×1, the number of channels is 8, the step size is 1, and the padding is 0, and the convolution layer Conv4 outputs the seventh feature map with a dimension of 64×64×8; use the seventh feature map with a size of 64×64×8 as the input of the deep convolution layer DW2, the convolution kernel size of the deep convolution layer DW2 is 3×3, the number of channels is 8, the step size is 1, and the padding is 1, and the deep convolution layer DW2 outputs the eighth feature map with a dimension of 64×64×8; use the eighth feature map with a size of 64×64×8 as the input of the convolution layer Conv5, the convolution kernel size of the convolution layer Conv5 is 1×1, the number of channels is 16, the step size is 1, and the padding is 0, and the convolution layer Conv5 outputs the ninth feature map with a dimension of 64×64×16;
[0016] The processing flow of the third branch SP2 is to use the first feature map with a size of 64×64×16 as the input of the third branch SP2, first pass through the convolution layer Conv6, the convolution kernel size of the convolution layer Conv6 is 1×1, the number of channels is 4, the step size is 1, and the padding is 0, and the convolution layer Conv6 outputs the tenth feature map with a dimension of 64×64×4; the tenth feature map with a size of 64×64×4 is used as the input of the convolution layer Conv7, the convolution kernel size of the convolution layer Conv7 is 1×3, the number of channels is 8, the step size is 1, and the padding is 0, and the convolution layer Conv7 outputs the eleventh feature map with a dimension of 64×64×8; the eleventh feature map with a size of 64×64×8 is used as the input of the convolution layer Conv8, and the convolution layer Con The convolution kernel size of v8 is 3×1, the number of channels is 8, the step size is 1, and the padding is 0. The convolution layer Conv8 outputs the twelfth feature map with a dimension of 64×64×8; the twelfth feature map with a size of 64×64×8 is used as the input of the convolution layer Conv9, the convolution kernel size of the convolution layer Conv9 is 1×3, the number of channels is 8, the step size is 1, and the padding is 0. The convolution layer Conv9 outputs the thirteenth feature map with a dimension of 64×64×8; the thirteenth feature map with a size of 64×64×8 is used as the input of the convolution layer Conv10, the convolution kernel size of the convolution layer Conv10 is 3×1, the number of channels is 16, the step size is 1, and the padding is 0. The convolution layer Conv10 outputs the fourteenth feature map with a dimension of 64×64×16;
[0017] The processing flow of the fourth branch SP3 is to first divide the channels of the first feature map into 2 groups by channel split, wherein the first group of 64×64×8 first feature maps passes through the convolution layer Conv11, the convolution kernel size of the convolution layer Conv11 is 1×1, the number of channels is 8, the step size is 1, and the padding is 0, and the convolution layer Conv11 outputs the fifteenth feature map with a dimension of 64×64×8;
[0018] The fifteenth feature map with a size of 64×64×8 is used as the input of the deep convolutional layer DW3. The convolution kernel size of the deep convolutional layer DW3 is 3×3, the number of channels is 16, the step size is 1, and the padding is 1. The deep convolutional layer DW3 outputs the sixteenth feature map with a dimension of 64×64×16; the sixteenth feature map with a size of 64×64×16 is used as the input of the convolutional layer Conv12. The convolution kernel size of the convolutional layer Conv12 is 1×1, the number of channels is 8, the step size is 1, and the padding is 0. The convolutional layer Conv12 outputs the seventeenth feature map with a dimension of 64×64×8; the seventeenth feature map is fused with the first feature map of the second group of 64×64×8 to output the eighteenth feature map with a dimension of 64×64×16; finally, the eighteenth feature map is channel shuffled to output the nineteenth feature map with a dimension of 64×64×16.
[0019] Preferably, the classification results include four types, namely 0, 1+, 2+, and 3+. 0 and 1+ are negative, 3+ is positive, and Her2+ indicates positive or negative cannot be determined.
[0020] The present invention provides a breast lesion MRI image recognition method based on a multi-branch CNN, which can automatically realize the four-category detection of breast MRI images, save manpower, and help imaging doctors better identify the type of lesions. An innovative deep network model based on a multi-branch convolutional neural network (CNN) is proposed. The model captures richer and higher-order breast cancer MRI features by processing image information at different levels in parallel, and focuses on lesion objects. The deep network model can adapt to small-scale data sets, ensure model performance, avoid overfitting problems, ensure that the model can achieve good generalization ability under limited data, and improve classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0022] Figure 1 is a flow chart of the method of the present invention;
[0023] Figure 2 It is a classification network structure model diagram of the present invention;
[0024] Figure 3 It is a structural diagram of the four branches DP1, SP1, SP2, SP3 and the deep feature extraction module of the present invention;
[0025] Figure 4 It is a structural diagram of the channel and spatial attention module CSAB of the present invention. DETAILED DESCRIPTION
[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0027] Figure 1 It is a flow chart of the method of the present invention, such as Figure 1 As shown, the method of this embodiment may include:
[0028] S1. Obtain a breast MRI image dataset, have a doctor outline the tumor contour in the breast MRI image, crop the breast MRI image based on the tumor contour to obtain a contour area image, perform data enhancement on the contour area image, and construct a training set based on the contour area image after data enhancement.
[0029] S2. Construct a multi-branch CNN network model, which includes a multi-branch module, a channel and spatial attention module, and a deep feature extraction module. The multi-branch module includes at least two types of branches, one type of branch is used to extract shallow features of the contour area image, and the other type of branch is used to extract deep features of the contour area image. The channel and spatial attention module includes a channel attention module and a spatial attention module. The channel attention module is used to learn the channel attention weights of the channels in the deep features, and filter the channels of the deep features according to the channel attention weights as channel features. The spatial attention module is used to learn the spatial attention weights of the spatial positions in the channel features, and filter the spatial positions of the channel features according to the spatial attention weights as channel spatial features. The deep feature extraction module is used to extract deep features from shallow features by increasing the number of channels, and fuse the channel spatial features with the deep features as classification features. The contour area image is classified according to the classification features.
[0030] S3, train the multi-branch CNN network model based on the training set, and obtain the trained multi-branch CNN network model,
[0031] S4. Obtain a breast MRI image to be classified, classify the breast MRI image to be classified based on the trained multi-branch CNN network model and obtain a classification result.
[0032] Based on the above scheme, the present invention provides a breast lesion MRI image recognition method based on multi-branch CNN, which can automatically realize the four-category detection of breast MRI images, save manpower, and help imaging doctors better identify the type of lesions. An innovative deep network model based on a multi-branch convolutional neural network (CNN) is proposed. The model captures richer and higher-order breast cancer MRI features by processing image information at different levels in parallel, and focuses on lesion objects. The deep network model can adapt to small-scale data sets, ensure model performance, avoid overfitting problems, and ensure that the model can achieve good generalization ability even with limited data.
[0033] Specifically, this embodiment provides the details of the method for recognizing breast lesions in MRI images. The process of the breast cancer classification learning method based on multi-branch CNN is as follows: Figure 2 Shown, including
[0034] S1. Obtain a breast MRI image dataset. Doctors use medical image annotation software to outline the tumor contour in the breast MRI image. The medical image annotation software includes 3D Slicer, ITK Snap, and Pair. The breast MRI image is cropped based on the tumor contour to obtain a contour area image. The contour area image is data enhanced, and a training set is constructed based on the contour area image after data enhancement.
[0035] Specifically, according to the breast tumor outline drawn by the doctor, the edge points in the four directions of the outline are extracted to obtain the minimum matrix covering the tumor. These four points are added 10 pixels in their respective directions to crop their contour areas, and the cropped image size is adjusted to 64×64 pixels through bilinear interpolation. At the same time, in order to improve the robustness of CNN and avoid overfitting, data enhancement operations are performed on the training data set: including random rotation, image translation, image cropping, etc. Finally, the image is normalized by formula (1).
[0036]
[0037] Where xi represents the image pixel value, and max(x) and min(x) represent the maximum and minimum values of the image pixels, respectively.
[0038] S2. Construct a multi-branch CNN network model, wherein the multi-branch CNN network model includes a multi-branch module, a channel and spatial attention module, and a deep feature extraction module.
[0039] The multi-branch module includes at least two types of branches, one type of branch is used to extract shallow features of the contour area image, and the other type of branch is used to extract deep features of the contour area image. Specifically, the multi-branch module is a four-branch module, and the four-branch module includes a first branch DP1, a second branch SP1, a third branch SP2 and a fourth branch SP3. The first branch DP1 is used to extract shallow features of the contour area image, and the second branch SP1, the third branch SP2 and the fourth branch SP3 are used to extract deep features of the contour area image. The structure diagram of the four branches DP1, SP1, SP2, SP3 and the deep extraction module is as shown in FIG. Figure 3 As shown,
[0040] Specifically, the network structure includes a four-branch framework, a channel and spatial attention module CSAB, and a deep feature extraction module DFEB. The two branches SP1 and SP2 in the four-branch framework fuse the extracted features and then pass them through the channel and spatial attention modules, and then fuse the image features passing through the attention module with the shallow features extracted by DP1 and the deep features extracted by SP3, thereby improving the classification performance of MRI images for breast cancer.
[0041] The preprocessed MRI image of size 64×64×3 is input into the multi-branch CNN network and passes through a convolution layer Conv1, where the convolution kernel size of the convolution layer Conv1 is 3×3, the number of channels is 16, the step size is 1, and the padding is 1. The convolution layer Conv1 outputs the first feature map of dimension 64×64×16. This part increases the number of channels and uses the shallow breast feature map after one convolution as the input of the next four branches.
[0042] The processing flow of the first branch DP1 in the four-branch module is to obtain a first feature map of the contour area image, the size of the first feature map is 64×64×16, the first feature map is passed through the convolution layer Conv2 to obtain a second feature map, wherein the convolution kernel size of the convolution layer Conv2 is 1×1, the number of channels is 128, the step size is 1, and the padding is 0, the dimension of the second feature map is 64×64×128, and the second feature map with a size of 64×64×128 is used as the input of the deep convolution layer DW1, the convolution kernel size of the deep convolution layer DW1 is 3×3, the number of channels is 128, the step size is 1, and the padding is 1, and the deep convolution layer DW1 outputs a third feature map with a dimension of 64×64×128; the size of 64 The third feature map of size 64×64×128 is used as the input of the squeeze and excitation block SE, and the squeeze and excitation block SE outputs the fourth feature map of size 64×64×128; the fourth feature map of size 64×64×128 is used as the input of the convolution layer Conv3, the convolution kernel size of the convolution layer Conv3 is 1×1, the number of channels is 16, the step size is 1, and the padding is 0, and the convolution layer Conv3 outputs the fifth feature map of size 64×64×16; the first feature map of size 64×64×16 is short-connected with the fifth feature map of size 64×64×16 to output the sixth feature map of size 64×64×16, which mainly contains shallow information such as texture and shape of the breast image. The first branch DP1 is used to extract shallow features of the image.
[0043] The processing flow of the second branch SP1 is to use the first feature map with a size of 64×64×16 as the input of the second branch SP1. First, it passes through the convolution layer Conv4, the convolution kernel size of the convolution layer Conv4 is 1×1, the number of channels is 8, the step size is 1, and the padding is 0. The convolution layer Conv2 outputs the seventh feature map of dimension 64×64×8; the seventh feature map of dimension 64×64×8 is used as the input of the deep convolution layer DW2, the convolution kernel size of the deep convolution layer DW2 is 3×3, the number of channels is 8, the step size is 1, and the padding is 1, and the deep convolution layer DW2 outputs the eighth feature map of dimension 64×64×8; the eighth feature map of dimension 64×64×8 is used as the input of the convolution layer Conv5, the convolution kernel size of the convolution layer Conv5 is 1×1, the number of channels is 16, the step size is 1, and the padding is 0, and the convolution layer Conv5 outputs the ninth feature map of dimension 64×64×16, which contains deep information such as global structure. In view of the low contrast between the tumor and the background, the features extracted by the second branch SP1 and the third branch SP2 are superimposed and fused to reduce the loss of information;
[0044] The processing flow of the third branch SP2 is to use the first feature map of size 64×64×16 as the input of the third branch SP2. First, it passes through the convolution layer Conv6, the convolution kernel size of the convolution layer Conv6 is 1×1, the number of channels is 4, the step size is 1, and the padding is 0. The convolution layer Conv6 outputs the tenth feature map of dimension 64×64×4; the tenth feature map of dimension 64×64×4 is used as the input of the convolution layer Conv7, the convolution kernel size of the convolution layer Conv7 is 1×3, the number of channels is 8, the step size is 1, and the padding is 0. The convolution layer Conv7 outputs the eleventh feature map of dimension 64×64×8; the eleventh feature map of dimension 64×64×8 is used as the input of the convolution layer Conv8, the convolution kernel size of the convolution layer Conv8 is 3×1, the number of channels is 8, the step size is 1, and the padding is 0. , the convolution layer Conv8 outputs the twelfth feature map of dimension 64×64×8; the twelfth feature map of dimension 64×64×8 is used as the input of the convolution layer Conv9, the convolution kernel size of the convolution layer Conv9 is 1×3, the number of channels is 8, the step size is 1, and the padding is 0, and the convolution layer Conv9 outputs the thirteenth feature map of dimension 64×64×8; the thirteenth feature map of dimension 64×64×8 is used as the input of the convolution layer Conv10, the convolution kernel size of the convolution layer Conv10 is 3×1, the number of channels is 16, the step size is 1, and the padding is 0, and the convolution layer Conv10 outputs the fourteenth feature map of dimension 64×64×16, which contains the deep features of the breast image. The third branch SP2 is used to extract the deep features of the image. The main structure of the network uses asymmetric 1×3 and 3×1 filters instead of traditional n×n filters, which can reduce the parameter size and calculation amount, can downscale, introduce more nonlinearity, and improve the generalization effect;
[0045] The processing flow of the fourth branch SP3 is to first divide the channels of the first feature map into 2 groups by channel split, wherein the first feature map of the first group of 64×64×8 passes through the convolution layer Conv11, the convolution kernel size of the convolution layer Conv11 is 1×1, the number of channels is 8, the step size is 1, and the padding is 0, and the convolution layer Conv11 outputs the fifteenth feature map of the dimension 64×64×8;
[0046] The fifteenth feature map with a size of 64×64×8 is used as the input of the deep convolution layer DW3. The convolution kernel size of the deep convolution layer DW3 is 3×3, the number of channels is 16, the step size is 1, and the padding is 1. The deep convolution layer DW3 outputs the sixteenth feature map with a dimension of 64×64×16; the sixteenth feature map with a size of 64×64×16 is used as the input of the convolution layer Conv12. The convolution kernel size of the convolution layer Conv12 is 1×1, the number of channels is 8, the step size is 1, and the padding is 0. The convolution layer Conv12 outputs the seventeenth feature map with a dimension of 64×64×8; the seventeenth feature map is fused with the first feature map of the second group of 64×64×8 to output the eighteenth feature map with a dimension of 64×64×16; finally, the eighteenth feature map is channel shuffled to output the nineteenth feature map with a dimension of 64×64×16, which contains the spatial resolution of the breast image and more detail information. The fourth branch SP3 uses channel rearrangement to enrich the information of each channel, further promoting the mixing of features and the flow of information.
[0047] The channel and spatial attention module includes a channel attention module and a spatial attention module. The channel attention module is used to learn the channel attention weights of the channels in the deep features, and the channels of the deep features are screened according to the channel attention weights as channel features. The spatial attention module is used to learn the spatial attention weights of the spatial positions in the channel features, and the spatial positions of the channel features are screened according to the spatial attention weights as channel spatial features. The structure diagram of the channel and spatial attention module CSAB is as follows: Figure 4 shown.
[0048] Specifically, the ninth feature map of the second branch with a size of 64×64×16 and the fourteenth feature map of the third branch with a size of 64×64×16 are added, and the twentieth feature map with a dimension of 64×64×16 is output, and the twentieth feature map is input into the channel and spatial attention module, which is composed of a channel attention module and a spatial attention module. The channel attention module suppresses channels with less information by learning the channel attention weights in the channel, and the spatial attention module obtains the importance of each spatial position by learning the spatial attention weights, so that the network can identify key features by spatial position.
[0049] The processing flow of the channel attention module is to use the twentieth feature map with a size of 64×64×16 as the input of the channel attention module CAB, first pass through the maximum pooling layer MaxPool1 to output the twenty-first feature map with a dimension of 1×1×16, and at the same time use the twentieth feature map with a size of 64×64×16 as the input of the average pooling layer AvgPool1, and output the twenty-second feature map with a dimension of 1×1×16; then use the twenty-first feature map with a size of 1×1×16 as the input of the convolution layer Conv13, the convolution kernel size of the convolution layer Conv13 is 1×1, the number of channels is 1, the step size is 1, the padding is 0, and the twenty-third feature map of 1×1×1 is output; the twenty-third feature map with a size of 1×1×1 is used as the input of the convolution layer Conv14, the convolution kernel size of the convolution layer Conv14 is 1×1, the number of channels is 16, the step size is 1, and the padding is 0, output the twenty-fourth feature map of size 1×1×16; use the twenty-second feature map of size 1×1×16 as the input of the convolution layer Conv13, and output the twenty-fifth feature map of size 1×1×1; use the twenty-fifth feature map of size 1×1×1 as the input of the convolution layer Conv14, and output the twenty-sixth feature map of size 1×1×16; add the twenty-fourth feature map of size 1×1×16 and the twenty-sixth feature map of size 1×1×16, and output the twenty-seventh feature map of size 1×1×16; perform sigmoid activation on the twenty-seventh feature map of size 1×1×16, and multiply it with the twentieth feature map of size 64×64×16, and output the twenty-eighth feature map of size 64×64×16. The deep features contained in this feature map can reflect the positional relationship of breast lesions.
[0050] The processing flow of the spatial attention module is to use the 28th feature map with a size of 64×64×16 as the input of the spatial attention module SAB, first pass through the maximum pooling layer MaxPool2 to output the 29th feature map with a dimension of 64×64×1, and at the same time use the 28th feature map with a size of 64×64×16 as the input of the average pooling layer AvgPool2, and output the 30th feature map with a dimension of 64×64×1; then fuse the 29th feature map with a size of 64×64×1 and the 30th feature map with a size of 64×64×1, and output the 30th feature map with a size of 64×64×1. The thirty-first feature map of size 4×64×2 is used as the input of the convolution layer Conv15. The convolution kernel size of the convolution layer Conv15 is 3×3, the number of channels is 1, and the padding is 1. The convolution layer Conv15 outputs the thirty-second feature map of dimension 64×64×1. The thirty-second feature map of size 64×64×1 is sigmoid activated and multiplied with the twenty-eighth feature map of size 64×64×16 to output the thirty-third feature map (refined feature) of dimension 64×64×16. The deep features contained in this feature map can reflect the spatial distribution of breast lesions.
[0051] The deep feature extraction module is used to extract deep features from shallow features by increasing the number of channels, fuse channel space features with deep features as classification features, and classify contour area images according to the classification features.
[0052] The processing flow of the deep feature extraction module is to add the 19th feature map of the fourth branch SP3 with a size of 64×64×16 and the 33rd feature map of the channel space attention module with a size of 64×64×16, and output the 34th feature map of the size of 64×64×16. Then, the 34th feature map of the size of 64×64×16 is fused with the 6th feature map of the first branch DP1 with a size of 64×64×16, and the 35th feature map of the size of 64×64×32 is output, and the 35th feature map of the size of 64×64×32 is used as the input of the deep feature extraction module DFEB. First, after the maximum pooling MaxPool3, the pooling window is 2×2, the step size is 2, and the output dimension is 32×32×32 of the 36th feature map; the 36th feature map of size 32×32×32 is used as the input of the convolution layer Conv16, the convolution kernel size of the convolution layer Conv16 is 1×1, the number of channels is 128, the step size is 1, and the padding is 0. The convolution layer Conv16 outputs the 37th feature map of dimension 32×32×128; the 37th feature map of size 32×32×128 is used as the input of the deep convolution layer DW4, the convolution kernel size of the deep convolution layer DW4 is 3×3, the number of channels is 1 28, stride 2, padding 0, the deep convolution layer DW4 outputs the 38th feature map of dimension 16×16×128; the 38th feature map of dimension 16×16×128 is used as the input of the squeeze and excitation block SE, and the squeeze and excitation block SE outputs the 39th feature map of dimension 16×16×128; the 39th feature map of dimension 16×16×128 is used as the input of the convolution layer Conv17, the convolution kernel size of the convolution layer Conv17 is 1×1, the number of channels is 64, the stride is 1, and the padding is 0, and the convolution layer Conv3 outputs the 40th feature map of dimension 16×16×64. The deep feature extraction module DFEB further extracts deep features by increasing the number of channels, and finally uses the extracted deep features for the classification of breast cancer.
[0053] Finally, the 40th feature map with a size of 16×16×64 is input into the global average pooling layer GAP, and the 41st feature map with a dimension of 1×1×64 is output. The 41st feature map is used as the input of the fully connected layer FC1. The number of neurons in the fully connected layer is equal to 4, and the 42nd feature map with a dimension of 4 is output. The 42nd feature map is then softmax processed and the probability of the four classification results is output.
[0054] S3, train the multi-branch CNN network model based on the training set, and obtain the trained multi-branch CNN network model,
[0055] S4, obtaining a breast MRI image to be classified, classifying the breast MRI image to be classified based on the trained multi-branch CNN network model and obtaining a classification result, wherein the classification result includes four types, namely 0, 1+, 2+, and 3+, 0 and 1+ are negative, and 3+ is positive. Her2+ indicates that it is positive or negative and cannot be determined.
[0056] Overall beneficial effects:
[0057] The present invention provides a breast lesion MRI image recognition method based on a multi-branch CNN, which can automatically realize the four-category detection of breast MRI images, save manpower, and help imaging doctors better identify the type of lesions. An innovative deep network model based on a multi-branch convolutional neural network (CNN) is proposed. The model captures richer and higher-order breast cancer MRI features by processing image information at different levels in parallel, and focuses on lesion objects. The deep network model can adapt to small-scale data sets, ensure model performance, avoid overfitting problems, and ensure that the model can achieve good generalization ability even with limited data.
[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for breast lesion MRI image recognition based on multi-branch CNN, It is characterized in that include S1. Obtain a breast MRI image dataset, have a doctor outline the tumor contour in the breast MRI image, crop the breast MRI image based on the tumor contour to obtain a contour area image, perform data enhancement on the contour area image, and construct a training set based on the contour area image after data enhancement. S2. Construct a multi-branch CNN network model, which includes a multi-branch module, a channel and spatial attention module, and a deep feature extraction module. The multi-branch module includes at least two types of branches, one type of branch is used to extract shallow features of the contour area image, and the other type of branch is used to extract deep features of the contour area image. The channel and spatial attention module includes a channel attention module and a spatial attention module. The channel attention module is used to learn the channel attention weights of the channels in the deep features, and filter the channels of the deep features according to the channel attention weights as channel features. The spatial attention module is used to learn the spatial attention weights of the spatial positions in the channel features, and filter the spatial positions of the channel features according to the spatial attention weights as channel spatial features. The deep feature extraction module is used to extract deep features from shallow features by increasing the number of channels, and fuse the channel spatial features with the deep features as classification features. The contour area image is classified according to the classification features. S3, train the multi-branch CNN network model based on the training set, and obtain the trained multi-branch CNN network model, S4. Obtain a breast MRI image to be classified, classify the breast MRI image to be classified based on the trained multi-branch CNN network model and obtain a classification result.
2. According to claim 1, a method for recognizing breast lesions MRI images based on multi-branch CNN, It is characterized in that The multi-branch module is a four-branch module.
3. According to claim 2, a method for recognizing breast lesions MRI images based on multi-branch CNN, It is characterized in that The four-branch module includes a first branch DP1, a second branch SP1, a third branch SP2 and a fourth branch SP3. The first branch DP1 is used to extract shallow features of the contour area image, and the second branch SP1, the third branch SP2 and the fourth branch SP3 are used to extract deep features of the contour area image.
4. The method for breast lesion MRI image recognition based on multi-branch CNN according to claim 3, It is characterized in that The processing flow of the first branch DP1 in the four-branch module is to obtain a first feature map of the contour area image, wherein the first feature map is obtained by inputting the contour area image after data enhancement into the convolution layer Conv1 for processing, wherein the convolution kernel size of the convolution layer Conv1 is 3×3, the number of channels is 16, the step size is 1, and the padding is 1, and the size of the first feature map is 64×64×16. The first feature map is passed through the convolution layer Conv2 to obtain a second feature map, wherein the convolution kernel size of the convolution layer Conv2 is 1×1, the number of channels is 128, the step size is 1, and the padding is 0, and the dimension of the second feature map is 64×64×128. The second feature map of size 64×64×128 is used as the input of the deep convolution layer DW1, and the convolution kernel size of the deep convolution layer DW1 is 3×3, the number of channels is 12 8. The step size is 1 and the padding is 1. The deep convolution layer DW1 outputs the third feature map of dimension 64×64×128; the third feature map of dimension 64×64×128 is used as the input of the squeeze and excitation block SE, and the squeeze and excitation block SE outputs the fourth feature map of dimension 64×64×128; the fourth feature map of dimension 64×64×128 is used as the input of the convolution layer Conv3, the convolution kernel size of the convolution layer Conv3 is 1×1, the number of channels is 16, the step size is 1, and the padding is 0, and the convolution layer Conv3 outputs the fifth feature map of dimension 64×64×16; the first feature map of dimension 64×64×16 is short-connected with the fifth feature map of dimension 64×64×16 to output the sixth feature map of dimension 64×64×16. The processing flow of the second branch SP1 is to use the first feature map with a size of 64×64×16 as the input of the second branch SP1, first pass through the convolution layer Conv4, the convolution kernel size of the convolution layer Conv4 is 1×1, the number of channels is 8, the step size is 1, and the padding is 0, and the convolution layer Conv4 outputs the seventh feature map with a dimension of 64×64×8; use the seventh feature map with a size of 64×64×8 as the input of the deep convolution layer DW2, the convolution kernel size of the deep convolution layer DW2 is 3×3, the number of channels is 8, the step size is 1, and the padding is 1, and the deep convolution layer DW2 outputs the eighth feature map with a dimension of 64×64×8; use the eighth feature map with a size of 64×64×8 as the input of the convolution layer Conv5, the convolution kernel size of the convolution layer Conv5 is 1×1, the number of channels is 16, the step size is 1, and the padding is 0, and the convolution layer Conv5 outputs the ninth feature map with a dimension of 64×64×16; The processing flow of the third branch SP2 is to use the first feature map with a size of 64×64×16 as the input of the third branch SP2, first pass through the convolution layer Conv6, the convolution kernel size of the convolution layer Conv6 is 1×1, the number of channels is 4, the step size is 1, and the padding is 0, and the convolution layer Conv6 outputs the tenth feature map with a dimension of 64×64×4; the tenth feature map with a size of 64×64×4 is used as the input of the convolution layer Conv7, the convolution kernel size of the convolution layer Conv7 is 1×3, the number of channels is 8, the step size is 1, and the padding is 0, and the convolution layer Conv7 outputs the eleventh feature map with a dimension of 64×64×8; the eleventh feature map with a size of 64×64×8 is used as the input of the convolution layer Conv8, and the convolution layer Con The convolution kernel size of v8 is 3×1, the number of channels is 8, the step size is 1, and the padding is 0. The convolution layer Conv8 outputs the twelfth feature map with a dimension of 64×64×8; the twelfth feature map with a size of 64×64×8 is used as the input of the convolution layer Conv9, the convolution kernel size of the convolution layer Conv9 is 1×3, the number of channels is 8, the step size is 1, and the padding is 0. The convolution layer Conv9 outputs the thirteenth feature map with a dimension of 64×64×8; the thirteenth feature map with a size of 64×64×8 is used as the input of the convolution layer Conv10, the convolution kernel size of the convolution layer Conv10 is 3×1, the number of channels is 16, the step size is 1, and the padding is 0. The convolution layer Conv10 outputs the fourteenth feature map with a dimension of 64×64×16; The processing flow of the fourth branch SP3 is to first divide the channels of the first feature map into 2 groups by channel split, wherein the first group of 64×64×8 first feature maps passes through the convolution layer Conv11, the convolution kernel size of the convolution layer Conv11 is 1×1, the number of channels is 8, the step size is 1, and the padding is 0, and the convolution layer Conv11 outputs the fifteenth feature map with a dimension of 64×64×8; The fifteenth feature map with a size of 64×64×8 is used as the input of the deep convolution layer DW3. The convolution kernel size of the deep convolution layer DW3 is 3×3, the number of channels is 16, the step size is 1, and the padding is 1. The deep convolution layer DW3 outputs the sixteenth feature map with a dimension of 64×64×16; the sixteenth feature map with a size of 64×64×16 is used as the input of the convolution layer Conv12. The convolution kernel size of the convolution layer Conv12 is 1×1, the number of channels is 8, the step size is 1, and the padding is 0. The convolution layer Conv12 outputs the seventeenth feature map with a dimension of 64×64×8; The seventeenth feature map is fused with the first feature map of the second group of 64×64×8 to output the eighteenth feature map with a dimension of 64×64×16. Finally, the eighteenth feature map is channel shuffled to output the nineteenth feature map with a dimension of 64×64×16.
5. The method for breast lesion MRI image recognition based on multi-branch CNN according to claim 1, It is characterized in that The classification results include four types, namely 0, 1+, 2+, and 3+. 0 and 1+ are negative, 3+ is positive, and Her2+ indicates positive or negative cannot be determined.