Breast cancer pathology image grading method based on multi-neural network fusion

Abstract

The present invention relates to a breast cancer pathology image grading method based on multi-neural network fusion, which belongs to the field of computer technology, and comprises the following steps: step one: cropping and screening the original pathology image, and performing image enhancement on it; step two: using the backbone network of ResNet-50, DenseNet-121 and InceptionNet to extract the low-level and high-level features of the pathology image, and using the global average pooling layer and the softmax layer to output the grading result of each neural network respectively; step three: using the splicing layer to splice the output features of the three neural networks, adding a fusion network by stacking, and using the fusion network to learn the output features of the neural network again; step four: using the multi-layer perception strategy to output the final pathology image grading result.

Description

Technical Field

[0001] The invention belongs to the field of computer technology and relates to a breast cancer pathology image grading method based on multi-neural network fusion. Background Art

[0002] Breast cancer is one of the most common cancers in women, with about 10% of women suffering from it every year. Due to its high mortality rate, early detection and evaluation have become indispensable means. Although common technologies such as breast ultrasound, ductal endoscopy, ultrasound-guided breast puncture biopsy and X-ray-guided localization biopsy are currently used in breast cancer detection, histopathological analysis is still the gold standard for its diagnosis. The internationally used breast cancer histological grading system is the Nottingham histological grading system, which defines three different levels of malignancy through a semi-quantitative comprehensive evaluation of the proportion of glandular duct formation, nuclear pleomorphism and mitotic count of invasive ductal carcinoma (IDC) of the breast: at level 1, cancer cells are close to the corresponding normal tissue of origin and have a low degree of malignancy; at level 3, cancer cells are very different from the corresponding normal tissue, poorly differentiated, and highly malignant; and level 2 is between level 1 and level 3, with an intermediate degree of malignancy. In clinical practice, if breast cancer can be accurately graded before surgery, it is of great significance for the treatment and prognosis of patients. However, in actual work, pathologists may be influenced by subjective factors, resulting in inefficient grading or misdiagnosis.

[0003] In recent years, with the continuous development of deep learning technology, it has achieved success in a large number of medical image problems. Compared with traditional machine learning, deep learning technology can capture the feature information of the entire image more accurately and efficiently. As for the convolutional neural network (CNN), it has been widely used in tissue pathology images. CNN can extract shallow and deep features in the image through the convolution layer. Compared with the classification of breast cancer pathology images, the grading task is still challenging. Due to the complexity of pathological images and the subtle differences between each level of images, although a single CNN has lower complexity and calculation parameters, it cannot achieve good grading results for complex pathological images. Model fusion technology can fully extract effective features in complex images by combining multiple different sub-networks, so that the network has better recognition and classification performance. Summary of the invention

[0004] In view of this, the object of the present invention is to provide a breast cancer pathology image grading method based on multi-neural network fusion.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A breast cancer pathology image grading method based on multi-neural network fusion includes the following steps:

[0007] Step 1: cropping and screening the original pathological image, and performing image enhancement on it;

[0008] Step 2: Use the backbone networks of ResNet-50, DenseNet-121, and InceptionNet to extract low-level and high-level features of pathological images, and use the global average pooling layer and softmax layer to output the classification results of each neural network respectively;

[0009] Step 3: Use the concatenation layer to concatenate the output features of the three neural networks, add a fusion network by stacking, and use the fusion network to learn the output features of the neural network again;

[0010] Step 4: Use multi-layer perception strategy to output the final pathological image grading results.

[0011] Furthermore, the cropping and screening of the original pathological image in step 1 includes: cropping the original pathological image into a patch image suitable for the neural network input size, and then screening out slices with labels and cell areas less than 50% in the patch image.

[0012] Furthermore, step one also includes: downsampling the screened pathological images, enriching the number of pathological images by random rotation, random change of image offset, vertical and horizontal mirror flipping, and finally sending the processed image data into the fusion network after normalization.

[0013] Furthermore, the main structures of InceptionNet, ResNet-50 and DenseNet-121 networks were used to learn the features of pathological images, where:

[0014] InceptionNet uses a 1×1 convolution bottleneck layer to reduce the number of input channels;

[0015] ResNet-50 uses a residual structure to extract deep features of the image while retaining shallow features to form the global features of the image;

[0016] DenseNet-121 repeatedly uses the features extracted from each layer of the network and adds a transition layer to reduce the number of unnecessary features;

[0017] Each sub-network uses a global average pooling layer to further reduce the dimensionality of the feature map, and then uses a softmax layer to normalize the network's predictions.

[0018] Further, the image input to the network is defined as Branch 1 uses the InceptionNet network and asymmetric convolution. The formula is as follows:

[0019] I m =InceptionNet(F m )

[0020] Among them I m is the output feature of the InceptionNet network;

[0021] Branch 2 uses the ResNet-50 network, which uses a special residual structure to extract deeper features R m :

[0022] R m =ResNet(F m )

[0023] Branch 3 uses the DenseNet-121 network to output the extracted features D m :

[0024] D m =DenseNet(F m )

[0025] Next, the features of each branch are subjected to global average pooling GAP, and then the classification results of each sub-network are obtained through the softmax layer, which is calculated as follows:

[0026]

[0027]

[0028]

[0029] The softmax function can be expressed as:

[0030]

[0031] where z i is the output value of the ith category, and C is the number of categories.

[0032] Furthermore, in step 3, the output features of the three sub-networks are concatenated together and calculated as follows:

[0033]

[0034] in b is the batch size for model training, w is and The size of the last dimension; n is the number of sub-models;

[0035] Design two adaptive weight matrices W x and Wy To fuse the features of the sub-networks, the fusion process is as follows:

[0036] H x =ReLU(H m ×W x +b x )

[0037] H y =ReLU(H x ×W y +b y )

[0038] in is the bias vector of the fully connected layer x, and d is the output dimension of the fully connected layer x; after the ReLU activation function, we get H x As the input of the fully connected layer y, the output of the fully connected layer y is obtained in the same way m is the number of classifications.

[0039] Furthermore, in step 4, four classification results are output after normalization through the softmax layer. The calculation results are as follows:

[0040] y=softmax(H y )

[0041] The model is trained using the back propagation algorithm. Through each round of training, W x and W y It can be continuously adjusted and optimized; the optimized model is obtained by minimizing the loss function, and the loss function adopts the cross entropy loss function as follows:

[0042]

[0043] Where M is the number of levels, y ic is the sample function (0 or 1), p ic is the predicted probability that sample i belongs to category c.

[0044] The beneficial effects of the present invention are:

[0045] 1) Use ResNet-50, DenseNet-121 and InceptionNet to fully extract high-level and low-level features of pathological images;

[0046] 2) Use the fusion network to learn the output features of the neural network again, achieve feature information interaction and model fusion, and eliminate the uncertainty of a single model;

[0047] 3) Use the designed adaptive weight matrix to optimize the performance of the model. By using ResNet-50, DenseNet-121 and InceptionNet, the high-level and low-level features of the pathological image can be fully extracted, and the output features of the neural network can be repeatedly learned through the fusion network to achieve feature information interaction and model fusion, eliminating the uncertainty of a single model, so that the present invention can more accurately grade breast cancer pathological images.

[0048] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0050] Figure 1 The process of grading breast cancer pathology images based on multi-neural network fusion;

[0051] Figure 2 This is a model diagram of a breast cancer pathology image grading system based on multi-neural network fusion;

[0052] Figure 3 It is the pathological image preprocessing module. DETAILED DESCRIPTION

[0053] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0054] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0055] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0056] like Figure 1 As shown, the present invention provides a breast cancer pathology image grading method based on multi-neural network fusion, such as Figure 2 The system model diagram used in this method is shown, which specifically includes:

[0057] Preprocessing module: Crop the original pathological images to a size suitable for neural network input and filter out images with more than 50% blank areas. Perform image enhancement on the filtered images by using random rotation, random translation, and random flipping techniques;

[0058] Feature extraction module: Use the backbone networks of ResNet-50, DenseNet-121, and InceptionNet to extract low-level and high-level features of pathological images, and use the global average pooling layer and softmax layer to output the classification results of each neural network respectively;

[0059] Fusion network module: Use the splicing layer to splice the output features of the three neural networks, add a fusion network by stacking, and use the fusion network to learn the output features of the neural network again to achieve feature information interaction and model fusion;

[0060] Image grading module: uses a multi-layer perception strategy to output the final pathological image grading results to improve the classification performance of the model.

[0061] Optionally, in a preprocessing module, such as Figure 3As shown in the figure, since the size of each original pathological image ranges from 1000×1000 pixels to 2048×1536 pixels, it cannot be directly sent to the model for training, so it needs to be cropped. In order to adapt to the deep learning model, the original image is first cropped into a patch image with a resolution of 250×250 pixels. The patch images are then screened to remove those slices with labels and less than 50% cell area. The screened pathological images are then downsampled to 224×224 pixels to match the input size of the convolutional neural network and reduce memory and computational overhead. Data augmentation techniques are then used to enrich the number of pathological images through random rotation, random changes in image offset, and vertical and horizontal mirror flipping. The processed image data is then normalized and sent to the fusion network.

[0062] Optionally, in the feature extraction module, the main structures of InceptionNet, ResNet-50 and DenseNet-121 networks are used to learn the features of pathological images. InceptionNet uses a bottleneck layer (1×1 convolution) to reduce the number of input channels, thereby reducing the amount of calculation. In order to aggregate in dimension, when performing convolution calculations in the model, the width and height remain unchanged except that the number of data processing channels will change; ResNet-50 uses a residual structure that can extract deep features of the image while retaining shallow features to form global features of the image; DenseNet-121 can repeatedly use the features extracted by each layer of the network, and add a transition layer to reduce the number of unnecessary features and reduce the complexity of the model. Each sub-network uses a global average pooling layer to further reduce the dimension of the feature map, and then uses a softmax layer to normalize the network's predicted values, so that they can independently process pathological images and output different grading results, simulating different doctors' different diagnosis results for the same disease. While achieving key feature learning, it can prevent unnecessary loss of non-critical information and improve the learning ability of the model.

[0063] Therefore, the present invention defines the image input to the network as Branch 1 uses the InceptionNet network, which uses asymmetric convolution and can control the number of channels, reduce redundant information in the features, and make the network lightweight, as shown in the formula.

[0064] I m =InceptionNet(F m )

[0065] Among them I m It is the output feature of the InceptionNet network.

[0066] Branch 2 uses the ResNet-50 network, which uses a special residual structure to solve the problem of network gradient disappearance, enabling it to extract deeper features R m :

[0067] R m =ResNet(F m )

[0068] Branch 3 uses the DenseNet-121 network to output the extracted features D m :

[0069] D m =DenseNet(F m )

[0070] Next, the features of each branch are subjected to global average pooling (GAP) and then to the softmax layer to obtain the classification results of each sub-network, which is calculated as follows:

[0071]

[0072]

[0073]

[0074] The softmax function can be expressed as:

[0075]

[0076] where z i is the output value of the ith category, and C is the number of categories.

[0077] Optionally, in the fusion network module, since each sub-network extracts different features from the pathological image, the grading results are also different, making it difficult to ensure objectivity and accuracy. To this end, a fusion network is added after the backbone network in a stacking manner, and the fusion network is used to learn the output features of the spliced ​​three sub-networks to achieve feature information interaction and model fusion of the sub-networks.

[0078] The output features of the three sub-networks need to be concatenated together and calculated as follows:

[0079]

[0080] in b is the batch size for model training, w is and The size of the last dimension. n is the number of sub-models. After concatenating the features of the sub-networks, two adaptive weight matrices W are designed.x and W y To fuse the features of the sub-networks. The fusion process is as follows:

[0081] H x =ReLU(H m ×W x +b x )

[0082] H y =ReLU(H x ×W y +b y )

[0083] in is the bias vector of the fully connected layer x, and d is the output dimension of the fully connected layer x. After the ReLU activation function, we get H x As the input of the fully connected layer y, the output of the fully connected layer y is obtained in the same way m is the number of classifications.

[0084] Optionally, in the image grading module, four grading results are output after normalization through the softmax layer. The calculation results are as follows:

[0085] y=softmax(H y )

[0086] The model is trained using the back propagation algorithm. Through each round of training, W x and W y It can be continuously adjusted and optimized. The optimized model is obtained by minimizing the loss function. The loss function adopts the cross entropy loss function as follows:

[0087]

[0088] Where M is the number of levels, y ic is the sample function (0 or 1), p ic is the predicted probability that sample i belongs to category c.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A breast cancer pathology image grading method based on multi-neural network fusion, characterized by: The following steps are involved: Step 1: cropping and screening the original pathological image, and performing image enhancement on it; Step 2: Use the backbone networks of ResNet-50, DenseNet-121, and InceptionNet to extract low-level and high-level features of pathological images, and use the global average pooling layer and softmax layer to output the classification results of each neural network respectively; Define the image input to the network as Branch 1 uses the InceptionNet network and asymmetric convolution. The formula is as follows: I m =InceptionNet(F m ) Among them I m is the output feature of the InceptionNet network; Branch 2 uses the ResNet-50 network, which uses a residual structure to extract deeper features R m : R m =ResNet(F m ) Branch 3 uses the DenseNet-121 network to output the extracted features D m : <h2 style=";text-align:left;direction:ltr">D<h2 style=";text-align:left;direction:ltr"> m <h2 style=";text-align:left;direction:ltr"> =DenseNet(F<h2 style=";text-align:left;direction:ltr"> m <h2 style=";text-align:left;direction:ltr"> ) Next, the features of each branch are subjected to global average pooling GAP, and then the classification results of each sub-network are obtained through the softmax layer, which is calculated as follows: The softmax function can be expressed as: where z i is the output value of the i-th category, and C is the number of classified categories; Step 3: Use the concatenation layer to concatenate the output features of the three neural networks, add a fusion network by stacking, and use the fusion network to learn the output features of the neural network again; The output features of the three sub-networks are concatenated together and calculated as follows: in b is the batch size for model training, w is and The size of the last dimension; n is the number of sub-models; Design two adaptive weight matrices W x and W y To fuse the features of the sub-networks, the fusion process is as follows: A x =ReLU(H m ×W x +b x ) A y =ReLU(H x ×W y +b y ) in is the bias vector of the fully connected layer x, and d is the output dimension of the fully connected layer x; after the ReLU activation function, we get H x As the input of the fully connected layer y, the output of the fully connected layer y is obtained in the same way m is the number of classifications; Step 4: Use multi-layer perception strategy to output the final pathological image grading results.

2. The method for grading breast cancer pathology images based on multi-neural network fusion according to claim 1, characterized in that: The cropping and screening of the original pathological image in step 1 includes: cropping the original pathological image into a patch image suitable for the neural network input size, and then screening out slices with labels and cell areas less than 50% in the patch image.

3. The method for grading breast cancer pathology images based on multi-neural network fusion according to claim 1, characterized in that: Step one also includes: downsampling the screened pathological images, enriching the number of pathological images by random rotation, random changes in image offset, vertical and horizontal mirror flipping, and finally sending the processed image data into the fusion network after normalization.

4. The method for grading breast cancer pathology images based on multi-neural network fusion according to claim 1, characterized in that: The main structures of InceptionNet, ResNet-50 and DenseNet-121 networks are used to learn the features of pathological images, where: InceptionNet uses a 1×1 convolution bottleneck layer to reduce the number of input channels; ResNet-50 uses a residual structure to extract deep features of the image while retaining shallow features to form the global features of the image; DenseNet-121 repeatedly uses the features extracted from each layer of the network and adds a transition layer to reduce the number of unnecessary features; Each sub-network uses a global average pooling layer to further reduce the dimensionality of the feature map, and then uses a softmax layer to normalize the network's predictions.

5. The method for grading breast cancer pathology images based on multi-neural network fusion according to claim 1, characterized in that: In step 4, four classification results are output after normalization through the softmax layer. The calculation results are as follows: and softmax(H y ) The model is trained using the back propagation algorithm. Through each round of training, W x and W y It can be continuously adjusted and optimized; the optimized model is obtained by minimizing the loss function, and the loss function adopts the cross entropy loss function as follows: Where M is the number of levels, y ic is the sample function 0 or 1, p ic is the predicted probability that sample i belongs to category c.

Images