A breast cancer pathology image processing method and system based on deep learning

Abstract

The invention proposes a breast cancer pathology image processing method and system based on deep learning, belonging to the field of image processing technology. Hot spot areas are identified on an original image, and the identified hot spot areas are sliced ​​and cut to obtain a picture block set; the picture blocks in the set are input into a trained improved DenseNet network for color recognition to obtain color categories and corresponding probability values ​​of the picture blocks; the picture blocks in the set are input into a trained improved VGG16 network small target classifier model for envelope classification recognition to obtain envelope classification results of the picture blocks; the percentage and coloring intensity of cell membrane coloring are scored according to the color recognition result and the envelope classification result to obtain a final rating result; the invention uses a deep learning model to intelligently identify hot spot areas, performs low-dimensional color classification and envelope classification on the hot spot areas, obtains breast cancer pathology image processing results, improves the efficiency of image processing, and provides more accurate data for future diseases.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a breast cancer pathology image processing method and system based on deep learning. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of deep learning and computer vision, target classification, segmentation and detection technologies have been widely used in the field of computer-aided diagnosis. As an important step in image understanding, target detection is one of the core issues in the field of computer vision. Its main task is to delineate all regions of interest from the image and determine their location and size. In China, the health burden of cancer is increasing year by year, with more than 1.6 million people diagnosed with cancer and 1.2 million deaths from cancer each year. Like most other countries, breast cancer has become the most common cancer in Chinese women. The number of new cases and deaths of breast cancer in China each year accounts for 12.2% and 9.6% of the world's total, respectively. Correct detection and assessment of HER2 protein expression and gene amplification status in breast cancer is crucial to the clinical treatment and prognosis of breast cancer. HER2 test results not only involve whether the patient is suitable for HER2 targeted therapy, but also guide the selection of endocrine therapy and chemotherapy regimens and prognosis evaluation.

[0004] The traditional manual identification method of Her2 indicators is inefficient and is very limited by the professional knowledge and energy of pathologists. At present, due to the uneven distribution of medical resources in my country, there is an extreme shortage of pathologists, especially in grassroots hospitals. Many laboratories in grassroots hospitals do not have the conditions for HER2 testing, which leads to some HER2-positive patients not being able to receive timely and effective targeted treatment. Therefore, it is very necessary and of great significance to adopt intelligent detection methods to realize intelligent HER2 detection. Summary of the invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a breast cancer pathology image processing method and system based on deep learning. Through the deep learning model, the hotspot area is intelligently identified, and the hotspot area is subjected to low-dimensional color classification and envelope classification to obtain the breast cancer pathology image processing result, thereby improving the efficiency of image processing and providing more accurate data for future diseases.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0007] The first aspect of the present invention provides a method for processing breast cancer pathology images based on deep learning;

[0008] A breast cancer pathology image processing method based on deep learning, comprising:

[0009] Identify the hot spots in the original image, slice and dice the identified hot spots to obtain a set of image blocks;

[0010] Input the image blocks in the set into the trained improved DenseNet network for color recognition to obtain the color recognition results of the image blocks;

[0011] Input the image blocks in the set into the trained improved VGG16 network small object classifier model for envelope classification to obtain the envelope classification results of the image blocks;

[0012] According to the color recognition results and envelope classification results, the percentage and coloring intensity of the cell membrane coloring are scored to obtain the final rating result.

[0013] Furthermore, the cancer nest hotspot area recognition model based on the Mask-R-CNN algorithm was used to identify the hotspot area.

[0014] Furthermore, the slicing is to segment the identified hot spot area from the original image to obtain a slice image;

[0015] The block cutting is to divide the slice image into pieces according to the specified size to obtain a set of picture blocks.

[0016] Furthermore, the improved DenseNet network cancels the network layer directly connected to the DenseNet part.

[0017] Furthermore, the improvement of VGG16 is to replace the softmax classifier with sigmoid.

[0018] Furthermore, the color recognition result is a color category and a corresponding probability value;

[0019] The envelope classification result is a probability value of the envelope.

[0020] Further, the scores are divided into 0, 1+, 2+ and 3+.

[0021] A second aspect of the present invention provides a breast cancer pathology image processing system based on deep learning.

[0022] A breast cancer pathology image processing system based on deep learning, including a hotspot area module, a color classification module, an envelope classification module, and a scoring and rating module:

[0023] The hotspot area module is configured to: identify the hotspot areas of the original image, slice and dice the identified hotspot areas to obtain a set of image blocks;

[0024] The color classification module is configured to: input the image blocks in the set into the trained improved DenseNet network for color recognition, and obtain the color category of the image blocks and the corresponding probability value;

[0025] The envelope classification module is configured to: input the image blocks in the set into the trained improved VGG16 network small object classifier model for envelope classification and recognition, and obtain the envelope classification results of the image blocks;

[0026] The scoring and rating module is configured to score the percentage and intensity of cell membrane coloring based on the color recognition results and envelope classification results to obtain a final rating result.

[0027] The third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in a method for processing breast cancer pathology images based on deep learning as described in the first aspect of the present invention.

[0028] The fourth aspect of the present invention provides an electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the method for processing breast cancer pathology images based on deep learning as described in the first aspect of the present invention are implemented.

[0029] One or more of the above technical solutions have the following beneficial effects:

[0030] The present invention adopts the calculation method of computer vision and applies computer imaging to the processing of pathological data to perform cancer nest hotspot area recognition, color recognition, and cell envelope recognition, so as to further determine the composition of Her2 related indicators and perform ratings, realize end-to-end training, combine traditional algorithms with deep learning algorithms, reduce the amount of calculation, improve processing efficiency, and improve the accuracy of image processing.

[0031] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0033] Figure 1 This is a flow chart of the method of the first embodiment.

[0034] Figure 2 It is a system structure diagram of the second embodiment.

[0035] Figure 3 This is a diagram of the improved DenseNet network structure in the first embodiment.

[0036] Figure 4 This is a diagram of the improved VGG16 network structure in the first embodiment. DETAILED DESCRIPTION

[0037] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0038] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0039] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0040] Embodiment 1

[0041] This embodiment discloses a method for processing breast cancer pathology images based on deep learning;

[0042] like Figure 1 As shown, a breast cancer pathology image processing method based on deep learning includes:

[0043] S1: Identify the hotspot areas of the original image, slice and dice the identified hotspot areas to obtain a set of image blocks. The specific steps are as follows:

[0044] S1-1: Use the trained cancer nest hotspot area recognition model to identify the hotspot area of ​​the original image to obtain a mask image of the cancer nest hotspot area, and superimpose the original image and the mask image to obtain a cancer nest hotspot area image.

[0045] Among them, the cancer nest hotspot area recognition model is built based on the Mask-R-CNN algorithm.

[0046] S1-2: Slice the superimposed image, segment the cancer nest hotspot area image to obtain a slice image, and cut the slice image into blocks according to a specified size to obtain a set of image blocks;

[0047] The cutting is to start from the upper left corner of the slice image, and gradually move (slide) the 128*128 box from left to right and from top to bottom, and cut out the local image corresponding to the 128*128 box as a picture block, thereby obtaining a series of picture blocks.

[0048] S2: Input the image blocks in the set into the trained improved DenseNet network for color recognition to obtain the color category of the image blocks and the corresponding probability value;

[0049] The improved DenseNet network structure is as follows Figure 3 As shown in the figure, compared with the original network, some directly connected network layers of DenseNet are removed. In this new architecture, the input of each layer is not completely composed of the feature maps of all previous layers, but part of its output is transmitted to each subsequent layer, which effectively solves the gradient vanishing problem, strengthens feature propagation, supports feature reuse, and greatly reduces the number of parameters.

[0050] The improved DenseNet network uses a 3*3 convolution kernel with a step size of 2 to convolve the image data and extract relevant feature data; each layer performs "zero padding" on the input features when performing convolution. After convolution operations of multiple convolutional layers, a feature mapping image with a depth of h of 64 is finally obtained.

[0051] The color categories include light, medium and dark.

[0052] S3: Input the image blocks in the set into the trained improved VGG16 network small object classifier model for envelope classification to obtain the envelope classification results of the image blocks;

[0053] Improved VGG16 network, replacing the softmax classifier with a sigmoid classifier.

[0054] The network structure of VGG16 is as follows Figure 4 As shown in the figure, it consists of six parts: the first and second parts consist of two convolutional layers and one maxpool, and the output of each convolutional layer is connected to a ReLu activation function; the third to fifth parts consist of four convolutional layers and one maxpool layer.

[0055] The input of the model is a 227×227 image. The size of the first convolution kernel is set to 9×9, the convolution kernel step is 1, and the size of all other convolution kernels is 3×3, with a step size of 1; the size of all maxpool pooling layers is 2×2, with a step size of 2; the sixth part is the expanded fully connected layer, which consists of 3 fully connected layers with 4096, 4096, and 1000 nodes respectively.

[0056] The process of training the model is as follows: the network is trained on the ImageNet large natural image dataset. The features extracted by the convolutional layer are passed to the sigmoid classifier after passing through the fully connected layer, and the predicted label is output. The prediction function of sigmoid is Among them, sign is sigmoid, v is weight, d is input data, c is bias, and then predict label The true label y i In comparison, the classifier uses the cross entropy loss function, expressed as:

[0057]

[0058] Among them, the Sgn function is a step function, a symbolic function in mathematics or a return function in computer language. The Sgn function returns a Variant (Integer) indicating the positive and negative signs of the parameters. After multiple iterations, all parameters in the neural network are optimized to reduce the loss function J. hinge The value of and save the trained parameters, which are weights;

[0059] Based on the modified VGG16 network, the network is trained using the preprocessed training data to obtain the envelope classifier model;

[0060] The envelope classification result is the probability value of the envelope. Two thresholds w1 and w2 are set. w2>w1. If the probability value is less than w1, it is an incomplete envelope. If the probability value is greater than w2, it is a complete envelope. The envelope between w1 and w2 is uncertain.

[0061] S4: Based on the color recognition results and envelope classification results, the percentage and coloring intensity of the cell membrane are scored to obtain the final rating result.

[0062] The judgment levels are the percentage of cell membrane staining and the staining intensity score, which are divided into 0, 1+, 2+ and 3+. The specific scoring rules are shown in the following table.

[0063] Full Envelope Incomplete Envelope Uncertain Envelope Dark 3+ 2+ uncertain medium color 2+ 1+ uncertain Light 2+ 0 uncertain

[0064] Embodiment 2

[0065] This embodiment discloses a breast cancer pathology image processing system based on deep learning;

[0066] like Figure 2 As shown in the figure, a breast cancer pathology image processing system based on deep learning includes a hotspot area module, a color classification module, an envelope classification module and a scoring and rating module:

[0067] The hotspot area module is configured to: identify the hotspot areas of the original image, slice and dice the identified hotspot areas to obtain a set of image blocks;

[0068] The color classification module is configured to: input the image blocks in the set into the trained improved DenseNet network for color recognition, and obtain the color category of the image blocks and the corresponding probability value;

[0069] The envelope classification module is configured to: input the image blocks in the set into the trained improved VGG16 network small object classifier model for envelope classification, and obtain the envelope classification results of the image blocks;

[0070] The scoring and rating module is configured to score the percentage and intensity of cell membrane coloring based on the color recognition results and envelope classification results to obtain a final rating result.

[0071] Embodiment 3

[0072] The purpose of this embodiment is to provide a computer-readable storage medium.

[0073] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in a method for processing breast cancer pathology images based on deep learning as described in Example 1 of the present disclosure.

[0074] Embodiment 4

[0075] The purpose of this embodiment is to provide an electronic device.

[0076] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in a method for processing breast cancer pathology images based on deep learning as described in Example 1 of the present disclosure are implemented.

[0077] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0078] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0079] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0080] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0081] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0082] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A breast cancer pathology image processing method based on deep learning, characterized in that: include: Identify the hot spots in the original image, slice and dice the identified hot spots to obtain a set of image blocks; Input the image blocks in the set into the trained improved DenseNet network for color recognition to obtain the color recognition results of the image blocks; Input the image blocks in the set into the trained improved VGG16 network small object classifier model for envelope classification to obtain the envelope classification results of the image blocks; The improvement of VGG16 is to replace the softmax classifier with sigmoid; The network structure of VGG16 consists of six parts: the first and second parts consist of two convolutional layers and a maxpool, and the output of each convolutional layer is connected to a ReLu activation function; The third to fifth parts consist of four convolutional layers and one maxpool layer; The input of the model is a 227×227 image. The size of the first convolution kernel is set to 9×9, the convolution kernel step is 1, and the size of all other convolution kernels is 3×3, with a step size of 1; the size of all maxpool pooling layers is 2×2, with a step size of 2; the sixth part is the expanded fully connected layer, which consists of 3 fully connected layers with 4096, 4096, and 1000 nodes respectively; According to the color recognition results and envelope classification results, the percentage and coloring intensity of the cell membrane coloring are scored to obtain the final rating result.

2. A method for processing breast cancer pathology images based on deep learning as claimed in claim 1, characterized in that: The cancer nest hotspot area recognition model based on the Mask-R-CNN algorithm is used to identify the hotspot area.

3. A method for processing breast cancer pathology images based on deep learning as claimed in claim 1, characterized in that: The slicing is to segment the identified hot spot area from the original image to obtain a slice image; The block cutting is to divide the slice image into pieces according to the specified size to obtain a set of picture blocks.

4. A method for processing breast cancer pathology images based on deep learning as claimed in claim 1, characterized in that: The improved DenseNet network cancels the network layer directly connected to the DenseNet part.

5. A method for processing breast cancer pathology images based on deep learning as claimed in claim 1, characterized in that: The color recognition result is the color category and the corresponding probability value; The envelope classification result is a probability value of the envelope.

6. A method for processing breast cancer pathology images based on deep learning as claimed in claim 1, characterized in that: The scores are 0, 1+, 2+ and 3+.

7. A breast cancer pathology image processing system based on deep learning, characterized by: Including hotspot area module, color classification module, envelope classification module and scoring and rating module: The hotspot area module is configured to: identify the hotspot areas of the original image, slice and dice the identified hotspot areas to obtain a set of image blocks; The color classification module is configured to: input the image blocks in the set into the trained improved DenseNet network for color recognition, and obtain the color category of the image blocks and the corresponding probability value; The envelope classification module is configured to: input the image blocks in the set into the trained improved VGG16 network small object classifier model for envelope classification, and obtain the envelope classification results of the image blocks; The improvement of VGG16 is to replace the softmax classifier with sigmoid; The network structure of VGG16 consists of six parts: the first and second parts consist of two convolutional layers and a maxpool layer, and the output of each convolutional layer is connected to a ReLu activation function; the third to fifth parts consist of four convolutional layers and a maxpool layer; The input of the model is a 227×227 image. The size of the first convolution kernel is set to 9×9, the convolution kernel step is 1, and the size of all other convolution kernels is 3×3, with a step size of 1; the size of all maxpool pooling layers is 2×2, with a step size of 2; the sixth part is the expanded fully connected layer, which consists of 3 fully connected layers with 4096, 4096, and 1000 nodes respectively; The scoring and rating module is configured to score the percentage and intensity of cell membrane coloring based on the color recognition results and envelope classification results to obtain a final rating result.

8. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the breast cancer pathology image processing method based on deep learning are implemented as described in any one of claims 1 to 6.

9. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the breast cancer pathology image processing method based on deep learning are implemented as described in any one of claims 1 to 6.

Images