A pathological image segmentation and grading method, device, equipment and medium

Abstract

The present application provides a pathological image segmentation and grading method, device and equipment, which generally processes the original pathological image containing specific pathological staining markers through HE-Norm to obtain H channel image and E channel image, and performs data enhancement and standardization processing respectively; inputs the image into the convolutional neural network model to automatically segment the lesion area to obtain the segmentation result; inputs the image and the segmentation result into the classifier to obtain the grading result of the lesion degree; and generates segmented images corresponding to different lesion levels according to the segmentation result and the grading result. The present application can improve the efficiency and accuracy of pathological diagnosis, support metastasis judgment, prognosis prediction, medication guidance, etc., improve the productivity of clinical departments and bring good news to patients.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to a pathological image segmentation and grading method, device and equipment. Background Art

[0002] Pathological diagnosis has always been the "gold standard" for clinical diagnosis and classification of malignant tumors, but pathologists' manual reading of films faces problems such as high subjectivity, low repeatability, insufficient quantitative and information utilization, time-consuming and labor-intensive, and difficulty in knowledge and experience inheritance. At present, the number of beds in China's medical and health institutions has reached tens of millions, and the demand for pathologists is 100,000 to 200,000. There is currently a gap of 100,000 people. Judging from the number of cancer patients discharged from hospitals in recent years, the number of cancer patients visiting hospitals has shown explosive growth, with an average annual growth rate of more than 10%.

[0003] At present, morphological pathology diagnosis relies on manual interpretation, which is not only labor-intensive, but also extremely dependent on the interpretation level and experience of pathologists for accuracy. With the development and improvement of artificial intelligence technology, AI algorithms represented by deconvolution have rapidly emerged in the fields of image analysis and recognition, providing a technical basis for intelligent judgment of pathological images.

[0004] Intelligent diagnosis of pathological images includes deep learning application of convolutional networks to extract features, combined with fully connected layers, to complete classification and prediction. However, its application is subject to the following limitations: 1. The feature modeling stage is limited by the size of the data set; 2. There is no full-process system from automatic reading to prognosis prediction; 3. There is a lack of reliable molecular markers. Summary of the invention

[0005] In view of the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a pathological image segmentation and grading method, device and equipment to solve the problems in the prior art.

[0006] To achieve the above-mentioned purpose and other related purposes, the present application provides a pathological image segmentation and grading method, the method comprising: subjecting an original pathological image containing specific pathological staining markers to HE-Norm processing to obtain an H channel image and an E channel image, respectively, and performing data enhancement processing and normalization processing respectively; inputting the images into a convolutional neural network model to automatically segment the lesion area to obtain a segmentation result; inputting the image and the segmentation result into a classifier to obtain a grading result of the degree of lesion; and generating segmented images corresponding to different lesion levels according to the segmentation result and the grading result.

[0007] In one embodiment of the present application, the specific pathological staining marker is obtained by screening out specific pathological markers that can perform early diagnosis of cancer cells or predict metastasis prognosis based on big data from multiple centers, and then through staining processing.

[0008] In one embodiment of the present application, the specific pathological markers include: any one or more of PD1, PD-L1, EZH2, EGFR, Cyclin D1 and CD166.

[0009] In one embodiment of the present application, the data enhancement processing operation includes: any one or more combinations of no flipping, horizontal flipping, vertical flipping, and rotation.

[0010] In one embodiment of the present application, the standardization process includes: before staining, using the same stain from the same manufacturer; after staining, unifying the time for observing the samples and uniformly adjusting the H channel image and the E channel image to a preset size.

[0011] In one embodiment of the present application, the convolutional neural network model includes: a UNet network and a ResNet34 network; the UNet network is used to realize automatic segmentation of the lesion area through multi-task training; the ResNet34 network is a classification network, which is used to perform classification in combination with the segmentation results of the UNet network.

[0012] In one embodiment of the present application, the UNet network is used to: input an overlay image of a processed H channel image and an E channel image; use the UNet network to outline the cancer nest area on the overlay image, and use Dice Loss as a loss function to train automatic segmentation of the cancer nest lesion area; use the features obtained by the UNet encoder to perform three-level classification according to the degree of staining, and use Cross Entropy Loss as a loss function so that the UNet network can learn more hierarchical semantic features; wherein the training of the UNet network uses the Adam optimizer for parameter optimization.

[0013] In one embodiment of the present application, the ResNet34 network is used to: input the segmentation results and overlay map of the cancer nest area based on the UNet network; use the ResNet34 network to map each segmentation result to the overlay map to obtain probability vectors corresponding to multiple levels, and perform hierarchical training on the segmentation results of the cancer nest area according to the corresponding highest level probability vector; wherein the training of the ResNet34 network uses Cross Entropy Loss as the loss function, and uses the Adam optimizer for parameter optimization.

[0014] To achieve the above-mentioned purpose and other related purposes, the present application provides a pathological image segmentation and grading device, which includes: a normalization module, which is used to process the original pathological image containing specific pathological staining markers through HE-Norm to obtain H channel image and E channel image respectively, and perform standardization processing and data enhancement processing respectively; a training module, which is used to input into a convolutional neural network model to automatically segment the lesion area to obtain a segmentation result; input the image and the segmentation result into a classifier to obtain a grading result of the lesion degree; and generate segmented images corresponding to different lesion levels according to the segmentation result and the grading result.

[0015] To achieve the above-mentioned purpose and other related purposes, the present application provides a computer device, which includes: a memory and a processor; the memory is used to store computer instructions; the processor executes the computer instructions to implement the above-mentioned method.

[0016] To achieve the above objectives and other related objectives, the present application provides a computer-readable storage medium storing computer instructions, which, when executed, execute the functions of the system described above.

[0017] In summary, the pathological image segmentation and grading method, device and apparatus of the present application have the following beneficial effects:

[0018] This application combines specific tumor-related molecular markers and innovatively adopts the technology of combining with artificial intelligence for interpretation, realizing the research and development of intelligent molecular diagnostic equipment for tumors. It fully combines the automated acquisition of pathological images, feature extraction and modeling based on neural networks, and prognosis prediction based on mathematical models. It is of great significance for the research and development and industrialization of artificial intelligence tumor classification systems. This application can improve the efficiency and accuracy of pathological diagnosis, support metastasis judgment, prognosis prediction, medication guidance, etc., improve the productivity of clinical departments, and bring good news to patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Shown is a flow chart of a pathological image segmentation and grading method in one embodiment of the present application.

[0020] Figure 2 Shown is a model schematic diagram of the UNet network structure in one embodiment of the present application.

[0021] Figure 3 Shown is a schematic diagram of low, medium and high scenes of staining of two protein markers EZH2 and PD-L1 in one embodiment of the present application.

[0022] Figures 4A-4B Shown is a curve diagram of the relationship between the marker expression level and prognosis in one embodiment of the present application.

[0023] Figure 5 Shown is a model schematic diagram of the ResNet34 network structure in one embodiment of the present application.

[0024] Figures 6A-6C Shown is a curve diagram of the AUC of pathological molecules in three categories: low, medium, and high in one embodiment of the present application.

[0025] Figure 7 Shown is a module schematic diagram of a pathological image segmentation and grading device in one embodiment of the present application.

[0026] Figure 8 Shown is a schematic diagram of the structure of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0027] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0028] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. Although the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation, the type, quantity and proportion of each component in actual implementation may be changed arbitrarily, and the component layout may also be more complicated.

[0029] Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case of "direct connection" but also the case of "indirect connection" by placing other elements therebetween. In addition, when a part is said to "include" a certain constituent element, unless otherwise stated, it does not exclude other constituent elements, but means that other constituent elements may be included.

[0030] The terms first, second and third are used to describe various parts, components, regions, layers and / or segments, but are not limited thereto. These terms are only used to distinguish a certain part, component, region, layer or segment from other parts, components, regions, layers or segments. Therefore, the first part, component, region, layer or segment described below may refer to the second part, component, region, layer or segment within the scope of this application.

[0031] Furthermore, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless there is an indication to the contrary in the context. It should be further understood that the terms "comprise", "include" indicate the presence of the described features, operations, elements, components, items, kinds, and / or groups, but do not exclude the presence, occurrence or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". Exceptions to this definition will only occur when the combination of elements, functions or operations is inherently mutually exclusive in some way.

[0032] To solve the above problems, the present application proposes a pathological cell segmentation and grading method based on deep learning, which realizes the automatic segmentation of the lesion area and the grading of the degree of lesions in the pathological cell microscopic image. This method is mainly composed of image staining normalization, image segmentation and grading, and image post-processing. In the image staining normalization link, the image is decomposed into two stained image components, hematoxylin and eosin. Then the normalized image is input into the convolutional neural network of the encoding-decoding structure, and the encoding vector generated by the encoding network is respectively input into the classifier and the decoding network to realize grading and lesion area segmentation. Finally, the grading results and segmentation results are used to generate a segmented image.

[0033] At the same time, this application combines specific tumor-related molecular markers and innovatively adopts the technology of combining with artificial intelligence for interpretation, realizing the research and development of intelligent molecular diagnostic equipment for tumors. It fully combines the automated acquisition of pathological images, feature extraction and modeling based on neural networks, and prognosis prediction based on mathematical models. It is of great significance for the research and development and industrialization of artificial intelligence tumor classification systems.

[0034] It should be noted that the pathological image segmentation and grading method described in the present application is preferably aimed at head and neck squamous cell carcinoma, which may include oral squamous cell carcinoma and maxillofacial squamous cell carcinoma.

[0035] like Figure 1 As shown, a flow chart of a pathological image segmentation and grading method in an embodiment of the present application is shown. As shown in the figure, the method includes:

[0036] Step 101: The original pathological image containing specific pathological staining markers is processed by HE-Norm to obtain an H channel image and an E channel image, and then normalized and data enhanced respectively.

[0037] Among them, the specific pathological staining markers are screened out based on big data from multiple centers, which can perform early diagnosis or metastasis prognosis prediction of cancer cells (such as oral cancer, head and neck cancer, etc.), and then obtained through staining treatment.

[0038] For example, this application relies on our hospital's Department of Oral and Maxillofacial Head and Neck Oncology and oral and maxillofacial tumor tissue sample and bioinformatics database sharing service platform, and the number and scale of samples in our hospital are at the leading level worldwide.

[0039] Preferably, the specific pathological markers include any one or more of PD1, PD-L1, EZH2, EGFR, Cyclin D1 and CD166.

[0040] For example, PD-1 (programmed death receptor 1), also known as CD279 (cluster of differentiation 279), is an important immunosuppressive molecule. It regulates the immune system and promotes self-tolerance by down-regulating the immune system's response to human cells and by inhibiting T cell inflammatory activity. This can prevent autoimmune diseases, but it can also prevent the immune system from killing cancer cells. For another example, EZH2 is a gene located on chromosome 7. Mutations or overexpression of EZH2 are associated with many types of cancer, such as breast cancer, prostate cancer, melanoma, and bladder cancer. Because abnormally activated EZH2 can inhibit the normal expression of tumor suppressor genes, inhibiting the activity of EZH2 can slow tumor growth.

[0041] It should be noted that the description of other specific pathological markers will not be expanded here, but it should be noted that the above-mentioned specific pathological markers selected in this application are all capable of early diagnosis of cancer cells or prediction of metastasis prognosis, and have the function of facilitating the identification of pathological cells. In order to facilitate the observation of these specific pathological markers, this application stains them.

[0042] In short, the present application processes the original pathological image through HE-Norm to obtain an H channel image (image vector) and an E channel image.

[0043] Hematoxylin and eosin combined staining, referred to as HE staining, is the most basic staining method in routine pathology preparation and is widely used. Hematoxylin is a light yellow-brown crystal extracted from the logwood native to Central and South America. It is an alkaline stain that generates hematoxylin after being oxidized. When used together with a mordant (commonly trivalent aluminum or iron salt), it can stain the cell nucleus. In pathological diagnosis, teaching and scientific research, HE staining is often used to observe the morphological structure of normal and diseased tissues. In order to determine or identify certain abnormal substances and special components in diseased tissues and cells, special staining methods, enzyme histochemistry methods, immunohistochemistry methods, etc. are also used on the basis of observing HE-stained tissue sections. In HE-stained tissue sections, the cell nucleus is blue and the cytoplasm is red, which forms a sharp contrast and is easy to observe and analyze.

[0044] Dyeing principle:

[0045] 1) Principle of cell nucleus staining: Hematoxylin is an alkaline natural dye that can stain the cell nucleus. The main component of chromatin in the cell nucleus is DNA. In the DNA double helix structure, the phosphate groups on the two nucleotide chains face outward, making the outer side of the DNA double helix negatively charged and acidic. It is easy to be stained by the positively charged hematoxylin alkaline dye through ionic bonds or hydrogen bonds. Hematoxylin is blue in alkaline solution, so the cell nucleus is stained blue.

[0046] 2) Principle of cytoplasm staining: Eosin is a chemically synthesized acidic dye that can stain the cytoplasm under certain conditions. The main component of the cytoplasm is protein, which is an amphoteric compound. The staining of the cytoplasm is closely related to the pH value of the staining solution. When the pH value of the staining solution is below the isoelectric point of the cytoplasmic protein (4.7-5.0), the cytoplasmic protein is ionized in an alkaline form, and the cytoplasm is positively charged, which can be stained with negatively charged acidic dyes. Eosin dissociates into negatively charged anions in water, which combine with the positively charged cations of the cytoplasmic protein to stain the cytoplasm and appear red.

[0047] 3) Differentiation: After staining, certain specific solutions are used to remove the excessively bound dyes from the tissues. This process is called differentiation, and the solution used is called differentiation solution. 1% hydrochloric acid ethanol is often used as a differentiation solution in HE staining, because the acid can destroy the quinone structure of hematoxylin, causing the tissue to separate from the pigment and fade. After most tissues are stained with hematoxylin, they must be differentiated with 1% hydrochloric acid ethanol to remove the excessively bound hematoxylin dyes to the cell nucleus and the hematoxylin dyes adsorbed to the cytoplasm, and then eosin staining is performed to ensure that the staining of the cell nucleus and cytoplasm is clear.

[0048] 4) Blueing: After differentiation, hematoxylin is in a red ion state under acidic conditions and appears red; it is in a blue ion state under alkaline conditions and appears blue. After differentiation of tissue sections with acidic ethanol, they appear red or pink. Immediately use water to remove the acid on the tissue sections to stop differentiation, and then use weak alkaline water to make the nuclei stained with hematoxylin appear blue. This process is called blueing or bluing. In addition, rinsing with tap water can also make the nuclei blue, but it takes a long time.

[0049] In order to increase the amount of data for subsequent neural network training and allow the network to learn more semantic information, data augmentation processing is provided in this application. Specifically, the operations of the data augmentation processing include: no flipping, horizontal flipping, vertical flipping, and any one or more combinations of rotation; wherein the flip angle or rotation angle is -10° to 10°.

[0050] In addition, due to the inconsistency of the preparation of histological sections, it is difficult to quantitatively analyze the results. The standardized operation used in this application can overcome many known inconsistencies in the staining process, thereby better quantitatively analyzing it. For example, the standardized process includes: before staining, using the same stain from the same manufacturer; after staining, unifying the observation time of the sample, and uniformly adjusting the H channel image and the E channel image to a preset size. For example, image normalization will be uniformly scaled to 1920x1070 size and input into the network.

[0051] Step 102: Input the convolutional neural network model to automatically segment the lesion area to obtain a segmentation result; input the image and the segmentation result into a classifier to obtain a grading result of the degree of lesion.

[0052] Simply put, the convolutional neural network model is a multi-task training model, which includes: UNet network and ResNet34 network.

[0053] In the present application, the UNet network is used to achieve automatic segmentation of lesion areas through multi-task training.

[0054] The UNet network is a well-known image segmentation network in medical image processing. The process is as follows: the input is an image, and the output is the target segmentation result. To simplify further, an image is encoded, or downsampled, and then decoded, that is, upsampled, and then a segmentation result is output. The segmentation network is trained by back propagation based on the difference between the result and the true segmentation.

[0055] For example, the UNet network structure is as follows Figure 2As shown, it can be seen that the network structure is mainly divided into three parts: downsampling, upsampling, and jump connection. First, the left part of the UNet network is the compression process, namely the Encoder. Through convolution and downsampling, the image size is reduced and some superficial features are extracted. The right part is the corresponding decoding process, namely the Decoder. Some deep features are obtained through convolution and upsampling. Among them, the convolution can use the valid filling method to ensure that the results are based on no missing context features, so that the size of the image will be reduced after each convolution. In the middle, the feature map obtained in the encoding stage is combined with the feature map obtained in the decoding stage by concat, and the deep and shallow features are combined to refine the image, and the prediction segmentation is performed according to the final feature map obtained. Among them, the feature maps of the two layers here are of different sizes and may need to be cut. The last layer is classified by 1x1 convolution.

[0056] In this embodiment, the training of the UNet network uses the Adam optimizer for parameter optimization. The UNet network is used to:

[0057] A. Overlay of the H channel image and the E channel image after input processing.

[0058] This application superimposes the H channel image and the E channel image, which can facilitate the observation of cell nuclei by slicing. Simply put, the superimposed image input into the network is the H channel part (hematoxylin) stained image and the E channel part (eosin) stained image after data enhancement and standardization, which are superimposed on the channel. It is equivalent to a normal color image (stained image) with RGB and 3 channels. This application superimposes the H channel and the E channel on the channel to become 6 channels. For example, it is equivalent to two (3, h, w) matrices spliced ​​in the first dimension to become (6, h, w).

[0059] B. Use the UNet network to outline the cancer nest area on the overlay image, and use Dice Loss as the loss function to train the automatic segmentation of the cancer nest lesion area.

[0060] In brief, the UNet network is used to scan the grayscale values ​​on the overlay image to outline the preset range of grayscale values ​​(corresponding to the cancer nest pixel range), and one or more regions are connected through the boundary grayscale values, thereby achieving the outline of the cancer nest area on the overlay image.

[0061] Here Dice Loss comes from Dice coefficient, which is a metric function used to evaluate the similarity between two samples. Its value range is between 0 and 1. The larger the value, the more similar they are.

[0062] C. The features obtained by the UNet encoder are used to perform three-level classification according to the degree of coloring, and Cross EntropyLoss is used as the loss function to allow the UNet network to learn more hierarchical semantic features.

[0063] Among them, the third-level classification mainly refers to the degree of staining that characterizes the expression amount. Figure 3 As shown, the expression levels of two protein markers EZH2 and PD-L1 in immunohistochemical staining (weak, moderate, high), i.e., the staining degree, are exemplified in one embodiment of the present application.

[0064] like Figure 4A and 4B As shown, it shows the relationship between marker expression and prognosis. The lower line represents high expression and the upper line represents low expression. It can be seen that the higher the expression, the worse the prognosis.

[0065] The role or purpose of the UNett network in this application is segmentation, but it is also used for hierarchical learning during the training process to supplement more classification knowledge. This is equivalent to not only learning "segmentation annotation" during the training of UNet in this application, but also letting it learn "classification" related knowledge at the same time, so that better semantic features can be learned. Finally, the first part of the network UNet is used for segmentation and transmits the segmentation results to the next network.

[0066] In this application, the ResNet34 network is a classification network, which is used to perform classification in combination with the segmentation results of the UNet network.

[0067] In short, the ResNet34 network belongs to the ResNet network. Among them, the main features of the ResNet network include: 1) The network contains a total of 5 convolution groups, each of which contains one or more basic convolution calculation processes (Conv->BN->ReLU); 2) Each convolution group contains 1 downsampling operation to halve the size of the feature map. Downsampling is achieved in the following two ways: maximum pooling, with a step size of 2, which is only used for the second convolution group (Conv2_x); convolution, with a step size of 2, which is used for the 4 convolution groups except the second convolution group; 3) The first convolution group contains only one convolution calculation operation. The first convolution groups of the five typical ResNet structures are exactly the same, with convolution kernels of 7x7 and a step size of 2; 4) The 2nd to 5th convolution groups all contain multiple identical residual units. In many code implementations, the 2nd to 5th convolution groups are usually called Stage1, Stage2, Stage3, and Stage4 respectively.

[0068] For example, the structure of the ResNet34 network is as follows Figure 5As shown in the figure, each large rectangular box represents a stage of ResNet, from top to bottom they are Stage1 (Conv2_x), Stage2 (Conv3_x), Stage3 (Conv4_x), Stage4 (Conv5_x); each small rectangular box in the large rectangular box represents one or more standard residual units, and the value on the left side of the small rectangular box represents the number of residual unit cascades, such as 5x represents 5 cascaded residual units; the number of channels changes: the input channel is 3, and the number of channels of the 4 stages is 64, 128, 256, and 512, respectively, that is, after each S tage channels doubled; Layer calculation: The number of residual units contained in each Stage is 3, 4, 6, and 3 respectively. Each residual unit contains 2 convolutional layers. Including the first 7x7 convolutional layer and the 3x3 maximum pooling layer, the total number of layers = (3+4+6+3)*2+1+1=34; Downsampling: The purple part in the small rectangular box represents the downsampling operation, that is, the feature map size is halved, and the arrow on the right is the size of the feature map after downsampling (taking the input 224x224 as an example); The orange rectangular box in the first large rectangular box represents the maximum pooling, where the first downsampling occurs; Explanation of convolutional layer parameters: Taking Conv as an example, Taking 3x3,c512,s2,p1 as an example, 3x3 represents the size of the convolution kernel, c512 represents the number of convolution kernels / output channels is 512, s2 represents the convolution step size is 2, and p1 represents the padding of the convolution is 1; pooling layer parameter explanation: Max_pool 3x3,c64,s2,p1, 3x3 represents the size of the pooling area (similar to the convolution kernel size), c64 represents the input and output channels is 64, s2 represents the pooling step size is 2, and p1 represents the padding is 1; after having a general understanding of the structure, you will find that the downsampling operation in ResNet occurs in the first residual unit or maximum pooling layer of each Stage, and the implementation method is that the operation is achieved by taking a step size of 2 in the convolution or pooling.

[0069] In this embodiment, the training of the ResNet34 network uses Cross Entropy Loss as the loss function and uses the Adam optimizer for parameter optimization. The ResNet34 network is used for:

[0070] A. Input the segmentation results and overlay images of the cancer nest area based on the UNet network.

[0071] B. Use the ResNet34 network to map each segmentation result to the overlay map to obtain vectors corresponding to the three levels of probability. Perform hierarchical training on the segmentation results of the cancer nest area based on the corresponding highest level probability vector.

[0072] Simply put, ResNet34 is a classification model, which can be understood as a mapping F from X->Y. When image X is input into F, a vector representing probability will be output, such as (0.1, 0.7, 0.2), which correspond to the probabilities of three levels (low, medium, and high) respectively. Then it can be considered that the network team judges the image X as medium (Y).

[0073] Step 103: Generate segmented images corresponding to different lesion grades according to the segmentation results and the grading results.

[0074] In short, segmented images corresponding to different lesion levels can be obtained based on the segmentation results obtained by the UNet network and the grading results obtained by the ResNet34 network. The present application has a high diagnostic efficiency for the segmented images of the three categories of low, medium and high.

[0075] like Figures 6A-6C As shown, the AUC curves of pathological molecules in the three categories of low, medium and high are displayed. It should be noted that the neural network learning process in this application can be looped using the for function, that is, 100 operations are performed to obtain 100 final logistic regression prediction values, and the pROC function can be used to calculate the AUC value of the predicted value and the true value.

[0076] The AUC value is the area under the curve, which is an evaluation indicator for classification problems. The area under the curve comprehensively measures the effects of all possible classification thresholds. One way to interpret the area under the curve is to regard it as the probability that the model will rank a random positive class sample above a random negative class sample. For example, the general judgment criteria for AUC are: 0.5-0.7: low effect; 0.7-0.85: average effect; 0.85-0.95: very good effect; 0.95-1: very good effect, but generally unlikely. Figures 6A-6C As shown, the AUC values ​​corresponding to the segmented images of the three categories of low, medium and high in this application are all above 0.95, so they all have high diagnostic efficiency.

[0077] In summary, the present invention will improve the efficiency and accuracy of pathological diagnosis, support metastasis judgment, prognosis prediction, medication guidance, etc., improve the productivity of clinical departments and bring good news to patients. This application combines specific tumor-related molecular markers and innovatively adopts the technology of combining with artificial intelligence for interpretation, realizing the research and development of intelligent molecular diagnostic equipment for tumors, and fully combines the automated collection of pathological images, feature extraction and modeling based on neural networks, and prognosis prediction based on mathematical models. It is of great significance for the research and development and industrialization of artificial intelligence tumor typing systems.

[0078] like Figure 7FIG. 7 is a schematic diagram of a module of a pathological image segmentation and grading device in one embodiment of the present application. As shown in the figure, the device 700 includes:

[0079] Normalization module 701, used for subjecting the original pathological image containing specific pathological staining markers to HE-Norm processing, obtaining an H channel image and an E channel image, and performing normalization processing and data enhancement processing respectively;

[0080] The training module 702 is used to input the convolutional neural network model to automatically segment the lesion area to obtain a segmentation result; input the image and the segmentation result into the classifier to obtain a grading result of the lesion degree; and generate a segmented image corresponding to different lesion levels based on the segmentation result and the grading result.

[0081] It should be noted that the information interaction, execution process, etc. between the modules / units of the above-mentioned device are based on the same concept as the method embodiment described in the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown in the previous description of the present application, and will not be repeated here.

[0082] It should also be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these units can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, each module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0083] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processors that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0084] like Figure 8 As shown, it is a schematic diagram of the structure of a computer device in an embodiment of the present application. As shown in the figure, the computer device 800 includes: a memory 801 and a processor 802; the memory 801 is used to store computer instructions; the processor 802 executes the computer instructions to implement the following Figure 1 The method described.

[0085] In some embodiments, the number of the memory 801 in the computer device 800 can be one or more, the number of the processor 802 can be one or more, and Figure 8 Take one as an example.

[0086] In one embodiment of the present application, the processor 802 in the computer device 800 will follow the following steps: Figure 1 In the above steps, one or more instructions corresponding to the process of the application are loaded into the memory 801, and the processor 802 runs the application stored in the memory 801, thereby achieving the following Figure 1 The method described.

[0087] The memory 801 may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 801 stores an operating system and operating instructions, executable modules or data structures, or a subset thereof, or an extended set thereof, wherein the operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic services and processing hardware-based tasks.

[0088] The processor 802 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0089] In some specific applications, the various components of the computer device 800 are coupled together through a bus system, wherein the bus system may include a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, Figure 8 In the text, all kinds of buses are called bus systems.

[0090] In one embodiment of the present application, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following Figure 1 The method described.

[0091] At any possible level of technical detail combination, the present application may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present application.

[0092] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0093] The computer-readable program described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0094] The computer program instructions for performing the operation of the present application can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as python, Smalltalk, C++, etc., and procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). In some embodiments, by using the state information of computer-readable program instructions to personalize electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (PLAs), the electronic circuits can execute computer-readable program instructions, thereby realizing various aspects of the present application.

[0095] In summary, the present application provides a pathological image segmentation and grading method, device and equipment, which obtains H channel image and E channel image respectively by HE-Norm processing of the original pathological image containing specific pathological staining markers, and performs data enhancement processing and standardization processing respectively; inputs the image into a convolutional neural network model to automatically segment the lesion area to obtain a segmentation result; inputs the image and the segmentation result into a classifier to obtain a grading result of the degree of lesion; and generates segmented images corresponding to different lesion levels according to the segmentation result and the grading result.

[0096] The present invention effectively overcomes various shortcomings of the prior art and has high industrial utilization value.

[0097] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present invention. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by those of ordinary skill in the art without departing from the spirit and technical ideas disclosed by the present invention shall still be covered by the claims of the present application.

Claims

1. A pathological image segmentation and grading method, characterized in that: The method comprises: The original pathological images containing specific pathological staining markers were processed by HE-Norm to obtain H channel images and E channel images, and data enhancement and standardization were performed respectively; Input the image into the convolutional neural network model to automatically segment the lesion area and obtain the segmentation result; input the image and the segmentation result into the classifier to obtain the grading result of the lesion degree; Generate segmented images corresponding to different lesion grades according to the segmentation results and grading results; The specific pathological staining markers are obtained by screening out specific pathological markers that can perform early diagnosis of cancer cells or predict metastasis prognosis based on big data from multiple centers, and then obtained through staining; The convolutional neural network model includes: a UNet network and a ResNet34 network; the UNet network is used to realize automatic segmentation of the lesion area through multi-task training; the ResNet34 network is a classification network, which is used to perform classification in combination with the segmentation result of the UNet network; The UNet network is used to: Input the overlay image of the processed H channel image and E channel image; Using the UNet network to delineate the cancer nest area on the overlay image, and using Dice Loss as a loss function to train automatic segmentation of the cancer nest lesion area; The features obtained by the UNet encoder are used to classify the images into three levels according to the degree of coloring, and the Cross Entropy Loss is used as the loss function to enable the UNet network to learn more hierarchical semantic features. The training of the UNet network uses the Adam optimizer for parameter optimization; The ResNet34 network is used to: Input the segmentation results and overlay images of the cancer nest area based on the UNet network; The ResNet34 network is used to map each segmentation result to an overlay map to obtain probability vectors corresponding to multiple levels. The segmentation results of the cancer nest area are graded and trained according to the probability vector corresponding to the highest level. The training of the ResNet34 network uses Cross Entropy Loss as the loss function and uses the Adam optimizer for parameter optimization.

2. The method according to claim 1, characterized in that The specific pathological markers include: any one or more of PD1, PD-L1, EZH2, EGFR, Cyclin D1 and CD166.

3. The method according to claim 1, characterized in that The data enhancement processing operation includes: any one or more combinations of no flipping, horizontal flipping, vertical flipping, and rotation.

4. A pathological image segmentation and grading device, used to implement the method according to any one of claims 1 to 3, characterized in that: The device comprises: The normalization module is used to process the original pathological image containing specific pathological staining markers through HE-Norm to obtain H channel image and E channel image, and perform normalization processing and data enhancement processing respectively; The training module is used to input the convolutional neural network model to automatically segment the lesion area to obtain the segmentation result; the image and the segmentation result are input into the classifier to obtain the grading result of the lesion degree; and the segmentation image corresponding to different lesion levels is generated according to the segmentation result and the grading result.

5. A computer device, characterized in that: The device comprises: a memory and a processor; the memory is used to store computer instructions; the processor executes the computer instructions to implement the method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that: Computer instructions are stored, and when the computer instructions are executed, the method according to any one of claims 1 to 3 is executed.

Images