A pathological image classification device, method, and usage method of the device

Abstract

The present invention discloses a pathological image classification device, which comprises: an image data set generation unit for sampling morphological digital slices to generate a first small-block image data set; and also for sampling morphological digital slices and generating a second small-block image data set in combination with set data set rules; a region classification unit for calculating predicted region classification labels of the small-block image data in the first small-block image data set; and combining with predetermined region classification rules to synthesize regions of the small-block images on the morphological digital slices; a pathological classification unit for calculating predicted pathological classification labels of the small-block abnormal image data in the second small-block image data set; and then obtaining the pathological classification label of the abnormal region by calculating the predicted pathological classification labels of the small-block abnormal image data located in the same abnormal region. The present invention also discloses a pathological image classification method; and a usage method of the pathological image classification device. Through the present invention, more rapid and accurate pathological image classification can be achieved.

Description

Technical Field

[0001] The present invention relates to the fields of image recognition and deep learning, and more particularly, to a pathological image classification technology. Background Art

[0002] Pathological images are the gold standard for the final diagnosis of cancer. However, the current classification of pathological images based on manual operation by doctors not only has the problems of time-consuming and laborious, but also the diagnostic results are easily affected by subjective human factors such as doctors' experience and level; introducing a computer-aided diagnosis system can not only improve the diagnostic efficiency, but also assist in providing more objective and accurate diagnostic results. In recent years, some convolutional neural network models have been used in the computer-aided diagnosis system for automatic classification of pathological images, and the ResNet convolutional neural network is the most widely used. However, since the ResNet convolutional neural network is designed for natural images, if a relatively accurate image recognition result is to be obtained, the number of layers required is generally relatively deep, and the calculation is time-consuming; on the other hand, the model parameters are numerous, and a large amount of training data is required to train the parameters to obtain a high degree of accuracy. However, due to the small scale of the pathological image dataset, directly applying the above model to the computer-aided diagnosis system for automatic classification of pathological images is prone to overfitting, reducing the accuracy and reliability of classification.

[0003] Therefore, a new device and method for quickly and accurately classifying pathological images are urgently needed. Summary of the Invention

[0004] The present invention discloses a pathological image classification device, which comprises:

[0005] An image dataset generation unit, configured to sample morphological digital slices to generate a first small-block image dataset, wherein the morphological digital slices are carriers of pathological images, and the first small-block image dataset contains all small-block image data; and is further configured to sample the morphological digital slices and generate a second small-block image dataset in combination with a set dataset rule, and the second small-block image dataset only contains small-block abnormal image data;

[0006] A region classification unit, configured to receive the small-block image data in the first small-block image dataset generated by the image dataset generation unit, construct a first shallow convolutional neural network model, and calculate the predicted region classification labels of all the small-block image data in the first small-block image dataset according to a set parameter one; combine a predetermined region classification rule to synthesize regions of the small-block images on the morphological digital slices, and write the region classification labels of the regions into the annotations of the morphological digital slices; the regions include normal regions and abnormal regions;

[0007] The pathological classification unit is used to receive the small abnormal image data in the small image data set II generated by the image data set generation unit, and through the convolutional neural network model II, according to the set parameter II, calculate the predicted pathological classification labels of all the small abnormal image data in the small image data set II; input the predicted pathological classification labels of the small abnormal image data located in the same abnormal area into the linear regression model, and according to the set parameter III, calculate the predicted pathological classification labels of the small abnormal image data located in the same abnormal area, obtain the pathological classification label of the abnormal area and write it into the annotation of the morphological digital slice.

[0008] Among them, the image data set generation unit further includes:

[0009] The image acquisition module is used to tile and sample the morphological digital slice using a sliding window to obtain multiple small images; the parameters of the sliding window are configurable;

[0010] The data set generation module is used to receive the small images obtained by the image acquisition module, read and record the annotations of the small images on the morphological digital slice, generate small image data, and add it to the small image data set I; it is also used to receive the small images obtained by the image acquisition module, read and record the annotations of the small images on the morphological digital slice, and combine the set data set rules to generate small abnormal image data and add it to the small image data set II.

[0011] Among them, the region classification unit further includes:

[0012] The region prediction module is used to receive the small image data in the small image data set I generated by the data set generation module, construct a shallow convolutional neural network model I, which is composed of a specific number of convolutional layers, shortcut connection modules, average pooling layers, convolutional layers and classifiers stacked, and according to the set parameter I, calculate the predicted region classification labels of all the small image data in the small image data set I and write them into the small image data, obtain the updated small image data, and output it to the region classification module;

[0013] The region classification module is used to receive all the small image data in the small image data set I obtained by the region prediction module; combine the predetermined region classification rules, synthesize regions of the small images on the morphological digital slice, and write the region classification labels of the regions into the annotations of the morphological digital slice.

[0014] Among them, the pathological classification unit further includes:

[0015] A pathological prediction module, which is used to receive the small abnormal image data in the small-block image dataset II generated by the dataset generation module, and through the convolutional neural network model II, combine the preset pathological classification criteria to set parameter II. According to the set parameter II, calculate the predicted pathological classification labels of all the small abnormal image data in the small-block image dataset II and write them into the small abnormal image data to obtain updated small abnormal image data, and output it to the pathological classification module;

[0016] A pathological classification module, which is used to receive all the small abnormal image data in the small-block image dataset II obtained by the pathological prediction module; input the predicted pathological classification labels of the small abnormal image data located in the same abnormal area into the linear regression model, and according to the set parameter III, calculate the predicted pathological classification labels of the small abnormal image data located in the same abnormal area, obtain the pathological classification label of the abnormal area and write it into the annotation of the morphological digital slice.

[0017] To improve the accuracy of the device, preferably, the device further includes:

[0018] A training unit, which is used to calculate the regional classification error according to the predicted regional classification label and the original regional classification label obtained for the small-block image data in the small-block image dataset I, and update and set the parameter I according to the regional classification error; and is also used to calculate the pathological classification error according to the predicted pathological classification label and the original pathological classification label obtained for the small abnormal image data in the small-block image dataset II, and update and set the parameter II according to the pathological classification error; and is also used to calculate the regional-level pathological classification error according to the obtained pathological classification label of the abnormal area and the original pathological classification label, and update and set the parameter III according to the regional-level pathological classification error;

[0019] Among them, the training unit further includes:

[0020] A parameter I training module, which is used to calculate the regional classification error according to the predicted regional classification label and the original regional classification label obtained for the small-block image data in the small-block image dataset I, and update and set the parameter I according to the regional classification error;

[0021] A parameter II training module, which is used to calculate the pathological classification error according to the predicted pathological classification label and the original pathological classification label obtained for the small abnormal image data in the small-block image dataset II, and update and set the parameter II according to the pathological classification error;

[0022] A parameter III training module, which is used to calculate the regional-level pathological classification error according to the obtained pathological classification label of the abnormal area and the original pathological classification label, and update and set the parameter III according to the regional-level pathological classification error;

[0023] The small piece image dataset one can be divided into a training set, a validation set, and a test set;

[0024] The small piece image dataset two can be divided into a training set, a validation set, and a test set.

[0025] The present invention also discloses a pathological image classification method, and the method includes the steps:

[0026] (11) Sampling the morphological digital section to generate a small piece image dataset one, where the morphological digital section is the carrier of the pathological image, and the small piece image dataset one contains all small piece image data;

[0027] (12) Constructing a shallow convolutional neural network model one, and according to the set parameter one, calculating the predicted region classification labels of all small piece image data in the small piece image dataset one; combining the predetermined region classification rules, synthesizing regions of the small piece images on the morphological digital section, and writing the region classification labels of the regions into the annotation of the morphological digital section; the regions include normal regions and abnormal regions;

[0028] (13) Sampling the morphological digital section and combining the set dataset rules to generate a small piece image dataset two, where the small piece image dataset two only contains small piece abnormal image data;

[0029] (14) Through the convolutional neural network model two, and according to the set parameter two, calculating the predicted pathological classification labels of all small piece abnormal image data in the small piece image dataset two; inputting the predicted pathological classification labels of the small piece abnormal image data located in the same abnormal region into a linear regression model, and according to the set parameter three, calculating the predicted pathological classification labels of the small piece abnormal image data located in the same abnormal region to obtain the pathological classification label of the abnormal region and writing it into the annotation of the morphological digital section.

[0030] Further, the method for generating the small piece image dataset one in the step (11) specifically includes:

[0031] Using a sliding window to tile and sample the morphological digital section to obtain a plurality of small piece images; the parameters of the sliding window are configurable;

[0032] During sampling, reading and recording the annotation of the small piece image on the morphological digital section, and the annotation includes position information;

[0033] Generating small piece image data and adding it to the small piece image dataset one; the small piece image data includes image information and labels, and the labels include annotations.

[0034] Further, the method in the step (12) specifically includes:

[0035] Construct a shallow convolutional neural network model 1, which is composed of a specific number of convolutional layers, shortcut connection modules, average pooling layers, convolutional layers, and classifiers stacked. According to the set parameter 1, calculate the predicted regional classification labels of all small piece image data in the small piece image dataset 1 and write them into the small piece image data to obtain updated small piece image data;

[0036] Combine the predetermined regional classification rules to synthesize regions from the small piece images on the morphological digital slice, and write the regional classification labels of the regions into the annotations of the morphological digital slice.

[0037] Further, the method for generating the small piece image dataset 2 in step (13) specifically includes:

[0038] Use a sliding window to tile and sample the morphological digital slice to obtain multiple small piece images; the parameters of the sliding window are configurable;

[0039] Read and record the annotations of the small piece images on the morphological digital slice during sampling; the annotations include position information, regional classification labels, and information on the regions to which they belong;

[0040] Combine the set dataset rules to generate small piece abnormal image data with an abnormal regional classification label and add it to the small piece image dataset 2; the small piece abnormal image data includes image information and labels, and the labels include annotations.

[0041] Further, the method in step (14) specifically includes:

[0042] Through the convolutional neural network model 2, combine the preset pathological classification criteria to set parameter 2. According to the set parameter 2, calculate the predicted pathological classification labels of all small piece abnormal image data in the small piece image dataset 2 and write them into the small piece abnormal image data to obtain updated small piece abnormal image data;

[0043] Input the predicted pathological classification labels of the small piece abnormal image data located in the same abnormal region into a linear regression model. According to the set parameter 3, calculate the predicted pathological classification labels of the small piece abnormal image data located in the same abnormal region to obtain the pathological classification label of the abnormal region and write it into the annotation of the morphological digital slice.

[0044] To improve the classification accuracy, preferably, the method further includes:

[0045] Calculate the regional classification error based on the predicted regional classification labels and the original regional classification labels obtained for the small piece image data in the small piece image dataset 1, and update and set the parameter 1 according to the regional classification error;

[0046] Calculate the pathological classification error based on the predicted pathological classification label and the original pathological classification label obtained for the small abnormal image data in the small image dataset II, and update and set the parameter II according to the pathological classification error;

[0047] Calculate the regional-level pathological classification error based on the pathological classification label and the original pathological classification label of the obtained abnormal region, and update and set the parameter III according to the regional-level pathological classification error;

[0048] The small image dataset I can be divided into a training set, a validation set, and a test set;

[0049] The small image dataset II can be divided into a training set, a validation set, and a test set.

[0050] The present invention also discloses a usage method of a pathological image classification device. According to this method, the device can be trained into a high-precision classification device. The method includes:

[0051] Select a group of morphological digital slices whose annotations already contain the original region classification label and the original pathological classification label, and divide them into a training set, a validation set, and a test set according to a certain ratio;

[0052] Step (21), the image dataset generation unit samples the morphological digital slices of the training set to obtain multiple small images, and at the same time reads and records the annotations of the small images on the morphological digital slices, generates small image data, and adds it to the small image dataset I of the training set; each small image data in the small image dataset I of the training set includes image information and a label; the label includes the position information, the original region classification label, and the original pathological classification label obtained from the annotation;

[0053] Preferably, the dataset rule can be preset to screen out the small images located in a single region, generate small image data, and add it to the small image dataset I of the test set;

[0054] Step (22), the region classification unit receives the small image data in the small image dataset I of the training set generated by the image dataset generation unit, constructs a shallow convolutional neural network model I, and calculates the predicted region classification label of all the small image data in the small image dataset I of the training set according to the set parameter I and writes it into the small image data to obtain the updated small image data;

[0055] Step (23), the training unit calculates the region classification error according to the obtained predicted region classification label and the original region classification label, and updates and sets the parameter I according to the region classification error;

[0056] Repeat the above steps (22) to (23) until the corresponding number of times is completed according to the number of training generations set by the training unit; obtain the optimal parameter one;

[0057] Step (24), the image dataset generation unit samples the morphological digital slices of the training set to obtain multiple small-piece images, and at the same time reads and records the annotations of the small-piece images on the morphological digital slices. Combining the set dataset rules, small-piece abnormal image data is generated and added to the small-piece image dataset two of the training set; each small-piece abnormal image data in the small-piece image dataset two of the training set contains image information and a label; the label contains position information, original region classification label, region information, and original pathological classification label obtained from the annotation;

[0058] Step (25), the pathological classification unit receives the small-piece abnormal image data in the small-piece image dataset two of the training set generated by the image dataset generation unit, and through the convolutional neural network model two, according to the set parameter two, calculates the predicted pathological classification label of the small-piece abnormal image data in the small-piece image dataset two of the training set and writes it into the small-piece abnormal image data to obtain updated small-piece abnormal image data;

[0059] Step (26), the training unit calculates the pathological classification error according to the obtained predicted pathological classification label and the original pathological classification label, and updates the set parameter two according to the pathological classification error;

[0060] Repeat the above steps (25) to (26) until the corresponding number of times is completed according to the number of training generations set by the training unit; obtain the optimal parameter two;

[0061] Step (27), the image dataset generation unit samples the morphological digital slices of the validation set to obtain multiple small-piece images, and at the same time reads and records the annotations of the small-piece images on the morphological digital slices. Combining the set dataset rules, small-piece abnormal image data is generated and added to the small-piece image dataset two of the validation set; each small-piece abnormal image data in the small-piece image dataset two of the validation set contains image information and a label; the label contains position information, original region classification label, region information, and original pathological classification label obtained from the annotation;

[0062] Step (28), the pathological classification unit receives the small-piece abnormal image data in the small-piece image dataset two of the validation set generated by the image dataset generation unit, and through the convolutional neural network model two, according to the set parameter two, calculates the predicted pathological classification label of the small-piece abnormal image data in the small-piece image dataset two of the validation set and writes it into the small-piece abnormal image data to obtain updated small-piece abnormal image data;

[0063] Step (29): The pathological classification unit inputs the predicted pathological classification labels of the small abnormal image data located within the same abnormal region into a linear regression model, and calculates the predicted pathological classification labels of the small abnormal image data located within the same abnormal region according to the set Parameter 3, so as to obtain the pathological classification label of the abnormal region.

[0064] Step (210): The training unit calculates the region-level pathological classification error according to the obtained pathological classification label of the abnormal region and the original pathological classification label, and updates and sets the Parameter 3 according to the region-level pathological classification error.

[0065] Repeat the above steps (29) to (210) until the corresponding number of times is completed according to the number of training epochs set by the training unit; obtain the optimal Parameter 3.

[0066] The present invention also discloses a usage method of a pathological image classification device. By applying this method, the performance of this device can be evaluated. The method includes:

[0067] Select a set of morphological digital slides whose annotations already contain the original region classification label and the original pathological classification label, and divide them into a training set, a validation set, and a test set according to a certain ratio.

[0068] The image dataset generation unit samples the morphological digital slides of the test set to obtain multiple small images, and at the same time reads and records the annotations of the small images on the morphological digital slides, generates small image data, and adds it to the small image dataset one of the test set; each small image data in the small image dataset one of the test set includes image information and a label; the label includes the position information, the original region classification label, and the original pathological classification label obtained from the annotation.

[0069] The region classification unit receives the small image data in the small image dataset one of the test set generated by the image dataset generation unit, constructs a shallow convolutional neural network model one, and calculates the predicted region classification labels of all the small image data in the small image dataset one of the test set according to the set Parameter 1.

[0070] The image dataset generation unit samples the morphological digital slides of the test set to obtain multiple small images, and at the same time reads and records the annotations of the small images on the morphological digital slides, combines the set dataset rules, generates small abnormal image data, and adds it to the small image dataset two of the test set; each small abnormal image data in the small image dataset two of the test set includes image information and a label; the label includes the position information, the original region classification label, the region information to which it belongs, and the original pathological classification label obtained from the annotation.

[0071] The pathological classification unit receives the small abnormal image data in the small image data set II of the test set generated by the image data set generation unit, and calculates the predicted pathological classification labels of all the small abnormal image data in the small image data set II of the test set through the convolutional neural network model II according to the set parameter II; inputs the predicted pathological classification labels of the small abnormal image data located in the same abnormal area into the linear regression model, and calculates the predicted pathological classification labels of the small abnormal image data located in the same abnormal area according to the set parameter III to obtain the pathological classification label of the abnormal area.

[0072] Compared with the prior art, the present invention achieves the beneficial effect of quickly and accurately classifying pathological images through the following innovations, significantly improving the pathological image classification performance of the computer-aided diagnosis system.

[0073] 1. A two-level structure is designed to optimize the original one-step pathological image classification method. The first-level classification first classifies the pathological image into normal and abnormal areas, and the second-level classification only classifies the pathological conditions of the abnormal areas. The complex N-classification (multi-classification) problem is split into a binary classification problem and an N-1 classification problem.

[0074] 2. During the first-level classification, a new shallow convolutional neural network model is constructed. The new shallow convolutional neural network has fewer layers, which can reduce the amount of computation and computing time, improve the speed; the new shallow convolutional neural network model also requires fewer parameters, and the optimal parameters can be obtained by training with a smaller-scale data set, which is beneficial to reducing the overfitting of the model; the prediction accuracy is improved.

[0075] 3. During the second-level classification, only the pathological images of the abnormal areas are classified, reducing the amount of input data, which is beneficial to improving the computing speed. The convolutional neural network model and the linear regression model are connected in series; the linear regression model is trained to adaptively perform weighted voting on the predicted labels of the small images; the small images located at the center of the abnormal area, which contain less noise information and play a stronger role in pathological representation, are given greater weights, effectively improving the prediction accuracy of the model for the pathological classification labels. Description of the Drawings

[0076] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0077] Figure 1 It is a schematic structural diagram of a pathological image classification device provided in Embodiment 1 of the present application;

[0078] Figure 2-1 Schematic structural diagram of another pathological image classification device provided in the second embodiment of the present application;

[0079] Figure 2-2 Schematic structural diagram of the region prediction module provided in the second embodiment of the present application;

[0080] Figure 2-3 Schematic structural diagram of the pathological prediction module provided in the second embodiment of the present application;

[0081] Figure 3 Schematic flowchart of a pathological image classification method provided in the third embodiment of the present application;

[0082] Figure 4 Schematic flowchart of a usage method of a pathological image classification device provided in the fourth embodiment of the present application;

[0083] Figure 5 Schematic flowchart of a usage method of another pathological image classification device provided in the fifth embodiment of the present application;

[0084] Figure 6 Schematic flowchart of the method provided in the sixth embodiment of the present application. Detailed implementation manners

[0085] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0086] Next, in conjunction with the drawings and embodiments, the specific implementation manners of the present invention will be further described in detail. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0087] As Figure 1 shown, a pathological image classification device, the device includes:

[0088] An image data set generation unit M1, configured to sample morphological digital slices to generate a small image data set one, where the morphological digital slices are carriers of pathological images, and the small image data set one includes all small image data; and is further configured to sample the morphological digital slices and generate a small image data set two in combination with set data set rules, where the small image data set two only includes small abnormal image data.

[0089] Morphological digital slides (WSI) are digital slides that use a fully automated microscope scanning system in combination with virtual slide software to scan and seamlessly stitch traditional glass slides to generate a whole-field (Whole Slide Image), abbreviated as WSI. In addition to the image, morphological digital slides carry annotations. The annotation information includes position information in the axis coordinate system, region classification information, and pathological classification information.

[0090] Regions are generally divided into abnormal regions and normal regions. Abnormal regions are areas of interest to users that need to be further classified. For example, when classifying breast cancer pathologically, the cancerous area is the abnormal region, and the non-cancerous area is the normal region; the cancerous area is further divided into three categories: benign, micro-invasive cancer, and invasive cancer. The present invention does not limit the criteria and methods for determining and classifying abnormal regions.

[0091] Generally, the steps for sampling a morphological digital slide to generate a small image dataset are as follows: tile and sample the morphological digital slide with a sliding window of size l and step size s to obtain small images. According to the position of the small image in the WSI, read and record the annotation of the WSI at the corresponding position as the label of the small image, generate small image data, and add it to the small image dataset X. X = [x1, x2,..., x N represents the dataset composed of all small image data. Each sample x i , {i = 1, 2,..., N} in the dataset X contains image information and labels. Where N is the number of small image data samples.

[0092] Preferably, a dataset rule can be set to screen out samples that meet the dataset rule and add them to the small image dataset. For example: it is stipulated that when the area of the region with a central side length of / belonging to a single region reaches the proportionality coefficient r during sampling, generate small image data and add it to the small image dataset; it can also be further stipulated to only screen out small abnormal image data within a single abnormal region. Setting the dataset rule is beneficial for screening out small image data with less noise information and stronger pathological representation.

[0093] The region classification unit M2 is used to receive the small image data in the small image dataset one generated by the image dataset generation unit M1, construct a shallow convolutional neural network model one, calculate the predicted region classification labels of all the small image data in the small image dataset one according to the set parameter one; combine the predetermined region classification rules, synthesize regions of the small images on the morphological digital slide, and write the region classification labels of the regions into the annotation of the morphological digital slide; the regions include normal regions and abnormal regions.

[0094] Construct a shallow convolutional neural network model 1. According to the set parameter 1, calculate the predicted regional classification labels of all small piece images in the small piece image dataset 1. Parameter 1 can be initialized according to actual experience or updated according to the regional classification error to improve the accuracy of the predicted regional classification labels.

[0095] Define the regional classification rule. For example, it can be defined that small piece images with the same predicted regional classification label and connected position information belong to the same region. Combine the small piece images that meet the rule into the same region in the WSI and write the regional classification label into the annotation.

[0096] The pathological classification unit M3 is used to receive the small piece abnormal image data in the small piece image dataset 2 generated by the image dataset generation unit M1. Through the convolutional neural network model 2, according to the set parameter 2, calculate the predicted pathological classification labels of all small piece abnormal image data in the small piece image dataset 2. Input the predicted pathological classification labels of the small piece abnormal image data located in the same abnormal region into the linear regression model. According to the set parameter 3, calculate the predicted pathological classification labels of the small piece abnormal image data located in the same abnormal region, obtain the pathological classification label of the abnormal region and write it into the annotation of the morphological digital slice.

[0097] Through the convolutional neural network model 2, combine the preset pathological classification criteria to set parameter 2. According to the set parameter 2, calculate the predicted pathological classification labels of all small piece abnormal image data in the small piece image dataset 2. For example, the cancer area of breast cancer is further divided into three categories: benign, micro-invasive cancer, and invasive cancer. Then select the ResNet-50 network and set the number of nodes in the fully connected layer to 3 to determine the cancer type of the cancer area. Parameter 2 can be initialized according to actual experience or updated according to the pathological classification error to improve the accuracy of the predicted pathological classification labels.

[0098] Input the predicted pathological classification labels of the small piece abnormal image data located in the same abnormal region into the linear regression model. According to the set parameter 3, obtain the pathological classification label of the abnormal region after weighted voting. Parameter 3 can be initialized according to actual experience or updated according to the regional-level pathological classification error to improve the accuracy of the regional-level pathological classification labels.

[0099] As can be seen from the above embodiments, the present application provides a pathological image classification device. The first-level structure obtains small-piece image data by sampling WSI; synthesizes regions on the WSI according to the predicted region classification labels of the calculated small-piece image data and writes the region classification labels into the annotation of the WSI; the second-level structure samples the WSI again to obtain small-piece abnormal image data; and obtains the region-level pathological classification label of the WSI through the predicted pathological classification labels of the calculated small-piece abnormal image data. Through the two-level structure, the original one-step pathological image classification method is optimized. The first-level classification first classifies the pathological image into normal and abnormal regions, and the second-level classification only classifies the abnormal regions pathologically. The complex N-classification (multi-classification) problem is split into a binary classification problem and an N-1 classification problem. This is beneficial to quickly and accurately classify pathological images.

[0100] To better illustrate the present invention, Embodiment 2 is given to elaborate in detail the working principles of each unit and module, as Figure 2-1 shown.

[0101] The image dataset generation unit M1 may further include:

[0102] The image acquisition module M11 is used to tile and sample the morphological digital slice using a sliding window to obtain multiple small-piece images; the parameters of the sliding window are configurable;

[0103] The dataset generation module M12 is used to receive the small-piece images obtained by the image acquisition module M11, read and record the annotations of the small-piece images on the morphological digital slice, generate small-piece image data, and add it to the first small-piece image dataset; it is also used to receive the small-piece images obtained by the image acquisition module M11, read and record the annotations of the small-piece images on the morphological digital slice, and combine the set dataset rules to generate small-piece abnormal image data and add it to the second small-piece image dataset.

[0104] Select a set of morphological digital slices that have been correctly classified and whose annotations already contain the original region classification labels and the original pathological classification labels, and divide them into a training set, a validation set, and a test set according to a certain ratio. When using the training set as the target data of the device, complete the training of the convolutional neural network according to the set training parameters. After reaching a certain level of accuracy, determine the parameters of the convolutional neural network to form a fixed convolutional neural network model. When using the validation set as the target data of the device, complete the training of the regression model according to the set training parameters. After reaching a certain level of accuracy, determine the parameters of the regression model to form a fixed regression model. When using the test set as the target data of the device, the performance of the device can be evaluated by comparing the obtained labels with the original labels.

[0105] The region classification unit M2 may further include:

[0106] The region prediction module M21 is configured to receive the small image data in the small image data set I generated by the data set generation module M12, construct a first shallow convolutional neural network model, which is composed of a specific number of convolutional layers, shortcut connection modules, average pooling layers, convolutional layers, and classifiers stacked in sequence. According to the set parameter I, calculate the predicted region classification labels of all the small image data in the small image data set I and write them into the small image data to obtain updated small image data, and output it to the region classification module M22.

[0107] The region prediction module M21 uses the newly constructed first shallow convolutional neural network model for prediction. The first shallow convolutional neural network model is composed of a specific number of convolutional layers, shortcut connection modules, average pooling layers, convolutional layers, and classifiers; the classifier is used to output corresponding classification labels according to the features obtained by the convolutional neural network in combination with the preset classification criteria. As Figure 2-2 shown, in this embodiment, a first shallow convolutional neural network model stacked in sequence by a 7×7 convolutional layer, three shortcut connection modules, an average pooling layer, a 1×1 convolutional layer, and a Softmax classifier is adopted; among them, the shortcut connection module is composed of two 3×3 convolutional layers and a shortcut connection bypass. The number of each component can be flexibly set according to the actual situation. The present invention does not limit the selection of the classifier.

[0108] The region classification module M22 is configured to receive all the small image data in the small image data set I obtained by the region prediction module M21; combine the predetermined region classification rules, synthesize regions of the small images on the morphological digital slices, and write the region classification labels of the regions into the annotations of the morphological digital slices.

[0109] The pathological classification unit M3 may further include:

[0110] The pathological prediction module M31 is configured to receive the small abnormal image data in the small image data set II generated by the data set generation module M12, pass through the second convolutional neural network model, set parameter II in combination with the preset pathological classification criteria, and according to the set parameter II, calculate the predicted pathological classification labels of all the small abnormal image data in the small image data set II and write them into the small abnormal image data to obtain updated small abnormal image data, and output it to the pathological classification module M32.

[0111] The pathological prediction module M31 uses the second convolutional neural network model for prediction. The second convolutional neural network model is composed of a convolutional neural network and a classifier; the classifier is used to output corresponding classification labels according to the features obtained by the convolutional neural network in combination with the preset classification criteria. As Figure 2-3As shown in the figure, in this embodiment, a convolutional neural network model two is adopted, which is composed of a ResNet-50 network and a Softmax classifier in series. The present invention does not limit the selection of the convolutional neural network and the classifier.

[0112] The pathological classification module M32 is used to receive all the small abnormal image data in the small image data set two obtained by the pathological prediction module M31; input the predicted pathological classification labels of the small abnormal image data located in the same abnormal area into the linear regression model, and calculate the predicted pathological classification labels of the small abnormal image data located in the same abnormal area according to the set parameter three, obtain the pathological classification label of the abnormal area and write it into the annotation of the morphological digital slice. The calculation formula is:

[0113]

[0114] where, L r represents the pathological classification label of the abnormal area, L i represents the predicted pathological classification label of the small image in the abnormal area, and θ represents parameter three, that is, the coefficient of the linear regression model.

[0115] The training unit M4 may further include:

[0116] The parameter one training module M41 is used to receive the predicted region classification labels obtained by the region prediction module M21 for the small image data in the small image data set one and the original region classification labels obtained by the data set generation module M12, calculate the region classification error, and update and set the parameter one according to the region classification error. Specifically:

[0117] Use the training set as the target data of the device;

[0118] Set the training parameters; the training parameters mainly include: learning rate, weight decay rate, training batch, training epoch;

[0119] Initialize parameter one;

[0120] Use the predicted region classification labels of all the small image data in the small image data set one of the training set obtained by the region prediction module M21 and the original region classification labels obtained by the data set generation module M12 to calculate the region classification error, that is, the loss function of the classifier. Use the stochastic gradient descent method to train the convolutional neural network to obtain the optimal parameters;

[0121] Perform backpropagation according to the initial region classification error and update parameter one. During the iteration process, the region classification error gradually decreases with the increase of the training epoch until it reaches the convergence state. Complete the training epoch and determine the optimal convolutional neural network model parameter one.

[0122] The parameter two training module M42 is used to receive the predicted pathological classification labels of all the small abnormal image data in the small image data set two of the training set obtained by the pathological prediction module M31 for the small abnormal image data in the small image data set two, and calculate the pathological classification error with the original pathological classification labels obtained by the data set generation module M12, and update and set the parameter two according to the pathological classification error. Specifically:

[0123] Use the training set as the target data of the device;

[0124] Set the training parameters. The training parameters mainly include: learning rate, weight decay rate, training batch, number of training epochs;

[0125] Initialize parameter two;

[0126] Use the predicted pathological classification labels of all the small abnormal image data in the small image data set two of the training set obtained by the pathological prediction module M31 and the original pathological classification labels obtained by the data set generation module M12 to calculate the pathological classification error, that is, the loss function of the classifier. Use the stochastic gradient descent method to train the convolutional neural network to obtain the optimal parameters.

[0127] Perform backpropagation according to the initial pathological classification error and update parameter two. During the iteration process, the pathological classification error gradually decreases as the number of training epochs increases until it reaches the convergence state. Complete the number of training epochs and determine the optimal convolutional neural network model parameter two.

[0128] The parameter three training module M43 is used to receive the pathological classification labels of the abnormal regions obtained by the pathological classification module M32 and the original pathological classification labels of the abnormal regions obtained from the WSI to calculate the region-level pathological classification error, that is, the loss function of the linear regression model. Update and set the parameter three according to the region-level pathological classification error. Specifically:

[0129] Use the validation set as the target data of the device;

[0130] Set the training parameters. The training parameters mainly include the number of training epochs;

[0131] Initialize parameter three;

[0132] Use the pathological classification labels of the abnormal regions obtained by the pathological classification module M32 and the original pathological classification labels of the abnormal regions obtained from the WSI to calculate the region-level pathological classification error, that is, the loss function of the linear regression model. Its loss function C is defined as:

[0133]

[0134] Among them, represents the original pathological classification label of the j-th abnormal region, Represents the pathological classification label of the j-th abnormal region.

[0135] Update parameter three until the loss function C reaches the convergence state. Complete the number of training epochs and determine the optimal parameter three of the regression model.

[0136] It can be seen that in this embodiment, when performing the first-level classification, a new shallow convolutional neural network model is constructed. The new shallow convolutional neural network has fewer layers, which can reduce the amount of computation and computing time, and improve the speed. The new shallow convolutional neural network model also requires fewer parameters, and the optimal parameters can be obtained by training with a smaller-scale dataset, which is beneficial to reducing overfitting of the model and improving the prediction accuracy. When performing the second-level classification, only the images of the abnormal regions are classified, which reduces the amount of input data and can improve the computing speed. The convolutional neural network model and the linear regression model are cascaded; the linear regression model is trained to adaptively perform weighted voting on the predicted labels of small patch images; a larger weight is given to the small patch images located at the center of the abnormal region, which contain less noise information and have a stronger effect on pathological representation, effectively improving the prediction accuracy of the model for the pathological classification label.

[0137] Embodiment 3 of the present invention discloses a pathological image classification method, as Figure 3 shown.

[0138] Step S31: Sample the morphological digital slice to generate a first small patch image dataset. The morphological digital slice is the carrier of the pathological image, and the first small patch image dataset contains all small patch image data;

[0139] Step S32: Construct a first shallow convolutional neural network model. According to the set parameter one, calculate the predicted region classification labels of all small patch image data in the first small patch image dataset; combine the predetermined region classification rules to synthesize regions of the small patches on the morphological digital slice, and write the region classification labels of the regions into the annotation of the morphological digital slice. The regions include normal regions and abnormal regions;

[0140] Step S33: Sample the morphological digital slice and generate a second small patch image dataset in combination with the set dataset rules. The second small patch image dataset only contains small patch abnormal image data;

[0141] Step S34: Through the second convolutional neural network model, calculate the predicted pathological classification labels of all small abnormal image data in the second small image data set according to the set second parameter; input the predicted pathological classification labels of the small abnormal image data located in the same abnormal area into the linear regression model, and calculate the predicted pathological classification labels of the small abnormal image data located in the same abnormal area according to the set third parameter, obtain the pathological classification label of the abnormal area and write it into the annotation of the morphological digital slice.

[0142] The inventive content of this method part is similar to that of the foregoing device part, and the specific description can refer to the foregoing device part and will not be repeated here.

[0143] As can be seen from the above embodiments, the present application provides a new pathological image classification method. A two-level structure is designed to optimize the original one-step pathological image classification method. The first-level classification first classifies the pathological image into normal and abnormal areas, and the second-level classification only classifies the abnormal areas pathologically. The complex N-classification (multi-classification) problem is split into a binary classification problem and an N-1 classification problem; in the first-level classification, a new shallow convolutional neural network model is constructed to reduce model parameters, improve model speed, and reduce model overfitting; in the second-level classification, only the images in the abnormal areas are classified, reducing the amount of input data and improving the calculation speed. The convolutional neural network model and the linear regression model are connected in series; the linear regression model is trained to adaptively perform weighted voting on the predicted labels of small images; a larger weight is given to the small images located at the center of the abnormal area, which contain less noise information and play a stronger role in pathological representation, effectively improving the prediction accuracy of the model for pathological classification labels.

[0144] This method effectively improves the classification speed and classification accuracy, and significantly improves the pathological image classification performance.

[0145] To elaborate on the working principle and usage method of a pathological image classification device for the training set and validation set, a fourth embodiment of the present invention is specifically given, including the following steps:

[0146] Select a group of morphological digital slices that have been correctly classified and whose annotations already contain the original area classification labels and original pathological classification labels, and divide them into a training set, a validation set, and a test set according to a certain ratio. Using the training set and the validation set as the target data of the device, the optimal device parameters are determined through training.

[0147] Step S401, the image data set generation unit samples the morphological digital slices of the training set to generate the first small image data set;

[0148] The image dataset generation unit samples the morphological digital slices of the training set to obtain multiple small block images, and at the same time reads and records the annotations of the small block images on the morphological digital slices, generates small block image data, and adds it to the small block image dataset one of the training set; each small block image data in the small block image dataset one of the training set contains image information and a label; the label contains position information, original region classification label, and original pathological classification label obtained from the annotation.

[0149] Preferably, in combination with a preset dataset rule, small block images located within a single region can be screened out, small block image data is generated, and added to the small block image dataset one of the test set; setting the dataset rule is beneficial to screening out small block images with less noise information and stronger pathological representation. It helps to obtain the optimal parameters and improve the classification accuracy of the device.

[0150] Step S402, the region classification unit receives the small block image data in the small block image dataset one of the training set generated by the image dataset generation unit, constructs a shallow convolutional neural network model one, and according to the set parameter one, calculates the predicted region classification labels of all the small block image data in the small block image dataset one of the training set and writes them into the small block image data to obtain updated small block image data.

[0151] Step S403, the training unit calculates the region classification error according to the obtained predicted region classification label and the original region classification label, and updates and sets the parameter one according to the region classification error.

[0152] Step S404, determine whether the corresponding number of training generations set by the training unit has been completed. If not, return to step S402; if completed, determine the optimal parameter one.

[0153] Step S405, the image dataset generation unit samples the morphological digital slices of the training set to generate the small block image dataset two of the training set;

[0154] The image dataset generation unit samples the morphological digital slices of the training set to obtain multiple small block images, and at the same time reads and records the annotations of the small block images on the morphological digital slices. In combination with the set dataset rule, small block abnormal image data with an abnormal region classification label is generated and added to the small block image dataset two of the training set; each small block abnormal image data in the small block image dataset two of the training set contains image information and a label; the label contains position information, original region classification label, region information to which it belongs, and original pathological classification label obtained from the annotation.

[0155] Step S406: The pathological classification unit receives the small abnormal image data in the small image data set II of the training set generated by the image data set generation unit, and through the convolutional neural network model II, according to the set parameter II, calculates the predicted pathological classification label of the small abnormal image data in the small image data set II of the training set and writes it into the small abnormal image data, obtaining updated small abnormal image data.

[0156] Step S407: The training unit calculates the pathological classification error according to the obtained predicted pathological classification label and the original pathological classification label, and updates the set parameter II according to the pathological classification error.

[0157] Step S408: Determine whether the corresponding number of training generations set by the training unit has been completed. If not, return to Step S406; if completed, determine the optimal parameter II.

[0158] Step S409: The image data set generation unit samples the morphological digital slices of the validation set to generate the small image data set II of the validation set;

[0159] The image data set generation unit samples the morphological digital slices of the validation set to obtain multiple small images, and at the same time reads and records the annotations of the small images on the morphological digital slices. Combining the set data set rules, small abnormal image data is generated and added to the small image data set II of the validation set; each small abnormal image data in the small image data set II of the validation set contains image information and a label; the label contains position information, original region classification label, region information to which it belongs, and original pathological classification label obtained from the annotation.

[0160] Step S410: The pathological classification unit receives the small abnormal image data in the small image data set II of the validation set generated by the image data set generation unit, and through the convolutional neural network model II, according to the set parameter II, calculates the predicted pathological classification label of the small abnormal image data in the small image data set II of the validation set and writes it into the small abnormal image data, obtaining updated small abnormal image data.

[0161] Step S411: The pathological classification unit inputs the predicted pathological classification labels of the small abnormal image data located in the same abnormal region into the linear regression model, and according to the set parameter III, calculates the predicted pathological classification labels of the small abnormal image data located in the same abnormal region, obtaining the pathological classification label of the abnormal region.

[0162] Step S412: The training unit calculates the region-level pathological classification error according to the obtained pathological classification label of the abnormal region and the original pathological classification label, and updates the set parameter III according to the region-level pathological classification error.

[0163] Step S413: Determine whether the corresponding number of training epochs set by the training unit has been completed. If not, return to Step S411; if completed, determine the optimal Parameter Three.

[0164] In this embodiment, the training process is completed using a pathological image classification device. By using the data of the training set and the validation set, classifying the images according to the initialized parameters, and adjusting the parameters with reference to the classification error, the parameters of the pathological image classification device are finally in the best state, with the smallest classification result error and the highest accuracy.

[0165] To illustrate in detail the working principle and usage method of a pathological image classification device for the test set, a fifth embodiment of the present invention is specifically given, including the following steps:

[0166] Select a set of morphological digital slices that have been correctly classified and whose annotations already contain the original region classification label and the original pathological classification label, and divide them into a training set, a validation set, and a test set according to a certain ratio. Using the test set as the target data of the device, the performance of the device can be evaluated by comparing the obtained labels with the original labels.

[0167] Step S51: The image dataset generation unit samples the morphological digital slices of the test set to generate a small image dataset one of the test set;

[0168] The image dataset generation unit samples the morphological digital slices of the test set to obtain multiple small images, and at the same time reads and records the annotations of the small images on the morphological digital slices, generates small image data, and adds it to the small image dataset one of the test set; each small image data in the small image dataset one of the test set contains image information and a label; the label contains position information, the original region classification label, and the original pathological classification label obtained from the annotation.

[0169] Step S52: The region classification unit receives the small image data in the small image dataset one of the test set generated by the image dataset generation unit, constructs a shallow convolutional neural network model one, and calculates the predicted region classification labels of all the small image data in the small image dataset one of the test set according to the set Parameter One.

[0170] Calculate the classification accuracy of the predicted region classification according to whether the predicted region classification label of the sample in the small image dataset one of the test set is consistent with the original region classification label.

[0171] Step S53: The image dataset generation unit samples the morphological digital slices of the test set to generate a small image dataset two of the test set;

[0172] The image dataset generation unit samples the morphological digital slices of the test set to obtain multiple small-piece images, and at the same time reads and records the annotations of the small-piece images on the morphological digital slices. Combining the set dataset rules, it generates small-piece abnormal image data and adds it to the small-piece image dataset two of the test set; each small-piece abnormal image data in the small-piece image dataset two of the test set includes image information and a label; the label includes position information, original region classification label, belonging region information, and original pathological classification label obtained from the annotation.

[0173] In step S54, the pathological classification unit receives the small-piece abnormal image data in the small-piece image dataset two of the test set generated by the image dataset generation unit, and through the convolutional neural network model two, according to the set parameter two, calculates the predicted pathological classification labels of all the small-piece abnormal image data in the small-piece image dataset two of the test set; inputs the predicted pathological classification labels of the small-piece abnormal image data located in the same abnormal region into the linear regression model, and according to the set parameter three, calculates the predicted pathological classification labels of the small-piece abnormal image data located in the same abnormal region to obtain the pathological classification label of the abnormal region.

[0174] According to whether the pathological classification label of the abnormal region is consistent with the original region-level pathological classification label, calculate the classification accuracy of the predicted pathological classification.

[0175] This embodiment is a process of testing a pathological image classification device using a test set. According to the training steps in the fourth embodiment above, a pathological image classification device has reached a relatively accurate state. Then, use the samples in the test set to test its accuracy to evaluate the performance of the device.

[0176] To more detailedly illustrate the working principle and working process of this device, the following gives Embodiment Six of the present invention and explains it in combination with examples:

[0177] This embodiment uses the breast cancer clinical image dataset as an example sample set. This dataset contains high-quality morphological digital slices of 186 patients. The regions of these WSIs have been correctly annotated as cancerous regions and non-cancerous regions, and the cancerous regions are further subdivided and annotated into three categories: benign, micro-invasive carcinoma, and invasive carcinoma. The WSIs are divided into a training set, a validation set, and a test set according to a certain ratio. In this example, 88 are used to construct the training set, 21 are used to construct the validation set, and the remaining 77 are used to construct the test set.

[0178] In this embodiment, a shallow convolutional neural network model one constructed by a pathological image classification device is used to first determine whether a small piece of image is a cancerous area; then, a convolutional neural network model two adopted by a pathological image classification device is used to determine the pathological type of the small cancerous area image; then, a linear regression model is used to obtain a regional-level pathological classification label.

[0179] Step S61, the image dataset generation unit samples the morphological digital slices to generate a first small image dataset;

[0180] Use a sliding window with a size of 224 and a step size of 112 to tile and sample to obtain small images, and at the same time read and record the annotations of the small images on the WSI to generate small image data. When the 100% central area with a side length of 112 of the small image is located in a single area, this small image data will be put into the first small image dataset. The first small image dataset sampled from the training set WSI is denoted as X train1 The first small image dataset sampled from the validation set WSI is denoted as X val1 The first small image dataset sampled from the test set WSI is denoted as X test1 .

[0181] Step S62, the image dataset generation unit samples the morphological digital slices to generate a second small image dataset;

[0182] Use a sliding window with a size of 1200 and a step size of 600 to tile and sample to obtain small images, and at the same time read and record the annotations of the small images on the WSI to generate small image data. When the 100% central area with a side length of 600 of the small image is located in a single abnormal area, this small image will be put into the second small image dataset. The second small image dataset sampled from the training set WSI is denoted as X train2 The second small image dataset sampled from the validation set WSI is denoted as X val2 The second small image dataset sampled from the test set WSI is denoted as X test2 ;

[0183] In this example, it is judged whether it belongs to a single area through the original classification label of the sample set, that is, the single area is divided into four categories: non-cancerous area, benign, micro-invasive cancer, and invasive cancer; in particular, to adapt to the convolutional neural network model used in this example, the image size in the second small image dataset is adjusted to 224×224.

[0184] Step S63, construct a model

[0185] The shallow convolutional neural network model 1 is used to determine the regional classification of small images. The shallow convolutional neural network model 1 is composed of a specific number of convolutional layers, shortcut connection modules, average pooling layers, convolutional layers, and a classifier. In this embodiment, a 7×7 convolutional layer, three shortcut connection modules, an average pooling layer, a 1×1 convolutional layer, and a Softmax classifier are stacked in sequence; among them, the shortcut connection module is composed of two 3×3 convolutional layers and a shortcut connection bypass. The Softmax classifier pre-sets the classification criteria and outputs a 2×1 predicted probability vector, that is, the values corresponding to the two category labels of the cancer area and the non-cancer area.

[0186] The convolutional neural network model 2 is used to determine the pathological classification of small images, specifically divided into three categories: benign, micro-invasive carcinoma, and invasive carcinoma. The convolutional neural network model 2 used in this example is composed of a ResNet-50 network and a Softmax classifier connected in series. The nodes of the fully connected layer of the ResNet-50 network are modified to 3; the Softmax classifier pre-sets the classification criteria and outputs a 3×1 predicted probability vector, that is, the values corresponding to the three category labels of benign, micro-invasive carcinoma, and invasive carcinoma.

[0187] Step S64, train the shallow convolutional neural network model 1 to determine parameter 1

[0188] (1) Set the training parameters: Set the learning rate of training to 10 -4 , the weight decay rate is 0.9, set the training batch to 100, and the number of training epochs to 50;

[0189] (2) Initialize parameter 1;

[0190] (3) Input the small image data in X train1 into the shallow convolutional neural network model 1, perform forward propagation of image features, and the Softmax classifier outputs the predicted regional classification label;

[0191] (4) Calculate the Softmax loss, perform backpropagation according to this loss, and update parameter 1. Complete the number of training epochs to determine the optimal parameters of the shallow convolutional neural network model 1.

[0192] Step S65, train the convolutional neural network model 2 to determine parameter 2

[0193] (1) Set the training parameters: Set the learning rate of training to 10 -4 , the weight decay rate is 0.9, set the training batch to 100, and the number of training epochs to 50;

[0194] (2) Initialize parameter 2;

[0195] (3) Input X train2The small-piece image data is input into the second convolutional neural network model, the image features are propagated forward, and the Softmax classifier outputs the predicted pathological classification label;

[0196] (4) Calculate the Softmax loss, perform backpropagation according to this loss, and update Parameter Two. Complete the number of training epochs and determine the optimal parameters of the second convolutional neural network model.

[0197] Step S66: Train the linear regression model to determine Parameter Three

[0198] (1) Set the training parameters: Set the number of training epochs to 50;

[0199] (2) Initialize Parameter Three;

[0200] (3) Input the small-piece image data in X val2 into the second convolutional neural network model to obtain the predicted pathological classification label; Input the predicted pathological classification labels of the small-piece abnormal image data located in the same abnormal area into the linear regression model to calculate the pathological classification label L of the abnormal area r ;

[0201] (3) Calculate the loss function of the linear regression model. Update the set regression model coefficients according to the loss function of the regression model, that is, the Parameter Three. Complete the number of training epochs and determine the optimal regression model Parameter Three.

[0202] Step S67: Test the model

[0203] The parameters of the model are set as the optimal parameters determined after being trained and adjusted by the training unit;

[0204] (1) Input the small-piece image data in X test1 into the first shallow convolutional neural network model, the image features are propagated forward, and the Softmax classifier outputs the predicted region classification label;

[0205] (2) Calculate the screening cancer area accuracy rate of the device according to whether the predicted region classification label of the small-piece image data in X test1 is consistent with the original region classification label;

[0206] (3) Input the small-piece image data in X test2 into the cascaded second convolutional neural network model and linear regression model to obtain the pathological classification label of the abnormal area;

[0207] (4) Calculate the cancer type classification accuracy rate of the device according to whether the pathological classification label of the abnormal area is consistent with the original pathological classification label.

[0208] In this embodiment, first, the training set data is used to train the first parameter of the shallow convolutional neural network model one for determining the cancerous area and non-cancerous area of small-piece images; then, the training set data is used to train the second parameter of the convolutional neural network two for determining the cancer type of small-piece abnormal images; then, the validation set data is used to train the third parameter of the linear regression model for obtaining the pathological classification labels of the abnormal areas; finally, the test set data is used to test the performance of a pathological image classification device that has reached a relatively accurate state.

[0209] Using the samples of the same breast cancer clinical image dataset, the classification method based on ResNet-50 is compared with the method disclosed in the present invention. The accuracy and time consumption of the two methods are shown in Table 1 below. It can be seen that the results of the present invention are more advantageous.

[0210] Table 1 Comparison of classification accuracy and time of breast cancer clinical image dataset

[0211]

[0212] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other. For the methods disclosed in the embodiments, since they correspond to the devices disclosed in the embodiments, the description is relatively simple. For the relevant parts, please refer to the description of the device part.

[0213] The steps of the methods or algorithms described in combination with the embodiments disclosed herein can be directly implemented by hardware, software modules executed by a processor, or a combination of both. The software modules can be placed in a random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims, and the above-mentioned drawings are used to distinguish similar parts, and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application described herein can be implemented in an order other than that illustrated herein.

[0214] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A pathological image classification device, characterized in that, The device includes: An image dataset generation unit, which is used to sample morphological digital slices to generate a small image dataset one. The morphological digital slices are carriers of pathological images, and the small image dataset one contains all small image data. It is also used to sample the morphological digital slices and generate a small image dataset two in combination with the set dataset rules. The small image dataset two only contains small abnormal image data; A region classification unit, which is used to receive the small image data in the small image dataset one generated by the image dataset generation unit, construct a shallow convolutional neural network model one, and calculate the predicted region classification labels of all the small image data in the small image dataset one according to the set parameter one. In combination with the predetermined region classification rules, the small images are synthesized into regions on the morphological digital slices, and the region classification labels of the regions are written into the annotations of the morphological digital slices; The regions include normal regions and abnormal regions; A pathological classification unit, which is used to receive the small abnormal image data in the small image dataset two generated by the image dataset generation unit, and calculate the predicted pathological classification labels of all the small abnormal image data in the small image dataset two through a convolutional neural network model two according to the set parameter two; The predicted pathological classification labels of the small abnormal image data located in the same abnormal region are input into a linear regression model, and according to the set parameter three, the predicted pathological classification labels of the small abnormal image data located in the same abnormal region are calculated, and the pathological classification label of the abnormal region is obtained and written into the annotation of the morphological digital slice.

2. The device according to claim 1, characterized in that, The image dataset generation unit further includes: An image acquisition module, which is used to tile and sample morphological digital slices using a sliding window to obtain multiple small images. The parameters of the sliding window are configurable; A dataset generation module, which is used to receive the small images obtained by the image acquisition module, read and record the annotations of the small images on the morphological digital slices, generate small image data, and add it to the small image dataset one. It is also used to receive the small images obtained by the image acquisition module, read and record the annotations of the small images on the morphological digital slices, and generate small abnormal image data in combination with the set dataset rules, and add it to the small image dataset two.

3. The device according to claim 2, characterized in that, The region classification unit further includes: A region prediction module, which is used to receive the small image data in the small image dataset one generated by the dataset generation module, construct a shallow convolutional neural network model one. The shallow convolutional neural network model one is composed of a specific number of convolutional layers, shortcut connection modules, average pooling layers, convolutional layers and classifiers stacked together. According to the set parameter one, the predicted region classification labels of all the small image data in the small image dataset one are calculated and written into the small image data, and the updated small image data is obtained and output to the region classification module; A region classification module, configured to receive all the small-block image data in the small-block image data set one obtained by the region prediction module; combine with a predetermined region classification rule, synthesize regions for the small-block images on the morphological digital section, and write the region classification labels of the regions into the annotation of the morphological digital section.

4. The device according to claim 3, characterized in that, The pathological classification unit further includes: A pathological prediction module, configured to receive the small-block abnormal image data in the small-block image data set two generated by the data set generation module, pass through the convolutional neural network model two, combine with the preset pathological classification standard to set parameter two, and according to the set parameter two, calculate the predicted pathological classification labels of all the small-block abnormal image data in the small-block image data set two and write them into the small-block abnormal image data, obtain updated small-block abnormal image data, and output them to the pathological classification module; A pathological classification module, configured to receive all the small-block abnormal image data in the small-block image data set two obtained by the pathological prediction module; Input the predicted pathological classification labels of the small-block abnormal image data located in the same abnormal region into a linear regression model, and according to the set parameter three, calculate the predicted pathological classification labels of the small-block abnormal image data located in the same abnormal region, obtain the pathological classification label of the abnormal region and write it into the annotation of the morphological digital section.

5. The device according to any one of claims 1-4, characterized in that, The device further includes: A training unit, configured to calculate a region classification error according to the predicted region classification label and the original region classification label obtained for the small-block image data in the small-block image data set one, and update and set the parameter one according to the region classification error; further configured to calculate a pathological classification error according to the predicted pathological classification label and the original pathological classification label obtained for the small-block abnormal image data in the small-block image data set two, and update and set the parameter two according to the pathological classification error; further configured to calculate a region-level pathological classification error according to the obtained pathological classification label of the abnormal region and the original pathological classification label, and update and set the parameter three according to the region-level pathological classification error; The training unit further includes: A parameter one training module, configured to calculate a region classification error according to the predicted region classification label and the original region classification label obtained for the small-block image data in the small-block image data set one, and update and set the parameter one according to the region classification error; A parameter two training module, configured to calculate a pathological classification error according to the predicted pathological classification label and the original pathological classification label obtained for the small-block abnormal image data in the small-block image data set two, and update and set the parameter two according to the pathological classification error; A parameter three training module, configured to calculate a region-level pathological classification error according to the obtained pathological classification label of the abnormal region and the original pathological classification label, and update and set the parameter three according to the region-level pathological classification error; The small-block image data set one can be divided into a training set, a validation set, and a test set; The small-block image data set two can be divided into a training set, a validation set, and a test set.

6. A pathological image classification method, characterized in that, The method includes the steps: (11) Sample the morphological digital slices to generate the first small image dataset. The morphological digital slices are the carriers of pathological images, and the first small image dataset contains all small image data; (12) Construct the first shallow convolutional neural network model. According to the set parameter one, calculate the predicted region classification labels of all small image data in the first small image dataset. Combine the predetermined region classification rules, synthesize regions of the small images on the morphological digital slices, and write the region classification labels of the regions into the annotations of the morphological digital slices; The regions include normal regions and abnormal regions; (13) Sample the morphological digital slices and generate the second small image dataset in combination with the set dataset rules. The second small image dataset only contains small abnormal image data; (14) Through the second convolutional neural network model, according to the set parameter two, calculate the predicted pathological classification labels of all small abnormal image data in the second small image dataset; Input the predicted pathological classification labels of the small abnormal image data located in the same abnormal region into the linear regression model. According to the set parameter three, calculate the predicted pathological classification labels of the small abnormal image data located in the same abnormal region, obtain the pathological classification label of the abnormal region and write it into the annotation of the morphological digital slice.

7. The method according to claim 6, characterized in that, The method for generating the first small image dataset in step (11) specifically includes: Use a sliding window to tile and sample the morphological digital slices to obtain multiple small images. The parameters of the sliding window are configurable; During sampling, read and record the annotations of the small images on the morphological digital slices. The annotations include position information; Generate small image data and add it to the first small image dataset. The small image data includes image information and labels, and the labels include annotations.

8. The method according to claim 7, wherein The method in step (12) specifically includes: Construct the first shallow convolutional neural network model. The first shallow convolutional neural network model is composed of a specific number of convolutional layers, shortcut connection modules, average pooling layers, convolutional layers, and classifiers stacked together. According to the set parameter one, calculate the predicted region classification labels of all small image data in the first small image dataset and write them into the small image data to obtain updated small image data; Combine the predetermined region classification rules, synthesize regions of the small images on the morphological digital slices, and write the region classification labels of the regions into the annotations of the morphological digital slices.

9. The method according to claim 8, wherein The method for generating the second small image dataset in step (13) specifically includes: Use a sliding window to tile and sample the morphological digital slices to obtain multiple small images. The parameters of the sliding window are configurable; During sampling, read and record the annotations of the small images on the morphological digital slices. The annotations include position information, region classification labels, and region information to which they belong; Combine the set dataset rules to generate small abnormal image data with an abnormal region classification label and add it to the second small image dataset. The small abnormal image data includes image information and labels, and the labels include annotations.

10. The method according to claim 9, wherein The method of step (14) specifically includes: By using the second convolutional neural network model, setting parameter two in combination with the preset pathological classification criteria, and according to the set parameter two, calculating the predicted pathological classification labels of all small abnormal image data in the small image data set two and writing them into the small abnormal image data to obtain updated small abnormal image data; Inputting the predicted pathological classification labels of the small abnormal image data located in the same abnormal area into the linear regression model, and according to the set parameter three, calculating the predicted pathological classification labels of the small abnormal image data located in the same abnormal area to obtain the pathological classification label of the abnormal area and writing it into the annotation of the morphological digital slice.

11. The method according to any one of claims 6 - 10, wherein The method further includes: Calculating the region classification error according to the predicted region classification label and the original region classification label obtained for the small image data in the small image data set one, and updating and setting the parameter one according to the region classification error; Calculating the pathological classification error according to the predicted pathological classification label and the original pathological classification label obtained for the small abnormal image data in the small image data set two, and updating and setting the parameter two according to the pathological classification error; Calculating the region-level pathological classification error according to the obtained pathological classification label and the original pathological classification label of the abnormal area, and updating and setting the parameter three according to the region-level pathological classification error; The small image data set one can be divided into a training set, a validation set, and a test set; The small image data set two can be divided into a training set, a validation set, and a test set.

12. A method for using a pathological image classification device, wherein The method includes: Selecting a group of morphological digital slices whose annotations already contain the original region classification label and the original pathological classification label, and dividing them into a training set, a validation set, and a test set according to a certain ratio; Step (21), the image data set generation unit samples the morphological digital slices in the training set to obtain multiple small images, and at the same time reads and records the annotations of the small images on the morphological digital slices to generate small image data, which is added to the small image data set one in the training set; each small image data in the small image data set one in the training set includes image information and a label; the label includes the position information, the original region classification label, and the original pathological classification label obtained from the annotation; Step (22), the region classification unit receives the small image data in the small image data set one in the training set generated by the image data set generation unit, constructs the first shallow convolutional neural network model, and according to the set parameter one, calculates the predicted region classification labels of all small image data in the small image data set one in the training set and writes them into the small image data to obtain updated small image data; Step (23), the training unit calculates the region classification error according to the obtained predicted region classification label and the original region classification label, and updates and sets the parameter one according to the region classification error; Repeatedly execute the above steps (22) to (23) until the corresponding number of times is completed according to the number of training epochs set by the training unit; obtain the optimal parameter one; Step (24), the image dataset generation unit samples the morphological digital slices of the training set to obtain multiple small piece images, and at the same time reads and records the annotations of the small piece images on the morphological digital slices. Combining the set dataset rules, small piece abnormal image data is generated and added to the second small piece image dataset of the training set; each small piece abnormal image data in the second small piece image dataset of the training set contains image information and a label; the label contains position information, original region classification label, region information to which it belongs, and original pathological classification label obtained from the annotation. Step (25), the pathological classification unit receives the small piece abnormal image data in the second small piece image dataset of the training set generated by the image dataset generation unit, and through the convolutional neural network model two, according to the set parameter two, calculates the predicted pathological classification label of the small piece abnormal image data in the second small piece image dataset of the training set and writes it into the small piece abnormal image data to obtain updated small piece abnormal image data. Step (26), the training unit calculates the pathological classification error according to the obtained predicted pathological classification label and the original pathological classification label, and updates the set parameter two according to the pathological classification error. Repeat the above steps (25) to (26) until the corresponding number of times is completed according to the number of training epochs set by the training unit; obtain the optimal parameter two. Step (27), the image dataset generation unit samples the morphological digital slices of the validation set to obtain multiple small piece images, and at the same time reads and records the annotations of the small piece images on the morphological digital slices. Combining the set dataset rules, small piece abnormal image data is generated and added to the second small piece image dataset of the validation set; each small piece abnormal image data in the second small piece image dataset of the validation set contains image information and a label; the label contains position information, original region classification label, region information to which it belongs, and original pathological classification label obtained from the annotation. Step (28), the pathological classification unit receives the small piece abnormal image data in the second small piece image dataset of the validation set generated by the image dataset generation unit, and through the convolutional neural network model two, according to the set parameter two, calculates the predicted pathological classification label of the small piece abnormal image data in the second small piece image dataset of the validation set and writes it into the small piece abnormal image data to obtain updated small piece abnormal image data. Step (29), the pathological classification unit inputs the predicted pathological classification labels of the small piece abnormal image data located in the same abnormal region into the linear regression model, and according to the set parameter three, calculates the predicted pathological classification labels of the small piece abnormal image data located in the same abnormal region to obtain the pathological classification label of the abnormal region. Step (210), the training unit calculates the region-level pathological classification error according to the obtained pathological classification label of the abnormal region and the original pathological classification label, and updates the set parameter three according to the region-level pathological classification error. Repeat the above steps (29) to (210) until the corresponding number of times is completed according to the number of training epochs set by the training unit; obtain the optimal parameter three.

13. A method for using a pathological image classification device, wherein The method includes: Select a set of morphological digital slices that already contain the original region classification label and the original pathological classification label in the annotation, and divide them into a training set, a validation set, and a test set according to a certain ratio; The image dataset generation unit samples the morphological digital slices of the test set to obtain multiple small block images, and at the same time reads and records the annotations of the small block images on the morphological digital slices, generates small block image data, and adds it to the first small block image dataset of the test set; each small block image data in the first small block image dataset of the test set contains image information and a label; the label contains the position information, the original region classification label, and the original pathological classification label obtained from the annotation; The region classification unit receives the small block image data in the first small block image dataset of the test set generated by the image dataset generation unit, constructs a shallow convolutional neural network model one, and calculates the predicted region classification labels of all the small block image data in the first small block image dataset of the test set according to the set parameter one; The image dataset generation unit samples the morphological digital slices of the test set to obtain multiple small block images, and at the same time reads and records the annotations of the small block images on the morphological digital slices, combines the set dataset rules, generates small block abnormal image data, and adds it to the second small block image dataset of the test set; each small block abnormal image data in the second small block image dataset of the test set contains image information and a label; the label contains the position information, the original region classification label, the region information to which it belongs, and the original pathological classification label obtained from the annotation; The pathological classification unit receives the small block abnormal image data in the second small block image dataset of the test set generated by the image dataset generation unit, and calculates the predicted pathological classification labels of all the small block abnormal image data in the second small block image dataset of the test set through a convolutional neural network model two according to the set parameter two; Input the predicted pathological classification labels of the small block abnormal image data located in the same abnormal region into a linear regression model, and calculate the predicted pathological classification labels of the small block abnormal image data located in the same abnormal region according to the set parameter three to obtain the pathological classification label of the abnormal region.

Images