Method and device for automatic detection of mitosis based on data and feature diversity

Abstract

The application discloses a mitosis automatic detection method and device based on data and feature diversity, which can quickly and efficiently detect mitosis by using point labeling; the method comprises the following steps: firstly, a hematoxylin staining-based detection method without training is adopted to obtain candidate samples; then, a balanced sampling strategy is used to remove redundant information in the samples, balance data information quantity, maintain sample diversity, and remove simple samples, so that balanced and representative training data are obtained, which is beneficial to learning of representative features by a classifier; and finally, a joint training classifier is designed according to the complex characteristics of mitosis cells, the cells are classified into sub-classes on the basis of binary classification, the sub-classes are taken as pseudo labels, and the parent class labels and the sub-class pseudo labels are jointly trained, so that the parent class and sub-class joint classifier is obtained, the joint classifier is based on prior knowledge of mitosis, learns more diverse feature information, and effectively improves the classification performance.

Description

Technical Field

[0001] This invention belongs to the technical field of image processing and mitosis detection, specifically relating to an automatic mitosis detection method and apparatus based on data and feature diversity. Background Technology

[0002] Mitosis reflects cell proliferation and is an important indicator for determining tumor grade and prognosis. Due to the rapid development of deep learning technology, various segmentation models, object detection models, and classification models have been applied to the automatic mitosis detection task, mainly categorized into single-stage and multi-stage methods. Single-stage methods directly use the detection results as the final result through end-to-end learning; for example, treating automatic mitosis detection as a semantic segmentation problem. Multi-stage methods typically include two stages: the first stage generates candidate cells while ensuring high recall; the second stage further classifies these candidate cells. In existing technologies, common multi-stage schemes for automatic mitosis detection are: ① generating pixel-level pseudo-labels through additional manual pixel annotation or using existing models (e.g., the HoVer net cell nuclear segmentation network); ② training the segmentation network with pixel-level pseudo-labels, inputting the image into the segmentation network to initially obtain candidate cells while ensuring high recall (identifying as many mitotic cells as possible); ③ classifying the candidate cells to obtain the final result.

[0003] However, the following drawbacks exist: 1) Existing methods use complex segmentation models to acquire candidate cells. Complex models often require higher-level annotations, so additional annotations (extra manual annotations or pseudo-annotations generated by existing models) are often needed to acquire candidate cells for training the segmentation model, resulting in low efficiency. 2) Existing methods still suffer from imbalanced data when training classification networks. They often use random sampling or specific loss methods to combat class imbalance, but this approach loses data diversity and is inefficient. The classification model cannot learn representative features well. For example, random sampling will lose some representative data during the screening process and cannot obtain samples with balanced information. Using specific loss methods on a large amount of data can only slightly alleviate the sample imbalance problem, and the model will still pay too much attention to high-frequency class samples. 3) Existing methods attempt to solve the mitosis detection problem by increasing the complexity of the classification model, using ensemble learning and increasing network depth to improve classification performance. However, using only complex classification models for detection can easily lead to overfitting to existing models, resulting in poor generalization ability. Summary of the Invention

[0004] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide an automatic detection method and device for mitotic cells based on data and feature diversity. Candidate cells are obtained based on hematoxylin staining detection, and simple samples are removed by sample screening based on diversity to obtain balanced samples. Finally, molecular classification is performed on the basis of binary classification, and the classification model is trained based on prior knowledge of mitosis to effectively improve classification performance.

[0005] To achieve the above objectives, the first objective of this invention is to employ an automatic mitosis detection method based on data and feature diversity, comprising the following steps:

[0006] Step 1: Detection based on hematoxylin staining: Obtain pathological images and their point annotations, stain the pathological images using HE staining, and then use color deconvolution to separate the stained pathological images to obtain hematoxylin staining channel images. Divide the image blocks to obtain candidate cells and classify them into positive and negative samples.

[0007] Step 2, Sample screening based on diversity: Redundant and simple samples are screened out from the negative samples, and the filtered samples are mixed with the positive samples to obtain training samples;

[0008] Step 3, Unsupervised staining enhancement: Expand the training samples to k color spaces and mix them with the filtered training samples to obtain training data;

[0009] Step 4: Training the joint classifier for parent and child classes: Cluster the training data using parent class labels based on the deep learning network to obtain pseudo-labels for child classes; Combine the parent class labels and pseudo-labels for child classes to optimize the deep learning network until the loss function converges, and obtain the joint classifier for parent and child classes;

[0010] Step 5: Mitosis detection: Use the trained parent-child joint classifier to perform mitosis detection on the pathological image to be detected, and obtain the detection results.

[0011] As a preferred technical solution, in step one, the detection based on hematoxylin staining specifically includes:

[0012] Obtain pathological image I and its point annotations, wherein the pathological image contains a cells;

[0013] The pathological images were stained using HE staining to obtain stained pathological images;

[0014] The stained pathological image is input into a color deconvolution process to separate and obtain the hematoxylin staining channel image I. h ;

[0015] Image I of the channel stained with hematoxylin h Obtain the centroid coordinates O of each cell;

[0016] Based on the centroid coordinates, pathological image I is cut to obtain image blocks D for each cell. H ={I1,I2,...,I a};

[0017] Based on the point markings in pathological image I, the image patch is divided into positive sample D. P and negative sample D N ;

[0018] The point annotations include mitotic point annotations and non-mitotic point annotations; if a mitotic point annotation is located in an image block, the image block is classified as a positive sample, otherwise it is classified as a negative sample.

[0019] As a preferred technical solution, in step two, the sample screening based on diversity specifically involves:

[0020] For negative sample D N The K-means clustering algorithm is used to obtain k subspace clusters C = {C1, C2, ..., C}. k},in Describes the a-th subspace cluster belonging to the k-th subspace. k One sample, a k This represents the number of samples in the k-th subspace cluster;

[0021] Select an equal number of m negative samples from each subspace cluster in, The m-th negative sample selected for the k-th subspace cluster;

[0022] Using positive sample D P and Train a classification network f easy-sampling Then use the classification network f easy-sampling To filter out The simple samples were left behind, while the negative samples were difficult to distinguish.

[0023] Mixed positive sample D P And negative samples that are difficult to distinguish Obtain the final training samples

[0024] As a preferred technical solution, the parent-child joint classifier is constructed based on a deep learning network, including an input layer, a feature extractor, a parent classifier, a child classifier, and an output layer; the input layer is connected to the feature extractor, and the feature extractor is connected to both the parent classifier and the child classifier; the fully connected layer of the parent classifier serves as the final fully connected layer, connected to the feature extractor, and then connected to the output layer.

[0025] The training process for the parent-child class joint classifier is as follows:

[0026] The training data is input into the input layer, where features are extracted by the feature extractor and then input into the parent classifier.

[0027] The parent classifier classifies the features of the training data to obtain parent class labels; the parent class labels include mitotic categories and non-mitotic categories;

[0028] Based on the parent class label, the feature extractor is trained by binary classification on the training data to obtain a preliminary feature extractor;

[0029] The preliminary feature extractor is used to extract features from the samples in the training data to obtain sample features;

[0030] The subclass classifier clusters the sample features, grouping the mitotic and non-mitotic categories in the parent class label into an equal number of subclasses, and using the subclass results as subclass pseudo-labels.

[0031] The initial feature extractor is trained using both parent class labels and child class pseudo-labels until the loss function converges or the required accuracy is achieved, resulting in the final joint parent and child class classifier.

[0032] As a preferred technical solution, for each parent class tag c = {c p ,c n}, c p For mitosis, c n For non-mitotic categories, an unsupervised clustering algorithm is used to cluster them into T subclasses. Use the unsupervised clustering results as pseudo-labels for subclasses;

[0033] Let Y be the parent class label of each sample in the training data. P Then the pseudo-label of each subclass in parent class c is Y. S The clustering objective for each parent class c is:

[0034]

[0035] Where, N c Y represents the number of samples in parent class c. S For subclass pseudo tags, For training data, The features extracted by the feature extractor from the training data, O is the matrix formed by the centroid coordinates of each cell in the training data, and 1 t Let be a t-dimensional identity matrix.

[0036] As a preferred technical solution, the parent classifier and the child classifier are supervised using both the focal loss function and the center loss function;

[0037] For the parent classifier f P Its loss function L P for:

[0038]

[0039] Among them, L Pf The focal loss function of the parent classifier. The center loss function is used for the parent classifier; for the child classifier f... S Its loss function L S for:

[0040]

[0041] in, The focal loss function for the subclass classifier. The center loss function is used for the subclass classifier.

[0042] The feature extractor is optimized using the loss functions of both the parent and child classifiers. The loss function of the feature extractor is expressed as follows:

[0043]

[0044] Where N is the number of samples in the training data. For training data, The features extracted by the feature extractor from the training data, λ is the balancing parameter, and θ is the balancing parameter. P and θ S These represent the parent classifier f respectively. P Subclass classifier f S The parameter, Y P For the parent category tag, Y S This is a pseudo-tag for a subclass.

[0045] As a preferred technical solution, the focal loss function is expressed as follows:

[0046]

[0047] Among them, y i and Let represent the true value and predicted value of the image label of the i-th sample in the training data, respectively. γ is an adjustable parameter used to control the weight of misclassified samples.

[0048] The center loss function is expressed as follows:

[0049]

[0050] Where, x i Let y be the feature vector of the i-th sample in the training data. i For x i The corresponding subclasses, where N is the number of samples. It is the center of the i-th subclass.

[0051] The second objective of this invention is to provide an automatic mitosis detection system based on data and feature diversity, which is applied to the above-mentioned automatic mitosis detection method based on data and feature diversity, including a staining detection module, a sample screening module, a data expansion module, a classifier training module, and a result detection module.

[0052] The staining detection module is used to acquire pathological images and their point annotations. The pathological images are stained using HE staining, and then the stained pathological images are separated using color deconvolution to obtain hematoxylin staining channel images. The image blocks are divided to obtain candidate cells and are divided into positive samples and negative samples.

[0053] The sample screening module is used to filter out simple samples from the negative samples, and then mix the filtered samples with the positive samples to obtain training samples.

[0054] The data expansion module is used to expand the training samples to k color spaces and mix them with the original data to obtain training data.

[0055] The classifier training module is used to cluster the training data using parent class labels based on the deep learning network to obtain child class pseudo labels; and to optimize the deep learning network by combining the parent labels and child class pseudo labels until the loss function converges, thus obtaining a joint parent-child class classifier.

[0056] The result detection module is used to perform mitosis detection on the pathological image to be detected using a trained parent-child joint classifier, and obtain the detection result.

[0057] A third objective of this invention is to provide an electronic device comprising:

[0058] At least one processor; and a memory communicatively connected to said at least one processor; wherein:

[0059] The memory stores computer program instructions that can be executed by the at least one processor, which enables the at least one processor to perform the above-described automatic mitosis detection method based on data and feature diversity.

[0060] The fourth objective of this invention is to provide a computer-readable storage medium storing a program that, when executed by a processor, implements the aforementioned automatic mitosis detection method based on data and feature diversity.

[0061] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0062] 1. The detection method based on hematoxylin staining in this invention has the advantages of requiring no training and no additional annotation: Existing methods obtain mitotic candidate cells by segmenting or detecting models, which requires more time and additional annotation. They train the model with additional border-level or pixel-level annotations, use a complex model to detect mitosis, and set the detected cells as candidate cells. However, this application, based on prior knowledge of mitosis, namely that mitosis often occurs in the cell nucleus, separates the hematoxylin staining (H) channel through color deconvolution. The H channel can obtain the distribution map of the cell nucleus, and the centroid coordinates of the cell nucleus can be located according to the H channel to obtain mitotic candidate cells. No training is required, and the detection method based on hematoxylin staining can detect the vast majority of mitotic cells, naturally having a high recall rate.

[0063] 2. The diversity-based sample selection method in this invention can obtain more balanced and representative training samples: Based on the extremely imbalanced nature of mitosis, this application specifically balances the quantity, diversity, and difficulty of training samples. It selects mitotic samples in the clustered feature space, divides the clustered samples into multiple subclasses, selects an equal amount of data from each subclass, removes data redundancy while ensuring sample diversity, and then uses a classifier to remove simple samples from the selected samples, balancing the amount of sample information and making it easier for the model to learn representative features.

[0064] 3. This invention proposes a parent-child joint classifier that adds a subclass classification task to the binary classification method. It trains the parent-child joint classifier using both parent and child class labels, thus learning more diverse information. The parent-child joint classifier is based on prior knowledge of mitosis. Since mitosis involves morphological differences in prophase, metaphase, anaphase, and telophase, and these differences are significant, this invention further subdivides the class into subclasses, allowing the model to notice more detailed information about mitosis, thereby improving classification performance. Compared to existing binary classification methods, this application considers the characteristics of mitosis and designs the model based on prior knowledge, achieving more efficient performance. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is a flowchart of an automated mitotic cell detection method based on data and feature diversity in an embodiment of the present invention.

[0067] Figure 2 This is a structural diagram of the automated mitotic cell detection system based on data and feature diversity in an embodiment of the present invention.

[0068] Figure 3 This is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. Detailed Implementation

[0069] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0070] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0071] Automated mitotic detection refers to the use of computer vision technology to locate and identify mitotic cells in pathological images. It faces two main technical challenges: firstly, extreme data imbalance; and secondly, the complex morphology and structure of mitotic cells. On the one hand, pathological images differ from natural images, possessing a large number of pixels and a wide field of view. Mitotic cells, however, are often only a few dozen pixels in size, making them small targets. Therefore, automated mitotic detection is a task of detecting small targets within a large field of view. Furthermore, mitotic cells have a low density, with most cells in the image being non-mitotic, resulting in an extremely imbalanced sample in automated mitotic detection, containing a large number of non-mitotic cells and a small number of mitotic cells. On the other hand, mitotic cells are difficult to distinguish due to their complex morphology and structure, encompassing multiple stages including prophase, metaphase, anaphase, and telophase, each with distinct morphological characteristics. Additionally, the nuclei of some dense non-mitotic cells and apoptotic cells are extremely similar to mitotic cells, further increasing the difficulty of identification and detection.

[0072] Based on this, such as Figure 1 As shown in the embodiments of this application, an automated detection method for mitotic cells based on data and feature diversity is provided, including the following steps:

[0073] Step 1: Detection based on hematoxylin staining: Obtain pathological images and their point annotations, use color deconvolution to separate the pathological images to obtain hematoxylin staining channel images, divide the image blocks to obtain candidate cells and classify them into positive samples and negative samples.

[0074] Hematoxylin-eosin (HE) staining is a commonly used histopathological image staining technique. Hematoxylin (H) staining stained the cell nucleus blue, and eosin (E) staining stained the cytoplasm red. Therefore, this embodiment separates the hematoxylin-stained (H) channel image from the stained pathological image, and then obtains candidate cell samples, specifically:

[0075] 1.1 Obtain pathological image I and its point annotations, wherein the pathological image contains a cells;

[0076] 1.2. The pathological images were stained using HE staining to obtain stained pathological images;

[0077] 1.3. Because the interaction between hematoxylin and the cell nucleus causes the nucleus to appear blue, the stained pathological image is input into a color deconvolution process for separation, resulting in a hematoxylin-stained (H) channel image I. h ;

[0078] 1.4 Therefore, based on the hematoxylin staining channel image I... h The centroid coordinates O of each cell can be obtained;

[0079] 1.5. Obtain image blocks D for each cell by cutting the pathological image I according to the centroid coordinates. H ={I1,I2,...,I a}, i.e., candidate cell set;

[0080] 1.6. Based on the point annotations in pathological image I, divide the image patch into positive sample D. P and negative sample D N .

[0081] Point annotations include mitotic point annotations and non-mitotic point annotations; if a mitotic point annotation is located in an image patch, the image patch is classified as a positive sample; otherwise, it is classified as a negative sample.

[0082] Step 2, Sample screening based on diversity: Redundant and simple samples are screened out from the negative samples, and the filtered samples are mixed with the positive samples to obtain training samples;

[0083] The image patch samples obtained in step one suffer from data imbalance, meaning that negative samples outnumber positive samples. Existing methods simply use random sampling to balance the samples without considering sample diversity, resulting in the loss of a small number of representative samples during the random selection process. Therefore, this application's embodiments use sample diversity for screening to eliminate data redundancy and obtain samples with high information content to ensure sample diversity. Specifically:

[0084] 2.1 Regarding negative sample D N The K-means clustering algorithm is used to obtain k subspace clusters C = {C1, C2, ..., C}. k},in Describes the a-th subspace cluster belonging to the k-th subspace. k One sample, a k This represents the number of samples in the k-th subspace cluster;

[0085] 2.2 Select an equal number of m negative samples from each subspace cluster. in, The m-th negative sample selected for the k-th subspace cluster;

[0086] 2.3. To filter out a large number of simple samples from the negative samples, positive sample D is used. P and Train a classification network f easy-sampling Then use the classification network f easy-sampling To filter out The simple samples were left behind, while the negative samples were difficult to distinguish.

[0087] 2.4 Mixed positive sample DP And negative samples that are difficult to distinguish Obtain the final training samples

[0088] Step 3, Unsupervised staining enhancement: Expand the training samples to k color spaces and mix them with the original data to obtain the training data;

[0089] Step 4: Training the joint classifier for parent and child classes: Cluster the training data using parent class labels based on the deep learning network to obtain pseudo-labels for child classes; Combine the parent class labels and pseudo-labels for child classes to optimize the deep learning network until the loss function converges, thus obtaining the joint classifier for parent and child classes.

[0090] Existing methods treat mitosis detection as a binary classification problem, ignoring the complex morphological characteristics of mitotic cells, which prevents the model from learning good feature representations. Therefore, this application adds a subclass classification task to the binary classification to allow the model to learn more diverse information.

[0091] The parent-child joint classifier is built based on a deep learning network and includes an input layer, a feature extractor, a parent classifier, a child classifier, and an output layer. The input layer is connected to the feature extractor, which is connected to both the parent classifier and the child classifier. The fully connected layer of the parent classifier serves as the final fully connected layer, which is connected to the feature extractor and then to the output layer.

[0092] The training process of the parent-child class joint classifier is as follows:

[0093] The training data is input into the input layer, where features are extracted by the feature extractor and then input into the parent classifier.

[0094] The parent classifier classifies the features of the training data to obtain parent class labels; the parent class labels include mitotic categories and non-mitotic categories;

[0095] Based on the parent class label, the feature extractor is trained by binary classification on the training data to obtain a preliminary feature extractor;

[0096] The preliminary feature extractor is used to extract features from the samples in the training data to obtain sample features;

[0097] The subclass classifier clusters the sample features, grouping the mitotic and non-mitotic categories in the parent class label into an equal number of subclasses, and using the subclass results as subclass pseudo-labels.

[0098] The initial feature extractor is trained using both parent class labels and child class pseudo-labels until the loss function converges or the required accuracy is achieved, resulting in the final joint parent and child class classifier.

[0099] Specifically, for each parent class c = {c p ,c n}, c p For mitosis, c N For non-mitotic categories, an unsupervised clustering algorithm is used to cluster them into T subclasses. Use the unsupervised clustering results as pseudo-labels for subclasses;

[0100] Let Y be the parent class label of each sample in the training data. P Then the pseudo-label of each subclass in parent class c is Y. S The clustering objective for each parent class c is:

[0101]

[0102] Where, N c Y represents the number of samples in parent class c. S For subclass pseudo tags, For training data, The features extracted by the feature extractor from the training data, O is the matrix formed by the centroid coordinates of each cell in the training data, and 1 t Let be a t-dimensional identity matrix.

[0103] Specifically, both the parent classifier and the child classifier are trained under supervised conditions using both the focal loss function and the center loss function:

[0104] For the parent classifier f P Its loss function L P for:

[0105]

[0106] in, The focal loss function of the parent classifier. The center loss function is used for the parent classifier; for the child classifier f... S Similar to the parent classifier, its loss function L S for:

[0107]

[0108] in, The focal loss function for the subclass classifier. The center loss function is used for the subclass classifier.

[0109] The feature extractor is optimized using the loss functions of both the parent and child classifiers. The loss function of the feature extractor is expressed as follows:

[0110]

[0111] Where N is the number of samples in the training data. For training data, The features extracted by the feature extractor from the training data, λ is the balancing parameter, and θ is the balancing parameter. P and θ S These represent the parent classifier f respectively. P Subclass classifier f S The parameter, Y P For the parent category tag, Y S This is a pseudo-tag for a subclass.

[0112] In this embodiment, to ensure the model focuses on difficult-to-distinguish samples, both focal loss and center loss functions are used to simultaneously widen the distance between different classes (parent and child classes) and narrow the distance between samples within the same class. The focal loss function reduces the weight of simpler samples and focuses more on difficult-to-distinguish samples, as shown below:

[0113]

[0114] Among them, y i and Let represent the true value and predicted value of the image label of the i-th sample in the training data, respectively. γ is an adjustable parameter used to control the weight of misclassified samples.

[0115] The center loss function reduces intra-class distance by encouraging feature vectors of the same class to move closer to their corresponding class center. It is expressed as:

[0116]

[0117] Where, x i Let y be the feature vector of the i-th sample in the training data. i For x i The corresponding subclasses, where N is the number of samples, and c yi It is the center of the i-th subclass.

[0118] To verify the effectiveness and advancement of the method in this application, this embodiment trains on the MIDOG2021 dataset. A 5412*7215 image is fed into a hematoxylin staining-based detector to obtain 80*80 mitotic candidate image blocks, which are divided into mitotic image blocks and non-mitotic image blocks according to point annotations. The non-mitotic image blocks are clustered into 10 classes using a ResNet38 network pre-trained by ImgeNet, and 3000 image blocks are selected from each class, for a total of 30000 non-mitotic image blocks. A ResNet38 classification network is trained using these non-mitotic and mitotic image patches. This network is then used to filter simple samples from the non-mitotic data. The filtered non-mitotic and mitotic image patches are then mixed to obtain training samples. These training samples are normalized to three different color domains by coloring to augment the training data. The augmented training data is then mixed with the training samples to obtain the final training data. The training data is fed into the new ResNet38 network for training to obtain a feature extractor. Features are extracted from the training data, and based on the features, the mitotic and non-mitotic data are each divided into four subclasses. The subclass results are used as subclass pseudo-labels for the training data. Focal loss and Center loss are then used to supervise the parent class labels and subclass pseudo-labels, respectively, thus obtaining a joint parent-subclass classifier.

[0119] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0120] Based on the same idea as the automatic mitosis detection method based on data and feature diversity in the above embodiments, the present invention also provides an automatic mitosis detection system based on data and feature diversity, which can be used to execute the above-described automatic mitosis detection method based on data and feature diversity. For ease of explanation, the structural schematic diagram of the embodiment of the automatic mitosis detection system based on data and feature diversity only shows the parts relevant to the embodiments of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and it may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0121] like Figure 2 As shown, another embodiment of the present invention provides an automatic mitosis detection system based on data and feature diversity, including a staining detection module, a sample screening module, a data expansion module, a classifier training module, and a result detection module;

[0122] The staining detection module is used to acquire pathological images and their point annotations. The pathological images are stained using HE staining, and then the stained pathological images are separated using color deconvolution to obtain hematoxylin staining channel images. The image blocks are divided to obtain candidate cells and are divided into positive and negative samples.

[0123] The sample screening module is used to filter out simple samples from the negative samples, and then mixes them with positive samples to obtain training samples.

[0124] The data expansion module is used to expand the training samples to k color spaces and mix them with the original data to obtain the training data;

[0125] The classifier training module is used to cluster the training data based on the deep learning network using parent class labels to obtain child class pseudo labels; the deep learning network is optimized by combining the parent labels and child class pseudo labels until the loss function converges, resulting in a joint parent-child classifier;

[0126] The result detection module is used to perform mitosis detection on the pathological images to be detected using a trained parent-child joint classifier, and obtain the detection results.

[0127] It should be noted that the automatic mitosis detection system based on data and feature diversity of the present invention corresponds one-to-one with the automatic mitosis detection method based on data and feature diversity of the present invention. The technical features and beneficial effects described in the embodiments of the automatic mitosis detection method based on data and feature diversity described above are applicable to the embodiments of the automatic mitosis detection system based on data and feature diversity. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.

[0128] Furthermore, in the above embodiments of the automatic mitosis detection system based on data and feature diversity, the logical division of each program module is merely illustrative. In practical applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or the convenience of software implementation. That is, the internal structure of the automatic mitosis detection system based on data and feature diversity can be divided into different program modules to complete all or part of the functions described above.

[0129] Please see Figure 3 In one embodiment, an electronic device is provided for implementing an automatic mitosis detection method based on data and feature diversity. The electronic device may include a first processor, a first memory, and a bus, and may also include a computer program stored in the first memory and executable on the first processor, such as an automatic mitosis detection program based on data and feature diversity.

[0130] The first memory includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the first memory can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the first memory can be an external storage device of the electronic device, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. Furthermore, the first memory can include both internal and external storage units of the electronic device. The first memory can be used not only to store application software and various types of data installed on the electronic device, such as the code of an automatic mitosis detection program based on data and feature diversity, but also to temporarily store data that has been output or will be output.

[0131] In some embodiments, the first processor may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the first memory (e.g., an automatic mitosis detection program based on data and feature diversity) and calls data stored in the first memory to perform various functions of the electronic device and process data.

[0132] Figure 3 Only electronic devices with components are shown; those skilled in the art will understand that... Figure 3 The structure shown does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0133] The automatic mitosis detection program based on data and feature diversity, stored in the first memory of the electronic device, is a combination of multiple instructions. When run in the first processor, it can achieve the following:

[0134] Step 1: Detection based on hematoxylin staining: Obtain pathological images and their point annotations, stain the pathological images using HE staining, and then use color deconvolution to separate the stained pathological images to obtain hematoxylin staining channel images. Divide the image blocks to obtain candidate cells and classify them into positive and negative samples.

[0135] Step 2, Sample screening based on diversity: Redundant and simple samples are screened out from the negative samples, and the filtered samples are mixed with the positive samples to obtain training samples;

[0136] Step 3, Unsupervised staining enhancement: Expand the training samples to k color spaces and mix them with the filtered training samples to obtain training data;

[0137] Step 4: Training the joint classifier for parent and child classes: Cluster the training data using parent class labels based on the deep learning network to obtain pseudo-labels for child classes; Combine the parent class labels and pseudo-labels for child classes to optimize the deep learning network until the loss function converges, and obtain the joint classifier for parent and child classes;

[0138] Step 5: Mitosis detection: Use the trained parent-child joint classifier to perform mitosis detection on the pathological image to be detected, and obtain the detection results.

[0139] Furthermore, if the modules / units integrated in the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0140] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0141] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0142] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. An automatic mitosis detection method based on data and feature diversity, characterized in that, Includes the following steps: Step 1: Detection based on hematoxylin staining: Obtain pathological images and their point annotations, stain the pathological images using HE staining, and then use color deconvolution to separate the stained pathological images to obtain hematoxylin staining channel images. Divide the image blocks to obtain candidate cells and classify them into positive and negative samples. Step 2: Sample screening based on diversity: Redundant and simple samples are screened out from the negative samples, and the screened negative samples are mixed with positive samples to obtain training samples; Step 3, Unsupervised staining enhancement: Expand the training samples to k color spaces and then mix them with the training samples to obtain training data; Step 4: Training the Parent-Child Joint Classifier: Construct and train a parent-child joint classifier based on a deep learning network; the parent-child joint classifier includes an input layer, a feature extractor, a parent classifier, a child classifier, and an output layer; the training process of the parent-child joint classifier is as follows: The training data is input into the input layer, where features are extracted by the feature extractor and then input into the parent classifier. The parent classifier classifies the features of the training data to obtain parent class labels; the parent class labels include mitotic categories and non-mitotic categories; Based on the parent class label, the feature extractor is trained by binary classification on the training data to obtain a preliminary feature extractor; The preliminary feature extractor is used to extract features from the samples in the training data to obtain sample features; The subclass classifier clusters the sample features, grouping the mitotic and non-mitotic categories in the parent class label into an equal number of subclasses, and using the subclass results as subclass pseudo-labels. The initial feature extractor is trained using both parent class labels and child class pseudo labels until the loss function converges or the accuracy requirement is met, thus obtaining a trained parent-child joint classifier. Step 5: Mitosis detection: Use the trained parent-child joint classifier to perform mitosis detection on the pathological image to be detected, and obtain the detection results.

2. The automatic mitosis detection method based on data and feature diversity according to claim 1, characterized in that, In step one, the detection based on hematoxylin staining specifically involves: Obtain pathological image I and its point annotations, wherein the pathological image contains a cells; The pathological images were stained using HE staining to obtain stained pathological images; The stained pathological image is input into a color deconvolution process to separate and obtain the hematoxylin staining channel image I. h ; Image I of the channel stained with hematoxylin h Obtain the centroid coordinates O of each cell; Based on the centroid coordinates, pathological image I is cut to obtain image blocks D for each cell. H ={I1,I2,...,I a }; Based on the point markings in pathological image I, the image patch is divided into positive sample D. P and negative sample D N ; The point annotations include mitotic point annotations and non-mitotic point annotations; if a mitotic point annotation is located in an image block, the image block is classified as a positive sample, otherwise it is classified as a negative sample.

3. The automatic mitosis detection method based on data and feature diversity according to claim 1, characterized in that, In step two, the diversity-based sample screening specifically involves: For negative sample D N The K-means clustering algorithm was used to obtain k subspace clusters C={C1,C2,...,C...} k },in , Describes the a-th subspace cluster belonging to the k-th subspace. k One sample, a k This represents the number of samples in the k-th subspace cluster; Select an equal number of m negative samples from each subspace cluster , ,in, The m-th negative sample selected for the k-th subspace cluster; Using positive sample D P and Train a classification network f easy-sampling Then use the classification network f easy-sampling To filter out The simple samples were left behind, while the negative samples were difficult to distinguish. ; Mixed positive sample D P And negative samples that are difficult to distinguish Obtain the final training samples .

4. The automatic mitosis detection method based on data and feature diversity according to claim 1, characterized in that, The input layer is connected to the feature extractor, which is connected to both the parent classifier and the child classifier. The fully connected layer of the parent classifier serves as the final fully connected layer, which is then connected to the feature extractor and finally to the output layer.

5. The automatic mitosis detection method based on data and feature diversity according to claim 1, characterized in that, For each parent class label c={c p ,c n }, c p For mitosis, c n For non-mitotic categories, an unsupervised clustering algorithm is used to cluster them into T subclasses. , t={1,2,...,T}, use the unsupervised clustering results as pseudo-labels for subclasses; Let Y be the parent class label of each sample in the training data. P,i Then the pseudo-label of each subclass in parent class c is Y. S The clustering objective for each parent class c is: ， Where, N c Y represents the number of samples in parent class c. S,i The pseudo-label for the subclass corresponding to the i-th sample. For training data, For the features extracted by the feature extractor for the i-th sample, O i Let be the centroid coordinates of the cell corresponding to the i-th sample, 1 t Let be a t-dimensional matrix of all 1s.

6. The automatic mitosis detection method based on data and feature diversity according to claim 1, characterized in that, The parent classifier and the child classifier are supervised using both the focal loss function and the center loss function. For the parent classifier f P Its loss function L P for: ， in, The focal loss function of the parent classifier. The center loss function is used for the parent classifier. For the subclass classifier f S Its loss function L S for: ， in, The focal loss function for the subclass classifier. The center loss function is used for the subclass classifier. The feature extractor is optimized using the loss functions of both the parent and child classifiers. The loss function of the feature extractor is expressed as follows: ， Where N is the number of samples in the training data. For training data, λ represents the feature extracted by the feature extractor for the i-th sample, λ is the balancing parameter, and θ is the feature of the i-th sample. P and θ S These represent the parent classifier f respectively. P Subclass classifier f S The parameter, Y P,i Y is the parent class label corresponding to the i-th sample. S,i The pseudo-label is the subclass label corresponding to the i-th sample.

7. The automatic mitosis detection method based on data and feature diversity according to claim 6, characterized in that, The focal loss function is expressed as follows: ， Among them, y i and These represent the first and second elements in the training data, respectively. The true and predicted values ​​of the labels for each sample image, where γ is an adjustable parameter used to control the weight of misclassified samples; The center loss function is expressed as follows: ， Where, x i Let y be the feature vector of the i-th sample in the training data. i For x i The corresponding subclasses, where N is the number of samples. It is the center of the i-th subclass.

8. An automatic mitosis detection system based on data and feature diversity, characterized in that, The automatic mitosis detection method based on data and feature diversity, applied to any one of claims 1-7, includes a staining detection module, a sample screening module, a data expansion module, a classifier training module, and a result detection module; The staining detection module is used to acquire pathological images and their point annotations. The pathological images are stained using HE staining, and then the stained pathological images are separated using color deconvolution to obtain hematoxylin staining channel images. The image blocks are divided to obtain candidate cells and are divided into positive samples and negative samples. The sample screening module is used to filter out simple samples from the negative samples, and the screened negative samples are mixed with positive samples to obtain training samples. The data expansion module is used to expand the training samples to k color spaces and then mix them with the training samples to obtain training data. The classifier training module is used to construct and train a joint parent-child classifier based on a deep learning network; the joint parent-child classifier includes an input layer, a feature extractor, a parent classifier, a child classifier, and an output layer; the training process of the joint parent-child classifier is as follows: The training data is input into the input layer, where features are extracted by the feature extractor and then input into the parent classifier. The parent classifier classifies the features of the training data to obtain parent class labels; the parent class labels include mitotic categories and non-mitotic categories; Based on the parent class label, the feature extractor is trained by binary classification on the training data to obtain a preliminary feature extractor; The preliminary feature extractor is used to extract features from the samples in the training data to obtain sample features; The subclass classifier clusters the sample features, grouping the mitotic and non-mitotic categories in the parent class label into an equal number of subclasses, and using the subclass results as subclass pseudo-labels. The initial feature extractor is trained using both parent class labels and child class pseudo labels until the loss function converges or the accuracy requirement is met, thus obtaining a trained parent-child joint classifier. The result detection module is used to perform mitosis detection on the pathological image to be detected using a trained parent-child joint classifier, and obtain the detection result.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to said at least one processor; wherein: The memory stores computer program instructions that can be executed by the at least one processor to enable the at least one processor to perform the automatic mitosis detection method based on data and feature diversity as described in any one of claims 1-7.

10. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the automatic mitosis detection method based on data and feature diversity as described in any one of claims 1-7.

Images