Mitosis automatic detection method and apparatus based on data and feature diversity

Abstract

Disclosed in the present invention are a mitosis automatic detection method and apparatus based on data and feature diversity. Rapid and efficient mitosis detection can be carried out only by using point annotation. In the method, a training-free hematoxylin staining-based detection method is first used to obtain candidate samples; then a balanced sampling strategy is used to remove redundant information in the samples to balance the data information amount and maintain the diversity of the samples; in addition, simple samples are removed to obtain balanced and representative training data, thereby facilitating a classifier learning representative features. In view of the characteristic that the morphology of mitotic cells is complex, a classifier for joint training is designed. Child class division is performed on the basis of binary classification, child classes obtained after division are used as pseudo labels, parent class labels and the child class pseudo labels are combined for training, and an obtained parent class-child class joint classifier is based on prior knowledge of mitosis, such that more diverse feature information is learned, effectively improving classification performance.

Description

Automatic mitosis detection method and device based on data and feature diversity Technical Field

[0001] The present invention belongs to the technical field of image processing and mitosis detection, and in particular relates to a method and device for automatic mitosis detection based on data and feature diversity. Background Art

[0002] Mitosis reflects cell proliferation and is an important indicator for determining tumor grade and prognosis. Due to the rapid development of deep learning technology, various segmentation, object detection, and classification models have been applied to the task of automatic mitosis detection. These models can be broadly categorized into single-stage and multi-stage approaches. Single-stage approaches utilize end-to-end learning to directly use the detection results as the final detection result; for example, automatic mitosis detection can be formulated as a semantic segmentation problem. Multi-stage approaches typically consist of two stages: the first stage generates candidate cells while ensuring high recall; the second stage further classifies these candidate cells. In the prior art, a common multi-stage approach for automatic mitosis detection is: ① pixel-level pseudo-annotations are generated through additional manual pixel annotation or by using existing models (e.g., the HoVer net cell nucleus segmentation network); ② pixel-level pseudo-annotations are used to train a segmentation network. The image is then fed into the segmentation network to initially generate candidate cells while ensuring high recall (identifying as many mitotic cells as possible); and ③ the candidate cells are then classified to obtain the final result.

[0003] However, there are the following disadvantages: 1) Existing methods use complex segmentation models to complete the task of obtaining candidate cells. Complex models often require higher-level annotations. Therefore, in order to train the segmentation model, additional annotations (additional manual annotations or pseudo-annotations generated by existing models) are often required to obtain candidate cells, resulting in low efficiency. 2) Existing methods still have data imbalance when training classification networks. They often use random sampling or specific loss methods to combat class imbalance, but such practices lose data diversity and are inefficient. The classification model cannot well learn representative features. For example, random sampling methods will lose some representative data during the screening process and cannot obtain information-balanced samples. Using specific loss methods on large amounts of data can only slightly alleviate the sample imbalance problem. The model still pays too much attention to high-frequency class samples. 3) The classification models used by existing methods attempt to solve the mitosis detection problem by adjusting the model complexity. They use ensemble learning, increase network depth, and other methods to improve classification performance. However, using only complex classification models for detection can easily lead to overfitting of the existing data, resulting in poor generalization ability.

[0004] Summary of the Invention

[0005] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and provide a method and device for automatic detection of mitotic cells based on data and feature diversity. Candidate cells are obtained based on hematoxylin staining detection, and then simple samples are removed based on sample screening based on diversity to obtain balanced samples; finally, molecular classification is performed on the basis of binary classification, and based on prior knowledge of mitosis, the classification model is trained to effectively improve the classification performance.

[0006] To achieve the above objectives, the first objective of the present invention is to use a method for automatic mitosis detection based on data and feature diversity, comprising the following steps:

[0007] Step 1: Detection based on hematoxylin staining: Obtain a pathological image and its point annotations, stain the pathological image using HE staining, then use color deconvolution to separate the stained pathological image to obtain a hematoxylin-stained channel image, divide the image blocks to obtain candidate cells and classify them into positive samples and negative samples;

[0008] Step 2: Diversity-based sample screening: filter out redundant and simple samples from negative samples, and then mix them with positive samples to obtain training samples;

[0009] Step 3: Unsupervised color enhancement: Expand the training samples to k color spaces and mix them with the filtered training samples to obtain training data;

[0010] Step 4: Training the parent-child joint classifier: Cluster the training data using the parent class labels based on the deep learning network to obtain the child class pseudo labels; Optimize the deep learning network by combining the parent class labels and the child class pseudo labels until the loss function converges to obtain the parent-child joint classifier;

[0011] Step 5: Mitosis detection: Use the trained parent-child joint classifier to perform mitosis detection on the pathological image to be detected to obtain the detection result.

[0012] As a preferred technical solution, in step 1, the detection based on hematoxylin staining is specifically as follows:

[0013] Obtain a pathological image I and its point annotations, where the pathological image contains a cells;

[0014] The pathological image is stained using HE staining to obtain a stained pathological image;

[0015] The stained pathological image is input into the color deconvolution to separate the hematoxylin stained channel image I h ;

[0016] Image of hematoxylin staining channel I h Get the center of mass coordinates O of each cell;

[0017] Cut the pathological image I according to the centroid coordinates to obtain the image block D of each cell H ={I1,I2,...,I a};

[0018] According to the point annotation of the pathological image I, the image block is divided into positive samples D P and negative sample D N ;

[0019] The point annotations include mitosis point annotations and non-mitosis point annotations; if the mitosis point annotation is located in the image block, the image block is classified as a positive sample, otherwise it is classified as a negative sample.

[0020] As a preferred technical solution, in step 2, the diversity-based sample screening is specifically as follows:

[0021] For negative samples D N Use K-means clustering algorithm to get k subspace clusters C={C1,C2,…,C k},in represents the ath cluster belonging to the kth subspace cluster k samples, a k represents the number of samples in the kth subspace cluster;

[0022] Select an equal number of m negative samples in each subspace cluster in, The mth negative sample selected for the kth subspace cluster;

[0023] Using positive sample D P and Train a classification network feasy-sampling, and then use the classification network feasy-sampling to filter out Simple samples in the , leaving behind difficult-to-distinguish negative samples

[0024] Mixed positive sample D P and difficult-to-distinguish negative samples Get the final training sample

[0025] As a preferred technical solution, the parent-subclass joint classifier is constructed based on a deep learning network, including an input layer, a feature extractor, a parent classifier, a subclass classifier, and an output layer; the input layer is connected to the feature extractor, and the feature extractor is connected to the parent classifier and the subclass classifier respectively; the fully connected layer of the parent class classifier is connected to the feature extractor as the final fully connected layer and then to the output layer;

[0026] The training process of the parent-child joint classifier is as follows:

[0027] Input the training data into the input layer, extract features through the feature extractor, and input into the parent class classifier;

[0028] The parent class classifier classifies the features of the training data to obtain a parent class label; the parent class label includes a mitosis class and a non-mitosis class;

[0029] Based on the parent class label, the feature extractor is trained on the training data for binary classification to obtain a preliminary feature extractor;

[0030] Use the preliminary feature extractor to extract features from samples in the training data to obtain sample features;

[0031] The subclass classifier clusters the sample features, clustering the mitosis and non-mitosis categories in the parent class label into multiple subclasses of equal number, and uses the subclass results as subclass pseudo labels;

[0032] The preliminary feature extractor is trained together with the parent class label and the child class pseudo label until the loss function converges or meets the accuracy requirement, and the final parent class and child class joint classifier is obtained.

[0033] As a preferred technical solution, for each parent class label c={c p ,c n}, c p is the mitosis category, c n For non-mitotic categories, use unsupervised clustering algorithm to cluster them into T subcategories Use the unsupervised clustering results as subclass pseudo labels;

[0034] Let the parent class label of each sample in the training data be Y P , then the subclass pseudo label corresponding to each subclass in the parent class c is Y S , the clustering target of each parent class c is:

[0035] Among them, N c is the number of samples in parent class c, Y S is the subclass pseudo label, is the training data, is the feature of the training data extracted by the feature extractor, O is the matrix composed of the centroid coordinates of each cell in the training data, 1 t is the t-dimensional identity matrix.

[0036] As a preferred technical solution, the parent class classifier and the child classifier are supervised using both the focal loss function and the center loss function.

[0037] For the parent classifier f P , its loss function L P for:

[0038] in, is the focal loss function of the parent class classifier, is the center loss function of the parent class classifier;

[0039] For the subclass classifier f S , its loss function L S for:

[0040] in, is the focal loss function of the subclass classifier, is the center loss function of the subclass classifier;

[0041] The loss function of the parent classifier and the child classifier is used to jointly optimize the feature extractor. The loss function of the feature extractor is expressed as:

[0042] Where N is the number of samples in the training data, is the training data, is the feature of the training data extracted by the feature extractor, λ is the balance parameter, θ P and θ S Represents the parent class classifier f P and subclass classifier f S Parameters, Y P is the parent class label, Y S is the pseudo label of the subclass.

[0043] As a preferred technical solution, the focal loss function is expressed as:

[0044] Among them, y i and Represent the true value and predicted value of the i-th sample image label in the training data, respectively. γ is an adjustable parameter used to control the weight of misclassified samples.

[0045] The center loss function is expressed as:

[0046] Among them, x i is the feature vector of the i-th sample in the training data, y i is x iThe corresponding subclass category, N is the number of samples, is the center of the ith subclass.

[0047] A second object of the present invention is to provide an automatic mitosis detection system based on data and feature diversity, which is applied to the above-mentioned automatic mitosis detection method based on data and feature diversity, and includes a staining detection module, a sample screening module, a data expansion module, a classifier training module, and a result detection module;

[0048] The staining detection module is used to obtain a pathological image and its point annotations, stain the pathological image using HE staining, separate the stained pathological image using color deconvolution to obtain a hematoxylin staining channel image, divide the image blocks to obtain candidate cells and divide them into positive samples and negative samples;

[0049] The sample screening module is used to screen out simple samples from negative samples and mix them with positive samples to obtain training samples.

[0050] The data expansion module is used to expand the training samples into k color spaces and mix them with the original data to obtain training data;

[0051] The classifier training module is used to cluster the training data using the parent class labels based on the deep learning network to obtain the child class pseudo labels; the deep learning network is optimized by combining the parent class labels and the child class pseudo labels until the loss function converges to obtain a parent class and child class joint classifier;

[0052] The result detection module is used to use the trained parent-subclass joint classifier to perform mitosis detection on the pathological image to be detected to obtain a detection result.

[0053] A third object of the present invention is to provide an electronic device, comprising:

[0054] at least one processor; and a memory communicatively coupled to the at least one processor; wherein:

[0055] The memory stores computer program instructions that can be executed by the at least one processor. The computer program instructions are executed by the at least one processor to enable the at least one processor to perform the above-mentioned automatic mitosis detection method based on data and feature diversity.

[0056] A fourth object of the present invention is to provide a computer-readable storage medium storing a program, which, when executed by a processor, implements the above-mentioned automatic mitosis detection method based on data and feature diversity.

[0057] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0058] 1. The detection method based on hematoxylin staining in the present invention has the advantages of no need for training and no additional labeling: the existing method of obtaining mitotic candidate cells through segmentation or detection model requires more time and additional labeling. It trains the model through additional border-level labeling or pixel-level labeling, uses a complex model to detect mitosis, and sets the detected cells as candidate cells; while the present application is based on the prior knowledge of mitosis, that is, mitosis often occurs in the cell nucleus, and separates the hematoxylin staining (H) channel through color deconvolution. The H channel can obtain the distribution map of the cell nucleus, and locates the center of mass coordinates of the cell nucleus according to the H channel to obtain the mitotic candidate cells. No training is required, and the detection method based on hematoxylin staining can detect the vast majority of mitotic cells, and naturally has a high recall rate.

[0059] 2. The diversity-based sample screening method in the present invention can obtain more balanced and representative training samples: Based on the extremely unbalanced characteristics of mitosis, this application balances the number, diversity and difficulty of training samples in a targeted manner, selects mitosis samples in the feature space after clustering, divides the clustered samples into multiple subclasses, selects an equal amount of data from each subclass, removes data redundancy while ensuring sample diversity, and then uses a classifier to remove simple samples from the selected samples, thereby balancing the amount of sample information and making it easier for the model to learn representative features.

[0060] 3. The present invention proposes a parent-subclass joint classifier that adds subclass classification tasks on the basis of binary classification, and simultaneously trains the parent-subclass joint classifier by combining parent-class labels and subclass labels to learn more diverse information. The parent-subclass joint classifier is based on prior knowledge of mitosis. Since there are morphological differences in the early, middle, late, and late stages of mitosis, and the morphological differences of mitosis in different stages are relatively large, the present invention further subdivides subclasses within the class, allowing the model to pay attention to more detailed information about mitosis, thereby improving classification performance. Compared with the existing binary classification method, this application pays attention to the characteristics of mitosis and designs the model based on prior knowledge to achieve more efficient performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0062] FIG1 is a flow chart of a method for automatically detecting mitotic cells based on data and feature diversity in an embodiment of the present invention.

[0063] FIG2 is a structural diagram of an automatic mitotic cell detection system based on data and feature diversity in an embodiment of the present invention.

[0064] FIG3 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0066] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0067] Automatic mitosis detection uses computer vision to locate and identify mitotic cells in pathology images. This approach faces two major technical challenges: extreme data imbalance and the complex morphological structure of mitotic cells. Unlike natural images, pathology images have a higher number of pixels and a wider field of view, whereas mitotic cells are often only a few dozen pixels across and are therefore relatively small. Therefore, automatic mitosis detection is a task of detecting small targets within a large field of view. Furthermore, the density of mitotic cells is low, with the majority of cells in an image being non-mitotic. Consequently, the sample size for automatic mitosis detection is extremely unbalanced, with a large number of non-mitotic cells and a small number of mitotic cells. Furthermore, mitotic cells are difficult to distinguish due to their complex morphological structure, encompassing multiple phases: prophase, metaphase, anaphase, and telophase, each with distinct morphological features. Furthermore, some dense non-mitotic nuclei and apoptotic cells can closely resemble mitotic cells, further complicating detection.

[0068] Based on this, as shown in FIG1 , an embodiment of the present application provides a method for automatically detecting mitotic cells based on data and feature diversity, comprising the following steps:

[0069] Step 1: Detection based on hematoxylin staining: Obtain a pathological image and its point annotations, use color deconvolution to separate the pathological image to obtain a hematoxylin staining channel image, divide the image blocks to obtain candidate cells and classify them into positive samples and negative samples.

[0070] Hematoxylin-eosin (HE) staining is a commonly used histopathology image staining technique. Hematoxylin (H) staining stains the cell nucleus blue, and eosin (E) staining stains the cytoplasm red. Therefore, this embodiment separates the stained pathology image to obtain a hematoxylin-stained (H) channel image, and then obtains a candidate cell segmentation sample, specifically:

[0071] 1.1. Obtain a pathological image I and its point annotations, where the pathological image contains a cells;

[0072] 1.2. Use HE staining to stain the pathological image to obtain a stained pathological image;

[0073] 1.3. Since the interaction between hematoxylin and cell nucleus makes the cell nucleus appear blue, the stained pathological image is input into color deconvolution for separation to obtain the hematoxylin stained (H) channel image I h ;

[0074] 1.4, therefore, the image of channel I is stained with hematoxylin h The center of mass coordinates O of each cell can be obtained;

[0075] 1.5. Cut the pathological image I according to the centroid coordinates to obtain the image block D of each cell H ={I1,I2,...,I a}, i.e., candidate cell set;

[0076] 1.6. Divide the image blocks into positive samples D according to the point annotations of the pathological image I P and negative sample D N .

[0077] Point annotations include mitosis point annotations and non-mitosis point annotations; if the mitosis point annotation is located in the image block, the image block is classified as a positive sample, otherwise it is classified as a negative sample.

[0078] Step 2: Diversity-based sample screening: filter out redundant and simple samples from negative samples, and then mix them with positive samples to obtain training samples;

[0079] The image block samples obtained in step 1 have a data imbalance problem, that is, the negative samples are larger than the positive samples. The existing method only uses simple random sampling to balance the samples, without considering the diversity of the samples, resulting in the loss of a small number of representative samples in the random selection process. Therefore, the embodiment of the present application screens the samples based on their diversity to eliminate data redundancy, obtain samples with high information content and ensure sample diversity. Specifically:

[0080] 2.1. For negative sample D N Use K-means clustering algorithm to get k subspace clusters C={C1,C2,…,C k},in represents the ath cluster belonging to the kth subspace cluster k samples, a k represents the number of samples in the kth subspace cluster;

[0081] 2.2. Select an equal number of m negative samples in each subspace cluster in, The mth negative sample selected for the kth subspace cluster;

[0082] 2.3. In order to filter out a large number of simple samples from negative samples, positive samples D are used P and Train a classification network feasy-sampling, and then use the classification network feasy-sampling to filter out Simple samples in the , leaving behind difficult-to-distinguish negative samples

[0083] 2.4. Mixed positive sample D P and difficult-to-distinguish negative samples Get the final training sample

[0084] Step 3: Unsupervised color enhancement: Expand the training samples to k color spaces and mix them with the original data to obtain training data;

[0085] Step 4: Training of parent-subclass joint classifier: Cluster the training data using parent class labels based on the deep learning network to obtain subclass pseudo labels; optimize the deep learning network by combining parent class labels and subclass pseudo labels until the loss function converges to obtain the parent-subclass joint classifier.

[0086] Existing methods treat the mitosis detection task as a binary classification problem, ignoring the complex morphological characteristics of mitotic cells, which prevents the model from learning good feature expression. Therefore, this application adds subclass classification tasks on the basis of binary classification to allow the model to learn more diverse information.

[0087] Among them, the parent-subclass joint classifier is constructed based on a deep learning network, including an input layer, a feature extractor, a parent classifier, a subclass classifier and an output layer; the input layer is connected to the feature extractor, and the feature extractor is connected to the parent classifier and the subclass classifier respectively; the fully connected layer of the parent class classifier is connected to the feature extractor as the final fully connected layer and then to the output layer.

[0088] The training process of the parent-child joint classifier is:

[0089] Input the training data into the input layer, extract features through the feature extractor, and input into the parent class classifier;

[0090] The parent class classifier classifies the features of the training data to obtain the parent class label; the parent class label includes mitosis class and non-mitosis class;

[0091] Based on the parent class label, the feature extractor is trained on the training data for binary classification to obtain a preliminary feature extractor;

[0092] Use the preliminary feature extractor to extract features from samples in the training data to obtain sample features;

[0093] The subclass classifier clusters the sample features, clustering the mitosis and non-mitosis categories in the parent class label into multiple subclasses of equal number, and uses the subclass results as subclass pseudo labels;

[0094] The preliminary feature extractor is trained together with the parent class label and the child class pseudo label until the loss function converges or meets the accuracy requirement, and the final parent class and child class joint classifier is obtained.

[0095] Specifically, for each parent class c={c p ,c n}, c p is the mitosis category, c n For non-mitotic categories, use unsupervised clustering algorithm to cluster them into T subcategories Use the unsupervised clustering results as subclass pseudo labels;

[0096] Let the parent class label of each sample in the training data be Y P , then the subclass pseudo label corresponding to each subclass in the parent class c is Y S , the clustering target of each parent class c is:

[0097] Among them, N c is the number of samples in parent class c, Y S is the subclass pseudo label, is the training data, is the feature of the training data extracted by the feature extractor, O is the matrix composed of the centroid coordinates of each cell in the training data, 1 t is the t-dimensional identity matrix.

[0098] Specifically, the parent classifier and the child classifier use both the focal loss loss function and the center loss loss function for supervised training:

[0099] For the parent classifier f P , its loss function L P for:

[0100] in, is the focal loss function of the parent class classifier, is the center loss function of the parent class classifier;

[0101] For the subclass classifier f S , similar to the parent classifier, its loss function L S for:

[0102] in, is the focal loss function of the subclass classifier, is the center loss function of the subclass classifier;

[0103] The loss function of the parent classifier and the child classifier is used to jointly optimize the feature extractor. The loss function of the feature extractor is expressed as:

[0104] Where N is the number of samples in the training data, is the training data, is the feature of the training data extracted by the feature extractor, λ is the balance parameter, θ P and θ S Represents the parent class classifier f P and subclass classifier f S Parameters, Y P is the parent class label, Y S is the pseudo label of the subclass.

[0105] In order to make the model focus on samples that are difficult to distinguish, the embodiment of the present application uses the focal loss loss function and the center loss loss function to simultaneously widen the distance between different classes for the parent class and the child class, and narrow the distance between the same class; wherein, the focal loss loss function will reduce the weight of simple samples and pay more attention to difficult-to-distinguish samples, which is expressed as:

[0106] Among them, y i and Represent the true value and predicted value of the i-th sample image label in the training data, respectively. γ is an adjustable parameter used to control the weight of misclassified samples.

[0107] The center loss function shortens the intra-class distance by encouraging feature vectors of the same class to be close to their corresponding class center, which is expressed as:

[0108] Among them, x i is the feature vector of the i-th sample in the training data, y i is x i The corresponding subclass category, N is the number of samples, is the center of the ith subclass.

[0109] To verify the effectiveness and advancement of the method of this application, this embodiment is trained on the MIDOG2021 dataset, and the 5412*7215 image is sent to the detector based on hematoxylin staining to obtain 80*80 mitosis candidate image blocks, which are divided into mitosis image blocks and non-mitosis image blocks according to point annotations; the non-mitosis image blocks are clustered through the ResNet38 network pre-trained by ImgeNet, and clustered into 10 classes, and 3000 image blocks are selected from each class, for a total of 30,000 non-mitosis image blocks. A ResNet38 classification network is trained using these non-mitotic image blocks and mitotic image blocks. This classification network is used to screen simple samples in non-mitosis, and the screened non-mitotic image blocks and mitotic image blocks are mixed to obtain training samples. These training samples are standardized to three different color domains by dyeing to expand the training data, and the expanded training data and training samples are mixed to obtain the final training data. The training data is sent to a new ResNet38 network for training to obtain a feature extractor, and feature extraction is performed on the training data. Mitosis data and non-mitosis data are each divided into four subclasses based on the features. The subclass results are used as subclass pseudo labels of the training data, and Focalloss and Center loss are used to supervise the parent class labels and subclass pseudo labels, respectively, to obtain a parent-subclass joint classifier.

[0110] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0111] Based on the same concept as the automatic mitosis detection method based on data and feature diversity in the above-mentioned embodiment, the present invention also provides an automatic mitosis detection system based on data and feature diversity, which can be used to implement the above-mentioned automatic mitosis detection method based on data and feature diversity. For ease of explanation, the structural diagram of the embodiment of the automatic mitosis detection system based on data and feature diversity only shows the parts relevant to the embodiment of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation of the device, and the device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0112] As shown in FIG2 , another embodiment of the present invention provides an automatic mitosis detection system based on data and feature diversity, including a staining detection module, a sample screening module, a data expansion module, a classifier training module, and a result detection module;

[0113] The staining detection module is used to obtain pathological images and their point annotations, stain the pathological images using the HE staining method, and then use color deconvolution to separate the stained pathological images to obtain hematoxylin staining channel images. The image blocks are divided to obtain candidate cells and divided into positive samples and negative samples.

[0114] The sample screening module is used to filter out simple samples from negative samples and then mix them with positive samples to obtain training samples;

[0115] The data expansion module is used to expand the training samples into k color spaces and mix them with the original data to obtain training data;

[0116] The classifier training module is used to cluster the training data using the parent class labels based on the deep learning network to obtain the child class pseudo labels. The deep learning network is optimized by combining the parent class labels and the child class pseudo labels until the loss function converges, thus obtaining a parent-child class joint classifier.

[0117] The result detection module is used to use the trained parent-subclass joint classifier to perform mitosis detection on the pathological image to be detected to obtain the detection result.

[0118] It should be noted that the automatic mitosis detection system based on data and feature diversity of the present invention corresponds one-to-one to the automatic mitosis detection method based on data and feature diversity of the present invention. The technical features and beneficial effects described in the above-mentioned embodiment of the automatic mitosis detection method based on data and feature diversity are all applicable to the embodiment of the automatic mitosis detection system based on data and feature diversity. For specific contents, please refer to the description in the embodiment of the method of the present invention. No further details will be given here. This is hereby declared.

[0119] In addition, in the implementation of the automatic mitosis detection system based on data and feature diversity in the above-mentioned embodiment, the logical division of each program module is only an example. In actual application, the above-mentioned functions can be distributed to different program modules as needed, for example, for the configuration requirements of the corresponding hardware or the convenience of software implementation. That is, the internal structure of the automatic mitosis detection system based on data and feature diversity is divided into different program modules to complete all or part of the functions described above.

[0120] Please refer to Figure 3. In one embodiment, an electronic device that implements a method for automatic mitosis detection based on data and feature diversity is provided. The electronic device may include a first processor, a first memory and a bus. It may also include a computer program stored in the first memory and executable on the first processor, such as an automatic mitosis detection program based on data and feature diversity.

[0121] The first memory includes at least one type of readable storage medium, including a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the first memory may be an internal storage unit of the electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the first memory may also be an external storage device of the electronic device, such as a plug-in mobile hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Furthermore, the first memory may also include both an internal storage unit of the electronic device and an external storage device. The first memory can be used not only to store application software and various types of data installed in the electronic device, such as the code of an automatic mitosis detection program based on data and feature diversity, but also to temporarily store data that has been output or is about to be output.

[0122] In some embodiments, the first processor may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including a combination of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines, and executing programs or modules stored in the first memory (such as an automatic mitosis detection program based on data and feature diversity, etc.), as well as calling data stored in the first memory, to perform various functions of the electronic device and process data.

[0123] FIG3 only shows an electronic device with components. Those skilled in the art will understand that the structure shown in FIG3 does not constitute a limitation on the electronic device, and may include fewer or more components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0124] The automatic mitosis detection program based on data and feature diversity stored in the first memory of the electronic device is a combination of multiple instructions, and when executed in the first processor, can achieve:

[0125] Step 1: Detection based on hematoxylin staining: Obtain a pathological image and its point annotations, stain the pathological image using HE staining, then use color deconvolution to separate the stained pathological image to obtain a hematoxylin-stained channel image, divide the image blocks to obtain candidate cells and classify them into positive samples and negative samples;

[0126] Step 2: Diversity-based sample screening: filter out redundant and simple samples from negative samples, and then mix them with positive samples to obtain training samples;

[0127] Step 3: Unsupervised color enhancement: Expand the training samples to k color spaces and mix them with the filtered training samples to obtain training data;

[0128] Step 4: Training the parent-child joint classifier: Cluster the training data using the parent class labels based on the deep learning network to obtain the child class pseudo labels; Optimize the deep learning network by combining the parent class labels and the child class pseudo labels until the loss function converges to obtain the parent-child joint classifier;

[0129] Step 5: Mitosis detection: Use the trained parent-child joint classifier to perform mitosis detection on the pathological image to be detected to obtain the detection result.

[0130] Furthermore, if the module / unit integrated into the electronic device is implemented as a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0131] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0132] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0133] The above embodiments are preferred implementations of the present invention, but the implementations of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. An automatic mitosis detection method based on data and feature diversity, characterized in that, Including the following steps: Step 1, Detection based on hematoxylin staining: Obtain a pathological image and its point annotations, stain the pathological image using the HE staining method, then use color deconvolution to separate the stained pathological image to obtain a hematoxylin staining channel image, divide the image into patches to obtain candidate cells and classify them into positive and negative samples; Step 2, Sample screening based on diversity: Screen out redundant and simple samples in the negative samples, and mix the screened samples with the positive samples to obtain training samples; Step 3, Unsupervised staining enhancement: Expand the training samples to k color spaces and mix them with the training samples obtained after screening to obtain training data; Step 4, Training of the parent - child joint classifier: Cluster the training data using the parent label based on a deep learning network to obtain child pseudo - labels; jointly optimize the deep learning network with the parent label and the child pseudo - labels until the loss function converges to obtain the parent - child joint classifier; Step 5, Detection of mitosis: Use the trained parent - child joint classifier to perform mitosis detection on the pathological image to be detected to obtain the detection result.

2. The automatic mitosis detection method based on data and feature diversity according to claim 1, wherein In Step 1, the detection based on hematoxylin staining is specifically as follows: Obtain a pathological image I and its point annotations, where the pathological image contains a cells; Stain the pathological image using the HE staining method to obtain a stained pathological image; Input the stained pathological image into color deconvolution for separation to obtain the hematoxylin stained channel image I h ; From the hematoxylin-stained channel image I h Obtain the centroid coordinates O of each cell; Cut the pathological image I according to the centroid coordinates to obtain the image patches D of each cell H ={I1, I2, ..., I a}; Divide the image patches into positive samples D P and negative samples D N ; The point annotations include mitosis point annotations and non - mitosis point annotations; if the mitosis point annotation is located in the image patch, then the image patch is classified as a positive sample, otherwise it is classified as a negative sample.

3. The automatic mitosis detection method based on data and feature diversity according to claim 1, characterized in that, In Step 2, the sample screening based on diversity is specifically as follows: For negative sample D N Use the K-means clustering algorithm to obtain k sub-space clusters C = {C1, C2, …, C k}, where Denote the \(a\)-th sample belonging to the \(k\)-th subspace cluster k , where \(a\) k represents the number of samples in the \(k\)-th subspace cluster; Select an equal number of m negative samples in each subspace cluster Among them, The m - th negative sample selected for the k - th subspace cluster; Use positive sample D P and Train a classification network feasy - sampling, and then use the classification network Use feasy-sampling to filter out Simple samples among them, leaving indistinguishable negative samples Mixed positive sample D P and indistinguishable negative samples Obtain the final training samples 4. The automatic mitosis detection method based on data and feature diversity according to claim 1, characterized in that, The parent - child joint classifier is constructed based on a deep learning network, including an input layer, a feature extractor, a parent classifier, a child classifier, and an output layer; the input layer is connected to the feature extractor, and the feature extractor is respectively connected to the parent classifier and the child classifier; the fully - connected layer of the parent classifier is used as the final fully - connected layer and is connected to the output layer after being connected to the feature extractor; The training process of the parent - child joint classifier is as follows: Input the training data into the input layer, extract features through the feature extractor, and input them into the parent classifier; The parent classifier classifies the features of the training data to obtain parent labels; the parent labels include mitosis categories and non - mitosis categories; Based on the parent labels, perform binary classification training on the feature extractor on the training data to obtain a preliminary feature extractor; Use the preliminary feature extractor to extract features from the samples in the training data to obtain sample features; The child classifier clusters the sample features, clusters both the mitosis category and the non - mitosis category in the parent labels into an equal number of multiple sub - classes, and uses the sub - class results as child pseudo - labels; Jointly train the preliminary feature extractor with the parent labels and the child pseudo - labels until the loss function converges or meets the accuracy requirement to obtain the final parent - child joint classifier.

5. The automatic mitosis detection method based on data and feature diversity according to claim 4, characterized in that, For each parent class label c = {c p , c n}, where c p is a mitotic category and c n is a non - mitotic category, use an unsupervised clustering algorithm to cluster it into T sub - classes t = {1, 2, …, T}, and take the unsupervised clustering result as the pseudo - label of the sub - class; Let the parent class label of each sample in the training data be Y P , then the subclass pseudo-label corresponding to each subclass in the parent class c is Y S , and the clustering objective for each parent class c is as follows: Among them, N c is the number of samples in the parent class c, Y S is the pseudo-label of the subclass, as training data F is the feature of the training data extracted by the feature extractor, O is the matrix composed of the centroid coordinates of each cell in the training data, and I t is the t-dimensional identity matrix.

6. The automatic mitosis detection method based on data and feature diversity according to claim 4, characterized in that The parent classifier and the sub-classifier are both supervised using the focal loss function and the center loss function; For the parent classifier f P , its loss function L P is as follows: Among them, The focal loss function for the parent classifier is the center loss function of the parent classifier; For the subclass classifier f S , its loss function L S is as follows: Among them, The focal loss function for the sub-classifier is the center loss function of the sub-classifier; The loss functions of the parent classifier and the child classifier are used to jointly optimize the feature extractor, and the loss function of the feature extractor is expressed as: where N is the number of samples in the training data, For training data, is the feature of the training data extracted by the feature extractor, λ is the balance parameter, θ P and θ S respectively represent the parameters of the parent classifier f P and the child classifier f S , Y P is the parent class label, Y S is the child class pseudo-label.

7. The automatic mitosis detection method based on data and feature diversity according to claim 6, wherein The focal loss function is expressed as: Among them, y i and respectively represent the true value and the predicted value of the label of the i-th sample image in the training data, and γ is an adjustable parameter used to control the weight of misclassified samples; The center loss function is expressed as: where x i is the feature vector of the i-th sample in the training data, y i is the subclass category corresponding to x i , N is the number of samples, is the center of the i-th sub-class.

8. An automatic mitosis detection system based on data and feature diversity, characterized in that, Applied to the automatic mitosis detection method based on data and feature diversity according to any one of claims 1-7, including a staining detection module, a sample screening module, a data expansion module, a classifier training module, and a result detection module; The staining detection module is used to obtain a pathological image and its point annotation, stain the pathological image using the HE staining method, then separate the stained pathological image using color deconvolution to obtain a hematoxylin staining channel image, divide the image into patches to obtain candidate cells and classify them into positive samples and negative samples; The sample screening module is used to screen out simple samples in the negative samples, and after screening, mix them with the positive samples to obtain training samples; The data expansion module is used to expand the training samples to k color spaces and mix them with the original data to obtain training data; The classifier training module is used to cluster the training data using the parent label based on a deep learning network to obtain sub- class pseudo-labels; jointly optimize the deep learning network by combining the parent label and the sub-class pseudo-labels until the loss function converges to obtain a parent-subclass joint classifier; The result detection module is used to perform mitosis detection on the pathological image to be detected using the trained parent-subclass joint classifier to obtain a detection result.

9. An electronic device, characterized in that, The electronic device includes: at least one processor; and, a memory communicatively connected to the at least one processor; wherein: The memory stores computer program instructions executable by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the automatic mitosis detection method based on data and feature diversity according to any one of claims 1-7.

10. A computer-readable storage medium stores a program, characterized in that, When the program is executed by the processor, it implements the automatic mitosis detection method based on data and feature diversity according to any one of claims 1-7.

Images