Weak supervision cell nucleus classification method based on domain adaptation

Abstract

The invention relates to a weak supervision cell nucleus classification method based on domain adaptation. Firstly, a single cell nucleus is cut out, feature extraction and high-dimensional vector mapping are carried out based on a convolutional neural network, then data alignment and label transfer are carried out in a high-dimensional space to enable data in a target domain to have label values, and finally, a classifier is used for cell nucleus classification task prediction. The method has the innovation points that 1) data integration is carried out based on a domain adaptation method, and the data difference between a source domain and a target domain is reduced; 2) the training process is supervised by using weak supervised learning, so that the dependence on label data is reduced; and 3) single cells are cut to construct a new data set, and the training process of the model is simplified. Compared with an existing cell nucleus classification method, the method has the advantages that the calculation complexity is reduced, the workload of manually labeling cells by doctors is reduced, and the method has a relatively high practical value clinically.

Description

Technical Field

[0001] The present invention relates to the technical field of cell nucleus segmentation and classification of medical images, and in particular to a weakly supervised cell nucleus classification method based on domain adaptation. Background Art

[0002] Digital pathology images are considered an important basis for clinical diagnosis and treatment. High-resolution images created by scanning tissue sections can provide clear tissue structure information, allowing pathologists to clearly observe the internal structure of tissues and cells, and can improve the efficiency and accuracy of disease diagnosis. The morphological and structural changes of cell nuclei are important indicators of the occurrence of many diseases, and the accurate classification of cell nuclei helps pathologists determine tumor areas and tumor types. Therefore, the accurate segmentation and classification of cell nuclei in pathology images is of great significance for promoting precision treatment.

[0003] Since different types of cell nuclei may have similar morphological and structural characteristics, and the same type of cell nuclei also have large differences at different stages, the classification of cell nuclei is still a huge challenge. Traditional cell nucleus classification methods mainly extract the morphological, texture and topological features inside the cell nucleus, and then use machine learning algorithms (support vector machines, random forests, etc.) to complete the classification prediction of cell nuclei. However, these methods are overly dependent on the design of manual features. If the representativeness of the features is not strong or the number of manual features is not enough to support the judgment of the classifier, the accuracy of cell nucleus classification will be reduced, and there will be no clinical treatment value. With the development and maturity of deep learning technology, convolutional neural network models with good classification performance in natural images are gradually applied to cell nucleus classification. HoVer-Net uses the horizontal and vertical distances from nuclear pixels to the centroid to effectively solve the problem of overlapping boundaries of cell nuclei, and designs three branches to first separate nuclear pixels from the background, then separate cell nuclei with overlapping boundaries, and finally aggregate the nuclear pixels of cells to predict the type. This is the first deep neural network that can simultaneously segment and classify cell nuclei in tissue pathology images. However, this method requires a large amount of data with cell nucleus segmentation and classification annotations for training, which consumes a lot of time and economic costs in actual operation. In addition, for cell nucleus classification tasks in unknown fields, the generalization ability of the model needs to be further verified. In summary, there is still a lot of room for improvement in data utilization and method innovation in the existing cell nucleus classification work. Summary of the invention

[0004] In view of the problems existing in the prior art, the main purpose of the present invention is to propose a method for weakly supervised cell nucleus classification tasks based on domain adaptation, and the ultimate goal is to reduce the data difference between the source domain and the target domain, so as to achieve the effect of simultaneously classifying multiple cell nuclei in the source domain and the target domain data. First, two widely used pathological image cell nucleus datasets with labels are selected as the source domain and target domain data in the domain adaptation method, and single cells are cut out according to the label values ​​of the two datasets. Then, a convolutional neural network is used to extract features from single cell images, and a fully connected network is used as a classifier to classify the cell nuclei of the training data. At the same time, a weakly supervised learning method is used to supervise the entire training process to achieve the effect of reducing the data difference between the source domain and the target domain. Compared with the existing technology, the present invention has obvious progress and advantages, and can effectively solve many problems in the existing cell nucleus classification technology, such as small amount of labeled data and large differences between data sets, and can achieve more accurate and complete cell nucleus classification effect at the same time.

[0005] In order to achieve the above object, the present invention provides a weakly supervised cell nucleus classification method based on domain adaptation, comprising the following steps:

[0006] Step 1) perform cropping preprocessing on the two cell nucleus datasets and construct corresponding new datasets;

[0007] Step 2) Extract features using a convolutional neural network based on the source domain dataset and perform high-dimensional mapping;

[0008] Step 3) Extract features and perform high-dimensional mapping on the target domain data based on the pre-trained network of the source domain data;

[0009] Step 4) Aligning the target domain data to the source domain data and transferring the labels are realized through the Gaussian mixture model in the high-dimensional mapping space;

[0010] Step 5) Iterate the process of step 4) until all target domain data have completed data alignment and label transfer;

[0011] Step 6) using the classifier model to perform cell nucleus classification prediction on the target domain data that has completed data integration and label transfer;

[0012] Furthermore, the clipping and preprocessing of the cell nucleus data set in step 1) includes:

[0013] Determine the mask value through label for the input RGB cell nucleus image data;

[0014] The boundary contour of the cell nucleus is determined by the mask of a single cell nucleus, and the minimum circumscribed rectangle of the cell nucleus and the geometric moment of the contour are calculated based on this;

[0015] Moments were used to calculate the centroid coordinates of individual cell nuclei;

[0016] The nucleus image data of the source and target domains are cropped using the centroid coordinates and minimum bounding rectangle of the nucleus.

[0017] Furthermore, the feature extraction and high-dimensional mapping based on source domain data in step 2) include:

[0018] Image preprocessing based on data enhancement (translation, rotation, scaling) of source domain data;

[0019] Use the residual neural network ResNet to extract features from the cell nucleus image data with labeled values ​​in the source domain;

[0020] The generated feature maps are globally averaged pooled, and the spatial dimension of each feature map is reduced to a single value to form a high-dimensional feature vector;

[0021] Use the classifier to make initial category predictions for source domain data based on cell features, determine the discriminative ability of cell features, and evaluate the reliability of the cell classification model based on classification loss.

[0022] Furthermore, in step 3), feature extraction and high-dimensional mapping are performed on the target domain data, including:

[0023] Image preprocessing based on data enhancement (translation, rotation, scaling) of target domain data;

[0024] Based on the pre-trained network ResNet, feature extraction is performed on the cell nucleus image data without label values ​​in the target domain data;

[0025] Calculate the feature vector of the high-dimensional mapping of the feature map to obtain the feature extraction.

[0026] Furthermore, the data alignment and label transfer in step 4) include:

[0027] In high-dimensional space, the known cell categories with labels in the source domain are separated from each other, while the target domain data without labels are clustered together.

[0028] A Gaussian mixture model is established for the source domain data and the target domain data to help find abnormal samples in the target domain and improve the robustness of the system;

[0029] Compare the cell features of known cell categories in the source domain with the features calculated in the target domain;

[0030] Align the cell classification features of the source and target domains;

[0031] Transfer the aligned source domain labels to the target domain.

[0032] Further, the specific steps of step 5) include:

[0033] Count the target domain data that does not have cell labels yet;

[0034] Iterate the above data pair and source domain to target domain label transfer steps;

[0035] Gradually narrow the data gap between the source and target domains until all unknown types are labeled with cells.

[0036] Further, the specific steps of step 6) include:

[0037] Use the fully connected layer for cell classification on the source domain data as the classifier;

[0038] The final cell nucleus classification is performed on the target domain data based on the pre-trained classifier.

[0039] The design idea of ​​the present invention is: first, two widely used pathological image cell nucleus datasets with labels are selected as the source domain and target domain data in the domain adaptation method, and the single cell nucleus is cropped according to the label value to construct a new dataset with a single cell nucleus image and the label value of the corresponding cell type, and then the cell nucleus images of the source domain and the target domain are subjected to feature extraction. The feature extraction process can use a convolutional neural network as an encoder to establish a Gaussian mixture mathematical model for the extracted features to achieve data interaction and label transfer. In this process, the source domain label is transferred to the target domain data through multiple iterations to achieve the effect of reducing the difference between the source domain and the target domain data. Finally, a classifier is used to classify the cell nuclei of all the data in the target domain to solve the main difficulties in the current cell nucleus classification task, such as sparse cell nucleus annotations and large data differences.

[0040] Compared with the existing technology, the beneficial effects of the present invention are as follows: the weakly supervised nucleus classification method based on domain adaptation can reduce the data difference between the source domain and the target domain through domain adaptation, improve the problem of data integration, and can reduce the label dependence on cell annotation based on weakly supervised learning, improve the accuracy of nucleus classification, and better meet clinical needs. Compared with other nucleus classification methods, the hardware requirements, data label dependence, generalization ability and overall classification task effect are optimized. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is the operation flow chart of the present invention;

[0042] Figure 2 The model diagram proposed for the algorithm of the present invention;

[0043] Figure 3 Schematic diagram for data integration and label transfer. DETAILED DESCRIPTION

[0044] The technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0045] The present invention designs a weakly supervised cell nucleus classification method based on domain adaptation; first, a single cell nucleus is cut out, and feature extraction and high-dimensional vector mapping are performed based on a convolutional neural network. Then, data alignment and label transfer are performed in a high-dimensional space so that the data in the target domain all have label values. Finally, a classifier is used to predict the cell nucleus classification task. The innovative points of the present invention include: 1) data integration based on a domain adaptation method reduces the data difference between the source domain and the target domain; 2) weakly supervised learning is used to supervise the training process, reducing the dependence on label data; 3) a single cell is cut out to construct a new data set, simplifying the training process of the model. Compared with existing cell nucleus classification methods, it reduces computational complexity and reduces the workload of doctors in manually labeling cells, which has great practical value in clinical practice.

[0046] Implementation process:

[0047] A weakly supervised cell nucleus classification method based on domain adaptation, the specific steps are as follows:

[0048] S1: Perform cropping and preprocessing on the two cell nucleus datasets and construct corresponding new datasets;

[0049] The mask value is determined by the label in the input RGB cell nucleus image data, and the boundary contour of the cell nucleus is determined by the mask of a single cell nucleus. The minimum enclosing rectangle of the cell nucleus and the geometric moment of the contour are calculated based on this. The centroid coordinates of a single cell nucleus are calculated using the moment, and the cell nucleus image data of the source domain and the target domain are cropped using the centroid coordinates of the cell nucleus and the minimum enclosing rectangle.

[0050] S2: Use convolutional neural networks to extract features based on the source domain dataset and perform high-dimensional mapping;

[0051] The source domain data is preprocessed with data enhancement (translation, rotation, scaling), and the residual neural network ResNet is used to extract features from the cell nucleus image data with labeled values ​​in the source domain. The generated feature maps are globally pooled on average, and the spatial dimension of each feature map is reduced to a single value to form a high-dimensional feature vector. The classifier is used to make initial category predictions for the source domain data based on cell features, to determine the distinguishing ability of cell features, and to evaluate the reliability of the cell classification model based on classification loss.

[0052] S3: Extract features and perform high-dimensional mapping on the target domain data based on the pre-trained feature extraction network;

[0053] For the image preprocessing of data enhancement (translation, rotation, scaling) of the target domain data, the pre-trained network ResNet in the source domain data is used to extract the features of the cell nucleus image data without label values ​​in the target domain data, and the feature vector of the high-dimensional mapping of the feature map obtained by feature extraction is calculated.

[0054] S4: Based on the Gaussian mixture model in the high-dimensional mapping space, the alignment and label transfer between the target domain and the source domain data are realized;

[0055] In high-dimensional space, the known cell categories with labels in the source domain are separated, and the unknown categories in the target domain without labels are clustered. A Gaussian mixture model is established for the source domain data and the target domain data to help find abnormal samples in the target domain and improve the robustness of the system. The cell classification features of the known cell categories in the source domain and the features calculated in the target domain are compared, the cell features of the source domain and the target domain are aligned, and the aligned source domain labels are transferred to the target domain.

[0056] S5: Iterative process of data alignment and label transfer from source domain to target domain based on high-dimensional mapping space;

[0057] Count the target domain data that does not yet have cell labels, iterate the above data pairs and source domain to target domain label transfer steps, gradually narrow the data difference between the source domain and the target domain, until all unknown types have cell labels.

[0058] S6: Use the classifier model to predict the cell nucleus classification of the target domain data after data integration and label transfer;

[0059] The target domain data after data alignment and label transfer is statistically completed, and finally the classifier pre-trained in the source domain data is used to perform the final cell classification on the target domain data.

[0060] Example:

[0061] S1: Data preprocessing

[0062] PanNuke and CoNSeP are selected from common cell nucleus image datasets as the source domain and target domain datasets in this embodiment. In other usage scenarios, they can be replaced with other cell nucleus image datasets according to actual needs.

[0063] The mask and cell type are determined based on the label information. In this embodiment, the cell types include tumor cells, immune cells, epithelial cells, and inflammatory cells. They can be replaced with other cell types according to the specific scenario. The boundary contour of the cell nucleus is determined based on the mask, and the geometric moment M of the cell nucleus contour is calculated based on this. The moment is used to calculate the centroid coordinates of a single cell nucleus. The calculation method is as follows:

[0064]

[0065] Among them, M 00 is the zero-order moment, which represents the sum of all pixel values ​​in the image, M 10 is the first-order moment, which represents the sum of the product of the x-coordinate of each pixel and the pixel value, M 01 It is the first-order moment, which represents the sum of the products of the y coordinate of each pixel and the pixel value.

[0066] According to the coordinates of the cell's centroid and the cell outline, a single cell nucleus is cropped according to the minimum circumscribed rectangle to construct a new dataset D containing a single cell nucleus and corresponding label information. S and D T .

[0067] S2: Source domain data feature extraction and high-dimensional mapping

[0068] For the source domain dataset D S Cell image I S Perform image enhancement (translation, rotation, scaling), represented by I enhanced In this embodiment, ResNet-50 is used to enhanced Feature extraction is performed to obtain feature F. In other scenarios, it can be replaced with neural networks such as DenseNet and EfficientNet for feature extraction:

[0069] F = ResNet(I enhanced )

[0070] The generated features are averaged and pooled to reduce the spatial dimension of each feature map to a single value to form a high-dimensional feature vector, and the fully connected layer is used as a classifier to classify the source domain data into cells:

[0071] f=GAP(F)

[0072] Y S =FC(f)

[0073] Where GAP represents the global average pooling operation, FC represents the use of a fully connected layer for cell classification, and Y S Represents the results of cell classification.

[0074] S3: Target domain data feature extraction and high-dimensional mapping

[0075] For the target domain dataset D T Cell image I T Perform image enhancement (translation, rotation, scaling), represented by I e ' nhanced, use the pre-trained network ResNet-50 on the source domain data to extract features and map high-dimensional vectors to obtain a high-dimensional vector f'.

[0076] S4: Data alignment and label transfer between source and target domains

[0077] Using X s represents the labeled cell features in the source domain, X T represents the cell features without labels in the target domain, where (N s is the number of source domain samples), (N T is the number of samples in the target domain), and a Gaussian mixture model is fitted for the source domain features and the target domain features:

[0078]

[0079] Where θ is the parameter of the Gaussian mixture model, π is the weight of the Gaussian distribution, and K s represents the number of Gaussian distributions, μ and Σ represent the mean and covariance matrices, represents the probability density function.

[0080] Align the features of the source domain and the target domain, and then transfer the source domain label to the target domain based on the aligned features. This is achieved by the following method:

[0081]

[0082] Where A and B are alignment matrices used to map the source and target domain features to a common space, X s and X T Represents the characteristics of the source domain and the target domain, Represents the label of the target domain after label transfer.

[0083] S5: Iterate the S4 process

[0084] Count the target domain data that does not yet have cell labels, iterate the above data pairs and source domain to target domain label transfer steps, gradually narrow the data difference between the source domain and the target domain, until all unknown types have cell labels.

[0085] S6: Cell nucleus classification prediction

[0086] Calculate the feature vector of the target domain after all data alignment and label transfer are completed Final classification of target domain nuclei using the classifier trained on the source domain data:

[0087]

[0088] in represents the final prediction of the target domain cell.

[0089] The above embodiments are only preferred embodiments of the present invention and are not limitations of the technical solutions of the present invention. Any technical solution that can be implemented on the basis of the above embodiments without creative work should be deemed to fall within the scope of protection of the patent of the present invention.

Claims

1. A weakly supervised cell nucleus classification method based on domain adaptation, characterized in that: The following steps are involved: Step 1) Select two cell nucleus datasets from the cell nucleus image dataset as the source domain dataset and the target domain dataset respectively; crop and preprocess the two cell nucleus datasets, and construct a new source domain dataset D S and the target domain dataset D T ; Step 2) Based on the source domain dataset D S Use convolutional neural networks to extract features from source domain data and perform high-dimensional mapping; Step 3) Based on the target domain dataset D T Use the pre-trained network of the source domain data to extract features and perform high-dimensional mapping on the target domain data; Step 4) Aligning the target domain data to the source domain data and transferring the labels are realized through the Gaussian mixture model in the high-dimensional mapping space; Step 5) Iterate the process of step 4) until all target domain data have completed data alignment and label transfer; Step 6) Use the classifier model to predict the cell nucleus classification of the target domain data that has completed data alignment and label transfer.

2. According to the weakly supervised cell nucleus classification method based on domain adaptation in claim 1, it is characterized in that: The clipping and preprocessing process of the cell nucleus data set in step 1) is as follows: Determine the mask value through label for the input RGB cell nucleus image data; The boundary contour of the cell nucleus is determined by the mask of a single cell nucleus, and the minimum circumscribed rectangle of the cell nucleus and the geometric moment of the contour are calculated based on this; Moments were used to calculate the centroid coordinates of individual cell nuclei; The cell nucleus image data of the source domain and the target domain are cropped by the centroid coordinates and the minimum bounding rectangle of the cell nucleus to construct a new dataset D containing single cell nuclei and corresponding label information. S and D T .

3. The weakly supervised cell nucleus classification method based on domain adaptation according to claim 1, characterized in that: The specific process of step 2) is as follows: Image preprocessing based on data enhancement of source domain data, where data enhancement includes translation, rotation, and scaling; Use the residual neural network ResNet to train the source domain dataset D S Feature extraction is performed on the cell nucleus image data with label values; The generated feature maps are globally averaged pooled, and the spatial dimension of each feature map is reduced to a single value to form a high-dimensional feature vector; Use the classifier to make initial category predictions for source domain data based on cell features, determine the discriminative ability of cell features, and evaluate the reliability of cell classification based on classification loss.

4. The weakly supervised cell nucleus classification method based on domain adaptation according to claim 1, characterized in that: The specific process of step 3) is as follows: Image preprocessing based on data enhancement of target domain data, where data enhancement includes translation, rotation, and scaling; Based on the pre-trained network ResNet of the source domain data, the target domain dataset D T Feature extraction of cell nucleus image data without label values; Calculate the feature vector of the high-dimensional mapping of the feature map to obtain the feature extraction.

5. The weakly supervised cell nucleus classification method based on domain adaptation according to claim 1, characterized in that: The specific process of step 4) is as follows: In high-dimensional space, the known cell categories with labels in the source domain data are separated from each other, while the target domain data without labels are clustered together. A Gaussian mixture model is established for the source domain data and the target domain data to remove the cell nucleus image data in the target domain that is different from the source domain, thereby improving the robustness of the system. Compare the cell features of known cell categories in the source domain data with the features calculated in the target domain data; Align the cell classification features of the source and target domains; The source domain labels are transferred to the target domain based on the aligned features.

6. The weakly supervised cell nucleus classification method based on domain adaptation according to claim 1, characterized in that: The specific steps of step 5) include: Count the target domain data that does not have cell labels yet; Iterate the above data pair and source domain to target domain label transfer steps; Gradually narrow the data gap between the source and target domains until all unknown types are labeled with cells.

7. The weakly supervised cell nucleus classification method based on domain adaptation according to claim 1, characterized in that: The specific steps of step 6) include: Use the fully connected layer for cell classification on the source domain data as the classifier; The final cell nucleus classification is performed on the target domain data based on the pre-trained classifier.

Images