Pathological image analysis method and equipment based on self-supervised learning and multi-task learning

Abstract

The present invention discloses a pathological image analysis method and device based on self-supervised learning and multi-task learning, including: collecting a number of pathological images without label information and a number of pathological images with label information; dividing the collected pathological images without label information into a number of image blocks to form a label data set; dividing the original image and the corresponding annotation image of the collected pathological image with label information into a number of image blocks to form a label data set; establishing a self-supervised learning model and a multi-task learning model; using the unlabeled data set to train the self-supervised learning model; using the trained self-supervised learning model to extract features of the labeled data set, respectively obtaining block coding features and class coding features; then using the obtained block coding features and class coding features to train the multi-task learning model; applying the pathological images to be analyzed. The present invention can speed up the efficiency of pathological image processing, and is helpful for timely detection of diseases and formulation of treatment plans.

Description

Technical Field

[0001] The present invention belongs to the technical field of pathological image processing, and specifically relates to a pathological image analysis method and device based on self-supervised learning and multi-task learning. Background Art

[0002] In the field of tumor diagnosis, pathological image diagnosis is regarded as the unshakable gold standard. For colorectal cancer, the histopathological evaluation of tissue sections by pathologists is the key to diagnosis. Therefore, hematoxylin and eosin (H&E) stained tissue sections have become a routine procedure for diagnosing colorectal cancer patients.

[0003] Pathologists must use microscopes to carefully observe the morphological characteristics and cell structure of tissue sections to provide accurate pathological evaluations. However, due to the complexity and size of pathological images, this traditional diagnostic method faces many challenges: it is not only difficult and requires high professional skills, but also has the risk of missed diagnosis, and the diagnostic results are also affected to a certain extent by the subjective judgment of the pathologist. These issues have always been hot spots and difficulties in the field of tumor research, and they urgently need to be solved through technological innovation.

[0004] Numerous studies have shown that deep learning technology can directly analyze digital H&E-stained colorectal cancer tissue sections, which is helpful for the diagnosis and treatment of colorectal cancer.

[0005] With the development of artificial intelligence, a series of colorectal cancer analysis methods based on artificial intelligence technology have begun to emerge. By designing targeted network architectures and deep learning of H&E pathological images, the trained neural network model can analyze the microsatellite status of colorectal cancer, which to a certain extent makes up for the shortcomings of traditional methods.

[0006] However, this series of methods also has the following problems:

[0007] First, the labeling cost is huge, especially for segmentation tasks, which require doctors to manually outline the tumor area or the region of interest, or to use a classification model trained by doctors after manual outlines to help screen. This process will also introduce differences in the subjective outlines of pathologists, which will have a certain impact on the results.

[0008] In order to avoid the differences in the outlines drawn by pathologists, some methods introduce pre-trained tumor recognition models into the model to help determine whether image tiles are tumors, or segmentation models to identify tumor areas. However, these models still require pathologists to manually outline during their training.

[0009] Second, the task is single, the generalization is poor, and the accuracy cannot meet clinical requirements. Although more and more deep learning models have achieved better and better results in algorithm accuracy, most of them are based on specific data sets. Once clinical data sets are used, the accuracy will decrease and cannot achieve the purpose of clinical use. Summary of the invention

[0010] In order to solve the above technical problems, the present invention proposes a pathological image analysis method and device based on self-supervised learning and multi-task learning.

[0011] In order to achieve the above object, the technical solution of the present invention is as follows:

[0012] On the one hand, the present invention discloses a pathological image analysis method based on self-supervised learning and multi-task learning, comprising:

[0013] Step S1: collecting a number of pathological images without label information and a number of pathological images with label information, wherein the label information includes: a label map with segmentation labels for tumor regions or regions of interest and graded labels for cancer severity;

[0014] Step S2: Divide the collected pathological images without label information into several image blocks to form an unlabeled data set;

[0015] Divide the collected original pathological images with label information and the corresponding annotated images into a number of image blocks to form a labeled data set;

[0016] Step S3: Establish a self-supervised learning model and a multi-task learning model;

[0017] The self-supervised learning model is used to extract features from image blocks, obtaining block coding features and class coding features respectively;

[0018] The multi-task learning model is used to perform the segmentation task based on the block coding features and the class coding features, and the obtained segmentation prediction results of the tumor area or the region of interest are used to assist the classification task to obtain the classification prediction results of the severity of the cancer;

[0019] Step S4: training the self-supervised learning model using the unlabeled dataset;

[0020] Step S5: Use the trained self-supervised learning model to extract features from the labeled data set to obtain block coding features and class coding features respectively;

[0021] Then use the obtained block encoding features and class encoding features to train the multi-task learning model;

[0022] Step S7: Divide the pathological image to be analyzed into several image blocks, and use the trained self-supervised learning model and multi-task learning model to predict them, so as to obtain the segmentation prediction results of the tumor area or the area of ​​interest and the grading prediction results of the cancer severity.

[0023] Based on the above technical solution, the following improvements can be made:

[0024] As a preferred solution, step S2 includes:

[0025] Step S2.1: normalize all collected pathological images to the same pixel size and resolution;

[0026] Step S2.2: Divide each pathological image without label information into a number of non-overlapping image blocks according to a fixed size to form an unlabeled data set;

[0027] Step S2.3: Divide the original image and the annotated image of each pathological image with label information into a number of non-overlapping image blocks according to the same fixed size;

[0028] All image blocks after the annotation map is divided are screened, and image blocks whose pixel values ​​in the tumor area or the area of ​​interest are lower than the threshold are deleted to form a labeled data set.

[0029] As a preferred solution, step S5 includes:

[0030] Step S5.1: Use the trained self-supervised learning model to extract features from the labeled data set to obtain block coding features and class coding features respectively;

[0031] Step S5.2: Input the block encoding features and class encoding features obtained in step S5.1 into the multi-task learning model to perform segmentation tasks and classification tasks;

[0032] Segmentation task: The block coding features obtained in step S5.1 are decoded through the convolution layer, normalized by the softmax function, and the pixel-level segmentation prediction results are output;

[0033] Classification task: The block coding features obtained in step S5.1 are decoded through the convolution layer, pooled through the pooling layer, and obtained through the fully connected layer.

[0034] The query key and the block coding features obtained in step S5.1 are weighted by the attention mechanism, the correlation between the block coding features and the class coding features is calculated, and finally the hierarchical prediction results are obtained through the classification network.

[0035] As a preferred solution, in step S5, a dynamic loss function is used Train the multi-task learning model as follows:

[0036] ;

[0037] ;

[0038] ;

[0039] ;

[0040] ;

[0041] in:

[0042] and are the label value and predicted value of pixel i in the segmentation task, respectively, and N is the total number of pixels;

[0043] and are the true labels and prediction results of the classification task, C is the number of classification categories, and n is the number of test samples;

[0044] , are adjustment factors, epoch is the current iteration number, and epochs is the number of iterations required to train the multi-task learning model.

[0045] As a preferred solution, the pathological image analysis method further includes:

[0046] Step S6: Use several evaluation indicators to evaluate the segmentation task and classification task in the multi-task learning model.

[0047] In addition, on the other hand, the present invention also discloses a pathological image analysis device based on self-supervised learning and multi-task learning, comprising:

[0048] A pathological image collection module, used to collect a number of pathological images without label information and a number of pathological images with label information, wherein the label information includes: a label map with segmentation labels for tumor regions or regions of interest and a graded label for cancer severity;

[0049] A pathological image segmentation module is used to divide the collected pathological images without label information into several image blocks to form an unlabeled data set;

[0050] Divide the collected original pathological images with label information and the corresponding annotated images into a number of image blocks to form a labeled data set;

[0051] Model building module, used to build self-supervised learning models and multi-task learning models;

[0052] The self-supervised learning model is used to extract features from image blocks, obtaining block coding features and class coding features respectively;

[0053] The multi-task learning model is used to perform the segmentation task based on the block coding features and the class coding features, and the obtained segmentation prediction results of the tumor area or the region of interest are used to assist the classification task to obtain the classification prediction results of the severity of the cancer;

[0054] A self-supervised learning model training module, which is used to train the self-supervised learning model using an unlabeled dataset;

[0055] The multi-task learning model training module is used to extract features from labeled data sets using the trained self-supervised learning model to obtain block coding features and class coding features respectively;

[0056] Then use the obtained block encoding features and class encoding features to train the multi-task learning model;

[0057] The pathological image analysis module is used to divide the pathological image to be analyzed into several image blocks, and use the trained self-supervised learning model and multi-task learning model to predict them, so as to obtain the segmentation prediction results of the tumor area or the area of ​​interest and the grading prediction results of the cancer severity.

[0058] As a preferred solution, the pathological image segmentation module includes:

[0059] A normalization unit, used to normalize all collected pathological images to the same pixel size and resolution;

[0060] An unlabeled data set forming unit, used for dividing each pathological image without label information into a plurality of non-overlapping image blocks according to a fixed size to form an unlabeled data set;

[0061] A labeled data set forming unit is used to divide the original image and the annotated image of each pathological image with label information into a plurality of non-overlapping image blocks according to the same fixed size;

[0062] All image blocks after the annotation map is divided are screened, and image blocks whose pixel values ​​in the tumor area or the area of ​​interest are lower than the threshold are deleted to form a labeled data set.

[0063] As a preferred solution, the multi-task learning model training module includes:

[0064] A feature extraction unit is used to extract features from a labeled data set using a trained self-supervised learning model, and obtain block coding features and class coding features respectively;

[0065] A task execution unit, used for inputting the block coding features and class coding features obtained by the feature extraction unit into the multi-task learning model to perform segmentation tasks and classification tasks;

[0066] Segmentation task: The block coding features obtained by the feature extraction unit are decoded through the convolution layer, normalized by the softmax function, and the pixel-level segmentation prediction results are output;

[0067] Hierarchical task: The block encoding features obtained by the feature extraction unit are decoded through the convolution layer, pooled through the pooling layer, and the query key is obtained through the fully connected layer;

[0068] The query key and the block coding features obtained by the feature extraction unit are weighted by the attention mechanism, and the correlation between the block coding features and the class coding features is calculated. Finally, the hierarchical prediction results are obtained through the classification network.

[0069] As a preferred solution, the multi-task learning model training module uses a dynamic loss function Train the multi-task learning model as follows: ;

[0070] ;

[0071] ;

[0072] ;

[0073] ;

[0074] in:

[0075] and are the label value and predicted value of pixel i in the segmentation task, respectively, and N is the total number of pixels;

[0076] and are the true labels and prediction results of the classification task, C is the number of classification categories, and n is the number of test samples;

[0077] , are adjustment factors, epoch is the current iteration number, and epochs is the number of iterations required to train the multi-task learning model.

[0078] As a preferred solution, the pathological image analysis device further includes:

[0079] The model evaluation module is used to evaluate the segmentation task and classification task in the multi-task learning model using several evaluation indicators.

[0080] The present invention discloses a pathological image analysis method and device based on self-supervised learning and multi-task learning, which has the following beneficial effects:

[0081] First, the present invention establishes a self-supervised learning model and a multi-task learning model. The self-supervised learning model does not require a large amount of annotation. Through self-supervised learning, common features are extracted from a large number of pathological images, and then the multi-task learning model is used to closely link the segmentation task and the classification task, thereby improving the prediction accuracy of the two tasks.

[0082] Second, the present invention can speed up the efficiency of pathological image processing, help to timely discover diseases and formulate treatment plans, reduce the workload of pathologists, and improve work efficiency. It has important practical significance, especially for large-scale pathological image data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0084] Figure 1 (a) is an original pathological image (healthy gland) provided in an embodiment of the present invention;

[0085] Figure 1 (b) is Figure 1 (a) The result after gland segmentation;

[0086] Figure 1 (c) is an original pathological image (adenoma) provided in an embodiment of the present invention;

[0087] Figure 1 (d) is Figure 1 (c) The result after gland segmentation;

[0088] Figure 1 (e) is an original pathological image (poorly differentiated) provided in an embodiment of the present invention;

[0089] Figure 1 In (f) Figure 1 Middle (e) is the result after gland segmentation;

[0090] Figure 1 Middle (g) is an original pathological image (low-moderate differentiation) provided in an embodiment of the present invention;

[0091] Figure 1 In (h) Figure 1Middle (g) Result after gland segmentation;

[0092] Figure 1 (i) is the original pathological image (medium differentiation) provided in the embodiment of the present invention;

[0093] Figure 1 In (j) Figure 1 (i) The result after gland segmentation.

[0094] Figure 2 A flow chart of a pathological image analysis method provided in an embodiment of the present invention.

[0095] Figure 3 A framework diagram of colorectal cancer pathology image analysis based on self-supervised learning and multi-task learning provided in an embodiment of the present invention.

[0096] Figure 4 A framework diagram of a self-supervised learning model provided in an embodiment of the present invention.

[0097] Figure 5 The visualization results of block coding features before and after pre-training using pathological images provided in an embodiment of the present invention.

[0098] Figure 6 A training block diagram of a multi-task learning model provided in an embodiment of the present invention.

[0099] Figure 7 A block diagram of a pixel-level segmentation task in a multi-task learning model provided in an embodiment of the present invention.

[0100] Figure 8 A block diagram of instance-level classification tasks in a multi-task learning model provided in an embodiment of the present invention.

[0101] Fig. 9 The dynamic adjustment factor change curve provided by the embodiment of the present invention.

[0102] Fig.10 (a) is an image block 1 to be analyzed provided in an embodiment of the present invention;

[0103] Fig.10 (b) is Fig.10 (a) The result after gland segmentation by the present invention Figure 1 ;

[0104] Fig.10 (c) Fig.10 (a) The result after gland segmentation by the present invention Figure 2 ;

[0105] Fig.10 (d) is the second image block to be analyzed provided by the embodiment of the present invention;

[0106] Fig.10 (e) is Fig.10 (d) The result after gland segmentation by the present invention Figure 1 ;

[0107] Fig.10 In (f) Fig.10 (d) The result after gland segmentation by the present invention Figure 2 . DETAILED DESCRIPTION

[0108] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0109] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0110] The expression of “comprising” an element is an “open” expression, which merely means that a corresponding component or step exists, and should not be interpreted as excluding additional components or steps.

[0111] In order to achieve the purpose of the present invention, a method and device for pathological image analysis based on self-supervised learning and multi-task learning are provided in some embodiments, such as Figure 1 As shown, Figure 1 (a), (c), (e), (g), and (i) are the original pathological images of patients with different colorectal cancer grades, specifically, five grades: healthy gland, adenoma, poorly differentiated, poorly-moderately differentiated, and moderately differentiated. Among them, healthy gland and adenoma are benign, and the other three are malignant. Figure 1 (b), (d), (f), (h), and (j) are the corresponding gland segmentation images.

[0112] The applicant has found that the grading of colorectal cancer is directly related to the morphology of its glands. Therefore, the segmentation task of a tumor region or region of interest for a certain cancer and the grading task of the severity of the cancer are closely related, and the improvement of the accuracy of either task may affect the other task.

[0113] like Figure 2 As shown, the present invention discloses a pathological image analysis method, which specifically includes the following steps:

[0114] Step S1: collecting a number of pathological images without label information and a number of pathological images with label information, wherein the label information includes: a label map with segmentation labels for tumor regions or regions of interest and graded labels for cancer severity;

[0115] Step S2: Divide the collected pathological images without label information into several image blocks to form an unlabeled data set;

[0116] Divide the collected original pathological images with label information and the corresponding annotated images into a number of image blocks to form a labeled data set;

[0117] Step S3: Establish a self-supervised learning model and a multi-task learning model;

[0118] The self-supervised learning model is used to extract features from image blocks, obtaining block coding features and class coding features respectively;

[0119] The multi-task learning model is used to perform the segmentation task based on the block coding features and the class coding features, and the obtained segmentation prediction results of the tumor area or the region of interest are used to assist the classification task to obtain the classification prediction results of the severity of the cancer;

[0120] Step S4: training the self-supervised learning model using the unlabeled dataset;

[0121] Step S5: Use the trained self-supervised learning model to extract features from the labeled data set to obtain block coding features and class coding features respectively;

[0122] Then use the obtained block encoding features and class encoding features to train the multi-task learning model;

[0123] Step S6: Use several evaluation indicators to evaluate the segmentation task and classification task in the multi-task learning model;

[0124] Step S7: Divide the pathological image to be analyzed into several image blocks, and use the trained self-supervised learning model and multi-task learning model to predict them, so as to obtain the segmentation prediction results of the tumor area or the area of ​​interest and the grading prediction results of the cancer severity.

[0125] Taking colorectal cancer as an example, each of the above steps is described in detail below to achieve colorectal cancer gland segmentation and colorectal cancer grading. Of course, the protection scope of the present invention is not limited to colorectal cancer.

[0126] Step S1 collects a large number of H&E stained pathological section images (whole-field pathological images WSI) of colorectal cancer patients without label information, and at the same time, collects pathological section images of colorectal cancer patients with label information, and the label information includes: annotated images with colorectal cancer gland segmentation annotations, and colorectal cancer grade annotations.

[0127] Specifically, in this implementation process, step S1 collects two open source datasets: the IMP-CRC-2024 dataset and the GlaS dataset.

[0128] The IMP-CRC-2024 dataset contains 5333 original pathological images of colorectal cancer patients without label information.

[0129] The GlaS dataset is a small dataset with label information, which contains annotation maps of gland segmentation and graded annotations of colorectal cancer. The graded annotations include five states: healthy glands, adenomas, poor differentiation, moderate-poor differentiation, and moderate differentiation. Healthy glands and adenomas are benign, and the other three are malignant.

[0130] Further, step S2 performs a preprocessing operation on the pathological image collected in step S1, including:

[0131] Step S2.1: normalize all collected pathological images to the same pixel size and resolution;

[0132] Step S2.2: Divide each pathological image without label information into a number of non-overlapping image blocks according to a fixed size to form an unlabeled data set;

[0133] Step S2.3: Divide the original image and the annotated image of each pathological image with label information into a number of non-overlapping image blocks according to the same fixed size;

[0134] All image blocks after the annotation map is divided are screened, and image blocks whose pixel values ​​in the tumor area or the area of ​​interest are lower than the threshold are deleted to form a labeled data set.

[0135] Specifically, in this implementation process, step S2 is for the dataset IMPT-CRC-2024 without label information, and is divided according to the set pixel size and image size to obtain a number of image blocks.

[0136] For example, preset Extract image blocks, each of which is of size , the image block size is obtained as follows:

[0137] ;

[0138] ;

[0139] ;

[0140] in: Indicates the preset MPP, represents the MPP of the dataset image, represents the scale factor, W is the image width before unification, represents the width of the image after unification, and H represents the height of the image before unification. Indicates the image height after unification.

[0141] In this implementation, for the GlaS dataset containing label information, the physical sizes of the original image and the annotated image are first normalized to 0.5 mpp according to the above formula, and then the dataset is cropped to obtain Image blocks of size.

[0142] All image blocks after the annotation map is divided are screened, and image blocks whose pixel values ​​in the tumor area or the area of ​​interest are lower than a threshold (e.g., 30%) are deleted.

[0143] After the above operations, the labeled dataset has 565 images.

[0144] Further, step S3 establishes a self-supervised learning model and a multi-task learning model. By using the self-supervised learning model to obtain the encoding vector, a multi-task learning model at the pixel level and the instance level is established. Figure 3 As shown, a colorectal cancer pathology image analysis framework based on self-supervised learning and multi-task learning.

[0145] The invention uses a large model trained on pathological images according to a self-supervised learning strategy as a multi-task feature sharing network, and proposes using patch coding features (Patch Token) and class coding features (Class Token) for multi-task learning respectively.

[0146] In this implementation, the main framework of the self-supervised learning model uses DINOv2. Figure 4 As shown, the framework based on the self-supervised learning model used in the present invention.

[0147] The image block is passed through the self-supervised learning model to obtain the encoding vector, which is split into the block encoding feature (Patch Token) and the class encoding feature (Class Token).

[0148] The block coding feature includes: the coding information of the image block, which contains the location information of the image, so the invention uses the block coding feature for the subsequent segmentation task. The class coding feature is a vector aggregated after the attention mechanism, and the invention uses the class coding feature for the subsequent classification task.

[0149] A multi-task learning model generally refers to a model that can achieve deep learning of multiple tasks through a single model. However, existing multi-task learning models only consider the correlation of tasks at the same level and perform fine-tuning. The present invention proposes an interpretable multi-task learning model based on pixel level and instance level. Considering the connection between the glandular area of ​​the tissue and the category of the tissue, the invention combines the pixel level and instance level tasks together. The purpose is to enable the two to influence each other so that the model has better generalization performance.

[0150] Furthermore, step S4 uses the unlabeled dataset to train the self-supervised learning model.

[0151] In this embodiment, step S4 uses a large amount of data obtained from the dataset IMP-CRC-2024 to pre-train the self-supervised learning model. Since the DINOv2 framework is used, further transfer learning can be performed based on the model trained on natural images.

[0152] The present invention uses the PCA dimensionality reduction method to perform visual analysis on block coding features. First, the first three principal components of the block coding features are obtained, and then the first three principal components are mapped to the R, G, and B channels so that the distribution of the features can be observed. Figure 5 As shown, it is the visualization result of block coding features before and after pre-training using pathological images.

[0153] After the self-supervised learning model is trained, the labeled data set is divided into a training set and a test set. The training set is used to train the multi-task learning model, and the test set is used to evaluate the multi-task learning model.

[0154] Specifically, the GlaS dataset after data preprocessing is divided into a training set and a test set. The original GlaS dataset includes 85 training images and 80 test images. After data preprocessing, the dataset includes 288 training images (training set) and 277 test images (test set).

[0155] Further, if Figure 6 As shown, step S5 includes:

[0156] Step S5.1: Use the trained self-supervised learning model to extract features from the labeled training set to obtain block coding features and class coding features respectively;

[0157] Step S5.2: Input the block encoding features and class encoding features obtained in step S5.1 into the multi-task learning model to perform segmentation tasks and classification tasks;

[0158] Segmentation task: The block coding features obtained in step S5.1 are decoded through the convolution layer, normalized by the softmax function, and the pixel-level segmentation prediction results are output;

[0159] Hierarchical task: The block coded features obtained in step S5.1 are decoded through the convolutional layer, pooled through the pooling layer (the purpose is to compress the spatial information of the block coded features), and obtained through the fully connected layer. The query key;

[0160] The query key and the block coding features obtained in step S5.1 are weighted by the attention mechanism, and the correlation between the block coding features and the class coding features is calculated using the following formula; ;

[0161] in:

[0162] They are the weight matrices of query key, key value, and value in the attention mechanism, and Score is the attention score.

[0163] Finally, the attention-weighted class encoding features are passed through the classification network to output the predicted classification results (e.g., predicted classification probability).

[0164] When the neural network gradient is updated, the two levels of tasks are associated to allow the model to learn more knowledge.

[0165] In this implementation, the constructed multi-task learning model is used to train the gland segmentation task and the colorectal cancer grading task. Figure 5 and Figure 6 It is an expansion of the specific structure of the segmentation and grading model.

[0166] like Figure 7 As shown in the figure, the dimension of the block coding feature obtained by the self-supervised learning model is 1536×14×14, where 1536 represents the coding feature dimension and 14 represents the number of image blocks. The segmentation task is decoded and finally a 2×224×224 image matrix is ​​obtained. The values ​​in the matrix represent the category probability of the tumor area or the area of ​​interest (such as gland) to which each pixel value belongs.

[0167] like Figure 8 As shown in the figure, the 2×224×224 image matrix is ​​obtained through the pooling layer to obtain the 2×112×112 image matrix, and the 2×112×112 image matrix uses the fully connected layer to obtain the 2×1536 query key. The attention mechanism is weighted according to the query key and the aggregated 1×1536 class encoding features, and the correlation between the block encoding features and the class encoding features is calculated. Finally, the classification probability is obtained through the classification network (MLP / multi-layer perceptron is used in this implementation).

[0168] In step S5, a dynamic loss function is used Train the multi-task learning model as follows: ;

[0169] ;

[0170] ;

[0171] ;

[0172] ;

[0173] in:

[0174] and are the label value and predicted value of pixel i in the segmentation task, respectively, and N is the total number of pixels;

[0175] and are the true labels and prediction results of the classification task, C is the number of classification categories, and n is the number of test samples;

[0176] , are adjustment factors, epoch is the current iteration number, and epochs is the number of iterations required to train the multi-task learning model.

[0177] Fig. 9 It shows the changing trend of the adjustment factor with the change of epoch.

[0178] Furthermore, step S6 uses the evaluation indicators Object Dice, Object Hausdorff, and F1 score to evaluate the segmentation task in the multi-task learning model, and uses AUROC to evaluate the classification task in the multi-task learning model. The formula is as follows:

[0179] ;

[0180] ;

[0181] ;

[0182] ;

[0183] ;

[0184] in:

[0185] is the i-th segmentation object, Representation and The maximum overlap.

[0186] is the weighting coefficient of the i-th segmentation object.

[0187] is the i-th actual label, Representation and Segment objects with maximum overlap.

[0188] is the weight coefficient of the i-th actual label.

[0189] and are the total number of segmented objects and actual label objects, respectively.

[0190] is a set distance measurement formula, which is specifically:

[0191] ;

[0192] represents the maximum value function, Represents the Euclidean distance calculation formula.

[0193] is a set similarity measurement function used to calculate the similarity of two samples, which is specifically:

[0194] .

[0195] Object Dice, Object F1 score, and Object Hausdorff are used to evaluate the segmentation task, as shown in Table 1.

[0196] Table 1 Segmentation tasks

[0197]

[0198] As can be seen from Table 1, the multi-task learning model has better results and has improved in all evaluation indicators.

[0199] AUROC is selected as the evaluation indicator for the classification task, as shown in Table 2.

[0200] Table 2 Grading tasks

[0201]

[0202] As can be seen from Table 2, the multi-task learning model has an overall improvement in AUROC at five levels.

[0203] According to the evaluation results, it can be seen that the method proposed in the present invention can improve the performance of two tasks at the same time and obtain better results.

[0204] Furthermore, in step S7, for colorectal cancer patients who need to be analyzed, their H&E stained tissue pathology images are collected, and the trained self-supervised learning model and multi-task learning model are applied to the actual colorectal cancer pathology image analysis process to obtain gland segmentation prediction results and grading prediction results.

[0205] like Fig.10 As shown, Fig.10 (a) is the image block to be analyzed. Fig.10 (b) and (c) are segmentation annotation images of gland segmentation in two different forms, and their graded prediction results are healthy glands; Fig.10 (d) in the figure is another image block to be analyzed. Fig.10 (e) and (f) are segmentation annotation images of gland segmentation in two different forms, and the graded prediction results are adenoma.

[0206] In summary, the present invention has the following effects:

[0207] First, the present invention uses a self-supervised learning model as a model encoder to extract features of image blocks, and obtains class coding features and block coding features of the image respectively.

[0208] The present invention improves the model's ability to represent colorectal cancer pathological images through self-supervised learning, reduces dependence on labeled data, and enhances the model's generalization ability.

[0209] Self-supervised learning can learn useful feature representations from unlabeled data, which is especially important for few-shot tasks because it can help the model learn effective features even with only a small amount of labeled data;

[0210] In the few-shot learning scenario, the scarcity of labeled data limits the application of supervised learning. Self-supervised learning generates supervision signals by designing proxy tasks, reducing the reliance on manually labeled data.

[0211] Self-supervised learning can help the model learn more generalized features, which is crucial for few-shot tasks because it allows the model to perform well even when faced with new, unseen categories.

[0212] Second, the present invention establishes a multi-task learning model at the pixel level and the instance level to decode the class encoding features and the block encoding features.

[0213] The present invention uses multiple task learning to segment colorectal cancer glands on the one hand and to predict the grading of colorectal cancer on the other hand, making full use of the features proposed by self-supervised learning and the correlation between the two clinical tasks, which can effectively improve the accuracy of the two tasks.

[0214] Third, through the deep learning method of multi-task learning, a multi-task model is trained using a dynamically cascaded loss function.

[0215] The present invention combines the colorectal cancer gland segmentation task and the colorectal cancer grading task through a multi-task learning model. During the training process, taking into account the particularity of the clinical task itself, a dynamic loss function is used for training to further improve the accuracy of the model.

[0216] Fourth, by loading the trained model, the gland segmentation of colorectal cancer pathology images and the colorectal cancer grading tasks are achieved.

[0217] The present invention can be used for gland segmentation and grading tasks in clinical colorectal cancer pathological images to help pathologists reduce their workload and improve work efficiency, which has great clinical significance.

[0218] The embodiment of the present invention further discloses a pathological image analysis device based on self-supervised learning and multi-task learning, comprising:

[0219] A pathological image collection module, used to collect a number of pathological images without label information and a number of pathological images with label information, wherein the label information includes: a label map with segmentation labels for tumor regions or regions of interest and a graded label for cancer severity;

[0220] A pathological image segmentation module is used to divide the collected pathological images without label information into several image blocks to form an unlabeled data set;

[0221] Divide the collected original pathological images with label information and the corresponding annotated images into a number of image blocks to form a labeled data set;

[0222] Model building module, used to build self-supervised learning models and multi-task learning models;

[0223] The self-supervised learning model is used to extract features from image blocks, obtaining block coding features and class coding features respectively;

[0224] The multi-task learning model is used to perform the segmentation task based on the block coding features and the class coding features, and the obtained segmentation prediction results of the tumor area or the region of interest are used to assist the classification task to obtain the classification prediction results of the severity of the cancer;

[0225] A self-supervised learning model training module, which is used to train the self-supervised learning model using an unlabeled dataset;

[0226] The multi-task learning model training module is used to extract features from labeled data sets using the trained self-supervised learning model to obtain block coding features and class coding features respectively;

[0227] Then use the obtained block encoding features and class encoding features to train the multi-task learning model;

[0228] The pathological image analysis module is used to divide the pathological image to be analyzed into several image blocks, and use the trained self-supervised learning model and multi-task learning model to predict them, so as to obtain the segmentation prediction results of the tumor area or the area of ​​interest and the grading prediction results of the cancer severity.

[0229] Furthermore, the pathological image segmentation module includes:

[0230] A normalization unit, used to normalize all collected pathological images to the same pixel size and resolution;

[0231] An unlabeled data set forming unit, used for dividing each pathological image without label information into a plurality of non-overlapping image blocks according to a fixed size to form an unlabeled data set;

[0232] A labeled data set forming unit is used to divide the original image and the annotated image of each pathological image with label information into a plurality of non-overlapping image blocks according to the same fixed size;

[0233] All image blocks after the annotation map is divided are screened, and image blocks whose pixel values ​​in the tumor area or the area of ​​interest are lower than the threshold are deleted to form a labeled data set.

[0234] Furthermore, the multi-task learning model training module includes:

[0235] A feature extraction unit is used to extract features from a labeled data set using a trained self-supervised learning model, and obtain block coding features and class coding features respectively;

[0236] A task execution unit, used for inputting the block coding features and class coding features obtained by the feature extraction unit into the multi-task learning model to perform segmentation tasks and classification tasks;

[0237] Segmentation task: The block coding features obtained by the feature extraction unit are decoded through the convolution layer, normalized by the softmax function, and the pixel-level segmentation prediction results are output;

[0238] Hierarchical task: The block encoding features obtained by the feature extraction unit are decoded through the convolution layer, pooled through the pooling layer, and the query key is obtained through the fully connected layer;

[0239] The query key and the block coding features obtained by the feature extraction unit are weighted by the attention mechanism, and the correlation between the block coding features and the class coding features is calculated. Finally, the hierarchical prediction results are obtained through the classification network.

[0240] Furthermore, the multi-task learning model training module uses a dynamic loss function Train the multi-task learning model as follows: ;

[0241] ;

[0242] ;

[0243] ;

[0244] ;

[0245] in:

[0246] and are the label value and predicted value of pixel i in the segmentation task, respectively, and N is the total number of pixels;

[0247] and are the true labels and prediction results of the classification task, C is the number of classification categories, and n is the number of test samples;

[0248] , are adjustment factors, epoch is the current iteration number, and epochs is the number of iterations required to train the multi-task learning model.

[0249] Furthermore, the pathological image analysis device also includes:

[0250] The model evaluation module is used to evaluate the segmentation task and classification task in the multi-task learning model using several evaluation indicators.

[0251] In this embodiment, the specific content of the pathological image analysis device based on self-supervised learning and multi-task learning is similar to the content of the pathological image analysis method based on self-supervised learning and multi-task learning disclosed in the above embodiment, and will not be repeated here.

[0252] The present invention discloses a pathological image analysis method and device based on self-supervised learning and multi-task learning, which has the following beneficial effects:

[0253] First, the present invention establishes a self-supervised learning model and a multi-task learning model. The self-supervised learning model does not require a large amount of annotation. Through self-supervised learning, common features are extracted from a large number of pathological images, and then the multi-task learning model is used to closely link the segmentation task and the classification task, thereby improving the prediction accuracy of the two tasks.

[0254] Second, the present invention can speed up the efficiency of pathological image processing, help to timely discover diseases and formulate treatment plans, reduce the workload of pathologists, and improve work efficiency. It has important practical significance, especially for large-scale pathological image data processing.

[0255] It should be understood that the various techniques described herein can be implemented in combination with hardware or software, or a combination thereof. Thus, the method and apparatus of the present invention, or some aspects or portions of the method and apparatus of the present invention, can take the form of program codes (i.e., instructions) embedded in a tangible medium, such as a floppy disk, a CD-ROM, a hard disk drive, or any other machine-readable storage medium, wherein when the program is loaded into a machine such as a computer and executed by the machine, the machine becomes a device for practicing the present invention.

[0256] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for illustrating the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which shall fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A pathological image analysis method based on self-supervised learning and multi-task learning, characterized in that: include: Step S1: collecting a number of pathological images without label information and a number of pathological images with label information, wherein the label information includes: a label map with segmentation labels for tumor regions or regions of interest and graded labels for cancer severity; Step S2: Divide the collected pathological images without label information into several image blocks to form an unlabeled data set; Divide the collected original pathological images with label information and the corresponding annotated images into a number of image blocks to form a labeled data set; Step S3: Establish a self-supervised learning model and a multi-task learning model; The self-supervised learning model is used to extract features from image blocks, and obtain block coding features and class coding features respectively; The multi-task learning model is used to perform the segmentation task based on the block coding features and the class coding features, and use the obtained segmentation prediction results of the tumor area or the region of interest to assist the grading task to obtain the grading prediction results of the severity of the cancer; Step S4: training the self-supervised learning model using the unlabeled dataset; Step S5: Use the trained self-supervised learning model to perform feature extraction on the labeled data set to obtain block coding features and class coding features respectively; Then use the obtained block encoding features and class encoding features to train the multi-task learning model; Step S7: Divide the pathological image to be analyzed into several image blocks, and use the trained self-supervised learning model and multi-task learning model to predict them, so as to obtain the segmentation prediction results of the tumor area or the area of ​​interest and the grading prediction results of the cancer severity.

2. The pathological image analysis method according to claim 1, characterized in that: The step S2 comprises: Step S2.1: normalize all collected pathological images to the same pixel size and resolution; Step S2.2: Divide each pathological image without label information into a number of non-overlapping image blocks according to a fixed size to form an unlabeled data set; Step S2.3: Divide the original image and the annotated image of each pathological image with label information into a number of non-overlapping image blocks according to the same fixed size; All image blocks after the annotation map is divided are screened, and image blocks whose pixel values ​​in the tumor area or the area of ​​interest are lower than the threshold are deleted to form a labeled data set.

3. The pathological image analysis method according to claim 1, characterized in that: The step S5 comprises: Step S5.1: Use the trained self-supervised learning model to extract features from the labeled data set to obtain block coding features and class coding features respectively; Step S5.2: Input the block encoding features and class encoding features obtained in step S5.1 into the multi-task learning model to perform segmentation tasks and classification tasks; Segmentation task: The block coding features obtained in step S5.1 are decoded through the convolution layer, normalized by the softmax function, and the pixel-level segmentation prediction results are output; Classification task: The block coding features obtained in step S5.1 are decoded through the convolution layer, pooled through the pooling layer, and the query key is obtained through the fully connected layer; The query key and the block coding features obtained in step S5.1 are weighted by the attention mechanism, the correlation between the block coding features and the class coding features is calculated, and finally the hierarchical prediction results are obtained through the classification network.

4. The pathological image analysis method according to claim 1, characterized in that: In step S5, a dynamic loss function is used Train the multi-task learning model as follows: ； ； ； ； ； in: and are the label value and predicted value of pixel i in the segmentation task, respectively, and N is the total number of pixels; and are the true labels and prediction results of the classification task, C is the number of classification categories, and n is the number of test samples; , are adjustment factors, epoch is the current iteration number, and epochs is the number of iterations required to train the multi-task learning model.

5. The pathological image analysis method according to claim 1, characterized in that: The pathological image analysis method further comprises: Step S6: Use several evaluation indicators to evaluate the segmentation task and classification task in the multi-task learning model.

6. A pathological image analysis device based on self-supervised learning and multi-task learning, characterized in that: include: A pathological image collection module, used to collect a number of pathological images without label information and a number of pathological images with label information, wherein the label information includes: a label map with segmentation labels for tumor regions or regions of interest and graded labels for cancer severity; A pathological image segmentation module is used to divide the collected pathological images without label information into several image blocks to form an unlabeled data set; Divide the collected original pathological images with label information and the corresponding annotated images into a number of image blocks to form a labeled data set; Model building module, used to build self-supervised learning models and multi-task learning models; The self-supervised learning model is used to extract features from image blocks, and obtain block coding features and class coding features respectively; The multi-task learning model is used to perform the segmentation task based on the block coding features and the class coding features, and use the obtained segmentation prediction results of the tumor area or the region of interest to assist the grading task to obtain the grading prediction results of the severity of the cancer; A self-supervised learning model training module, which is used to train the self-supervised learning model using an unlabeled dataset; The multi-task learning model training module is used to extract features from labeled data sets using the trained self-supervised learning model to obtain block coding features and class coding features respectively; Then use the obtained block encoding features and class encoding features to train the multi-task learning model; The pathological image analysis module is used to divide the pathological image to be analyzed into several image blocks, and use the trained self-supervised learning model and multi-task learning model to predict them, so as to obtain the segmentation prediction results of the tumor area or the area of ​​interest and the grading prediction results of the cancer severity.

7. The pathological image analysis device according to claim 6, characterized in that: The pathological image segmentation module comprises: A normalization unit, used to normalize all collected pathological images to the same pixel size and resolution; An unlabeled data set forming unit, used for dividing each pathological image without label information into a plurality of non-overlapping image blocks according to a fixed size to form an unlabeled data set; A labeled data set forming unit is used to divide the original image and the annotated image of each pathological image with label information into a plurality of non-overlapping image blocks according to the same fixed size; All image blocks after the annotation map is divided are screened, and image blocks whose pixel values ​​in the tumor area or the area of ​​interest are lower than the threshold are deleted to form a labeled data set.

8. The pathological image analysis device according to claim 6, characterized in that: The multi-task learning model training module includes: A feature extraction unit is used to extract features from a labeled data set using a trained self-supervised learning model, and obtain block coding features and class coding features respectively; A task execution unit, used for inputting the block coding features and class coding features obtained by the feature extraction unit into the multi-task learning model to perform segmentation tasks and classification tasks; Segmentation task: The block coding features obtained by the feature extraction unit are decoded through the convolution layer, normalized by the softmax function, and the pixel-level segmentation prediction results are output; Hierarchical task: The block encoding features obtained by the feature extraction unit are decoded through the convolution layer, pooled through the pooling layer, and the query key is obtained through the fully connected layer; The query key and the block coding features obtained by the feature extraction unit are weighted by the attention mechanism, and the correlation between the block coding features and the class coding features is calculated. Finally, the hierarchical prediction results are obtained through the classification network.

9. The pathological image analysis device according to claim 6, characterized in that: The multi-task learning model training module uses a dynamic loss function Train the multi-task learning model as follows: ； ； ； ； ； in: and are the label value and predicted value of pixel i in the segmentation task, respectively, and N is the total number of pixels; and are the true labels and prediction results of the classification task, C is the number of classification categories, and n is the number of test samples; , are adjustment factors, epoch is the current iteration number, and epochs is the number of iterations required to train the multi-task learning model.

10. The pathological image analysis device according to claim 6, characterized in that: The pathological image analysis device also includes: The model evaluation module is used to evaluate the segmentation task and classification task in the multi-task learning model using several evaluation indicators.

Images