Pathological image cell population feature extraction method and system

Abstract

The invention relates to a pathological image cell population feature extraction method and system. The method comprises the following steps: acquiring a pathological image; inputting the pathological image into a cell nucleus segmentation model to obtain a cell nucleus segmentation result; cutting the pathological image based on the cell nucleus segmentation result to obtain a cell image of each cell in the pathological image; inputting the cell image into a cell feature extraction model based on comparative learning, and obtaining the feature and cell position information of each cell; the feature set of all the cells and the corresponding cell position information are input into a population feature extraction model based on comparative learning, cell population features are obtained, and the population feature extraction model generates positive samples of comparative learning by randomly selecting rectangular areas with different sizes and any rotation angles in the pathological image in the training process. Compared with the prior art, the method has the advantages of good robustness, generalization ability, high interpretability and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of digital pathological image processing, and in particular to a method and system for extracting cell population features from pathological images. Background Art

[0002] In the field of digital pathology image analysis based on computer vision, existing technologies mainly focus on the extraction of pixel-level features, overall image feature analysis, and classification and segmentation methods of targets or functional areas based on traditional machine learning or deep learning. However, with the continuous improvement of pathology image resolution and the rapid growth of sample numbers, existing technologies face many challenges in processing complex image data, especially in group feature analysis, and have the following major defects:

[0003] Pixel-level features are difficult to cope with the diversity of pathological images: Most existing technologies rely on pixel-level feature extraction, but pixel-level features are greatly affected by image scaling, staining conditions, and field of view size in different pathological images, resulting in obvious differences in feature performance under different experimental conditions, affecting the generalization ability and robustness of the model.

[0004] Lack of in-depth analysis of the overall characteristics of cell-level groups: Although some technologies have been able to extract and analyze the morphological characteristics of individual cells, the existing technologies have not provided effective solutions for the overall characteristics of cell groups that are clustered together, especially for group behavior and feature analysis in complex pathological environments such as tumors where cells are clustered in certain rules. This also leads to the global information of pathological images being ignored in many scenarios, limiting the accuracy of pathological diagnosis and the extended application of algorithms of artificial intelligence analysis methods under interpretable conditions.

[0005] Limitations of the contrastive learning sample construction method: Most existing contrastive learning-based methods rely on simple computer vision transformation operations (such as rotation, scaling, etc.) to generate samples for comparison during sample construction. This method does not consider the similarity of the aggregation characteristics of cell populations after changes in a certain image physical interval, and cannot fully capture the diversity of cell populations in pathological images, which in turn affects the performance of the model in practical applications.

[0006] Lack of interpretability: Although existing deep learning models have achieved good results in pathological image analysis, due to the high complexity of the model, its feature extraction and reasoning process lacks transparency and interpretability, which limits its promotion in actual medical applications and makes it difficult to use it widely. In particular, for the modeling of cell population characteristics, the existing technology fails to provide a clear explanation path, which makes the credibility and application value of the model questionable.

[0007] In summary, the existing technologies mainly face problems such as insufficient robustness, lack of population feature analysis, single sample construction method and poor model interpretability when processing pathological images. These defects have brought obstacles to the in-depth analysis of pathological images and their application in clinical practice, and a new technical solution that can overcome the above limitations is urgently needed. Summary of the invention

[0008] The purpose of the present invention is to overcome the defects of the prior art, such as insufficient robustness, lack of population feature analysis, single sample construction method and poor model interpretability, and to provide a method and system for extracting cell population features from pathological images.

[0009] The purpose of the present invention can be achieved by the following technical solutions:

[0010] A method for extracting cell population features from pathological images comprises the following steps:

[0011] Acquire pathological images;

[0012] The pathological image is input into the cell nucleus segmentation model to obtain the cell nucleus segmentation result;

[0013] The pathological image is cropped based on the cell nucleus segmentation result to obtain the cell image of each cell in the pathological image;

[0014] Input the cell image into the cell feature extraction model based on contrast learning to obtain the characteristics and cell location information of each cell;

[0015] The feature sets of all cells and the corresponding cell position information are input into a population feature extraction model based on contrastive learning to obtain cell population features, wherein the population feature extraction model generates positive samples for contrastive learning by randomly selecting rectangular areas of different sizes and arbitrary rotation angles within the pathological image during training.

[0016] The training process of the cell nucleus segmentation model includes the following steps:

[0017] Obtain a public dataset of pathological image cell nucleus segmentation, convert the annotation file of the dataset into an image format, wherein the black color in the converted annotation image represents the non-cell nucleus area and the white color represents the cell nucleus area, the converted annotation image corresponds to the pathological image one by one, and the pathological image and the annotation image are divided into a plurality of small images of a first preset size using a sliding window method;

[0018] The segmented small images are randomly divided into training sets and test sets, and the cell nucleus segmentation model is trained using the training set, where each training image contains an RGB pathological image and a black-and-white cell nucleus segmentation annotation image.

[0019] The process of using the cell nucleus segmentation model includes the following steps:

[0020] The input pathological image to be segmented is divided into a plurality of small images of a first preset size, and the divided small images are input as input data into a trained cell nucleus segmentation model to obtain cell nucleus segmentation results of the small images;

[0021] The cell nucleus segmentation results of all the small images are merged according to the original segmentation coordinates to obtain the cell nucleus segmentation result of the pathological image, and the segmentation result of a single cell nucleus is obtained using the watershed algorithm.

[0022] The cropping of the pathological image based on the cell nucleus segmentation result is specifically: according to the center positioning information of each cell in the cell nucleus segmentation result, taking the center positioning information as the center of the rectangular frame, cropping a rectangular frame of a second preset size as the cell image.

[0023] The cell feature extraction model uses a transformer encoding module to extract cell features, and the output of the transformer encoding module passes through multiple layers of fully connected layers to output the feature tensor of the cell image.

[0024] During the training process of the cell feature extraction model, data enhancement is performed on the cell image by randomly rotating, randomly flipping horizontally / vertically, and randomly color jittering to obtain a data-enhanced cell image. The original cell image and the corresponding data-enhanced cell image are taken as a pair of image pairs and are respectively input into a transformer encoding module and a fully connected layer with shared parameters to obtain image features. The cosine similarity between the image features of the original cell image and the data-enhanced cell image is calculated as a loss function. The loss function is minimized through iterative training to complete the training of the cell feature extraction model.

[0025] The population feature extraction model inputs the input data into the transformer encoder, then performs global pooling, and then obtains a fixed-dimensional tensor through a multi-layer fully connected layer. The fixed-dimensional tensor is the cell population feature.

[0026] The positive sample generation method of the population feature extraction model is specifically as follows: assuming that the population features calculated under completely random conditions for a local image randomly selected from an original image are approximately equal to the population features calculated for the entire original image, a square area of ​​a preset size is selected at a random position inside the pathological image, the positions of the two vertices above the square area are random in the overall image, the square area is cropped, and the original coordinate positions of the cells in the square area are transformed into relative coordinate positions in the new square area, the image features of the cells are kept unchanged, and positive samples participating in comparative learning training are constructed.

[0027] The training process of the population feature extraction model is as follows: construct sample pairs based on cell images and constructed positive samples, input them into the population feature extraction model to obtain two feature tensors respectively, calculate the cosine similarity between the two feature vectors as the loss function, minimize the loss function through iterative training, and complete the training of the population feature extraction model.

[0028] A pathological image cell population feature extraction system, used to implement the above method, the system comprising:

[0029] Image acquisition module: acquire pathological images;

[0030] Cell nucleus segmentation module: input the pathological image into the cell nucleus segmentation model to obtain the cell nucleus segmentation result;

[0031] Cell image acquisition module: crop the pathological image based on the cell nucleus segmentation result to obtain the cell image of each cell in the pathological image;

[0032] Cell feature acquisition module: input the cell image into the cell feature extraction model based on contrast learning to obtain the characteristics and cell location information of each cell;

[0033] Population feature extraction module: The feature sets of all cells and the corresponding cell position information are input into the population feature extraction model based on contrastive learning to obtain the cell population features. During the training process, the population feature extraction model generates positive samples for contrastive learning by randomly selecting rectangular areas of different sizes and arbitrary rotation angles within the pathological image.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] (1) Convert pixel-level features into cell population features, significantly improving robustness: The present invention converts the traditional pathological image pixel-level feature extraction method into an overall feature extraction based on the cell population in the image, effectively reducing the feature differences caused by image scaling, field of view size, and staining errors, greatly enhancing the robustness of the algorithm and making it more adaptable.

[0036] (2) A novel positive sample construction method improves contrastive learning effect: In the process of learning cell population characteristics based on contrastive learning, the present invention proposes an innovative positive sample construction method. By randomly selecting rectangular areas of different sizes and arbitrary rotation angles within the pathological image to generate positive samples for contrastive learning, under a large number of random conditions, the constructed samples maintain a high degree of consistency with the population characteristics of the original samples, thereby significantly improving the learning effect and the generalization ability of the model.

[0037] (3) The group feature extraction method is highly interpretable: The feature extraction method of the present invention is highly interpretable. Its innovation lies in treating the features of each cell in the image as a "word", and all cell features together form a "sentence", and the overall features of the "sentence" are the group features of the image. This method draws on the language model in the neural network, breaks through the limitation of the input word length, and makes the calculation process of group features more intuitive and scientifically meaningful. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of the method of the present invention;

[0039] Figure 2 It is a schematic diagram of the cell nucleus segmentation model of the present invention;

[0040] Figure 3 It is a schematic diagram of the cell feature extraction model of the present invention;

[0041] Figure 4 Schematic diagram of the positive sample construction process of the group feature extraction model of the present invention;

[0042] Figure 5 Schematic diagram of the group feature extraction model of the present invention. DETAILED DESCRIPTION

[0043] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0044] This embodiment provides a method for extracting cell population features from pathological images. Figure 1 As shown, the following steps are included:

[0045] S1, obtain pathological images.

[0046] S2, inputting the pathological image into the cell nucleus segmentation model to obtain the cell nucleus segmentation result.

[0047] Using the integrated model code module of python hub, the deeplabV3_resnet50 model without pre-trained parameters was loaded as the cell nucleus segmentation model. The model was trained using the BCE loss function. The model parameters were saved after training for 100 to 300 epochs until the loss function curve was close to stable.

[0048] like Figure 2 As shown in Figure 1, the training process of the cell nucleus segmentation model includes the following steps:

[0049] S21, obtain a public dataset of cell nucleus segmentation in pathological images, use the xml.dom package of Python to convert the xml annotation file of the dataset into an image format, in which the black color represents the non-cell nucleus area and the white color represents the cell nucleus area, the converted annotation image corresponds to the pathological image one by one, and the pathological image and the annotation image are divided into several 384*384 small images using a sliding window method;

[0050] S22, randomly dividing the segmented small images into a training set and a test set, and using the training set to train the cell nucleus segmentation model, wherein each training image includes a pathological image in RGB format and a black-and-white cell nucleus segmentation annotation image.

[0051] like Figure 2 As shown, the process of using the cell nucleus segmentation model includes the following steps:

[0052] S23, the size of a real full slice is usually very large. The image input size supported by the cell nucleus segmentation model is 384*384, so the pathological image to be segmented needs to be segmented first. The input pathological image to be segmented is segmented into several 384*384 small images, and the segmented small images are used as input data to input the trained cell nucleus segmentation model to obtain the cell nucleus segmentation results of the small images;

[0053] S24, merging the cell nucleus segmentation results of all the small images according to the original segmentation coordinates to obtain the cell nucleus segmentation result of the pathological image, and using the watershed algorithm to obtain the segmentation result of a single cell nucleus.

[0054] S3, cropping the pathological image based on the cell nucleus segmentation result to obtain a cell image of each cell in the pathological image.

[0055] Specifically, according to the center positioning information of each cell in the cell nucleus segmentation result, a rectangular frame of 58*58 pixels is cropped as a cell image with the center positioning information as the center of the rectangular frame.

[0056] S4, inputting the cell image into a cell feature extraction model based on contrastive learning to obtain the characteristics and cell position information of each cell.

[0057] like Figure 3 As shown in the figure, the cell feature extraction model uses the transformer encoding module to extract cell features. The output of the transformer encoding module passes through multiple layers of fully connected layers to output the feature tensor of the cell image.

[0058] During the training process of the cell feature extraction model, data enhancement is performed on the cell image by performing random rotation of 100-200 degrees, random horizontal / vertical flipping and random color jittering to obtain a data-enhanced cell image. The original cell image and the corresponding data-enhanced cell image are taken as a pair of image pairs and input into the transformer encoding module and the fully connected layer with shared parameters respectively to obtain image features. The cosine similarity between the image features of the original cell image and the data-enhanced cell image is calculated as the loss function. The loss function is minimized through iterative training. After the loss function curve is close to stability, the model parameters are saved to complete the cell feature extraction model training.

[0059] In the process of using the cell feature extraction model, a real pathological image first identifies the location of cells through the cell nucleus segmentation model, and then cuts to obtain 58*58 pixel cell images of all cells in the image, which also record their locations in the whole image; finally, these cell images are input into the trained cell feature extraction model to obtain cell features with a dimension of 1024. In this way, the feature set of all cells in a pathological image can be obtained, and the location information of the cells can also be obtained.

[0060] S5, inputting the feature sets of all cells and the corresponding cell position information into the population feature extraction model based on contrastive learning to obtain the cell population features, wherein the population feature extraction model generates positive samples for contrastive learning by randomly selecting rectangular areas of different sizes and arbitrary rotation angles within the pathological image during the training process.

[0061] like Figure 4 As shown in the figure, the positive sample generation method of the population feature extraction model is as follows: assuming that the population feature calculated under completely random conditions from a local image randomly selected from an original image is approximately equal to the population feature calculated from the entire original image, a square area of ​​a preset size is selected at a random position inside the pathological image, the positions of the two vertices above the square area are random in the overall image, the square area is cropped, and the original coordinate positions of the cells in the square area are transformed into relative coordinate positions in the new square area, keeping the image features of the cells unchanged, and constructing positive samples participating in contrastive learning training.

[0062] like Figure 5 As shown in the figure, the population feature extraction model inputs the input data into the transformer encoder, then performs global pooling, and then obtains a tensor with a dimension of 3072 through a multi-layer fully connected layer. This tensor is the cell population feature.

[0063] like Figure 5As shown in the figure, the training process of the population feature extraction model is as follows: sample pairs are constructed based on cell images and constructed positive samples, and two feature tensors are obtained by inputting them into the population feature extraction model respectively. The cosine similarity between the two feature vectors is calculated as the loss function, and the loss function is minimized through iterative training. After the loss function curve is close to stability, the model parameters are saved to complete the training of the population feature extraction model.

[0064] This embodiment also provides a pathological image cell population feature extraction system for implementing the above method, the system comprising:

[0065] Image acquisition module: acquire pathological images;

[0066] Cell nucleus segmentation module: input the pathological image into the cell nucleus segmentation model to obtain the cell nucleus segmentation result;

[0067] Cell image acquisition module: crop the pathological image based on the cell nucleus segmentation result to obtain the cell image of each cell in the pathological image;

[0068] Cell feature acquisition module: input the cell image into the cell feature extraction model based on contrast learning to obtain the characteristics and cell location information of each cell;

[0069] Population feature extraction module: The feature sets of all cells and the corresponding cell position information are input into the population feature extraction model based on contrastive learning to obtain the cell population features. During the training process, the population feature extraction model generates positive samples for contrastive learning by randomly selecting rectangular areas of different sizes and arbitrary rotation angles within the pathological image.

[0070] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0071] In order to verify the performance of the present invention, this embodiment conducts experiments on a public data set, selects the KIRC data therein, and downloads all the full-slice images (ie, pathological images).

[0072] Step 1: Preprocess the whole-slice images: Use the openslide package of python to extract the 40X field of view images from the downloaded svs format images and convert them into jpg images, obtaining a total of 513 images.

[0073] Step 2: Construct an image dataset: Randomly crop 100 regions of size 1000*1000 pixels from each image to obtain 51,300 images of smaller size.

[0074] Step 3: Cell identification and feature calculation: Use the method mentioned in step S2 to train and obtain a cell nucleus segmentation model, and calculate the segmentation and positioning information of the cell nucleus for each 1000*1000 pixel image; use the method mentioned in step S3 to crop a 58*58 pixel rectangular area with the center of the cell nucleus as the cell area, and obtain a cell image set and a cell location information set for each sub-image; use the method mentioned in step S4 to train and calculate the image features of each cell. Each sub-image obtains a data set of cell images, which contains the image features of the cell and the location features of the cell. The image features used here are 1024 dimensions.

[0075] Step 4: Calculation of group characteristics: Use the method mentioned in step S5 to train the group feature extraction model. First, construct the data sets constructed by all sub-graphs as positive samples for comparative learning, and then input them into the group feature extraction model for training and calculation of the group characteristics corresponding to each sub-graph.

[0076] Step 5: Data organization: Through the above steps, the KIRC data is calculated to obtain a group feature matrix of [51300,3072], where 51300 refers to 513 samples, each of which has 10 1000*1000 pixel sub-images, and 3072 represents the dimension of the group feature.

[0077] Step 6: Downstream application example: Correlation between population characteristics and KIRC tumor staging: Download the clinical data corresponding to the KIRC dataset in TCGA, extract the sample name and tumor staging information, and use machine learning algorithms to establish a tumor staging prediction model. The prediction accuracy of G1+G2 and G3+G4 high- and low-grade tumor staging on the test set exceeds 85%.

[0078] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning, or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for extracting cell population features from pathological images, characterized in that: The following steps are involved: Acquire pathological images; The pathological image is input into the cell nucleus segmentation model to obtain the cell nucleus segmentation result; The pathological image is cropped based on the cell nucleus segmentation result to obtain the cell image of each cell in the pathological image; Input the cell image into the cell feature extraction model based on contrast learning to obtain the characteristics and cell location information of each cell; The feature sets of all cells and the corresponding cell position information are input into a population feature extraction model based on contrastive learning to obtain cell population features, wherein the population feature extraction model generates positive samples for contrastive learning by randomly selecting rectangular areas of different sizes and arbitrary rotation angles within the pathological image during training.

2. A method for extracting cell population features from pathological images according to claim 1, characterized in that: The training process of the cell nucleus segmentation model includes the following steps: Obtain a public dataset of pathological image cell nucleus segmentation, convert the annotation file of the dataset into an image format, wherein the black color in the converted annotation image represents the non-cell nucleus area and the white color represents the cell nucleus area, the converted annotation image corresponds to the pathological image one by one, and the pathological image and the annotation image are divided into a plurality of small images of a first preset size using a sliding window method; The segmented small images are randomly divided into training sets and test sets, and the cell nucleus segmentation model is trained using the training set, where each training image contains an RGB pathological image and a black-and-white cell nucleus segmentation annotation image.

3. A method for extracting cell population features from pathological images according to claim 2, characterized in that: The process of using the cell nucleus segmentation model includes the following steps: The input pathological image to be segmented is divided into a plurality of small images of a first preset size, and the divided small images are input as input data into a trained cell nucleus segmentation model to obtain cell nucleus segmentation results of the small images; The cell nucleus segmentation results of all the small images are merged according to the original segmentation coordinates to obtain the cell nucleus segmentation result of the pathological image, and the segmentation result of a single cell nucleus is obtained using the watershed algorithm.

4. The method for extracting cell population features from pathological images according to claim 1, characterized in that: The cropping of the pathological image based on the cell nucleus segmentation result is specifically: according to the center positioning information of each cell in the cell nucleus segmentation result, taking the center positioning information as the center of the rectangular frame, cropping a rectangular frame of a second preset size as the cell image.

5. The method for extracting cell population features from pathological images according to claim 1, characterized in that: The cell feature extraction model uses a transformer encoding module to extract cell features, and the output of the transformer encoding module passes through multiple layers of fully connected layers to output the feature tensor of the cell image.

6. A method for extracting cell population features from pathological images according to claim 5, characterized in that: During the training process of the cell feature extraction model, data enhancement is performed on the cell image by randomly rotating, randomly flipping horizontally / vertically, and randomly color jittering to obtain a data-enhanced cell image. The original cell image and the corresponding data-enhanced cell image are taken as a pair of image pairs and are respectively input into a transformer encoding module and a fully connected layer with shared parameters to obtain image features. The cosine similarity between the image features of the original cell image and the data-enhanced cell image is calculated as a loss function. The loss function is minimized through iterative training to complete the training of the cell feature extraction model.

7. The method for extracting cell population features from pathological images according to claim 1, characterized in that: The population feature extraction model inputs the input data into the transformer encoder, then performs global pooling, and then obtains a fixed-dimensional tensor through a multi-layer fully connected layer. The fixed-dimensional tensor is the cell population feature.

8. The method for extracting cell population features from pathological images according to claim 1, characterized in that: The positive sample generation method of the population feature extraction model is specifically as follows: assuming that the population features calculated under completely random conditions for a local image randomly selected from an original image are approximately equal to the population features calculated for the entire original image, a square area of ​​a preset size is selected at a random position inside the pathological image, the positions of the two vertices above the square area are random in the overall image, the square area is cropped, and the original coordinate positions of the cells in the square area are transformed into relative coordinate positions in the new square area, the image features of the cells are kept unchanged, and positive samples participating in comparative learning training are constructed.

9. The method for extracting cell population features from pathological images according to claim 1, characterized in that: The training process of the population feature extraction model is as follows: construct sample pairs based on cell images and constructed positive samples, input them into the population feature extraction model to obtain two feature tensors respectively, calculate the cosine similarity between the two feature vectors as the loss function, minimize the loss function through iterative training, and complete the training of the population feature extraction model.

10. A pathological image cell population feature extraction system, characterized in that: For implementing the method according to any one of claims 1 to 9, the system comprises: Image acquisition module: acquire pathological images; Cell nucleus segmentation module: input the pathological image into the cell nucleus segmentation model to obtain the cell nucleus segmentation result; Cell image acquisition module: crop the pathological image based on the cell nucleus segmentation result to obtain the cell image of each cell in the pathological image; Cell feature acquisition module: input the cell image into the cell feature extraction model based on contrast learning to obtain the characteristics and cell location information of each cell; Population feature extraction module: The feature sets of all cells and the corresponding cell position information are input into the population feature extraction model based on contrastive learning to obtain the cell population features. During the training process, the population feature extraction model generates positive samples for contrastive learning by randomly selecting rectangular areas of different sizes and arbitrary rotation angles within the pathological image.

Images