A pathological image cell population feature extraction method and system

Abstract

The present application relates to a kind of pathological image cell population feature extraction method and system, wherein the method comprises the following steps: obtaining pathological image;Pathological image is input into cell nucleus segmentation model, and cell nucleus segmentation result is obtained;Pathological image is cropped based on cell nucleus segmentation result, and the cell image of each cell in pathological image is obtained;Cell image is input into the cell feature extraction model based on contrast learning, and the feature and cell location information of each cell are obtained;The feature set of all cells and corresponding cell location information are input into the population feature extraction model based on contrast learning, and the cell population feature is obtained, wherein, population feature extraction model generates positive sample of contrast learning by randomly selecting rectangular region of different size, arbitrary rotation angle in pathological image during training process.Compared with prior art, the present application has the advantages of good robustness, strong generalization ability and strong explainability.

Description

Technical Field

[0001] This invention relates to the field of digital pathological image processing technology, and in particular to a method and system for extracting cell population features from pathological images. Background Technology

[0002] In the field of computer vision-based digital pathology image analysis, existing technologies mainly focus on pixel-level feature extraction, overall image feature analysis, and classification and segmentation methods for target or functional regions based on traditional machine learning or deep learning. However, with the continuous improvement of pathology image resolution and the rapid increase in the number of samples, existing technologies face many challenges in processing complex image data, especially in population feature analysis, where they suffer from the following major shortcomings:

[0003] Pixel-level features are difficult to handle the diversity of pathological images: Most existing technologies rely on pixel-level feature extraction, but pixel-level features are greatly affected by image scaling, staining conditions and field of view in different pathological images, resulting in significant differences in feature performance under different experimental conditions, which affects the generalization ability and robustness of the model.

[0004] Lack of in-depth analysis of the overall characteristics of cell-level populations: While some technologies can extract and analyze the morphological features of individual cells, existing technologies do not offer effective solutions for analyzing the overall characteristics of aggregated cell populations, especially in complex pathological environments such as tumors where cells cluster according to certain rules, focusing on their group behavior and characteristics. This leads to the neglect of global information in pathological images in many scenarios, limiting the accuracy of pathological diagnosis and the expanded application of artificial intelligence analysis algorithms under interpretable conditions.

[0005] Limitations of contrastive learning sample construction methods: Existing contrastive learning-based methods mostly rely on simple computer vision transformation operations (such as rotation and scaling) to generate comparative samples during the sample construction process. This method does not consider the similarity of cell population aggregation characteristics after changes within a certain physical range of the image, and cannot fully capture the diversity of cell populations in pathological images, thus affecting the model's performance in practical applications.

[0006] Insufficient interpretability: While existing deep learning models have achieved good results in pathological image analysis, their high complexity and lack of transparency and interpretability in feature extraction and inference processes limit their widespread application in practical medicine. In particular, the modeling of cell population characteristics fails to provide a clear interpretive path, casting doubt on the model's credibility and application value.

[0007] In summary, existing technologies for processing pathological images mainly suffer from insufficient robustness, lack of population feature analysis, limited sample construction methods, and poor model interpretability. These shortcomings hinder in-depth analysis of pathological images and their clinical application, necessitating a novel technical solution that can overcome these limitations. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of existing technologies, such as insufficient robustness, lack of population feature analysis, single sample construction method, and poor model interpretability, and to provide a method and system for extracting cell population features from pathological images.

[0009] The objective of this invention can be achieved through the following technical solutions:

[0010] A method for extracting cell population features from pathological images includes the following steps:

[0011] Obtain pathological images;

[0012] The pathological image is input into the cell nucleus segmentation model to obtain the cell nucleus segmentation result;

[0013] Based on the cell nucleus segmentation results, the pathological image is cropped to obtain the cell image of each cell in the pathological image;

[0014] The cell images are input into a cell feature extraction model based on contrastive learning to obtain the features and cell location information of each cell;

[0015] The feature set of all cells and their corresponding cell location information are input into a population feature extraction model based on contrastive learning to obtain cell population features. During the training process, the population feature extraction model generates positive samples for contrastive learning by randomly selecting rectangular regions of different sizes and arbitrary rotation angles within the pathological image.

[0016] The training process of the cell nucleus segmentation model includes the following steps:

[0017] Obtain a public dataset of cell nucleus segmentation for pathological images, convert the dataset's annotation file into an image format, where black represents non-nucleus regions and white represents nucleus regions in the converted annotation images, and the converted annotation images correspond one-to-one with the pathological images. The pathological images and annotation images are then segmented into several smaller images of a first preset size using a sliding window method.

[0018] The segmented images are randomly divided into training and testing sets. The cell nucleus segmentation model is trained using the training set. Each training image contains an RGB pathological image and a black and white cell nucleus segmentation annotation image.

[0019] The process of using the cell nucleus segmentation model includes the following steps:

[0020] The input pathological image to be segmented is divided into several small images of a first preset size, and the segmented small images are used as input data to input the trained cell nucleus segmentation model to obtain the cell nucleus segmentation results of the small images;

[0021] The cell nucleus segmentation results of all the small images are merged according to the original segmentation coordinates to obtain the cell nucleus segmentation results of the pathological image, and the watershed algorithm is used to obtain the segmentation results of individual cell nuclei.

[0022] The specific method for cropping the pathological image based on the cell nucleus segmentation results is as follows: according to the center positioning information of each cell in the cell nucleus segmentation results, a rectangle of a second preset size is cropped as the cell image, with the center positioning information as the center of the rectangle.

[0023] The cell feature extraction model uses a transformer encoding module to extract cell features. The output of the transformer encoding module is passed through multiple fully connected layers to output the feature tensor of the cell image.

[0024] During the training process of the cell feature extraction model, data augmentation is performed on the cell image by random rotation, random horizontal / vertical flipping, and random color jitter to obtain data-augmented cell images. The original cell image and the corresponding data-augmented cell image are treated as a pair of images and input into the transformer encoding module and the fully connected layer with shared parameters, respectively, to obtain image features. The cosine similarity between the image features of the original cell image and the data-augmented cell image is calculated as the loss function. The loss function is minimized through iterative training to complete the training of the cell feature extraction model.

[0025] The population feature extraction model inputs the input data into a transformer encoder, then performs global pooling, and then obtains a fixed-dimensional tensor through a multi-layer fully connected layer. The fixed-dimensional tensor is the cell population feature.

[0026] The positive sample generation method of the population feature extraction model is as follows: assuming that the population features calculated by randomly selecting a local image in an original image under completely random conditions are approximately equal to the population features calculated by the entire original image, a square region of a preset size is randomly selected at a random position inside the pathological image. The positions of the two vertices above the square region are random within the overall image. The square region is cropped, and the original coordinate positions of the cells within the square region are transformed into relative coordinate positions within the new square region, while preserving the image features of the cells. Positive samples for comparison learning training are then constructed.

[0027] The training process of the population feature extraction model is as follows: Sample pairs are constructed based on cell images and constructed positive samples, and then input into the population feature extraction model to obtain two feature tensors. The cosine similarity between the two feature vectors is calculated as the loss function. The loss function is minimized through iterative training to complete the training of the population feature extraction model.

[0028] A system for extracting cell population features from pathological images, used to implement the method described above, the system comprising:

[0029] Image acquisition module: Acquires pathological images;

[0030] Cell nucleus segmentation module: Input pathological images into the cell nucleus segmentation model to obtain cell nucleus segmentation results;

[0031] Cell image acquisition module: Based on the cell nucleus segmentation results, the pathological image is cropped to obtain the cell image of each cell in the pathological image;

[0032] Cell feature acquisition module: Input cell images into a cell feature extraction model based on contrastive learning to obtain the features and cell location information of each cell;

[0033] Population feature extraction module: Input the feature set of all cells and the corresponding cell location information into the population feature extraction model based on contrastive learning to obtain cell population features. In the training process, the population feature extraction model generates positive samples for contrastive learning by randomly selecting rectangular regions of different sizes and arbitrary rotation angles inside the pathological image.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] (1) Converting pixel-level features into cell population features significantly improves robustness: This invention converts the traditional method of extracting pixel-level features from pathological images into extracting overall features based on cell populations within the image. This effectively reduces feature differences caused by image scaling ratio, field of view size, and staining errors, greatly enhancing the robustness and adaptability of the algorithm.

[0036] (2) Novel Positive Sample Construction Method Enhances Contrastive Learning Performance: In the process of cell population feature learning based on contrastive learning, this invention proposes an innovative positive sample construction method. By randomly selecting rectangular regions of different sizes and arbitrary rotation angles within the pathological image to generate positive samples for contrastive learning, under a large number of random conditions, the constructed samples maintain a high degree of consistency with the population features of the original samples, thereby significantly improving the learning effect and the generalization ability of the model.

[0037] (3) The group feature extraction method has high interpretability: The feature extraction method of this invention has strong interpretability. Its innovation lies in treating the features of each cell in the image as a 'word', and the features of all cells together form a 'sentence'. The overall features of the 'sentence' are the group features of the image. This method draws on the language model in neural networks, breaks through the limitation of input word length, and makes the calculation process of group features more intuitive and scientifically significant. Attached Figure Description

[0038] Figure 1 This is a flowchart of the method of the present invention;

[0039] Figure 2 This is a schematic diagram of the cell nucleus segmentation model of the present invention;

[0040] Figure 3 This is a schematic diagram of the cell feature extraction model of the present invention;

[0041] Figure 4 This is a schematic diagram illustrating the positive sample construction process of the population feature extraction model of the present invention;

[0042] Figure 5 This is a schematic diagram of the group feature extraction model of the present invention. Detailed Implementation

[0043] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0044] This embodiment provides a method for extracting cell population features from pathological images, such as... Figure 1 As shown, it includes the following steps:

[0045] S1, acquire pathological images.

[0046] S2: Input the pathological image into the cell nucleus segmentation model to obtain the cell nucleus segmentation result.

[0047] Using the integrated model code module of Python Hub, the deeplabV3_resnet50 model without pre-trained parameters was loaded as a cell nucleus segmentation model. The BCE loss function was used to train the model. After training for 100 to 300 epochs until the loss function curve was close to stationary, the model parameters were saved.

[0048] like Figure 2 As shown, the training process of the cell nucleus segmentation model includes the following steps:

[0049] S21. Obtain a public dataset of cell nucleus segmentation for pathological images. Use the xml.dom package in Python to convert the XML annotation file of the dataset into an image format. In the converted annotation image, black represents non-nucleus regions and white represents nucleus regions. The converted annotation image corresponds one-to-one with the pathological image. Use the sliding window method to divide the pathological image and annotation image into several small images of 384*384.

[0050] S22, the segmented small images are randomly divided into training set and test set, and the cell nucleus segmentation model is trained using the training set. Each training image contains a pathological image in RGB format and a black and white cell nucleus segmentation annotation image.

[0051] like Figure 2 As shown, the process of using the cell nucleus segmentation model includes the following steps:

[0052] S23. A real whole-slice image is usually very large. The cell nucleus segmentation model supports an image input size of 384*384. Therefore, the pathological image to be segmented needs to be segmented first. The input pathological image to be segmented is divided into several smaller images of 384*384. The segmented smaller images are then used as input data to the trained cell nucleus segmentation model to obtain the cell nucleus segmentation results for the smaller images.

[0053] S24: Merge the cell nucleus segmentation results of all small images according to the original segmentation coordinates to obtain the cell nucleus segmentation results of the pathological image, and use the watershed algorithm to obtain the segmentation results of individual cell nuclei.

[0054] S3, based on the cell nucleus segmentation results, crop the pathological image to obtain the cell image of each cell in the pathological image.

[0055] Specifically, based on the center location information of each cell in the cell nucleus segmentation results, a 58*58 pixel rectangle is cropped as the cell image, with the center location information as the center of the rectangle.

[0056] S4. Input the cell image into the cell feature extraction model based on contrastive learning to obtain the features and cell location information of each cell.

[0057] like Figure 3 As shown, the cell feature extraction model uses a transformer encoding module to extract cell features. The output of the transformer encoding module is output as a feature tensor of the cell image after passing through multiple fully connected layers.

[0058] During the training of the cell feature extraction model, data augmentation is performed on the cell image by randomly rotating it by 100-200 degrees, randomly flipping it horizontally / vertically, and randomly jittering its colors to obtain data-augmented cell images. The original cell image and the corresponding data-augmented cell image are treated as a pair of images and input into a transformer encoding module and a fully connected layer with shared parameters, respectively, to obtain image features. The cosine similarity between the image features of the original cell image and the data-augmented cell image is calculated as the loss function. The loss function is minimized through iterative training. After the loss function curve approaches a stationary state, the model parameters are saved, and the training of the cell feature extraction model is completed.

[0059] In the process of using the cell feature extraction model, a real pathological image is first used to identify the location of cells through a cell nucleus segmentation model. Then, it is segmented to obtain 58*58 pixel cell images of all cells in the image, and these images also record their positions in the whole image. Finally, these cell images are input into a trained cell feature extraction model to obtain cell features with a dimension of 1024. In this way, the feature set of all cells in a pathological image can be obtained, and the cell location information can also be obtained.

[0060] S5. Input the feature set of all cells and the corresponding cell location information into the population feature extraction model based on contrastive learning to obtain the cell population features. During the training process, the population feature extraction model generates positive samples for contrastive learning by randomly selecting rectangular regions of different sizes and arbitrary rotation angles inside the pathological image.

[0061] like Figure 4 As shown, the positive sample generation method of the population feature extraction model is as follows: assuming that the population features calculated by randomly selecting a local image in an original image under completely random conditions are approximately equal to the population features calculated by the entire original image, a square region of a preset size is randomly selected at a random position inside the pathological image. The positions of the two vertices above the square region are random within the overall image. The square region is cropped, and the original coordinate positions of the cells within the square region are transformed into the relative coordinate positions within the new square region, while preserving the image features of the cells. Positive samples for comparison learning training are then constructed.

[0062] like Figure 5 As shown, the population feature extraction model inputs the input data into the transformer encoder, then performs global pooling, and then obtains a tensor with a dimension of 3072 through a multi-layer fully connected layer. This tensor is the cell population feature.

[0063] like Figure 5As shown, the training process of the population feature extraction model is as follows: Sample pairs are constructed based on cell images and constructed positive samples, and then input into the population feature extraction model to obtain two feature tensors. The cosine similarity between the two feature vectors is calculated as the loss function. The loss function is minimized through iterative training. After the loss function curve approaches stationary, the model parameters are saved to complete the training of the population feature extraction model.

[0064] This embodiment also provides a pathological image cell population feature extraction system for implementing the method described above. The system includes:

[0065] Image acquisition module: Acquires pathological images;

[0066] Cell nucleus segmentation module: Input pathological images into the cell nucleus segmentation model to obtain cell nucleus segmentation results;

[0067] Cell image acquisition module: Based on the cell nucleus segmentation results, the pathological image is cropped to obtain the cell image of each cell in the pathological image;

[0068] Cell feature acquisition module: Input cell images into a cell feature extraction model based on contrastive learning to obtain the features and cell location information of each cell;

[0069] Population feature extraction module: Input the feature set of all cells and the corresponding cell location information into the population feature extraction model based on contrastive learning to obtain cell population features. In the training process, the population feature extraction model generates positive samples for contrastive learning by randomly selecting rectangular regions of different sizes and arbitrary rotation angles inside the pathological image.

[0070] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0071] To verify the performance of the present invention, this embodiment conducted an experiment on a public dataset, selecting the KIRC data and downloading all whole-slice images (i.e., pathological images).

[0072] Step 1: Preprocess the whole-slice image: Use the openslide package in Python to extract the 40X field of view from the downloaded SVS format image and convert it into a JPG image, resulting in a total of 513 images.

[0073] Step 2: Construct an image dataset: randomly crop 100 regions of size 1000*1000 pixels from each image to obtain 51,300 smaller images.

[0074] Step 3: Cell Identification and Feature Calculation: Using the method mentioned in step S2, a cell nucleus segmentation model is trained. For each 1000*1000 pixel image, the segmentation and localization information of the cell nucleus is calculated. Using the method mentioned in step S3, a 58*58 pixel rectangular region is cropped with the center of the cell nucleus as the cell region, resulting in a set of cell images and a set of cell location information for each sub-image. Using the method mentioned in step S4, the image features of each cell are trained and calculated. Each sub-image yields a set of cell image data, which includes the cell's image features and location features. The image features used here are 1024-dimensional.

[0075] Step 4: Calculation of group features: Train the group feature extraction model using the method mentioned in step S5. First, construct contrastive learning positive samples from the datasets constructed from all subgraphs, and then input them into the group feature extraction model to train and calculate the group features corresponding to each subgraph.

[0076] Step 5: Data processing: Through the above steps, the KIRC data calculation yielded a matrix of group characteristics [51300, 3072], where 51300 represents 513 samples, each sample having 10 sub-images of 1000*1000 pixels, and 3072 represents the dimension of the group characteristics.

[0077] Step 6: Downstream Application Example: Correlation between Population Characteristics and KIRC Tumor Staging: Download the clinical data corresponding to the KIRC dataset in TCGA, extract the sample names and tumor staging information, and use machine learning algorithms to build a tumor staging prediction model. The accuracy of predicting high and low grade tumor stages (G1+G2 and G3+G4) on the test set exceeds 85%.

[0078] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A pathological image cell population feature extraction method, characterized by, The method comprises the following steps: Obtaining a pathological image; Inputting the pathological image into a nucleus segmentation model to obtain a nucleus segmentation result; Cutting the pathological image based on the nucleus segmentation result to obtain a cell image of each cell in the pathological image; Inputting the cell image into a cell feature extraction model based on contrast learning to obtain the feature and cell position information of each cell; Inputting the feature set and corresponding cell position information of all cells into a population feature extraction model based on contrast learning to obtain cell population features, wherein the population feature extraction model generates positive samples for contrast learning by randomly selecting rectangular regions of different sizes and arbitrary rotation angles within the pathological image during the training process. The positive sample generation method of the population feature extraction model is specifically as follows: assuming that the population features calculated under completely random conditions of a randomly selected local image in an original image are approximately equal to the population features calculated from the entire original image, a square region of a preset size is selected at a random position within the pathological image, the positions of the two top points above the square region are random within the entire image, the square region is cut, and the original coordinate positions of the cells in the square region are transformed into relative coordinate positions within the new square region, the image features of the cells are kept unchanged, and a positive sample participating in contrast learning training is constructed.

2. The pathological image cell population feature extraction method of claim 1, wherein, The training process of the nucleus segmentation model comprises the following steps: Obtaining a public data set for pathological image nucleus segmentation, converting the annotation file of the data set into an image format, the black part of the converted annotation image representing a non-nucleus region and the white part representing a nucleus region, the converted annotation image corresponding to the pathological image one by one, and cutting the pathological image and the annotation image into a plurality of small images of a first preset size by using a sliding window method. Randomly dividing the cut small images into a training set and a test set, and training the nucleus segmentation model by using the training set, wherein each training image comprises a pathological image in RGB form and a black-and-white nucleus segmentation annotation image.

3. The pathological image cell population feature extraction method of claim 2, wherein, The use process of the nucleus segmentation model comprises the following steps: Cutting the input pathological image to be segmented into a plurality of small images of a first preset size, and inputting the cut small images as input data into the trained nucleus segmentation model to obtain the nucleus segmentation result of the small images; Combining the nucleus segmentation results of all small images according to the original cutting coordinates to obtain the nucleus segmentation result of the pathological image, and using a watershed algorithm to obtain the segmentation result of a single nucleus.

4. The pathological image cell population feature extraction method of claim 1, wherein, The cutting of the pathological image based on the nucleus segmentation result is specifically as follows: according to the center positioning information of each cell in the nucleus segmentation result, taking the center positioning information as the center of a rectangular frame, and cutting a rectangular frame of a second preset size as a cell image.

5. The pathological image cell population feature extraction method of claim 1, wherein, The cell feature extraction model uses a transformer encoding module to extract cell features, and the output of the transformer encoding module is output through a plurality of fully connected layers to output a feature tensor of the cell image.

6. The pathological image cell population feature extraction method of claim 5, wherein, In the training process of the cell feature extraction model, data augmentation is performed on the cell images by random rotation, random horizontal / vertical flipping and random color jittering to obtain data augmented cell images. The original cell images and the corresponding data augmented cell images are input into the transformer encoding module and the fully connected layer sharing the parameters, respectively, to obtain image features. The cosine similarity between the image features of the original cell images and the data augmented cell images is calculated as a loss function. The loss function is minimized through iterative training to complete the training of the cell feature extraction model.

7. The pathological image cell population feature extraction method of claim 1, wherein, The population feature extraction model inputs the input data into the transformer encoder, then performs global pooling, and then obtains a fixed-dimensional tensor through a multi-layer fully connected layer. The fixed-dimensional tensor is the cell population feature.

8. The pathological image cell population feature extraction method of claim 1, wherein, The training process of the population feature extraction model is as follows: based on the cell images and the constructed positive samples, a sample pair is constructed, and two feature tensors are obtained by inputting the sample pair into the population feature extraction model. The cosine similarity between the two feature vectors is calculated as a loss function. The loss function is minimized through iterative training to complete the training of the population feature extraction model.

9. A pathological image cell population feature extraction system characterized by, The system is used to implement the method of any one of claims 1-8. An image acquisition module: acquiring a pathological image; A nucleus segmentation module: inputting the pathological image into a nucleus segmentation model to obtain a nucleus segmentation result; A cell image acquisition module: cropping the pathological image based on the nucleus segmentation result to obtain a cell image of each cell in the pathological image; A cell feature acquisition module: inputting the cell image into a cell feature extraction model based on contrast learning to obtain the feature and the cell position information of each cell; A population feature extraction module: inputting the feature set and the corresponding cell position information of all cells into a population feature extraction model based on contrast learning to obtain a cell population feature. In the training process of the population feature extraction model, positive samples for contrast learning are generated by randomly selecting rectangular regions of different sizes and arbitrary rotation angles within the pathological image.

Images