System and image segmentation method for microsatellite instability prediction in colorectal cancer

Abstract

The present application belongs to the technical field of medical image processing, and particularly relates to a system for predicting microsatellite instability of colorectal cancer and an image segmentation method. The segmentation method comprises the following steps: reading a multi-resolution image from original full field of view slice image data, segmenting a high-resolution image after removing background, and obtaining a plurality of blocks; calculating image entropy of all blocks obtained in step 1; screening blocks according to image entropy, discarding blocks with excessively low image entropy, and obtaining an Entropy-mask composed of remaining blocks; and multiplying the Entropy-mask with a high-resolution image to obtain a segmented effective region. The present application also provides a system for predicting microsatellite instability by using a machine learning model trained through weak supervision learning based on the segmentation result. The present application has the advantages of high efficiency, reliability, wide applicability, no need for manual annotation and high cost-effectiveness, and has a good application prospect in clinical practice.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical image processing, and particularly relates to a system for predicting microsatellite instability of colorectal cancer and an image segmentation method. BACKGROUND

[0002] Colorectal cancer is the third most common cancer in the world today, and the second leading cause of death worldwide, with about 900,000 people dying from colorectal cancer each year. Microsatellite instability (MSI) refers to the increase or loss of simple repeat sequences due to mismatch repair or replication errors, resulting in changes in the length of microsatellites. The main cause of MSI is the defect in the function of genes involved in mismatch repair, which cannot correct replication errors normally, causing changes in microsatellite DNA, so that it cannot normally play a regulatory role. MSI can easily lead to abnormal cell proliferation and differentiation, and even promote the formation of malignant tumors. Compared with normal tissues, microsatellites in tumors are more likely to change the length of microsatellites. MSI can be divided into three categories according to the degree: microsatellite high instability (MSI-H), microsatellite low instability (MSI-L), and microsatellite stability (MSS). After MSI analysis of the pathological pictures of patient tissue samples, an MSI score is obtained, and the MSI category is further determined.

[0003] Studies have shown that MSI is closely related to the occurrence of colorectal cancer, gastric cancer, endometrial cancer and other tumors. High microsatellite instability MSI-H is also a marker of hereditary non-polyposis colorectal cancer, i.e. Lynch syndrome (LS), and is a favorable prognostic factor and negative predictive factor for Ⅱ stage colorectal cancer patients with fluorouracil-based adjuvant chemotherapy. About 15% of sporadic colorectal cancer is related to MSI, and more than 90% of hereditary non-polyposis colorectal cancer (HNPCC, also known as Lynch syndrome) is related to MSI, so MSI detection is of great significance in clinical practice.

[0004] The commonly used microsatellite site stability detection method at present is based on the unified detection standard for MSI detection approved by the National Cancer Institute of the United States: detecting the instability of 5 repeat regions (microsatellite sites) to determine the microsatellite site stability of the sample. However, due to the high cost, long time consumption, and the fact that small hospitals do not have the detection conditions, this method is limited by many reasons, so that a large number of colorectal cancer patients have no opportunity to do microsatellite instability detection.

[0005] The artificial intelligence technology-based colorectal cancer microsatellite status classification method classifies the colorectal cancer microsatellite status through the design of a targeted network architecture and the autonomous learning of the gastric cancer pathological image, and makes the trained neural network model capable of classifying the colorectal cancer microsatellite status, thereby making up for the shortcomings of traditional methods.

[0006] However, the whole slide image (WSI) for detecting the colorectal cancer microsatellite status is large. For a 1cm*1cm pathological section, at a resolution of 20X, each pixel size is 0.5um, and the resolution of the whole pathological section is about 20000x20000, and the image file size is about 1GB. Therefore, it is difficult for an ordinary computer to directly process it, and it is necessary to process it into 512x512 small patches / blocks / tiles for operation. However, after such processing, the number of patch images increases dramatically. In these patch images, there is redundant information irrelevant to the identification task, and therefore, it is necessary for a person to pay attention to the interested part of the image, obtain key information, and suppress other useless information.

[0007] The existing scheme selects the tumor region by manual selection, or selects the tumor region by a pre-trained tumor segmentation model, or judges whether it is a tumor by a tissue classification model, thereby reducing the number of effective patches, so that the model training can be performed by using a deep learning method.

[0008] However, whether it is a tumor segmentation model or a tissue classification model, the pathological image needs to be accurately labeled by a pathologist in advance, and then the model training can be performed. Due to the large image, the workload of the doctor's labeling is large. Moreover, the classification accuracy of the model is affected by the training data, and the labeling result of the pathologist has a certain degree of subjective influence, which further affects the difference of the subsequent MSI model judgment.

[0009] In addition, in most training processes, the same weight is used for the MSI status of the patch generated from the same WSI, and the heterogeneity between patches is ignored, which also affects the accuracy of the result to some extent.

[0010] Therefore, how to pay attention to the interested part of the image, automatically obtain key information, suppress other useless information, and solve the problem of inaccurate MSI label of each patch is a problem that needs to be solved urgently. SUMMARY

[0011] In view of the defects of the prior art, the present application provides a system and an image segmentation method for colorectal cancer microsatellite instability prediction, aiming to provide a method for automatically segmenting the effective area in a WSI, which can effectively extract key information and inhibit useless information, thereby reducing the workload of manual annotation and effectively improving the prediction performance of the final model.

[0012] A full-view slice image effective area segmentation method for machine learning, comprising the following steps:

[0013] Step 1. Read a multi-resolution image from the original full-view slice image data, and segment the high-resolution image after removing the background to obtain a plurality of blocks;

[0014] Step 2. Calculate the image entropy of all blocks obtained in step 1;

[0015] Step 3. Screen the blocks according to the image entropy, discard the blocks with too low image entropy, and obtain an Entropy-mask composed of the remaining blocks;

[0016] Step 4. Multiply the Entropy-mask with the high-resolution image to obtain the segmented effective area.

[0017] Preferably, in step 1, the method for removing the background is:

[0018] Step 1.1. Segment the effective area using a low-resolution image to obtain a mask image under low resolution;

[0019] Step 1.2. Remove the background of the high-resolution image according to the mask image under low resolution.

[0020] Preferably, in step 1.1, the resolution of the low-resolution image is (800-1200) x (800-1200); and in step 4, the resolution of the high-resolution image is (60000-140000) x (60000-140000).

[0021] Preferably, in step 1.1, the segmentation method is threshold segmentation.

[0022] Preferably, the size of the block is half of the tilesize in the full-view slice image.

[0023] Preferably, in step 4, after obtaining the segmented effective area, it is segmented into a plurality of examples with fixed size, each example containing the identification of the position information of the example in the segmented effective area.

[0024] The present application also provides a system for colorectal cancer microsatellite instability prediction, comprising:

[0025] an input module configured to input original whole field slice image data;

[0026] an example generation module configured to segment the whole field slice image data by using the segmentation method;

[0027] a model prediction module configured to input the segmentation result of the example generation module into a machine learning model to obtain a prediction result of colorectal cancer microsatellite instability and generate a microsatellite state heat map of the whole field slice image;

[0028] an output module configured to output the calculation result of the model prediction module.

[0029] Preferably, the training method of the machine learning model is as follows:

[0030] Step a: labeling each sample, and the label is microsatellite high instability, microsatellite low instability and microsatellite stability;

[0031] Step b: training the sample with microsatellite high instability as positive and the sample with microsatellite stability as negative to obtain an initial model Model_0;

[0032] Step c: predicting all samples by using the Model_0, and sorting them according to the probability of microsatellite instability from high to low;

[0033] Step d: training again by using the samples ranked in the top m% among the samples with microsatellite high instability as positive, the samples ranked in the top n% among the samples with microsatellite low instability as positive, and the samples with microsatellite stability as negative to obtain a model Model_1; wherein m is greater than n;

[0034] Step e: repeating the above steps c to d until the training reaches a stop condition to obtain a final prediction model of colorectal cancer microsatellite instability state.

[0035] Preferably, the system comprises a slide scanner control computer, a cloud server and a mobile terminal in communication with each other, the input module and the example generation module are located in the slide scanner control computer, the model prediction module and the output module are located in the cloud server, and the output module outputs the calculation result to the mobile terminal.

[0036] The application further provides a computer readable storage medium having a computer program for implementing the above segmentation method or system stored thereon.

[0037] The application provides a method for automatically segmenting WSI images, which can segment effective areas from WSI images, thereby reducing the number of patches input into a prediction model subsequently, compared with the method of the prior art, the segmentation method of the application does not need manual segmentation and manual annotation data for training a segmentation model, and can greatly reduce segmentation cost and manual workload.

[0038] In addition, in the preferred scheme, the application reduces the dependence of model training on sample annotation accuracy by optimizing the model training process, and overcomes the problem of ignoring the heterogeneity between patches in the prior art in the process of iterative training, so that a prediction model with better prediction performance can be obtained.

[0039] It can be seen that the application has the advantages of high efficiency, reliability, wide applicability, introduction of weak supervision learning so as to eliminate the need for manual annotation, and high cost-effectiveness, and by using the multi-instance learning method, the machine learning model is used to predict the microsatellite instability of colorectal cancer, the threshold of colorectal cancer microsatellite instability detection is greatly reduced, the widespread promotion of the detection of the microsatellite instability state of colorectal cancer patients becomes possible, and the doctor can help the patient to receive targeted treatment, which has high clinical significance.

[0040] Obviously, according to the above content of the application, according to the ordinary technical knowledge and conventional means in the art, other various forms of modifications, replacements or changes can be made without departing from the above basic technical idea of the application.

[0041] The above content of the application will be further described in detail through the specific embodiments in the form of examples. However, this should not be understood as limiting the scope of the above subject matter of the application to the following examples. Any technology realized based on the above content of the application belongs to the scope of the application. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 The flowchart for generating instances in Example 1 is shown in the figure;

[0043] Figure 2 The flowchart for generating instances in Example 1 is shown in the figure, wherein "LowResolutionImage" in the figure is the low-resolution image obtained in step S1, "GrayMask" is the image obtained after removing the background image in step S2, "Tissue Blocks" is the segmented block obtained in step S3, "Entropy-mask" is the image obtained in step S4, and "Tiles / instances" is the instance obtained in step S5.

[0044] Figure 3Figure for repeating the training process in the multiple-instance learning process in Example 2;

[0045] Figure 4 Figure and results for predicting microsatellite instability status of colorectal cancer from WSI image data in Example 2. DETAILED DESCRIPTION

[0046] It should be particularly noted that the algorithms of the data acquisition, transmission, storage and processing steps not specifically described in the examples, and the hardware structure, circuit connection, etc. not specifically described can be realized by the existing technology.

[0047] Example 1 WSI image effective area segmentation method for machine learning

[0048] This embodiment is a method for processing WSI images, aiming to segment the effective area that can be used to predict the microsatellite instability of colorectal cancer without manual operation or labeling. The specific steps are as shown in Figure 1 、 Figure 2 , including:

[0049] S1: From the original WSI, read the multi-resolution image, and record the original graph (high-resolution image) as HR-image and the lowest-resolution image as LR-image. For a general WSI, the physical slice of 2cmx2cm, under the conventional parameters of 40X, MPP=0.25 (each pixel size is 0.25 microns), the HR-image size can be selected as (60000-140000) x (60000-140000); the LR-image size can be selected as (800-1200) x (800-1200).

[0050] S2: Use the low-resolution image (LR-image) to segment the effective area, and the segmentation method can be realized by the existing technology. Specifically, this embodiment uses the simplest and most effective threshold segmentation method to obtain the mask image under low resolution: LR-mask.

[0051] At this time, a large number of background graphs will be excluded, and only the tissue part will be left. Processing the background under high resolution will be too large in calculation. Therefore, this embodiment uses the low-resolution image to remove the background, which can be processed quickly.

[0052] S3: According to the LR-mask, the HR-image is cut into small blocks, and the resolution of the block is 256x256. The entropy of the image is calculated.

[0053] In this embodiment, the size of the block is selected as half of the tile size in the WSI. In this way, the organization information contained in each block can be as much as possible, while also retaining enough size for subsequent calculation.

[0054] S4, screening the blocks according to image entropy, taking the median of all the calculated entropies as the threshold, and retaining those greater than the threshold and discarding those less than the threshold. Thus, the Entropy-mask is generated. It should be noted that the setting of the threshold can be adjusted according to the actual demand for model performance in other embodiments.

[0055] The part related to the MSI status is often tumor cells and inflammatory cells adjacent to the cancer. The data of this part is very rich in cell and texture information in the image, so the present application proposes to use image entropy to screen the blocks.

[0056] S5: multiplying the HR-image with the Entropy-MASK generated in the previous step to obtain the effective area, and dividing the effective area into fixed-size segments, which are generally set to 512x512. These small images are called instances, and each instance is encoded according to its row and column position in the WSI to identify its position in the original WSI. The set of instances of the entire WSI is called a package.

[0057] Embodiment 2: System for predicting microsatellite instability of colorectal cancer

[0058] The system of this embodiment includes a slide scanner control computer, a cloud server and a mobile terminal in communication with each other. The input module and the instance generation module are located in the slide scanner control computer, the model prediction module and the output module are located in the cloud server, and the output module outputs the calculation results to the mobile terminal. The mobile terminal is used by doctors and patients.

[0059] wherein,

[0060] The input module is used to input the original WSI image data.

[0061] The instance generation module is used to segment the WSI image data according to the segmentation method of embodiment 1.

[0062] The model prediction module is used to input the results of the instance generation module into a machine learning model to obtain the prediction results of the microsatellite instability of colorectal cancer, and generate a microsatellite status heat map of the whole field of view slice image.

[0063] The output module is used to output the calculation results of the model prediction module.

[0064] The process of predicting microsatellite instability of colorectal cancer using the above system is as follows:

[0065] 1. Use a scanner to scan the pathological section to obtain a target pathological picture, which is a picture of the tissue region obtained after cell staining of the patient's tissue sample.

[0066] 2. According to the obtained target pathological picture data WSI, generate example instance pictures that meet the conditions according to the WSI.

[0067] 3. The example picture obtained in step 3 is transmitted to the cloud server.

[0068] This operation can be synchronized with three, and each example can be sent to the cloud server simultaneously.

[0069] 4. Use the microsatellite instability prediction model on the cloud server to predict the microsatellite instability weight of each received example.

[0070] 5. The prediction results of each example are spliced to generate a microsatellite state heat map of the WSI.

[0071] The prediction results are spliced according to the position encoding of each example.

[0072] 6. The cloud server sends the prediction results to the doctor's mobile terminal for confirmation and signature.

[0073] After the doctor confirms the results, the cloud server sends the report to the patient.

[0074] As a further preferred solution, to solve the problem that the same weight is used for the MSI state of the Patch generated by the existing method for the same WSI, ignoring the heterogeneity between Patches, the embodiment adopts a weakly supervised learning method to train the model in the prediction module. The model used to predict microsatellite instability adopts a multiple-instance learning method, and the specific training process is as follows:

[0075] S1: Collect and organize the pathological section images of colorectal cancer and their MSI detection results.

[0076] S2: Divide the WSI image into examples using the example generation method in embodiment 1;

[0077] The image size is 512x512, and each small image is recorded as an example. For the collection of all examples under a patient, it is recorded as a package. For each package, the labels are microsatellite high instability (MSI-H), microsatellite low instability (MSI-L), and microsatellite stable (MSS).

[0078] S3: For the screened examples, we label each example according to the package label, and mark the examples in MSI-H package as positive, and mark the examples in MSS package as negative; MSI-L is temporarily not added to the training set.

[0079] According to the above data, the initial model Model_0 is trained.

[0080] The network structure used in this step can be a classic classification network such as ResNet, MobileNet, VGG, etc., or a custom classification network structure. In this embodiment, we choose shufflenet as the backbone network structure of this step.

[0081] S4: Using the trained Model_0, the probability of MSI of each example in each package is predicted, and the examples are sorted according to the probability from high to low.

[0082] S5: The samples ranked in the top m% of samples labeled as microsatellite instability MSI-H and the samples ranked in the top n% of samples labeled as microsatellite instability MSI-L are used as positive samples. For the values of m and n, if the amount of pathological samples is large, m and n can be small, for example, 20% and 10% respectively, if the amount of pathological samples is small, the value can be larger, such as 50% and 20%. Overall, since MSI-H has a higher microsatellite instability state than MSI-L, the value of m must be greater than n.

[0083] Through this operation, we can gradually screen positive samples into samples with obvious MSI characteristics, and also avoid non-tumor samples, samples with no obvious MSI characteristics, and samples as positive samples for further training in the ROI region.

[0084] And the microsatellite stable samples are always used as negative samples, and the model model_1 is obtained by training again.

[0085] S6: Repeat steps S4 and S5 until the training reaches the stopping condition (such as a fixed number of iterations, or a high enough prediction accuracy). Figure 3 is a process diagram of repeating training in this step.

[0086] S7: The obtained prediction model of the microsatellite instability state of colorectal cancer is uploaded to the cloud server.

[0087] Figure 4The process and result image of using the system of the embodiment for colorectal cancer microsatellite instability state prediction of WSI image data of a pathological section with a physical size of 17mm x 18.5mm, wherein the size of the HR-image is 68000 x 73728, and the size of the LR-image is 1062 x 1152.

[0088] As can be seen from the above embodiments, the system and image segmentation method for colorectal cancer microsatellite instability prediction provided by the present application have the advantages of high efficiency, reliability, wide use, no need for manual annotation, and high cost-effectiveness. Through the use of the multi-instance learning method, the threshold for colorectal cancer microsatellite instability detection is greatly reduced, making it possible to widely promote the detection of the microsatellite instability state of colorectal cancer patients. It can help doctors to carry out targeted treatment for patients, and has very high clinical significance.

Claims

1. A method for effective region segmentation of full-view slice images for machine learning, characterized in that, Includes the following steps: Step 1: Read multi-resolution images from the original full-view slice image data, remove the background from the high-resolution images and then segment them to obtain several blocks; Step 2: Calculate the image entropy of all blocks obtained in Step 1; Step 3: Filter blocks based on image entropy, discarding blocks with excessively low image entropy to obtain an entropy mask composed of the remaining blocks; Step 4: Multiply the Entropy-mask with the high-resolution image to obtain the segmented effective region; In step 1, the method for segmenting after removing the background is as follows: The effective region is segmented using a low-resolution image to obtain a low-resolution mask image; based on the low-resolution mask image, the high-resolution image is divided into several blocks.

2. The segmentation method according to claim 1, characterized in that: In step 1, the resolution of the low-resolution image is (800-1200) × (800-1200); in step 4, the resolution of the high-resolution image is (60000-140000) × (60000-140000).

3. The segmentation method according to claim 1, characterized in that: In step 1, the segmentation method is threshold segmentation.

4. The segmentation method according to claim 1, characterized in that: The size of the block is half the tile size in the full-view slice image.

5. The segmentation method according to claim 1, characterized in that: In step 4, after obtaining the segmented effective region, it is divided into several examples of a fixed size. Each example contains an identifier of its position information in the segmented effective region.

6. A system for predicting microsatellite instability in colorectal cancer, characterized in that, include: The input module is used to input raw full-view slice image data; Example generation module, used to segment the full-view slice image data according to the segmentation method of claim 5; The model prediction module is used to input the segmentation results of the example generation module into the machine learning model to obtain the microsatellite instability prediction results of colorectal cancer and generate a microsatellite state heat map of the full-view slice image; The output module is used to output the calculation results of the model prediction module.

7. The system according to claim 6, characterized in that: The training method for the machine learning model is as follows: Step a: Label each sample as microsatellite highly unstable, microsatellite poorly unstable, and microsatellite stable; Step b: Use highly unstable microsatellite samples as positive samples and stable microsatellite samples as negative samples for training to obtain the initial model Model_0; Step c: Predict all samples using Model_0 and sort them from high to low according to the probability of microsatellite instability; Step d: The top m% of samples labeled as highly unstable microsatellites are selected as positive samples, the top n% of samples labeled as lowly unstable microsatellites are selected as positive samples, and the samples labeled as stable microsatellites are selected as negative samples. The samples are then trained again to obtain model_1; where m is greater than n. Step e: Repeat steps c to d above until the training reaches the stopping condition, and obtain the final prediction model of colorectal cancer microsatellite instability.

8. The system according to claim 6, characterized in that: The system includes a slide scanner control computer, a cloud server, and a mobile terminal that communicate with each other. The input module and the example generation module are located on the slide scanner control computer, the model prediction module and the output module are located on the cloud server, and the output module outputs the calculation results to the mobile terminal.

9. A computer-readable storage medium, characterized in that: It stores a computer program for implementing the segmentation method according to any one of claims 1-5.

Images