Method for detecting tumor purity, model training method, equipment and medium

Abstract

The present application discloses a method for detecting tumor purity, a model training method, equipment and medium, wherein the original pathological image is cut and processed to obtain multiple sub-images to be detected; the multiple sub-images to be detected are input into a pre-trained cell counting model to obtain the detection density map of each sub-image to be detected output by the cell counting model; wherein the detection density map includes a cancer cell density map and a non-cancerous cell density map; the cell counting model is obtained by optimizing the parameters and hyperparameters of the initial model based on multiple training sub-images and their reference density maps, and the reference density map is generated based on the training sub-images and their annotation results, and the annotation results include the correspondence between the cell center annotation position and the color information; according to the detection density map of each sub-image to be detected, the tumor purity detection result is determined. It can simply and quickly realize the identification of cancer cells and non-cancerous cells and the detection of tumor purity, and improve the diagnostic efficiency and accuracy.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular, to a method for detecting tumor purity, a method for training a cell counting model, an electronic device and a computer-readable medium. Background Art

[0002] Currently, there are many methods for estimating tumor purity for papillary thyroid carcinoma (PTC), but these methods all have some disadvantages: (1) The accuracy of the tumor purity obtained by the pathologist's naked eye is limited. This is because there will be large subjective errors between different scholars when relying solely on hematoxylin-eosin (HE) staining to identify cell morphology. (2) Hashimoto's thyroiditis (HT) is an autoimmune disease that exists in the thyroid gland. PTC patients with HT account for about 10%-50%, and the pathological manifestations are infiltration by a large number of lymphocytes. If the tumor purity is assessed only by HE staining, it will undoubtedly greatly increase the difficulty of counting. (3) Although several methods including ABSOLUTE, ESTIMATE and purity score consensus (CPE) can reversely infer tumor purity from sequencing results, these methods are expensive and time-consuming, and require high professionalism, so they are difficult to popularize. (4) Microdissection technology has many problems such as complex operation, small sampling, long time consumption and high cost, and cannot be carried out on a large scale.

[0003] Therefore, how to accurately identify thyroid malignant tumor cells and quickly count them in high-throughput is an urgent problem that needs to be solved. Summary of the invention

[0004] The present application aims to solve one of the technical problems in the related art to a certain extent. To this end, the present application provides a method for detecting tumor purity, a method for training a cell counting model, an electronic device and a computer readable medium.

[0005] As a first aspect of the present application, a method for detecting tumor purity is provided, wherein the method comprises:

[0006] Performing cutting processing on the original pathological image to obtain multiple sub-images to be detected;

[0007] Input the multiple sub-images to be detected into a pre-trained cell counting model to obtain a detection density map of each sub-image to be detected output by the cell counting model; wherein the detection density map includes a cancer cell density map and a non-cancerous cell density map; the cell counting model is obtained by optimizing the parameters and hyperparameters of the initial model according to the multiple training sub-images and their reference density maps, the reference density map is generated according to the training sub-images and their annotation results, and the annotation results include the correspondence between the cell center annotation position and the color information;

[0008] The tumor purity detection result is determined according to the detection density map of each sub-image to be detected.

[0009] Optionally, determining the tumor purity detection result according to the detection density map of each sub-image to be detected includes:

[0010] For any of the sub-images to be detected, determining the number of cancer cells to be detected according to the cancer cell density map of the sub-image to be detected, and determining the number of non-cancer cells to be detected according to the non-cancer cell density map of the sub-image to be detected;

[0011] The tumor purity detection result is determined according to the detection number of cancer cells and the detection number of non-cancerous cells corresponding to each of the sub-images to be detected.

[0012] A second aspect of the present application provides a method for training a cell counting model, wherein the method comprises:

[0013] Preprocessing the sample pathological image to obtain multiple training sub-images;

[0014] According to the preset one-to-one correspondence between the color information and the cell type, the labeling result of each training sub-image is obtained; wherein the labeling result includes the correspondence between the cell center labeling position and the color information, and the cell type includes cancer cells and non-cancerous cells;

[0015] Generating a reference density map for each of the training sub-images according to the training sub-images and the annotation results thereof;

[0016] Input the plurality of training sub-images into the initial model to obtain a detection density map of each of the training sub-images output by the initial model; wherein the density map includes a cancer cell density map and a non-cancerous cell density map;

[0017] According to the loss function between the reference density map and the detection density map of each training sub-image, the parameters and hyperparameters of the initial model are optimized to obtain a fitted cell counting model.

[0018] Optionally, the preprocessing of the sample pathological image to obtain a plurality of training sub-images includes:

[0019] Performing segmentation processing on the sample pathological image to obtain a plurality of sub-pathological images;

[0020] Performing grayscale conversion processing on the multiple sub-pathological images to obtain multiple grayscale sub-images;

[0021] Binarizing the plurality of grayscale sub-images according to a preset grayscale threshold to obtain a plurality of binary sub-images;

[0022] For any of the binarized sub-images, the average grayscale value of each pixel in the binarized sub-image is calculated, and when the average value is not lower than a preset average value threshold, the sub-pathological image corresponding to the binarized sub-image is determined as a training sub-image.

[0023] Optionally, the method further comprises:

[0024] Performing cutting processing on the test pathological image to obtain multiple test sub-images;

[0025] Inputting the plurality of test sub-images into the fitted cell counting model to obtain a detection density map of each of the test sub-images output by the fitted cell counting model;

[0026] Determining the cell center detection position corresponding to each of the test sub-images according to the detection density map;

[0027] Obtaining the actual total number of cells and the actual position of the cell center corresponding to each of the test sub-images;

[0028] Determining error information of the fitted cell counting model according to the actual total number of cells, the actual position of the cell center and the detected position of the cell center corresponding to each of the test sub-images;

[0029] When the error information is less than a preset threshold, the fitted cell counting model is used as a trained cell counting model.

[0030] Optionally, the error information includes a mean absolute error value MAE and / or a mean square error value MSE.

[0031] Optionally, generating a reference density map for each of the training sub-images according to the training sub-images and the annotation results thereof includes:

[0032] For each pixel point at the marked position of the cell center, the grayscale values ​​of other pixels within a preset fuzzy radius of the pixel point are updated according to a preset function and the grayscale value of the pixel point;

[0033] For any of the training sub-images, a reference density map is generated for the training sub-image according to the grayscale value of each pixel in the training sub-image after the update.

[0034] Optionally, each of the models includes a cascaded backbone network, a pyramid scale network and an output layer; the backbone network includes ten convolutional layers and three pooling layers; the pyramid scale network includes a global attention module and a feature pyramid module in parallel; the output layer includes three cascaded second convolutional filters and one first convolutional filter;

[0035] The global attention module includes a cascaded global pooling layer and two first convolution filters;

[0036] The feature pyramid module includes a first unit, a second unit, a third unit and a second convolution filter, and the second convolution filter is used to fuse the connection features between the output features of the first unit, the output features of the second unit and the output features of the third unit;

[0037] The first unit includes one second convolution filter; the second unit includes one third convolution filter and one second convolution filter in cascade, and the second convolution filter in the second unit is used to fuse the output features of the first unit with the output features of the third convolution filter in the second unit; the third unit includes one fourth convolution filter and one second convolution filter in cascade, and the second convolution filter in the third unit is used to fuse the output features of the second unit with the output features of the fourth convolution filter in the third unit.

[0038] According to a third aspect of the present application, an electronic device is provided, wherein the electronic device includes:

[0039] one or more processors;

[0040] A memory having one or more computer programs stored thereon, wherein when the one or more computer programs are executed by the one or more processors, the one or more processors implement any of the following:

[0041] The first aspect of the present application provides a method for detecting tumor purity;

[0042] A second aspect of the present application provides a method for training a cell counting model.

[0043] According to a fourth aspect of the present application, a computer-readable medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, any of the following is implemented:

[0044] The first aspect of the present application provides a method for detecting tumor purity;

[0045] A second aspect of the present application provides a method for training a cell counting model.

[0046] In the method for detecting tumor purity provided in the embodiment of the present application, the original pathological image is cut and processed to obtain a plurality of sub-images to be detected; the plurality of sub-images to be detected are input into a pre-trained cell counting model to obtain the detection density map of each sub-image to be detected output by the cell counting model; wherein the detection density map includes a cancer cell density map and a non-cancer cell density map; the cell counting model is obtained by optimizing the parameters and hyperparameters of the initial model according to a plurality of training sub-images and their reference density maps, the reference density map is generated according to the training sub-images and their annotation results, the annotation results include the correspondence between the cell center annotation position and the color information; according to the detection density map of each sub-image to be detected, the tumor purity detection result is determined. Applying artificial intelligence algorithms to the processing of high-density cell images, its automated processing efficiency and strong adaptability help to improve objectivity and processing speed, can simply and quickly realize the identification of cancer cells and non-cancer cells and the detection of tumor purity, reduces labor costs, improves diagnostic efficiency and accuracy, and also provides important technical support for clinical diagnosis and pathological research. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The present application is further described below in conjunction with the accompanying drawings:

[0048] Figure 1 This is a flow chart of an implementation method of the method for detecting tumor purity provided in the examples of the present application;

[0049] Figure 2 This is a flow chart of an implementation method of determining the tumor purity test result provided in the examples of the present application;

[0050] Figure 3 It is a flowchart of an implementation method of the cell counting model training method provided in the examples of the present application;

[0051] Figure 4 This is a flowchart of an implementation method of preprocessing a sample pathological image to obtain multiple training sub-images provided in an embodiment of the present application;

[0052] Figure 5 It is a flowchart of another implementation of the training method of the cell counting model provided in the examples of the present application;

[0053] Figure 6 This is a flowchart of an implementation method of generating a reference density map for each training sub-image provided in an embodiment of the present application;

[0054] Figure 7 It is a schematic diagram of an implementation of the initial model and cell counting model provided in the examples of the present application;

[0055] Figure 8 It is a module diagram of an implementation of an electronic device provided in an embodiment of the present application;

[0056] Fig. 9 It is a schematic diagram of a computer-readable medium provided in an embodiment of the present application.

[0057] Description of Reference Numerals

[0058] 101: Processor 102: Memory

[0059] 103: I / O interface 104: bus DETAILED DESCRIPTION

[0060] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments based on the embodiments are intended to be used to explain the present application and cannot be understood as limiting the present application.

[0061] References to "one embodiment" or "an example" or "an example" in this specification mean that a particular feature, structure, or characteristic described in conjunction with the embodiment itself may be included in at least one embodiment disclosed in the present application. The appearance of the phrase "in one embodiment" in various places in the specification does not necessarily refer to the same embodiment.

[0062] At present, it is urgent to solve the problem of how to accurately identify thyroid malignant tumor cells and quickly perform high-throughput counting on them. In this regard, the applicant of this application believes that it is necessary to consider the following two aspects: first, how to accurately distinguish cancer cells from non-cancerous cells; second, how to quickly perform high-throughput counting. It can be considered to use artificial intelligence to assist the development process to solve this problem.

[0063] The traditional AI-assisted development process includes: target segmentation, feature calculation and classification. The deep learning-based model gradually recognizes and learns through the hierarchical structure of the deep neural network. In this process, irrelevant features will be attenuated and filtered out, and the important attributes of the image will be gradually identified and amplified, and identified and classified in a self-learning manner through a deep neural network model. Therefore, in the field of medical diagnosis and pathology research, it will be of great significance to realize the use of artificial intelligence to calculate the number of various types of cells in pathological images.

[0064] The applicant of this application further proposed that it can be implemented based on the crowd counting algorithm. The crowd counting algorithm was originally proposed to count the number of pedestrians in a scene, and has shown wide application potential in many fields such as video surveillance, traffic monitoring, public safety, urban planning, and smart supermarket construction. At present, there are three common schools of crowd counting algorithms: target detection-based methods, regression-based methods, and density map prediction-based methods.

[0065] Based on the target detection method, each pedestrian in the image is located and identified, and then the number of people is counted based on the identification results. Its advantage is that it can provide accurate pedestrian locations and is suitable for low-density crowd images. However, for high-density crowd images, due to occlusion and overlap between multiple targets, the detection effect of this method is often poor. In addition, since the location of each pedestrian needs to be manually marked, the labeling cost is also very high.

[0066] The regression-based method does not accurately locate the position of pedestrians, but makes a rough estimate of the number of people in the crowd. According to the specific implementation method, the crowd image is directly input into the deep learning neural network, and an estimated value of the number of pedestrians is output. Its advantage is that it works better for high-density crowd images, but its disadvantage is that it cannot provide accurate pedestrian locations.

[0067] Based on the density map prediction method, the approximate center position of each head is manually annotated, and then the size of the head at that position is estimated to obtain the coverage area of ​​the head. Through methods such as geometric adaptive Gaussian kernel, the area is converted into the probability that there may be a head in the area, and finally a crowd density map is obtained. The above method can output a density map of the same size as the input image through an input image, and the number of pedestrians can be obtained by integrating the density map. Compared with other methods, this method is better when processing high-density crowd images. It can not only estimate the number of pedestrians, but also reflect the spatial distribution characteristics of the crowd.

[0068] The applicant of the present application further proposes that the method based on density map prediction can be considered to be applied to pathological image analysis. By converting the pathological slice image into a density map, the number of different types of cells can be effectively counted, especially high-density cell images, such as the statistics of malignant tumor cells, i.e., cancer cells.

[0069] As a first aspect of the embodiment of the present application, a method for detecting tumor purity is provided, such as Figure 1 As shown, the method may include:

[0070] In step S110, the original pathological image is cut to obtain a plurality of sub-images to be detected;

[0071] In step S120, the plurality of sub-images to be detected are input into a pre-trained cell counting model to obtain a detection density map of each sub-image to be detected output by the cell counting model; wherein the detection density map includes a cancer cell density map and a non-cancerous cell density map; the cell counting model is obtained by optimizing the parameters and hyperparameters of the initial model according to the plurality of training sub-images and their reference density maps, the reference density map is generated according to the training sub-images and their annotation results, and the annotation results include the correspondence between the cell center annotation position and the color information;

[0072] In step S130, a tumor purity detection result is determined according to the detection density map of each sub-image to be detected.

[0073] The original pathological image is obtained by scanning HE-stained thyroid paraffin sections using a pathological instrument, and is a high-resolution digital image. For example, a 40X image is obtained by scanning using the pathological digital scanner KF-040-pro.

[0074] Among them, the embodiment of the present application does not make any special limitation on how to perform this step S110. For example, an open source Python data processing library OpenSlide can be used to read the original resolution of the original pathological image through the interface open_slide provided by OpenSlide. The target resolution is pre-set according to the original resolution, and the original pathological image is cut. In the process of cutting the original pathological image, the original pathological image is cut into multiple sub-pathological images, that is, multiple sub-images to be detected, according to the preset target resolution. The cutting process can ensure that there will be no repeated or missing parts of the image, and ensure the integrity of the image data. Finally, the multiple sub-images to be detected obtained by cutting can be saved as files in ".png" format.

[0075] In the embodiment of the present application, multiple sub-images to be detected can also be named according to the order of rows and columns in the original pathological image, and the name of each sub-image to be detected contains the index of the row and column where it is located. For example, the sub-image to be detected located in the 2nd row and the 3rd column may be named "image_2_3.png". This naming method allows the position of the sub-image to be detected in the original pathological image to be obtained through its name, which is not only convenient for image management and retrieval, but also can quickly correspond the sub-image to be detected with its position in the original pathological image during the subsequent analysis process.

[0076] Among them, the cell counting model is pre-trained based on the sample pathology image. Specifically, the sample pathology image is pre-processed to obtain multiple training sub-images; according to the preset one-to-one correspondence between color information and cell type, the annotation results of each training sub-image are obtained; wherein the annotation results include the correspondence between the cell center annotation position and the color information, and the cell types include cancer cells and non-cancerous cells; according to each training sub-image and its annotation results, a reference density map is generated for each training sub-image; multiple training sub-images are input into the initial model to obtain the detection density map of each training sub-image output by the initial model; wherein the density map includes the cancer cell density map and the non-cancerous cell density map; according to the loss function between the reference density map and the detection density map of each training sub-image, the parameters and hyperparameters of the initial model are optimized to obtain a fitted cell counting model.

[0077] After obtaining the cancer cell density map and non-cancer cell density map of each sub-image to be detected, the number of cancer cells and non-cancer cells can be obtained by integrating the density map, and then the tumor purity detection result can be obtained.

[0078] In high-density cell images, traditional methods based on target detection are often difficult to handle occlusion and overlapping phenomena, while methods based on density map prediction can better cope with these challenges and provide accurate cell count statistics. Density maps can not only reflect the number of cells, but also reflect the spatial distribution characteristics of cells, thereby providing more accurate cell counts. Through the cell counting model based on density maps, images can be automatically and quickly analyzed, the time cost and physical cost of manual identification counting can be greatly reduced, the diagnostic efficiency can be improved, and the errors caused by human subjective factors can be reduced, which helps to understand the pathological characteristics of the lesion area more deeply, and provide more abundant information for clinical diagnosis and pathological research. Therefore, the method for detecting tumor purity provided in the embodiment of the present application, its automated processing efficiency and strong adaptability, not only reduce labor costs, improve diagnostic efficiency and accuracy, but also provide important technical support for clinical diagnosis and pathological research.

[0079] In the method for detecting tumor purity provided in the embodiment of the present application, the original pathological image is cut and processed to obtain a plurality of sub-images to be detected; the plurality of sub-images to be detected are input into a pre-trained cell counting model to obtain the detection density map of each sub-image to be detected output by the cell counting model; wherein the detection density map includes a cancer cell density map and a non-cancer cell density map; the cell counting model is obtained by optimizing the parameters and hyperparameters of the initial model according to a plurality of training sub-images and their reference density maps, the reference density map is generated according to the training sub-images and their annotation results, the annotation results include the correspondence between the cell center annotation position and the color information; according to the detection density map of each sub-image to be detected, the tumor purity detection result is determined. Applying artificial intelligence algorithms to the processing of high-density cell images, its automated processing efficiency and strong adaptability help to improve objectivity and processing speed, can simply and quickly realize the identification of cancer cells and non-cancer cells and the detection of tumor purity, reduces labor costs, improves diagnostic efficiency and accuracy, and also provides important technical support for clinical diagnosis and pathological research.

[0080] It can be understood that the method for detecting tumor purity provided in the present application can be used not only to detect the tumor purity of papillary thyroid carcinoma (PTC), but also to detect the purity of other types of tumors.

[0081] As described above, the density map can reflect the spatial distribution characteristics of cells. Accordingly, in some embodiments, the tumor purity detection result is determined according to the detection density map of each sub-image to be detected (i.e., step S130). Figure 2 As shown, it may include:

[0082] In step S210, for any of the sub-images to be detected, the number of cancer cells to be detected is determined according to the cancer cell density map of the sub-image to be detected, and the number of non-cancer cells to be detected is determined according to the non-cancer cell density map of the sub-image to be detected;

[0083] In step S220, the tumor purity detection result is determined according to the number of cancer cell detections and the number of non-cancerous cell detections corresponding to each of the sub-images to be detected.

[0084] Among them, for each sub-image to be detected, the cell counting model outputs a cancer cell density map and a non-cancerous cell density map.

[0085] In the cancer cell density map, the detected cancer cells are represented by a dot-like area whose brightness is significantly different from other areas. A dot-like area may include only one pixel or several pixels that are tightly clustered. The same is true for the non-cancerous cell density map. By integrating the cancer cell density map or the non-cancerous cell density map, the number of cancer cell detections or the number of non-cancerous cell detections in the density map can be obtained.

[0086] For each sub-image to be detected, its tumor purity detection result can be expressed as: number of cancer cell detections / (number of cancer cell detections + number of non-cancer cell detections). For each original pathological image, its tumor purity detection result can be expressed as: the sum of the number of cancer cell detections of multiple sub-images to be detected / (the sum of the number of cancer cell detections of multiple sub-images to be detected + the sum of the number of non-cancer cell detections of multiple sub-images to be detected).

[0087] As a second aspect of the embodiment of the present application, a training method for a cell counting model is provided, such as Figure 3 As shown, the method may include:

[0088] In step S310, the sample pathological image is preprocessed to obtain a plurality of training sub-images;

[0089] In step S320, according to the preset one-to-one correspondence between color information and cell types, the labeling results of each training sub-image are obtained; wherein the labeling results include the correspondence between the cell center labeling position and the color information, and the cell types include cancer cells and non-cancerous cells;

[0090] In step S330, a reference density map is generated for each of the training sub-images according to the training sub-images and the annotation results thereof;

[0091] In step S340, the plurality of training sub-images are input into the initial model to obtain a detection density map of each of the training sub-images output by the initial model; wherein the density map includes a cancer cell density map and a non-cancerous cell density map;

[0092] In step S350, the parameters and hyperparameters of the initial model are optimized according to the loss function between the reference density map and the detection density map of each training sub-image to obtain a fitted cell counting model.

[0093] Among them, the embodiment of the present application does not specifically limit how to perform step S320. For example, it can be preset: use yellow with an RGB value of (255, 255, 0) to mark cancer cells, and use green with an RGB value of (0, 255, 0) to mark non-cancerous cells. By displaying the preset correspondence and the training sub-image, the annotator (who can be a more experienced pathologist) is prompted to use the color number marking method to mark the training sub-image. The yellow and green colors can visually make cancer cells and non-cancerous cells form a significant contrast with other cells and the background, which helps to speed up the marking speed.

[0094] After the annotator completes the annotation, the cell center annotation position can be further extracted and recorded according to the annotation color. The annotation results reflect the annotated cancer cells and their cell center annotation positions, non-cancerous cells and their cell center annotation positions. A reference density map can be generated for the training sub-images and their annotation results, which helps to characterize the spatial distribution of cells, facilitates the initial model to automatically learn the cell features in each reference density map, is applicable to cells of various sizes, and improves the accuracy of counting.

[0095] In the training method of the cell counting model provided in the embodiment of the present application, the sample pathological image is preprocessed to obtain multiple training sub-images; according to the one-to-one correspondence between the preset color information and the cell type, the annotation results of each of the training sub-images are obtained; wherein the annotation results include the correspondence between the cell center annotation position and the color information, and the cell types include cancer cells and non-cancerous cells; according to each of the training sub-images and their annotation results, a reference density map is generated for each of the training sub-images; the multiple training sub-images are input into the initial model to obtain a detection density map of each of the training sub-images output by the initial model; wherein the density map includes a cancer cell density map and a non-cancerous cell density map; according to the loss function between the reference density map and the detection density map of each of the training sub-images, the parameters and hyperparameters of the initial model are optimized to obtain a fitted cell counting model.

[0096] In order to make it easier to process and analyze the sample pathology image, it is preprocessed, for example, segmented and invalid information filtered. Accordingly, in some embodiments, the sample pathology image is preprocessed to obtain a plurality of training sub-images (i.e., step S310), such as Figure 4 As shown, it may include:

[0097] In step S410, the sample pathology image is segmented to obtain a plurality of sub-pathology images;

[0098] In step S420, grayscale conversion processing is performed on the multiple sub-pathological images to obtain multiple grayscale sub-images;

[0099] In step S430, the plurality of grayscale sub-images are binarized according to a preset grayscale threshold value to obtain a plurality of binarized sub-images;

[0100] In step S440, for any of the binarized sub-images, the average grayscale value of each pixel in the binarized sub-image is calculated, and when the average value is not lower than a preset average value threshold, the sub-pathological image corresponding to the binarized sub-image is determined as a training sub-image.

[0101] Among them, the embodiments of the present application do not make any special restrictions on how to cut the sample pathology image. For example, an open source Python data processing library OpenSlide can be used to read the original resolution of the sample pathology image through the open_slide interface provided by OpenSlide. The target resolution is pre-set according to the original resolution, and the sample pathology image is cut. In the process of cutting the sample pathology image, the sample pathology image is cut into multiple sub-pathology images according to the preset target resolution. The cutting process can ensure that there will be no repeated or missing parts of the image, thereby ensuring the integrity of the image data. Finally, the multiple sub-pathology images obtained by cutting can be saved as files in ".png" format.

[0102] In the embodiment of the present application, multiple sub-pathology images can also be named according to the order of rows and columns in the sample pathology image, and the name of each sub-pathology image contains the index of the row and column where it is located. For example, the sub-pathology image located in the 2nd row and the 3rd column may be named "image_2_3.png". This naming method allows the position of the sub-pathology image in the sample pathology image to be obtained through its name, which not only facilitates image management and retrieval, but also allows the sub-pathology image to be quickly matched with its position in the sample pathology image during subsequent analysis.

[0103] The embodiment of the present application does not specifically limit how to perform grayscale conversion on the sub-pathology images. For example, another open source Python data processing library OpenCV can be used to complete the grayscale conversion of each sub-pathology image through the interface provided by OpenCV to obtain multiple grayscale sub-images.

[0104] After obtaining multiple grayscale sub-images, the pixels in the grayscale sub-images can be divided into two categories, i.e., foreground pixels and background pixels, by comparing the pixel values ​​in the grayscale sub-images with the preset grayscale thresholds, thereby obtaining multiple binary sub-images. This method facilitates clear distinction between valid information and invalid information in the image.

[0105] After obtaining multiple binary sub-images, for each binary sub-image, the average value of the grayscale value of each pixel in the binary sub-image is calculated, and the average value can be used as a basis for whether the binary sub-image contains a large amount of invalid information. If the average value is lower than the preset average value threshold, it means that the sub-pathological image corresponding to the binary sub-image is mainly composed of invalid information, so the sub-pathological image corresponding to the binary sub-image can be filtered out, and the filtered sub-pathological image is not used for data processing, which can improve the efficiency of data processing and analysis. If the average value is not lower than the preset average value threshold, it means that the sub-pathological image corresponding to the binary sub-image contains a large amount of valid information, has more analytical value, and can be used for training, so the sub-pathological image corresponding to the binary sub-image is used as a training sub-image, which helps to improve the efficiency of data processing and analysis.

[0106] It can be seen that through the above steps S420-S440, the multiple sub-pathological images obtained by cutting in the above step S410 are screened, and the sub-pathological images containing more effective information are used as training sub-images.

[0107] In the training method of the cell counting model provided in the embodiment of the present application, the sample pathological image is cut to obtain multiple sub-pathological images; the multiple sub-pathological images are gray-scale converted to obtain multiple gray-scale sub-images; the multiple gray-scale sub-images are binarized according to a preset gray-scale threshold to obtain multiple binarized sub-images; for any of the binarized sub-images, the average gray-scale value of each pixel in the binarized sub-image is calculated, and when the average value is not lower than the preset average value threshold, the sub-pathological image corresponding to the binarized sub-image is determined as the training sub-image. The integrity of the image data can be guaranteed, the efficiency of data processing and analysis can be improved, and the speed of model training can be increased.

[0108] After obtaining the fitted cell counting model, the accuracy of the fitted cell counting model can be verified using the test set data. Figure 5 As shown, the method also includes:

[0109] In step S510, the test pathological image is segmented to obtain a plurality of test sub-images;

[0110] In step S520, the plurality of test sub-images are input into the fitted cell counting model to obtain a detection density map of each of the test sub-images output by the fitted cell counting model;

[0111] In step S530, according to the detection density map, the cell center detection position corresponding to each of the test sub-images is determined;

[0112] In step S540, the actual total number of cells and the actual position of the cell center corresponding to each of the test sub-images are obtained;

[0113] In step S550, error information of the fitted cell counting model is determined according to the actual total number of cells, the actual position of the cell center and the detected position of the cell center corresponding to each of the test sub-images;

[0114] In step S560, when the error information is less than a preset threshold, the fitted cell counting model is used as a trained cell counting model.

[0115] Among them, the embodiment of the present application does not make any special limitation on how to execute step S510. The execution method can be consistent with the above-mentioned step S110 and step S410. It has been described in detail above and will not be repeated here.

[0116] Among them, the embodiment of the present application does not specifically limit how to obtain the actual total number of cells and the actual position of the cell center corresponding to each test sub-image. For example, the labeling results of each test sub-image can be obtained based on the one-to-one correspondence between the preset color information and the cell type, and then the actual position of the cell center can be extracted and the actual total number of cells can be counted.

[0117] The smaller the error information of the fitted cell counting model is, the more accurate the cell positions and cell numbers detected by the cell counting model are. Thus, when the error information is less than a preset threshold, the fitted cell counting model can be used as a trained cell counting model.

[0118] It can be understood that, when the error information is not less than the preset threshold, it can be considered that the cell position and cell number detected by the cell counting model are not accurate enough, and the above steps S410-S440 can be continued to train the initial model until the error information of the obtained fitted cell counting model is determined to be less than the preset threshold using the test set data.

[0119] In some embodiments, the error information includes a mean absolute error value MAE and / or a mean square error value MSE.

[0120] Among them, the embodiment of the present application can use the mean absolute error value MAE to evaluate the effect of the cell counting model. The mean absolute error value MAE can be calculated by the following formula:

[0121] (1);

[0122] In formula (1), N represents the actual total number of cells in the test sub-image, Indicates the first The actual location of the cell center of each cell, Indicates the first The cell center detection position of each cell.

[0123] Among them, the embodiment of the present application can use the mean square error value MSE to evaluate the effect of the cell counting model, and the mean square error value MSE can be calculated by the following formula:

[0124] (2);

[0125] In formula (2), N represents the actual total number of cells in the test sub-image, Indicates the first The actual location of the cell center of each cell, Indicates the first The cell center detection position of each cell.

[0126] The process of generating a density map first requires obtaining the cell center annotation position represented by a single pixel point. However, a single pixel point cannot reasonably represent the space and influence range of a cell in the image. In order to more reasonably represent the space and influence range of each cell in the image, the grayscale value of the pixel point at the annotation position of each cell center can be diffused to other pixels within a certain range around it. Accordingly, in some embodiments, according to each of the training sub-images and their annotation results, a reference density map is generated for each of the training sub-images (i.e., step S330), such as Figure 6 As shown, it may include:

[0127] In step S610, for each pixel point at the marked position of the cell center, the grayscale values ​​of other pixels within a preset fuzzy radius of the pixel point are updated according to a preset function and the grayscale value of the pixel point;

[0128] In step S620, for any of the training sub-images, a reference density map is generated for the training sub-image according to the updated grayscale value of each pixel in the training sub-image.

[0129] The preset function may include a Gaussian kernel function.

[0130] Among them, the embodiment of the present application does not make any specific limitation on the preset blur radius. For example, it can be set according to the actual size of the cell or the center position distance of adjacent cells. For larger cells or adjacent cells with a larger center position distance, a larger blur radius can be set. This can more accurately reflect the actual size and spatial characteristics of each cell.

[0131] For each pixel at the marked position of the center of the cell, a Gaussian kernel function is applied with this point as the center, and the grayscale value of the pixel is diffused to other pixels within the preset fuzzy radius, and the grayscale values ​​of other pixels are updated to generate an area that conforms to the normal distribution. After all updates are completed, a reference density map is generated. In the reference density map, the grayscale value of each pixel represents the cell density at that position, and the larger the grayscale value, the denser the cells at that position. The density map generated in this way can more realistically reflect the spatial distribution and number of cells in the image, thereby improving the accuracy and effectiveness of the cell counting model.

[0132] The applicant of the present application further proposes to design a cell counting model including a cascaded backbone network, a pyramid scale network and an output layer to efficiently process various features in the image and provide accurate cell counting results. Accordingly, in some embodiments, each of the models includes a cascaded backbone network, a pyramid scale network and an output layer; the backbone network includes ten convolutional layers and three pooling layers; the pyramid scale network includes a global attention module and a feature pyramid module in parallel; the output layer includes three cascaded second convolution filters and one first convolution filter;

[0133] The global attention module includes a cascaded global pooling layer and two first convolution filters;

[0134] The feature pyramid module includes a first unit, a second unit, a third unit and a second convolution filter, and the second convolution filter is used to fuse the connection features between the output features of the first unit, the output features of the second unit and the output features of the third unit;

[0135] The first unit includes one second convolution filter; the second unit includes one third convolution filter and one second convolution filter in cascade, and the second convolution filter in the second unit is used to fuse the output features of the first unit with the output features of the third convolution filter in the second unit; the third unit includes one fourth convolution filter and one second convolution filter in cascade, and the second convolution filter in the third unit is used to fuse the output features of the second unit with the output features of the fourth convolution filter in the third unit.

[0136] It can be understood that the initial model and the cell counting model have the same model structure, and the cell counting model is obtained after optimizing the parameters and hyperparameters of the initial model.

[0137] like Figure 7 Shown is a schematic diagram of an implementation of the initial model and cell counting model provided in the examples of the present application.

[0138] Among them, the backbone network layer, as a feature extraction layer, is designed based on VGG-16, a deep convolutional neural network widely used in the field of computer vision, which is particularly good at extracting low-level features such as textures and edges. Based on the consideration of the trade-off between resource cost and model accuracy, the first ten convolutional layers and three pooling layers of VGG-16 are retained as the backbone network, which can take advantage of VGG-16's advantages in extracting fine-grained features, and by reducing the number of layers and pooling layers, the computational complexity and resource consumption can be effectively reduced.

[0139] With this design, the spatial width and height of the feature maps generated by the backbone network are reduced by 8 times relative to the input image. This means that after the input image is processed, the generated feature maps are significantly compressed in the spatial dimension to reduce the amount of computation and extract deep semantic features, but still retain important feature information. It is worth noting that each subsequent processing step will not change the resolution of these feature maps. This strategy of keeping the resolution unchanged ensures that key feature details in the image will not be lost during subsequent processing.

[0140] The main problem that the cell counting model needs to solve is that the cell distribution usually presents an uneven density, and some features of non-cancerous cells and cancer cells are similar. Therefore, multi-scale features are crucial for distinguishing target cells from background, cancer cells from non-cancerous cells. Therefore, the pyramid scale network is used to apply information transfer mechanism and attention mechanism to effectively extract high-level multi-scale features. The pyramid scale network includes a global attention module and a feature pyramid module.

[0141] Among them, the global attention module includes a cascaded global pooling layer and 2 first convolution filters. The first convolution filter refers to a 1x1 convolution filter. The global attention module provides global contextual attention as a guide for the multi-scale features generated by the feature pyramid module to weigh the importance of these features. Specifically, for a given input feature, the global attention module performs global average pooling on the input feature and generates a channel vector that soft-encodes the global context in each channel. In order to reduce parameter overhead, the global attention module first uses a 1×1 convolution filter to reduce the number of channels, and then estimates the attention across channels with another 1×1 convolution filter, and finally obtains the global contextual attention feature.

[0142] The feature pyramid module includes a first unit, a second unit and a third unit connected in parallel, and the first unit, the second unit and the third unit are all connected to a second convolution filter, that is, a 3x3 convolution filter.

[0143] The first unit, the second unit, and the third unit are equivalent to three branches. In order to construct a feature pyramid with diverse scales, we use 3×3 convolution filters, 5×5 convolution filters, and 7×7 convolution filters in the first unit, the second unit, and the third unit, respectively. Therefore, the first unit, the second unit, and the third unit can also be called: 3×3 branch, 5×5 branch, 7×7 branch, respectively. In addition, it is also possible to consider using the corresponding dilated convolution filters to replace these convolution filters, because the dilated convolution filters can achieve similar effects by expanding the receptive field without increasing the parameters and the amount of calculation. Finally, the information of different scales is gradually integrated through the message passing mechanism. That is, the connected output features of the lower branch and the adjacent upper-level branch are fused together using another 3×3 convolution filter as the final output feature of the lower branch. Finally, the first unit includes 1 second convolution filter (i.e., 3×3 convolution filter), the second unit includes 1 third convolution filter (i.e., 5×5 convolution filter) and 1 second convolution filter (i.e., 3×3 convolution filter) in cascade, and the third unit includes 1 fourth convolution filter (i.e., 7×7 convolution filter) and 1 second convolution filter (i.e., 3×3 convolution filter) in cascade. The second convolution filter (i.e., 3×3 convolution filter) in the second unit is used to fuse the output features of the first unit with the output features of the third convolution filter (i.e., 5×5 convolution filter) in the second unit, and the second convolution filter (i.e., 3×3 convolution filter) in the third unit is used to fuse the output features of the second unit with the output features of the fourth convolution filter (i.e., 7×7 convolution filter) in the third unit.

[0144] After the first unit, the second unit, and the third unit, the feature pyramid module uses a second convolution filter (i.e., a 3×3 convolution filter) to fuse the connection features between the output features of the first unit, the output features of the second unit, and the output features of the third unit.

[0145] Finally, the multi-scale features output by the feature pyramid module are multiplied by the global context attention features output by the global attention module and enter the output layer. The output layer includes 3 cascaded second convolution filters (i.e., 3×3 convolution filters) and 1 first convolution filter (i.e., 1×1 convolution filter). For each input image, the output layer outputs a cancer cell density map and a non-cancerous cell density map.

[0146] It is understandable that the structure of the neural network model is usually very complex, and the internal structure of the model involved in the embodiment of the present application is only the most critical part. In addition, some other conventional structures may also be involved in the model. For example, in order to follow the setting of the global attention module, a first convolution filter, i.e., a 1×1 convolution filter, may be added at the beginning of each of the three branches, the first unit, the second unit, and the third unit, to reduce the number of channels of the input feature to 1 / 3 of its input feature.

[0147] As a third aspect of the embodiments of the present application, an electronic device is provided, wherein: Figure 8 As shown, the electronic device includes:

[0148] One or more processors 101;

[0149] The memory 102 stores one or more computer programs. When the one or more computer programs are executed by the one or more processors 101, the one or more processors 101 implement any of the following:

[0150] The first aspect of the embodiment of the present application provides a method for detecting tumor purity;

[0151] A second aspect of an embodiment of the present application provides a method for training a cell counting model.

[0152] The electronic device may further include one or more I / O interfaces 103 connected between the processor 101 and the memory 102 and configured to implement information interaction between the processor 101 and the memory 102 .

[0153] Among them, the processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU), etc.; the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read-write interface) is connected between the processor and the memory, and can realize information exchange between the processor and the memory, including but not limited to a data bus (Bus), etc.

[0154] In some embodiments, the processor 101 , the memory 102 , and the I / O interface 103 are connected to each other via a bus 104 , and further connected to other components of the computing device.

[0155] As a fourth aspect of the embodiment of the present application, Fig. 9 As shown, a computer readable medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, any of the following is implemented:

[0156] The first aspect of the embodiment of the present application provides a method for detecting tumor purity;

[0157] A second aspect of an embodiment of the present application provides a method for training a cell counting model.

[0158] It will be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. Accordingly, the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can implement the method of any of the above-mentioned embodiments when executed. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the embodiments of the present application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0159] The above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Those skilled in the art should understand that the present application includes but is not limited to the contents described in the drawings and the above specific embodiments. Any modification that does not deviate from the functional and structural principles of the present application will be included in the scope of the claims.

Claims

1. A method for detecting tumor purity, characterized in that: The method comprises: Performing cutting processing on the original pathological image to obtain multiple sub-images to be detected; Input the multiple sub-images to be detected into a pre-trained cell counting model to obtain a detection density map of each sub-image to be detected output by the cell counting model; wherein the detection density map includes a cancer cell density map and a non-cancerous cell density map; the cell counting model is obtained by optimizing the parameters and hyperparameters of the initial model according to the multiple training sub-images and their reference density maps, the reference density map is generated according to the training sub-images and their annotation results, and the annotation results include the correspondence between the cell center annotation position and the color information; Determining a tumor purity detection result according to the detection density map of each sub-image to be detected; The model includes a cascaded backbone network, a pyramid scale network and an output layer; the backbone network includes ten convolutional layers and three pooling layers; the pyramid scale network includes a global attention module and a feature pyramid module in parallel; the output layer includes three cascaded second convolutional filters and one first convolutional filter; The global attention module includes a cascaded global pooling layer and two first convolution filters; The feature pyramid module includes a first unit, a second unit, a third unit and a second convolution filter, and the second convolution filter is used to fuse the connection features between the output features of the first unit, the output features of the second unit and the output features of the third unit; The first unit includes one second convolution filter; the second unit includes one third convolution filter and one second convolution filter in cascade, and the second convolution filter in the second unit is used to fuse the output features of the first unit with the output features of the third convolution filter in the second unit; the third unit includes one fourth convolution filter and one second convolution filter in cascade, and the second convolution filter in the third unit is used to fuse the output features of the second unit with the output features of the fourth convolution filter in the third unit.

2. The method according to claim 1, characterized in that: Determining the tumor purity detection result according to the detection density map of each sub-image to be detected includes: For any of the sub-images to be detected, determining the number of cancer cells to be detected according to the cancer cell density map of the sub-image to be detected, and determining the number of non-cancer cells to be detected according to the non-cancer cell density map of the sub-image to be detected; The tumor purity detection result is determined according to the detection number of cancer cells and the detection number of non-cancerous cells corresponding to each of the sub-images to be detected.

3. A method for training a cell counting model, characterized in that: The method comprises: Preprocessing the sample pathological image to obtain multiple training sub-images; According to the preset one-to-one correspondence between the color information and the cell type, the labeling result of each training sub-image is obtained; wherein the labeling result includes the correspondence between the cell center labeling position and the color information, and the cell type includes cancer cells and non-cancerous cells; Generating a reference density map for each of the training sub-images according to the training sub-images and the annotation results thereof; Input the plurality of training sub-images into the initial model to obtain a detection density map of each of the training sub-images output by the initial model; wherein the density map includes a cancer cell density map and a non-cancerous cell density map; Optimizing the parameters and hyperparameters of the initial model according to the loss function between the reference density map and the detection density map of each training sub-image to obtain a fitted cell counting model; The model includes a cascaded backbone network, a pyramid scale network and an output layer; the backbone network includes ten convolutional layers and three pooling layers; the pyramid scale network includes a global attention module and a feature pyramid module in parallel; the output layer includes three cascaded second convolutional filters and one first convolutional filter; The global attention module includes a cascaded global pooling layer and two first convolution filters; The feature pyramid module includes a first unit, a second unit, a third unit and a second convolution filter, and the second convolution filter is used to fuse the connection features between the output features of the first unit, the output features of the second unit and the output features of the third unit; The first unit includes one second convolution filter; the second unit includes one third convolution filter and one second convolution filter in cascade, and the second convolution filter in the second unit is used to fuse the output features of the first unit with the output features of the third convolution filter in the second unit; the third unit includes one fourth convolution filter and one second convolution filter in cascade, and the second convolution filter in the third unit is used to fuse the output features of the second unit with the output features of the fourth convolution filter in the third unit.

4. The method according to claim 3, characterized in that The preprocessing of the sample pathological image to obtain a plurality of training sub-images includes: Performing segmentation processing on the sample pathological image to obtain a plurality of sub-pathological images; Performing grayscale conversion processing on the multiple sub-pathological images to obtain multiple grayscale sub-images; Binarizing the plurality of grayscale sub-images according to a preset grayscale threshold to obtain a plurality of binary sub-images; For any of the binarized sub-images, the average grayscale value of each pixel in the binarized sub-image is calculated, and when the average value is not lower than a preset average value threshold, the sub-pathological image corresponding to the binarized sub-image is determined as a training sub-image.

5. The method according to claim 3, characterized in that: The method further comprises: Performing cutting processing on the test pathological image to obtain multiple test sub-images; Inputting the plurality of test sub-images into the fitted cell counting model to obtain a detection density map of each of the test sub-images output by the fitted cell counting model; Determining the cell center detection position corresponding to each of the test sub-images according to the detection density map; Obtaining the actual total number of cells and the actual position of the cell center corresponding to each of the test sub-images; Determining error information of the fitted cell counting model according to the actual total number of cells, the actual position of the cell center and the detected position of the cell center corresponding to each of the test sub-images; When the error information is less than a preset threshold, the fitted cell counting model is used as a trained cell counting model.

6. The method according to claim 5, characterized in that The error information includes a mean absolute error value MAE and / or a mean square error value MSE.

7. The method according to any one of claims 4 to 6, characterized in that: Generating a reference density map for each of the training sub-images according to the training sub-images and the annotation results thereof includes: For each pixel point at the marked position of the cell center, the grayscale values ​​of other pixels within a preset fuzzy radius of the pixel point are updated according to a preset function and the grayscale value of the pixel point; For any of the training sub-images, a reference density map is generated for the training sub-image according to the grayscale value of each pixel in the training sub-image after the update.

8. An electronic device, characterized in that: The electronic device comprises: one or more processors; A memory having one or more computer programs stored thereon, wherein when the one or more computer programs are executed by the one or more processors, the one or more processors implement any of the following: The method for detecting tumor purity according to any one of claims 1-2; A method for training a cell counting model according to any one of claims 3 to 7.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, any of the following is achieved: The method for detecting tumor purity according to any one of claims 1-2; A method for training a cell counting model according to any one of claims 3 to 7.

Images