Method, device and medium for training cell classification model

Abstract

The application discloses a cell classification model training method and device, equipment and medium, and relates to the field of machine learning. The method comprises the following steps: acquiring a sample image and a cell label of the sample image, the sample image comprising at least two cells, and the cell label being used for representing the type of the cells in the sample image; performing data processing on the sample image through a cell classification model, and outputting a sample prediction heat map, the sample prediction heat map being used for predicting the type of the cells in the sample image; restoring the contour of each cell in the sample image through the contour of each nucleus in the sample image, and obtaining a cell segmentation map; generating a sample classification heat map according to the cell label and the cell segmentation map, the sample classification heat map being used for representing the type of the cells in the cell segmentation map; and training the cell classification model according to the loss between the sample prediction heat map and the sample classification heat map. The application can realize weak supervision learning and improve the model accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of machine learning, and in particular to a cell classification model training method and device, equipment and medium. BACKGROUND

[0002] Modern medicine has new research and understanding of tumors, and the means of treating malignant tumors are constantly improving. Immunotherapy is an important way to fight cancer, and the implementation of immunotherapy requires that the user's tumor cell and mononuclear inflammatory cell count meet the requirements.

[0003] Related technologies require a technician to pre-annotate cell types. When training a cell classification model, the sample image is first divided into sub-regions, and the images corresponding to the sub-regions are sequentially input into the cell classification model. The difference between the predicted results output by the cell classification model and the morphology and type of the cells is calculated, and the cell classification model is trained in stages according to the difference to obtain a trained cell classification model.

[0004] However, the amount of information provided by the cell annotation of the sample image is limited, resulting in low accuracy of related technologies. SUMMARY

[0005] The present application provides a cell classification model training method, device, equipment and medium, which can restore the outline of the cells, extract more information from the sample image, and make the classification result more accurate. The technical solution is as follows:

[0006] According to one aspect of the present application, a cell classification model training method is provided, which comprises:

[0007] Obtaining a sample image and cell annotation of the sample image, the sample image comprising at least two cells, and the cell annotation being used to indicate the type of cells in the sample image;

[0008] Performing data processing on the sample image by the cell classification model, and outputting a sample prediction heat map, the sample prediction heat map being used to predict the type of cells in the sample image;

[0009] Restoring the outline of each cell in the sample image by the outline of each nucleus in the sample image, and obtaining a cell segmentation map;

[0010] Generating a sample classification heat map according to the cell annotation and the cell segmentation map, the sample classification heat map being used to indicate the type of cells in the cell segmentation map;

[0011] Training the cell classification model according to the loss between the sample prediction heat map and the sample classification heat map.

[0012] According to an aspect of the present application, a device for training a cell classification model is provided, the device comprising:

[0013] a sample acquisition module configured to acquire a sample image and a cell label of the sample image, the sample image comprising at least two types of cells, and the cell label being used to represent a type of the cells in the sample image;

[0014] a data processing module configured to perform data processing on the sample image by using the cell classification model, and output a sample prediction heat map, the sample prediction heat map being used to predict the type of the cells in the sample image;

[0015] the data processing module is further configured to restore a contour of each cell in the sample image by using a contour of each nucleus in the sample image, and obtain a cell segmentation map;

[0016] the data processing module is further configured to generate a sample classification heat map according to the cell label and the cell segmentation map, the sample classification heat map being used to represent the type of the cells in the cell segmentation map;

[0017] a training module configured to train the cell classification model according to a loss between the sample prediction heat map and the sample classification heat map.

[0018] According to an aspect of the present application, a cell classification method is provided, the method being performed by a computer device running a cell classification model as described above, the method comprising:

[0019] acquiring an input image, the input image comprising at least two types of cells;

[0020] performing data processing on the input image by using the cell classification model, and outputting a prediction heat map, the prediction heat map being used to represent a probability that a cell belongs to a target cell type;

[0021] determining a type of each cell in the input image according to the prediction heat map.

[0022] According to an aspect of the present application, a cell classification device is provided, the device running a cell classification model as described above, the device comprising:

[0023] an image acquisition module configured to acquire an input image, the input image comprising at least two types of cells;

[0024] a model calling module configured to perform data processing on the input image by using the cell classification model, and output a prediction heat map, the prediction heat map being used to represent a probability that a cell belongs to a target cell type;

[0025] a prediction module configured to determine the type of each cell in the input image according to the prediction hotspot map.

[0026] According to another aspect of the present application, a computer device is provided, which comprises a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, at least one program, code set or instruction set being loaded and executed by the processor to implement the classification method of the cell classification model or the classification method according to the above aspect.

[0027] According to another aspect of the present application, a computer storage medium is provided, which stores at least one program code, the program code being loaded and executed by a processor to implement the classification method of the cell classification model or the classification method according to the above aspect.

[0028] According to another aspect of the present application, a computer program product or computer program is provided, which comprises computer instructions, the computer instructions being stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the classification method of the cell classification model or the classification method according to the above aspect.

[0029] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects:

[0030] When the cell classification model is trained, the cell label of the sample image is converted into a more complex cell segmentation map, the cell segmentation map comprises information related to the cell morphology, the sample classification hotspot map is obtained through the cell segmentation map, and the cell classification model is trained according to the sample classification hotspot map and the sample prediction hotspot map of the sample image. Since the embodiments of the present application restore the contour of each cell in the sample image through simple cell labeling when training the cell classification model, and further extract more information from the sample image for training the cell classification model, the accuracy of the classification result of the cell classification model can be effectively improved, and since the training of the cell classification model only needs to use the label of the cell type, without providing the label of the cell morphology, weak supervision learning can be realized, and the training efficiency is improved. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical scheme in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0032] Figure 1 This is a schematic diagram of the structure of a computer system provided in an exemplary embodiment of this application;

[0033] Figure 2 This is a schematic diagram of a training method for a cell classification model provided in an exemplary embodiment of this application;

[0034] Figure 3 This is a flowchart illustrating a training method for a cell classification model provided in an exemplary embodiment of this application;

[0035] Figure 4 This is a schematic diagram of a sample image provided in an exemplary embodiment of this application;

[0036] Figure 5 This is a schematic diagram of cell annotation provided in an exemplary embodiment of this application;

[0037] Figure 6 This is a flowchart illustrating a training method for a cell classification model provided in an exemplary embodiment of this application;

[0038] Figure 7 This is a schematic diagram of an expansion operator provided in an exemplary embodiment of this application;

[0039] Figure 8 This is a schematic diagram of an expansion operator provided in an exemplary embodiment of this application;

[0040] Figure 9 This is a schematic diagram of the field of image generation provided by an exemplary embodiment of this application;

[0041] Figure 10 This is a schematic diagram of the field of image generation provided by an exemplary embodiment of this application;

[0042] Figure 11 This is a schematic diagram of a cell classification method provided in an exemplary embodiment of this application;

[0043] Figure 12 This is a schematic flowchart of a cell classification method provided in an exemplary embodiment of this application;

[0044] Figure 13 This is a schematic diagram showing the comparison of classification results provided in an exemplary embodiment of this application;

[0045] Figure 14 This is a schematic diagram showing the comparison of classification results provided in an exemplary embodiment of this application;

[0046] Figure 15 This is a block diagram of a training apparatus for a cell classification model provided in an exemplary embodiment of this application;

[0047] Figure 16 is a block diagram of a cell classification device provided by an example embodiment of the present application;

[0048] Figure 17 is a structural schematic diagram of a computer device provided by an example embodiment of the present application. DETAILED DESCRIPTION

[0049] For the purpose, technical solutions and advantages of the present application to be clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0050] First, the terms involved in the embodiments of the present application are introduced:

[0051] Artificial Intelligence (AI): is to use digital computers or digital computer controlled machine simulation, extension and expansion of human intelligence, perception of the environment, acquisition of knowledge and use of knowledge to obtain the best results of theory, method, technology and application system. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0052] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software level technology. Artificial intelligence basic technology generally includes such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning and other several major directions.

[0053] Computer Vision (CV): Computer vision is a scientific discipline that works on how to make a machine "see". More specifically, it refers to the use of cameras and computers to replace the human eye in identifying and measuring objects, and further processing the images so that the computer processing becomes more suitable for human observation or transmission to an instrument for detection. As a scientific discipline, computer vision researches related theories and technologies, trying to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and other technologies. It also includes common face recognition, fingerprint recognition, and other biometric identification technologies.

[0054] Machine Learning (ML): Machine learning is a multidisciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is a subject that studies how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent. Its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0055] IHC (immunohistochemistry): is a pathological method that uses specific primary antibodies to label cellular proteins in tissue samples and uses detection reagents to visualize the target. Protein expression can be evaluated using IHC and detected by colorimetric or fluorescent detection.

[0056] PD-L1 (Programmed Death-Ligand 1): is a protein in the human body. It is a transmembrane protein related to immune system suppression.

[0057] Pembrolizumab (Keytruda): is a humanized PD-1 (Programmed Death-1) monoclonal antibody used for cancer immunotherapy. It is used to treat non-small cell lung cancer, urothelial carcinoma, esophageal squamous cell carcinoma, triple-negative breast cancer, and other indications.

[0058] CPS (Combined Positive Score, integrated positive score): is the judgment of indication user whether to adopt Pembrolizumab for immunotherapy PD-L1 detection interpretation index.

[0059]

[0060] If the CPS value calculated according to the formula is greater than 100, it is considered that CPS is equal to 100. CPS is used for PD-L1 detection and patient classification of indications such as head and neck squamous cell carcinoma, gastric or gastroesophageal junction adenocarcinoma, cervical cancer, uroepithelial carcinoma, esophageal squamous cell carcinoma and triple negative breast cancer.

[0061] With the research and progress of artificial intelligence technology, artificial intelligence technology is researched and applied in many fields, such as common intelligent medical treatment, intelligent home, intelligent wearable device, virtual assistant, intelligent sound box, intelligent marketing, unmanned driving, automatic driving, unmanned aerial vehicle, robot, intelligent medical treatment, intelligent customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0062] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the present application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the images and user information involved in the present application are obtained under full authorization.

[0063] Figure 1 The structure schematic diagram of the computer system provided by an example embodiment of the present application is shown. The computer system 100 includes a terminal 120 and a server 140.

[0064] The application program related to cell classification is installed on the terminal 120. The application program can be a small program in the app (application), can be a special application program, or can be a web client. For example, in the case where the user is a doctor, the user obtains the result of the cell classification of the patient from the terminal 120, and determines the patient's condition through the result. The terminal 120 is at least one of a smart phone, a tablet computer, an electronic book reader, an MP3 player, an MP4 player, a laptop computer and a desktop computer.

[0065] The terminal 120 is connected to the server 140 through a wireless network or a wired network.

[0066] The server 140 can be a stand-alone physical server, a server cluster composed of multiple physical servers, or a distributed system, and can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms. The server 140 is used to provide background services for the cell classification application, and sends video and cell classification related information to the terminal 120. Alternatively, the server 140 undertakes the main computing work, and the terminal 120 undertakes the secondary computing work; or the server 140 undertakes the secondary computing work, and the terminal 120 undertakes the main computing work; or the server 140 and the terminal 120 both use a distributed computing architecture for collaborative computing.

[0067] Figure 2 A schematic diagram of a training method of a cell classification model provided by an example embodiment of the present application is shown.

[0068] The sample image 201 is subjected to color deconvolution to obtain a sample channel image 202. Color deconvolution is used to separate the target color in the sample image 201. For example, the target color is at least one of brownish yellow, blue, and pink. The target color can be adjusted by a technician according to actual needs. The sample image 201 and the sample channel image 202 are input into the cell classification model 200, and the cell classification model 200 classifies the cells in the sample image 201 to obtain a sample prediction heat map 203. The sample prediction heat map 203 is used to predict the heat map of the type of cells in the sample image 201. For example, the cell classification model 200 is a U-Net (U-shaped network), and the cell classification model 200 includes a first feature encoding module and a first up-sampling decoding module. The cell classification model 200 performs up-sampling and down-sampling on the sample image 201 to obtain the sample prediction heat map 203. The cell classification model 200 can also be other types of classification models.

[0069] The sample image 201 is subjected to cell nucleus segmentation to determine the cell nucleus contour of each cell in the sample image 201, thereby generating a cell nucleus schematic diagram 204. In the following steps, the cell nucleus schematic diagram 204 is subjected to image post-processing to generate a cell segmentation diagram 205. The image post-processing is used to restore the contour of the cell according to the contour of the cell nucleus. The cell segmentation diagram 205 is classified with reference to the cell annotation 206 to obtain a sample classification heat map 207. The sample classification heat map 207 is used to represent the heat map of the type of cells in the sample image 201.

[0070] The first loss 208 between the sample prediction heat map 203 and the sample classification heat map 207 is calculated, and the second loss 209 between the sample prediction heat map 203, the cell annotation 206, and the sample classification heat map 207 is calculated, the first loss 208 is used to represent the difference between the sample prediction heat map and the sample classification heat map, and the second loss 209 is used to optimize and learn the cell classification model 200 according to the position and texture features of the cell nucleus (or chromosome) during the training process of the cell classification model 200. The cell classification model 200 is trained according to the first loss 208 and the second loss 209.

[0071] Figure 3 A flowchart of a training method of a cell classification model provided by an example embodiment of the present application is shown. The method can be executed by the computer system 100 shown, and the method comprises: Figure 1

[0072] Step 302: Obtain a sample image and a cell annotation of the sample image, the sample image comprising at least two types of cells, and the cell annotation being used to represent the type of the cells in the sample image.

[0073] Optionally, the sample image is an electronic image of a stained pathological section. For example, as shown in FIG. 4, the sample image 401 shows cells after staining. Figure 4 It should be noted that, since the cell nucleus contains most of the chromosomes in the cell, the part of the cell that is stained after the cell is stained is the cell nucleus.

[0074] Optionally, the sample image is an RGB color digital image. In an optional design, the physical size of a pixel of the sample image is not greater than 0.5 μm / pixel (micrometer / pixel), and the physical size of a pixel is used to represent the actual physical size of a single pixel.

[0075] Different types of cells will exhibit different morphologies. Since the pathological section itself is transparent, the pathological section needs to be stained to observe the morphology of the cells, and the staining agent can also produce a reflection with a specific type of cell or a specific type of substance, thereby presenting different colors. For example, DAB as a staining agent can make positive cells or substances present a brownish yellow color. Specifically, DAB can make the diseased cells present a brownish yellow color.

[0076] Optionally, the type of the cell comprises at least one of a positive tumor cell, a negative tumor cell, a positive mononuclear inflammatory cell, and a negative mononuclear inflammatory cell.

[0077] Optionally, the cell annotation is point annotation, that is, a point is used to represent a cell or a cell nucleus in the sample image. Optionally, the cell annotation is an annotation at the center point of the cell nucleus. For example, as shown in FIG. 4, the cell annotation 402 is located at the center point of the cell nucleus. Figure 4 ​and Figure 5 As shown in the cell labeling 501, a dot is used to replace the stained nucleus in the sample image 401.

[0078] Optionally, all cells in the sample image are labeled, and different colors are used to label different types of cells. For example, all pixels on the cell are labeled. For example, all pixels on the nucleus are labeled. For example, the center point of the cell is taken, and the center point is labeled.

[0079] Optionally, the cell label includes the horizontal coordinate value, the vertical coordinate value and the category label value of each dot.

[0080] Optionally, different colors are used to represent different types of cells. For example, red is used to represent positive tumor cells, green is used to represent negative tumor cells, yellow is used to represent positive mononuclear cells, and blue is used to represent negative mononuclear cells. Optionally, different shapes are used to represent different types of cells. For example, a rectangle is used to represent a positive tumor cell, and a circle is used to represent a negative tumor cell.

[0081] Step 304: performing data processing on the sample image by using the cell classification model, and outputting a sample prediction heat map, the sample prediction heat map being used for predicting the type of the cell in the sample image.

[0082] The cell classification model is a model for predicting the type of the cell. Optionally, the cell classification model belongs to a U-Net (U-shaped network), and the cell classification model includes a first feature encoding module and a first up-sampling decoding module. The first feature encoding module is used for performing image down-sampling on the sample image, and extracting an image feature from the sample image, the image feature being related to the classification of the type of the cell. The first up-sampling decoding module is used for restoring the size of the image down-sampling result to the size of the sample image. For example, the sample image is a 32x32 image, the first feature encoding module is used to perform image down-sampling on the sample image, and four 4x4 image features are obtained. The first up-sampling decoding module is called to perform image up-sampling on the four 4x4 image features, and four 32x32 image prediction results are obtained, wherein the image prediction result is used to represent the prediction result of a type of cell.

[0083] Optionally, the cell classification model is a model for target detection or a model for image segmentation. For example, the cell classification model can also be a Transformer model. The model type of the cell classification model is not limited in the embodiments of the present application.

[0084] Optionally, the sample prediction hotspot map is a hotspot map for predicting a cell type in the sample image, wherein the bth hotspot map is used to display the bth cell type, and a represents the number of cell types, and b is a positive integer less than a. For example, only cells of cell type A are displayed in the hotspot Figure 1 , and only cells of cell type B are displayed in the hotspot Figure 2 .

[0085] Optionally, a target color of the sample image is separated to obtain a sample channel image, the sample channel image being an image obtained by expressing the sample image through a target color; and the sample image and the sample channel image are processed by calling the cell classification model to output a sample prediction hotspot map. The target color is a color corresponding to a target cell type in the sample image. For example, when the target cell type is a positive tumor cell, the positive tumor cell will display brownish yellow after being dyed, and the target color refers to brownish yellow.

[0086] In a possible implementation, color decomposition is performed on a Hematoxylin-Eosin-DAB staining space, and the following can be obtained: a positive expression channel representing DAB (brownish yellow), a negative expression channel representing Hematoxylin (blue) re-staining, and an Eosin (pink) channel (not appearing in PD-L1 staining). The output result of the positive expression channel of DAB is taken as the sample channel image, and the sample image of the RGB channel and the sample channel image of the DAB channel are taken as the input of the cell classification model. Optionally, the sample image of the RGB channel and the sample image of the hue channel are taken as the input of the cell classification model. Alternatively, the sample image of the RGB channel and the sample image of the gray channel are taken as the input of the cell classification model.

[0087] The number of prediction hotspot maps is the same as the number of color channels. For example, after the input image is processed by using the RGB channel and the DAB channel, four groups of prediction hotspot maps are obtained.

[0088] Step 306: The contour of each cell in the sample image is restored through the contour of each nucleus in the sample image to obtain a cell segmentation map.

[0089] Optionally, the contour of each nucleus in the sample image is output by processing the sample image by using a nucleus segmentation model. For example, the nucleus segmentation model belongs to a U-Net neural network. The nucleus segmentation model includes a second feature encoding module and a second up-sampling decoding module.

[0090] Optionally, the contour of each cell in the sample image is restored through the contour of each nucleus in the sample image by using a morphological dilation operator.

[0091] The cell segmentation map refers to an image including cell contours corresponding to the sample image. For example, as shown in the figures, by comparing the figure and the figure, it can be obtained that the cell segmentation map is a prediction of the contours of each cell in the sample image.

[0092] Step 308: generating a sample classification hotspot map according to the cell annotation and the cell segmentation map, the sample classification hotspot map being used to represent the types of cells in the cell segmentation map.

[0093] Optionally, the sample classification hotspot map is a hotspot map used to represent a cell type in the sample image, and the cth hotspot map is used to display the cth cell type. For example, only the cells of cell type A are displayed in the hotspot Figure 3 , and only the cells of cell type B are displayed in the hotspot Figure 4 .

[0094] Step 310: training the cell classification model according to the loss between the sample prediction hotspot map and the sample classification hotspot map.

[0095] Optionally, the loss between the sample prediction hotspot map and the sample classification hotspot map includes a first loss and a second loss, the first loss being used to represent the difference between the sample prediction hotspot map and the sample classification hotspot map, and the second loss being used to optimize and learn the cell classification model according to the position and texture features presented by the cell nucleus (or chromosome) during the training process of the cell classification model. For example, the cell classification model is trained according to the first loss between the sample prediction hotspot map and the sample classification hotspot map; or the cell classification model is trained according to the second loss between the sample prediction hotspot map and the sample classification hotspot map; or the cell classification model is trained according to the first loss and the second loss between the sample prediction hotspot map and the sample classification hotspot map.

[0096] Optionally, the mean square error between the sample prediction hotspot map and the sample classification hotspot map is calculated to obtain the first loss.

[0097] Optionally, a second loss is calculated according to the sample prediction heat map and the sample classification heat map according to the cell type provided by the cell annotation. The second loss is used to represent a loss term of a dense conditional random field. A conditional random field (CRF) is a conditional probability distribution model of an output random variable given a set of input random variables, and the output random variable constitutes a Markov random field, which can be used for image annotation or analysis. The dense conditional random field is a kind of conditional random field. If the dense conditional random field is used to classify the pixel points in the image, the dense conditional random field will associate the target pixel point with other pixel points to obtain the type of the target pixel point, and the other pixel points refer to the pixel points in the image that are associated with the target pixel point. In the embodiments of the present application, the loss term of the dense conditional random field is taken as one of the loss functions of the cell classification model, and the cell classification model is trained.

[0098] For example, the loss term of the dense conditional random field (i.e. the second loss) is denoted as Then we have:

[0099]

[0100] where M is the total number of pixels of each channel image, is the prediction heat map corresponding to the cell category c, W c is the similarity measurement matrix of the category. W c In the dense conditional random field, it is a fully connected Gaussian, and calculating its gradient in the corresponding loss optimization will become a bilateral filtering problem. TC N represents positive tumor cells, TC P represents negative tumor cells, MIC N represents positive mononuclear inflammatory cells, MIC P represents negative mononuclear inflammatory cells.

[0101] Optionally, the cell classification model is trained by an error back propagation algorithm according to the loss between the sample prediction heat map and the sample classification heat map.

[0102] In summary, in the training of the cell classification model, the cell annotation of the sample image is converted into a more complex cell segmentation graph, which includes information related to the cell morphology. The sample classification heat map is obtained from the cell segmentation graph, and the cell classification model is trained according to the sample classification heat map and the sample prediction heat map of the sample image. Since the cell classification model is trained in this embodiment, the contours of each cell in the sample image are restored through simple cell annotation, and more information is extracted from the sample image for training the cell classification model. Not only can the accuracy of the classification result of the cell classification model be effectively improved, but also since the training of the cell classification model only needs to use the annotation of the cell type, and does not need to provide the annotation of the cell morphology, weak supervision learning can be realized, and the training efficiency is improved.

[0103] In the following embodiment, a process for restoring the contour of the cell according to the contour of the nucleus will be introduced. This embodiment can convert the cell annotation of the sample image into a stronger pseudo-label of cell segmentation, so that more information can be obtained from the sample image in the training of the cell classification model, which is beneficial to the training of the cell classification model, and the trained cell classification model can classify the cells more accurately.

[0104] Figure 6 A flowchart of a training method of a cell classification model provided by an example embodiment of the present application is shown. The method can be executed by the computer system 100 shown in the figure, and the method comprises the following steps. Figure 1 The method can be executed by the computer system 100 shown in the figure, and the method comprises the following steps.

[0105] Step 601: using a dilation operator to traverse the contour of the i th nucleus.

[0106] The dilation operator is used to dilate the occupied area of the i th nucleus, and i is a positive integer.

[0107] The shape of the dilation operator includes but is not limited to at least one of a circle, a rectangle, a triangle, a regular hexagon, and a regular octagon. The specific shape of the dilation operator can be modified by the technician according to the actual needs. For example, when processing the sample image A, a circular dilation operator is used, and when processing the sample image B, a rectangular dilation operator is used.

[0108] Optionally, the shape of the dilation operator is related to the traversal distance of the dilation operator, and the traversal distance is used to represent the distance moved by the dilation operator in the process of traversing the contour of the i th nucleus. For example, after the dilation operator moves 0.1 microns, the dilation operator is a circle with a radius of 5 microns, and after the dilation operator moves 0.2 microns, the dilation operator is a circle with a radius of 5.1 microns. For example, the size of the dilation operator is y = f(x), x represents the traversal distance of the dilation operator, and f(x) is a self-defined function.

[0109] Optionally, the shape of the dilation operator is related to a radius of curvature of a center point of the dilation operator, the radius of curvature referring to a radius of curvature of the center point of the dilation operator on the contour of the ith nucleus in a process that the dilation operator traverses the contour of the ith nucleus. For example, when the center point of the dilation operator corresponds to a radius of curvature of 4 microns, the dilation operator is a circle with a radius of 5 microns, and when the center point of the dilation operator corresponds to a radius of curvature of 3 microns, the dilation operator is a square with a side length of 4 microns. It should be noted that the shape of the dilation operator mentioned above can refer to at least one of a radius, a diameter, a side length, an area, and a shape of the dilation operator.

[0110] For example, as shown in FIG. 7, taking the dilation operator 702 as a circle for example, the center of the circle of the dilation operator 702 is placed on the contour of the nucleus 701, and the center is controlled to move on the contour of the nucleus 701 so as to make the dilation operator 702 traverse the contour of the nucleus 701. Figure 7

[0111] Step 602: determining an image field corresponding to the ith nucleus according to a coverage area of the dilation operator.

[0112] The image field of the ith nucleus is used to predict an area occupied by the ith cell in the sample image. The ith cell refers to a cell containing the ith nucleus. In the embodiments of the present application, only the case that one cell contains one nucleus is considered.

[0113] The coverage area of the dilation operator is used to represent a union of occupied areas of the dilation operator in a process that the dilation operator traverses. For example, as shown in FIG. 7, in a process that the dilation operator 702 traverses the nucleus 701, when the dilation operator 702 is at position A, the dilation operator 702 occupies a first area. When the dilation operator 702 is at position B, the dilation operator 702 occupies a second area, and a union of the first area and the second area is taken as the coverage area of the dilation operator. It should be noted that the movement of the dilation operator is a continuous process, and here, two discrete points are used to illustrate the traversal process of the dilation operator for clarity. Figure 8 Step 603: determining a contour of the ith cell according to the image field of the ith nucleus.

[0114] In actual operation, since cells in part of the sample image can present an aggregated distribution, for example, when the sample image is used to display cells in a tumor cell nest or in an immune cell nest, it is necessary to divide the adjacent cells that can overlap.

[0115]

[0116] ​​Optionally, the watershed algorithm is used to determine the contour of the i-th cell according to the image field of the i-th nucleus. Exemplarily, the method comprises the following sub-steps:

[0117] 1. The image field of the i-th cell is expanded to obtain a target region.

[0118] The target region is larger than the image field of the i-th cell. That is, the image field of the i-th cell is located in the target region.

[0119] Optionally, the method of expanding the image field of the i-th cell can be to set a determination box with the image field of the i-th cell, and the area occupied by the determination box is recorded as the target region. The size of the determination box can be set by the technician according to the actual needs. The method of expanding the image field of the i-th cell can be to use the morphological dilation operator described above.

[0120] 2. Determine the pixel points in the target region that belong to the gray value interval.

[0121] Optionally, the pixel points in the target region that belong to the target gray value are determined. The target gray value is a constant.

[0122] Optionally, the gray value interval is dynamically generated. For example, the gray value interval is generated according to the gray values of the pixel points located at the edge of the image field. If the gray values of the pixel points located at the edge of the image field are in the interval [2, 8], the subset of the interval is taken as the gray value interval, or the target gray value in the interval is taken as the gray value interval.

[0123] Optionally, the gray value interval can be set by the technician. For example, the gray value interval is directly set to [1, 10].

[0124] 3. Connect the pixel points that belong to the gray value interval to form a closed region.

[0125] Exemplarily, as shown in FIG. 9B, for the image field 901 of the i-th cell, the pixel points 902 in the target region that belong to the target gray value are determined (for the convenience of description, only part of the pixel points that belong to the target gray value are shown), and the pixel points 902 are connected using a smooth curve to obtain a closed region 903. Figure 9

[0126] 4. The closed region is determined as the image field of the i-th cell.

[0127] Step 604: Repeat the above three steps until the contours of the cells are determined, and obtain a cell segmentation map.

[0128] ​Since the above three steps only provide a profile of one cell, and the sample image comprises multiple cells, the above three steps need to be repeated to obtain the profiles of the respective cells, and then generate a cell segmentation map.

[0129] Step 605: According to the cell annotation, separate the cells of the n cell types from the cell segmentation map to generate n cell type maps.

[0130] The n cell type maps correspond to the n cell types one-to-one.

[0131] For example, the cell segmentation map comprises cells of cell types A, B, C, and D. The cells of the four cell types are separated from the cell segmentation map to obtain a cell type map comprising only cells of cell type A, a cell type map comprising only cells of cell type B, a cell type map comprising only cells of cell type C, and a cell type map comprising only cells of cell type D.

[0132] Step 606: For the jth cell type map in the n cell type maps, assign a first gray value to the pixel points located outside the cells in the jth cell type map.

[0133] The first gray value is a constant. Optionally, the first gray value is 0.

[0134] For example, as shown in Figure 10 FIG. 10B, the pixel point 1002 is a pixel point located outside the cell 1001, and the gray value of the pixel point 1002 is set to 0.

[0135] Step 607: Assign a dynamic gray value to the pixel points located inside the cells in the jth cell type map.

[0136] The dynamic gray value is positively correlated with the distance of the pixel point located inside the cell to the edge of the cell. Optionally, the distance of the pixel point to the edge of the cell refers to the shortest distance of the pixel point to the edge of the cell. Alternatively, the distance of the pixel point to the edge of the cell refers to the average distance of the pixel point to the edge of the cell.

[0137] For example, the pixel point 1003 and the pixel point 1004 are pixel points located inside the cell 1001. The distance of the pixel point 1003 to the edge of the cell is greater than the distance of the pixel point 1004 to the edge of the cell. Therefore, the gray value of the pixel point 1003 is 255, and the gray value of the pixel point 1004 is 250.

[0138] Step 608: Generate a jth sample classification hotspot map according to the gray values of the respective pixel points in the jth cell type map.

[0139] In the jth sample classification hotspot map, the pixel points located outside the cells have a hotspot of 0, and the farther the pixel points located inside the cells are from the edge of the cell, the higher the hotspot.

[0140] Step 609: repeat the above three steps until the sample classification hotspot map is obtained.

[0141] Since the above three steps only provide a sample classification hotspot of one cell type, and the cell includes at least two, the above three steps need to be repeated to obtain a sample classification hotspot map of each cell type.

[0142] In summary, in the training of the cell classification model, the cell annotation of the sample image is converted into a more complex cell segmentation map, the cell segmentation map includes information related to the cell morphology, the sample classification hotspot map is obtained through the cell segmentation map, and the cell classification model is trained according to the sample classification hotspot map and the sample prediction hotspot map of the sample image. Since the cell classification model is trained in the embodiment of the present application, the outline of each cell in the sample image is restored through simple cell annotation, and more information is extracted from the sample image for training the cell classification model, which can not only effectively improve the accuracy of the classification result of the cell classification model, but also realize weak supervision learning and improve the training efficiency.

[0143] Figure 11 A schematic diagram of a cell classification method provided by an example embodiment of the present application is shown.

[0144] The input image 1101 is color deconvolved to obtain a channel image 1102, wherein color deconvolution is used to separate the target color in the input image 1101. The input image 1101 and the channel image 1102 are input into the cell classification model 1100, and the cells in the input image 1101 are classified by the cell classification model 1100 to obtain a prediction hotspot map 1103. After image processing of the prediction hotspot map 1103, the positions and quantities of the cells in the input image 1101 are obtained. Wherein, the image processing here is used to count the positions and quantities of different types of cells.

[0145] For example, the positions and quantities of the cells include the positions and quantities of negative tumor cells, the positions and quantities of positive tumor cells, the positions and quantities of negative mononuclear inflammatory cells, and the positions and quantities of positive mononuclear inflammatory cells. It should be noted that the skilled person can determine more or less positions and quantities of cells according to actual needs. Specifically, after image processing of the prediction hotspot map 1103, a result image 1104 is obtained.

[0146] Figure 12 A flowchart of a cell classification method provided by an example embodiment of the present application is shown. The method is executed by the cell classification provided by the above-mentioned embodiment. The method can be executed by the terminal 120 or the server 140 shown in the figure, and the terminal 120 or the server 140 runs the cell classification model provided by the above-mentioned embodiment. The method includes: Figure 1 The terminal 120 or the server 140 shown in the figure or the terminal 120 or the server 140 runs the cell classification model provided by the above-mentioned embodiment. The method includes:

[0147] Step 1202: obtaining an input image, the input image including at least two types of cells.

[0148] Optionally, the input image is an electronic image of a stained pathological section.

[0149] Optionally, the input image is an RGB color digital image. In an optional design, the physical size of a pixel of the input image is not greater than 0.5 μm / pixel.

[0150] Step 1204: performing data processing on the input image by a cell classification model, and outputting a predicted hot spot map, the predicted hot spot map being used to represent the probability of a cell belonging to a target cell type.

[0151] The cell classification model is a model for predicting cell types. Optionally, the cell classification model belongs to U-Net, and the cell classification model includes a first feature encoding module and a first up-sampling decoding module. The first feature encoding module is used to perform image down-sampling on the input image, and extract image features from the input image, the image features being related to the classification of cell types.

[0152] Optionally, a target color of the input image is separated to obtain a channel image, the channel image being an image obtained by a representation channel of the target color; the cell classification model is called to perform data processing on the input image and the channel image, and output a predicted hot spot map. In a possible implementation, color decomposition is performed in a Hematoxylin-Eosin-DAB staining space, and the following can be obtained: a positive expression channel representing DAB (brownish yellow), a negative expression channel representing Hematoxylin (blue) re-staining, and a channel representing Eosin (pink) (not appearing in PD-L1 staining), and the output result of the positive expression channel of DAB is taken as the channel image.

[0153] The number of predicted hot spot maps is the same as the number of color channels. For example, after the input image is processed by an RGB channel and a DAB channel, four groups of predicted hot spot maps are obtained.

[0154] Exemplarily, the predicted hot spot map includes a group of hot spot maps. Each hot spot map respectively gives the probability of each pixel belonging to a cell type corresponding to the aforementioned hot spot map. Exemplarily, the first predicted hot spot map is used to represent a hot spot map of positive tumor cells, and the second predicted hot spot map is used to represent a hot spot map of negative tumor cells.

[0155] Step 1206: determining the types of cells in the input image according to the predicted hot spot map.

[0156] Optionally, the predicted hot map is a hot map for representing a cell type in the input image, and the cth hot map is used to display the cth cell type. For example, only cells of cell type A are displayed in the hot map Figure 3 , and only cells of cell type B are displayed in the hot map Figure 4 .

[0157] In the embodiments of the present application, after determining the types and quantities of cells in the input image, the quantities of cells of the n cell types in the predicted hot map can be counted, where n is a positive integer greater than 2; and a comprehensive positive score is calculated according to the quantity of cells of at least one of the n cell types. Let the comprehensive positive score be CPS, then:

[0158]

[0159] where N0 represents the quantity of negative tumor cells, N1 represents the quantity of positive tumor cells, and N3 represents the quantity of positive mononuclear inflammatory cells.

[0160] Optionally, the embodiments of the present application are applicable to staining methods of other PD-L1 clone numbers, for example, the detection method of PD-L1 can also be at least one of Ventana SP263, Ventana SP142, Dako 28-8, Cell Signaling Technology E1L3N, and WuXiDiagnostics WD160.

[0161] Optionally, the quantities of cells of different cell types are determined by local extreme points in different classification hot maps. For example, the method for counting the quantities of cells can include the following steps:

[0162] 1. Determine the local extreme points in the kth predicted hot map.

[0163] In the embodiments of the present application, since the predicted hot map is used to represent the probability that a cell belongs to a target cell type, the local extreme points in the predicted hot map are located at the edges of the cells.

[0164] 2. Obtain the center coordinates of the connected region formed by the local extreme points.

[0165] Optionally, the center coordinates of the connected region represent the coordinates of the center of the cell.

[0166] Optionally, the center coordinates of the connected region are determined according to the average value of the coordinates of the edges of the connected region.

[0167] 3. Determine the quantity of cells of the kth hot map according to the number of center coordinates.

[0168] Optionally, the center point of the center coordinates is assigned to the cell type corresponding to the kth hot map.

[0169] 4, repeat the above three steps until the number of cells of n cell types is obtained.

[0170] To sum up, the embodiment reduces the contours of each cell in the sample image through simple cell labeling, and further extracts more information from the sample image for training of the cell classification model, which can effectively improve the accuracy of the classification result of the cell classification model, and since the training of the cell classification model only needs to use the labeling of the cell type and does not need to provide the labeling of the cell morphology, weakly supervised learning can be realized and the training efficiency is improved.

[0171] Optionally, the cell classification model of the present application is applied to an AI pathology cloud platform: in a pathology department or medical center equipped with a digital pathology reading system, the cell classification model of the embodiment of the present application can be integrated into the reading interface as an AI algorithm plug-in. After the doctor opens the digital pathology slice, the classification result of the cell classification model of the present application can be presented on the digital pathology slice.

[0172] Optionally, the cell classification model of the present application is applied to an AI microscope: on a microscope equipped with a digital image acquisition module, the cell classification model of the present application can be implanted into the interpretation process of the pathologist under the microscope. The doctor presses the "start calculation" button or steps on the foot pedal, and the cell classification model classifies the current acquired (and also the doctor is currently viewing) field of view, and returns the image superimposed with the classification result to the microscope eyepiece optical path in real time, and the doctor sees the cell classification result in the eyepiece.

[0173] Figure 13 and Figure 14 The classification result comparison schematic diagram provided by an example embodiment of the present application is shown.

[0174] In Figure 13 , the interpretation image 1302 is the classification result of the cell classification of the original image 1301 by using the cell classification method provided in the above embodiment. In Figure 14 , the interpretation image 1402 is the classification result of the cell classification of the original image 1401. The positive TC (red dot), negative TC (green dot), positive MIC (yellow dot) and negative MIC (blue dot) are marked in the interpretation image 1302 and the interpretation image 1402 respectively. Among them, the classification results corresponding to the interpretation image 1302 and the interpretation image 1402 are accurate, so the cell classification model provided by the embodiment of the present application can well complete the cell classification task, and the classification effect is good.

[0175] The following is an apparatus embodiment of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the apparatus embodiments of the present application, please refer to the method embodiments of the present application.

[0176] Please refer to Figure 15 , which shows a block diagram of a training device of a cell classification model according to an embodiment of the present application. The above functions can be implemented by hardware, or by hardware executing corresponding software. The device 1500 includes:

[0177] A sample acquisition module 1501 is configured to acquire a sample image and cell labels of the sample image, the sample image including at least two cells, and the cell labels being used to represent types of cells in the sample image.

[0178] A data processing module 1502 is configured to perform data processing on the sample image by using the cell classification model, and output a sample prediction heat map, the sample prediction heat map being used to predict types of cells in the sample image.

[0179] The data processing module 1502 is further configured to restore contours of each cell in the sample image by using contours of each nucleus in the sample image, and obtain a cell segmentation map.

[0180] The data processing module 1502 is further configured to generate a sample classification heat map according to the cell labels and the cell segmentation map, the sample classification heat map being used to represent types of cells in the cell segmentation map.

[0181] A training module 1503 is configured to train the cell classification model according to a loss between the sample prediction heat map and the sample classification heat map.

[0182] In an optional design of the present application, the data processing module 1502 is further configured to traverse a contour of an i th nucleus by using a dilation operator, the dilation operator being used to dilate an occupied area of the i th nucleus, i being a positive integer; determine an image field corresponding to the i th nucleus according to a coverage area of the dilation operator, the image field of the i th nucleus being used to predict an area occupied by the i th cell in the sample image; determine a contour of the i th cell according to the image field of the i th nucleus; and repeat the above three steps until contours of the cells are determined, and the cell segmentation map is obtained.

[0183] In an optional design of the present application, the data processing module 1502 is further configured to expand the image field of the i th cell to obtain a target area; determine pixel points belonging to a gray value interval in the target area; connect the pixel points belonging to the gray value interval to form a closed area; and determine the closed area as the image field of the i th cell.

[0184] In an optional design of the present application, the data processing module 1502 is further configured to separate cells of n types of cells from the cell segmentation map according to the cell label, to generate n cell type maps, the n cell type maps corresponding to the n types of cells one by one, n being a positive integer.

[0185] In an optional design of the present application, the data processing module 1502 is further configured to, for a jth cell type map in the n cell type maps, j being a positive integer less than n+1, assign a first gray value to a pixel point located outside a cell in the jth cell type map, assign a dynamic gray value to a pixel point located inside the cell in the jth cell type map, the dynamic gray value being positively correlated with a distance from the pixel point located inside the cell to a cell edge, and generate a jth sample classification hotspot map according to the gray values of the pixel points in the jth cell type map; and repeat the above three steps until the sample classification hotspot map is obtained.

[0186] In an optional design of the present application, the data processing module 1502 is further configured to separate a target color of the sample image to obtain a sample channel image, the sample channel image being an image of the sample image obtained through a representation channel of the target color; and call the cell classification model to perform data processing on the sample image and the sample channel image, and output the sample prediction hotspot map.

[0187] In an optional design of the present application, the training module 1503 is further configured to calculate a first loss between the sample prediction hotspot map and the sample classification hotspot map, calculate a second loss between the cell label, the sample prediction hotspot map and the sample classification hotspot map, and train the cell classification model according to the first loss and the second loss.

[0188] In an optional design of the present application, the training module 1503 is further configured to calculate a mean square error between the sample prediction hotspot map and the sample classification hotspot map to obtain the first loss.

[0189] In an optional design of the present application, the training module 1503 is further configured to calculate the second loss from the sample prediction hotspot map and the sample classification hotspot map according to a type of cell provided by the cell label, the second loss being used to represent a loss term of a dense conditional random field.

[0190] In summary, in the training of the cell classification model, the cell labeling of the sample image is converted into a more complex cell segmentation map, the cell segmentation map includes information related to the cell morphology, the sample classification heat map is obtained through the cell segmentation map, and the cell classification model is trained according to the sample classification heat map and the sample prediction heat map of the sample image. Since the cell classification model is trained in the embodiment of the present application, the contours of each cell in the sample image are restored through simple cell labeling, and more information is extracted from the sample image for training the cell classification model. Not only can the accuracy of the classification result of the cell classification model be effectively improved, but also since the training of the cell classification model only needs to use the labeling of the cell type, and does not need to provide the labeling of the cell morphology, weak supervision learning can be realized, and the training efficiency is improved.

[0191] Reference is made to Figure 16 which shows a block diagram of a cell classification device provided by an embodiment of the present application. The above functions can be realized by hardware, or by hardware executing corresponding software. The device 1600 runs a cell classification model provided by the above embodiments, and the device 1600 includes:

[0192] An image acquisition module 1601 is configured to acquire an input image, the input image including at least two types of cells;

[0193] A model calling module 1602 is configured to perform data processing on the input image through the cell classification model, and output a prediction heat map, the prediction heat map being used to represent the probability that a cell belongs to a target cell type;

[0194] A prediction module 1603 is configured to determine the type of each cell in the input image according to the prediction heat map.

[0195] In an optional design of the present application, the model calling module 1602 is further configured to separate the colors of the input image to obtain a channel image, the channel image being an image obtained by the input image through an expression channel of the target color; the input image and the image are processed through the cell classification model to output the prediction heat map.

[0196] In an optional design of the present application, the prediction module 1603 is further configured to count the number of cells of n types of cells in the prediction heat map, n being a positive integer greater than 2; and calculate a comprehensive positive score according to the number of cells of at least one type of cell in the n types of cells.

[0197] In an optional design of the present application, the prediction module 1603 is further configured to determine a local extreme point in the kth predicted hot spot map, obtain a center coordinate of a connected region formed by the local extreme point, determine the number of cells in the kth hot spot map according to the number of the center coordinates, and repeat the above three steps until the number of cells of the n types of cells is obtained.

[0198] In summary, the present embodiment restores the contours of each cell in the sample image through simple cell labeling, and further extracts more information from the sample image for training of the cell classification model, which can effectively improve the accuracy of the classification result of the cell classification model, and since the training of the cell classification model only needs to use the labeling of the cell types and does not need to provide the labeling of the cell morphology, weakly supervised learning can be implemented and the training efficiency is improved.

[0199] Figure 17 is a structural schematic diagram of a computer device according to an example embodiment. The computer device 1700 includes a central processing unit (CPU) 1701, a system memory 1704 including a random access memory (RAM) 1702 and a read-only memory (ROM) 1703, and a system bus 1705 connecting the system memory 1704 and the central processing unit 1701. The computer device 1700 further includes a basic input / output system (I / O) 1706 to help transfer information between various devices in the computer device, and a mass storage device 1707 for storing an operating system 1713, application programs 1714, and other program modules 1715.

[0200] The basic input / output system 1706 includes a display 1708 for displaying information and an input device 1709 such as a mouse, keyboard, etc. for user input. The display 1708 and the input device 1709 are both connected to the central processing unit 1701 through an input / output controller 1710 connected to the system bus 1705. The basic input / output system 1706 can also include an input / output controller 1710 for receiving and processing input from a keyboard, mouse, or electronic stylus, and other devices. Similarly, the input / output controller 1710 also provides output to a display screen, printer, or other types of output devices.

[0201] The mass storage device 1707 is connected to the central processing unit 1701 through a mass storage controller (not shown) connected to the system bus 1705. The mass storage device 1707 and its associated computer device readable medium provide nonvolatile storage for computer device 1700. This mass storage device 1707 can comprise a computer device readable medium (not shown), such as a hard drive or a compact disk read-only memory (CD-ROM) drive.

[0202] In general, the computer device readable medium can include computer device storage media and communication media. Computer device storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer device readable instructions, data structures, program modules or other data. Computer device storage media include RAM, ROM, Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other solid state memory, CD-ROM, digital video disc (DVD), or other optical disk drives, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. It should be understood by those skilled in the art that computer device storage media does not limit the computer device to any one of the aforementioned examples, but instead includes any computer device storage media which is now known or which becomes known in the future. The system memory 1704 and the mass storage device 1707 described above can be collectively referred to as memory.

[0203] According to various embodiments of the present disclosure, the computer device 1700 can also operate in the networked environment using logical connections to one or more remote computer devices, such as a host computer device 1708. The host computer device 1708 can be located in a remote business office, a branch office, or at home. The computer device 1700 can connect to the host computer device 1708 using a network interface unit 1712 connected to the system bus 1705 through a network 1711. The network interface unit 1712 can connect to the network 1711 through additional network connections made to the system bus 1705. The network 1711 can be implemented as an Ethernet or other types of network.

[0204] The memory further includes one or more programs stored in the memory that, when executed by the central processing unit 1701, implement the training method of the cell classification model, or all or part of the steps of the cell classification method.

[0205] In an example embodiment, a computer readable storage medium is also provided, and the computer readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by a processor to implement the training method of the cell classification model or the cell classification method provided by each of the above method embodiments.

[0206] The present application also provides a computer readable storage medium, and the storage medium stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the training method of the cell classification model or the cell classification method provided by the above method embodiments.

[0207] The present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device execute the training method of the cell classification model or the cell classification method provided by the above aspect embodiments.

[0208] The above-mentioned sequence numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0209] Those of ordinary skill in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by a program instructing relevant hardware, and the program can be stored in a computer readable storage medium, and the storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0210] The above-mentioned only describes optional embodiments of the present application, and does not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for training a cell classification model, characterized in that, The method includes: Obtain a sample image and cell annotations for the sample image, wherein the sample image includes at least two types of cells, and the cell annotations are used to indicate the types of cells in the sample image; The sample image is processed by the cell classification model to output a sample prediction heat map, which is used to predict the type of cells in the sample image. By reconstructing the outlines of each cell in the sample image using the outlines of each cell nucleus, a cell segmentation map is obtained. A sample classification heatmap is generated based on the cell annotations and the cell segmentation map, and the sample classification heatmap is used to represent the cell types in the cell segmentation map; The cell classification model is trained based on the loss between the predicted heatmap and the classified heatmap of the samples.

2. The method according to claim 1, characterized in that, The process of reconstructing the outline of each cell in the sample image from the outline of each cell nucleus to obtain a cell segmentation map includes: The outline of the i-th cell nucleus is traversed using an expansion operator, where i is a positive integer. The image region corresponding to the i-th cell nucleus is determined based on the coverage area of ​​the dilation operator, and the image region of the i-th cell nucleus is used to predict the region occupied by the i-th cell in the sample image; The outline of the i-th cell is determined based on the image region of the i-th cell nucleus; The process resumes by re-entering the step of traversing the outline of the i-th cell nucleus using the dilation operator, continuing until the outline of each cell is determined, thus obtaining the cell segmentation map.

3. The method according to claim 2, characterized in that, Determining the contour of the i-th cell based on the image region of the i-th cell includes: Expanding the image region of the i-th cell yields the target region; Identify the pixels within the target area that belong to the grayscale value range; Connect the pixels belonging to the grayscale value range to form a closed region; The enclosed region is defined as the image domain of the i-th cell.

4. The method according to any one of claims 1 to 3, characterized in that, The step of generating a sample classification heatmap based on the cell annotations and the cell segmentation map includes: Based on the cell annotations, n cell types are separated from the cell segmentation map, generating n cell type maps. The n cell type maps correspond one-to-one with the n cell types, where n is a positive integer. For the j-th cell type image among the n cell type images, assign a first gray value to the pixels located outside the cell in the j-th cell type image, where j is a positive integer less than n+1; A dynamic grayscale value is assigned to the pixel located inside the cell in the j-th cell type image, and the dynamic grayscale value is positively correlated with the distance from the pixel located inside the cell to the cell edge; Based on the grayscale values ​​of each pixel in the j-th cell type image, generate the j-th sample classification heatmap; The process of re-entering the j-th cell type image among the n cell type images and assigning a first gray value to the pixels located outside the cell in the j-th cell type image begins, until the sample classification heatmap is obtained.

5. The method according to any one of claims 1 to 3, characterized in that, The step of processing the sample images using the cell classification model to output a predicted heatmap includes: Separate the target color of the sample image to obtain a sample channel image, wherein the sample channel image is an image obtained by passing the target color expression channel through the sample image; The cell classification model is invoked to process the sample image and the sample channel image, and the predicted heatmap of the sample is output.

6. The method according to any one of claims 1 to 3, characterized in that, The step of training the cell classification model based on the loss between the predicted heatmap and the classified heatmap of the samples includes: Calculate the first loss between the predicted heatmap of the sample and the classified heatmap of the sample; Calculate the second loss among the cell annotations, the sample prediction heatmap, and the sample classification heatmap; The cell classification model is trained based on the first loss and the second loss.

7. The method according to claim 6, characterized in that, The calculation of the first loss between the predicted heatmap and the classified heatmap includes: The mean squared error between the predicted heatmap and the classified heatmap is calculated to obtain the first loss.

8. The method according to claim 6, characterized in that, The calculation of the second loss between the cell annotation, the sample prediction heatmap, and the sample classification heatmap includes: The second loss is calculated based on the cell type provided by the cell labeling, using the sample prediction heatmap and the sample classification heatmap. The second loss is used to represent the loss term of the dense conditional random field.

9. A cell classification method, characterized in that, The method is performed by a computer device running a cell classification model as described in any one of claims 1 to 8, and the method includes: Acquire an input image, the input image comprising at least two types of cells; The input image is processed by the cell classification model to output a predicted heat map, which is used to represent the probability that a cell belongs to the target cell type. Based on the predicted heatmap, the type of each cell in the input image is determined.

10. The method according to claim 9, characterized in that, The step of processing the input image using a cell classification model to output a predicted heatmap includes: The colors of the input image are separated to obtain a channel image, which is an image obtained by passing the target color expression channel through the input image; The cell classification model processes the input image and the image to output the predicted heatmap.

11. The method according to claim 9, characterized in that, The method further includes: Count the number of cells of n cell types in the predicted hotspot map, where n is a positive integer greater than 2; A comprehensive positive score is calculated based on the number of cells of at least one of the n cell types.

12. The method according to claim 11, characterized in that, The statistical counting of the number of cells of n cell types in the predicted hotspot map includes: Determine the local extremum point in the k-th predicted heatmap, where k is a positive integer less than n+1; Obtain the center coordinates of the connected region formed by the local extreme points; The number of cells in the k-th predicted hotspot map is determined based on the number of center coordinates; The process of re-entering the step of determining the local extremum point in the k-th predicted hotspot map is executed until the number of cells of the n cell types is obtained.

13. A training device for a cell classification model, characterized in that, The device includes: A sample acquisition module is used to acquire a sample image and cell annotations of the sample image, wherein the sample image includes at least two types of cells, and the cell annotations are used to indicate the types of cells in the sample image; The data processing module is used to process the sample image using the cell classification model and output a sample prediction heatmap, which is used to predict the cell type in the sample image. The data processing module is also used to reconstruct the outline of each cell in the sample image based on the outline of each cell nucleus in the sample image, and obtain a cell segmentation map; The data processing module is further configured to generate a sample classification heatmap based on the cell annotation and the cell segmentation map, wherein the sample classification heatmap is used to represent the cell type in the cell segmentation map; The training module is used to train the cell classification model based on the loss between the sample prediction heatmap and the sample classification heatmap.

14. A cell sorting device, characterized in that, The device operates a cell classification model as described in any one of claims 1 to 8, the device comprising: An image acquisition module is used to acquire an input image, the input image including at least two types of cells; The model calling module is used to process the input image through the cell classification model and output a predicted heat map, which is used to represent the probability that a cell belongs to the target cell type. The prediction module is used to determine the type of each cell in the input image based on the predicted heatmap.

15. A computer device, characterized in that, The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the training method of the cell classification model as described in any one of claims 1 to 8, or the cell classification method as described in any one of claims 9 to 12.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one piece of program code, which is loaded and executed by a processor to implement the training method of the cell classification model as described in any one of claims 1 to 8, or the cell classification method as described in any one of claims 9 to 12.

17. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the training method of the cell classification model according to any one of claims 1 to 8, or the cell classification method according to any one of claims 9 to 12.

Images