Cell classification method, system, cell graph Transformer model training method and electronic device

Abstract

The present invention relates to a cell classification method, system, cell graph transformer model training method and electronic equipment, belonging to the field of cell classification technology, including: acquiring a cell image; inputting the cell image into a cell segmentation model to obtain a cell segmentation map or inputting the cell detection model to obtain a cell detection map; inputting the cell image into the feature extractor to obtain a feature map; using the cell graph construction module to generate a cell map according to the feature map and the cell segmentation map or generating a cell map according to the feature map and the cell detection map; using the cell graph symbolization module to generate a symbol vector according to the cell map; using the transformer encoder to obtain a cell classification according to the symbol vector. The present invention makes full use of the powerful classification performance of the transformer encoder and the global topological relationship information of the cell nucleus of the cell image to improve the accuracy of cell classification.

Description

Technical Field

[0001] The present invention relates to the technical field of cell classification, and in particular to a cell classification method, system, cell graph transformer model training method and electronic equipment. Background Art

[0002] To date, many methods for classifying cells have been published, such as using convolutional neural networks, where convolution operations are networked to extract features of cell nuclei, and then classifying the features to predict the category of each cell; modeling the relationship between cell nuclei to generate a cell graph, and then using a graph neural network to learn the cell graph to capture the position and feature connections between cells, thereby achieving cell classification.

[0003] The method of using convolutional neural network for cell classification cannot capture global features well due to the limitation of receptive field in convolution operation, resulting in poor cell classification effect; the method of using graph neural network for cell classification based on cell graph has poor cell classification effect because the mode of information transmission of graph neural network is to start from neighbor nodes and pass to the outer layers through edge connections. This process is limited by the original structure of cell graph construction. In summary, the existing cell classification methods all have the defect of poor cell classification effect. Summary of the invention

[0004] In order to solve the defects of the prior art, the present invention provides a cell classification method, system, cell graph transformer model training method and electronic equipment.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] In a first aspect, an embodiment of the present invention provides a cell classification method, which is implemented based on a cell graph transformer model, wherein the cell graph transformer model includes a feature extractor, a cell graph construction module, a cell graph symbolization module, and a transformer encoder; the method includes the following steps:

[0007] Acquire cell images;

[0008] Inputting the cell image into a cell segmentation model to obtain a cell segmentation map or inputting the cell image into a cell detection model to obtain a cell detection map;

[0009] Inputting the cell image into the feature extractor to obtain a feature map;

[0010] Using the cell map construction module to generate a cell map according to the feature map and the cell segmentation map, or generating a cell map according to the feature map and the cell detection map;

[0011] generating a symbol vector according to the cell map using the cell map symbolization module;

[0012] The transformer encoder is used to obtain a cell classification according to the symbol vector.

[0013] In some embodiments, the generating a cell map according to the feature map and the cell segmentation map using the cell map construction module or generating a cell map according to the feature map and the cell detection map comprises:

[0014] Using the cell map construction module to calculate the cell nucleus centroid position information according to the cell segmentation map or the cell detection map;

[0015] The cell map construction module is used to generate a cell map according to the feature map and the cell nucleus centroid position information.

[0016] In a second aspect, an embodiment of the present invention provides a cell graph transformer model training method, comprising the following steps:

[0017] Acquire a training sample set, wherein the training sample set includes a cell image and a cell instance pixel label and a cell instance category label of the cell image;

[0018] The training sample set is used to train a cell graph transformer model, wherein the cell graph transformer model includes a feature extractor, a cell graph construction module, a cell graph symbolization module and a transformer encoder.

[0019] In some embodiments, training a cell graph transformer model using the training sample set comprises:

[0020] Using the training sample set to train the graph neural network model;

[0021] After the graph neural network model training converges, the parameters of the feature extractor of the graph neural network model are loaded into the feature extractor of the cell graph transformer model, and then the training sample set is used to train the cell graph transformer model.

[0022] In some embodiments, the loss function of training the graph neural network model using the training sample set is:

[0023] L GCN =L dice (P s ,Y s )+L CE (P c ,Y c )

[0024] Among them, P s is the pixel prediction for the cell instance, P c is the cell instance category prediction, Y s is the pixel label of the cell instance, Y c is the cell instance category label, L dice is the Dice loss, L CE is the cross entropy loss.

[0025] In some embodiments, the loss function of training the cell graph transformer model using the training sample set is:

[0026]

[0027] Among them, r is a hyperparameter, B is the number of cell instance categories, and Y b is the category label of the b-th cell instance, P b For the b-th cell instance category prediction, T b is the loss weight, which is the ratio of the total number of cell instances to the number of class b cell instances.

[0028] In a third aspect, an embodiment of the present invention provides a cell classification system, comprising:

[0029] A cell image acquisition module, used for acquiring the cell image to be classified;

[0030] A cell image processing module, used for inputting the cell image to be classified into a cell segmentation model to obtain a cell segmentation map or inputting it into a cell detection model to obtain a cell detection map;

[0031] A cell classification module is used to input the cell image to be classified and the cell segmentation map into a cell map transformer model or input the cell image to be classified and the cell detection map into a cell map transformer model to obtain cell classification, wherein the cell map transformer model is trained using the cell map transformer model training method described in any one of the second aspects.

[0032] In a fourth aspect, an embodiment of the present invention provides an electronic device, including: a memory and a processor;

[0033] The memory is configured to store computer program instructions;

[0034] The processor is configured to execute the computer program instructions so that the electronic device implements the cell graph transformer model training method as described in any one of the second aspects.

[0035] In a fifth aspect, an embodiment of the present invention provides a readable storage medium, including: computer program instructions;

[0036] The electronic device executes the computer program instructions so that the electronic device implements the cell graph transformer model training method as described in any one of the second aspects.

[0037] In a sixth aspect, an embodiment of the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the cell classification method as described in any one of the first aspects.

[0038] The present invention provides a cell classification method, system, cell graph transformer model training method and electronic device. Compared with the prior art, the present invention has the following technical effects:

[0039] 1. The present invention extracts a feature map of a cell image through a feature extractor, and then uses a cell map construction module to generate a cell map with global features of a cell nucleus topological relationship according to the feature map and the position information of the cell nucleus centroid, and then converts the cell map into a symbol vector that can be processed by a transformer encoder through a cell map symbolization module, and finally uses the transformer encoder to process the symbol vector to finally obtain a cell classification. That is, the present invention makes full use of the powerful classification performance of the transformer encoder and the global topological relationship information of the cell nucleus of the cell image, thereby improving the accuracy of cell classification.

[0040] 2. Make full use of the feature extractor's ability to perceive topological information, the cell graph construction module's ability to use the adjacency matrix to represent the connection relationship between cell nucleus nodes, and the transformer encoder's powerful cell classification ability to train a cell graph transformer model with high cell classification accuracy.

[0041] 3. By training the graph neural network model, a pre-trained feature extractor is realized, so that the feature extractor can learn the topological logical relationship in the cell image, and then the parameters of the feature extractor of the graph neural network model are loaded into the feature extractor of the cell graph transformer model, so that the two feature extractors have the same ability to perceive the topological logical relationship of the cell image, thereby reducing the noise during the training of the cell graph transformer model, making the training process more robust and easier to converge, reducing the number of data required for the training sample set, and making the final trained cell graph transformer model have higher cell classification accuracy.

[0042] 4. Make full use of the classification performance of the transformer encoder and the global topological relationship information of the cell nucleus of the cell image to improve the accuracy of cell classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description only involve some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 This is a flow chart of a cell classification method according to an embodiment of the present invention;

[0045] Figure 2 is a flowchart of step S4 of an embodiment of the present invention;

[0046] Figure 3 This is a flow chart of a cell graph transformer model training method according to an embodiment of the present invention;

[0047] Figure 4 This is a flow chart of another cell graph transformer model training method according to an embodiment of the present invention;

[0048] Figure 5 is a flow chart of another cell classification method according to an embodiment of the present invention;

[0049] Figure 6 This is a schematic diagram of the structure of a cell classification system according to an embodiment of the present invention;

[0050] Figure 7 The figure is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, other solutions obtained by ordinary technicians in this field without creative work should belong to the protection scope of the present invention.

[0052] In order to facilitate those skilled in the art to understand the embodiments of the present invention, the technical terms involved in the present invention are explained below.

[0053] A cell graph is a graph structure in which each node is represented by a cell nucleus, and adjacent cells are connected by edges to build channels for information transmission.

[0054] Example 1

[0055] Figure 1 A flow chart of a cell classification method according to an embodiment of the present invention is shown. The cell classification method is implemented based on a cell graph transformer model, which includes a feature extractor, a cell graph construction module, a cell graph symbolization module, and a transformer encoder; Figure 1 It can be seen that the cell classification method comprises the following steps:

[0056] S1. Acquire cell images.

[0057] There are many ways to obtain cell images, such as using a microscope or other tissue slice observation equipment to obtain cell images, or obtaining them through the Internet, or purchasing them from professional institutions.

[0058] S2. Input the cell image into a cell segmentation model to obtain a cell segmentation map or input the cell image into a cell detection model to obtain a cell detection map.

[0059] The cell image is processed using an existing cell segmentation model to obtain a cell segmentation map, or the cell image is processed using an existing cell detection model to obtain a cell detection map. In this embodiment, it is better to use a cell nucleus segmentation model to replace the cell segmentation model and to use a cell nucleus detection model to replace the cell detection model, such as Hover-net and MCSpatnet, which can further improve the accuracy of subsequent cell map construction and the accuracy of generated symbol vectors, thereby improving the accuracy of the cell classification method.

[0060] S3. Inputting the cell image into the feature extractor to obtain a feature map.

[0061] S4, using the cell map construction module to generate a cell map according to the feature map and the cell segmentation map, or to generate a cell map according to the feature map and the cell detection map. Figure 2 As shown, the specific implementation sub-steps of step S4 are as follows:

[0062] S41, using the cell map construction module to calculate the cell nucleus centroid position information according to the cell segmentation map or the cell detection map.

[0063] S42. Generate a cell map according to the feature map and the cell nucleus centroid position information using the cell map construction module.

[0064] Assuming that there are n cell nucleus centroid positions in the cell image, an undirected cell graph G = (V, E) is constructed by connecting each cell nucleus centroid with its nearest K cell nucleus centroids, where V = {v1, v2, ..., vn} are n node features obtained from the feature graph by bilinear interpolation according to the cell nucleus centroid position information, E = {e1, e2, ..., e D} represents D pairs of connected nodes, that is, D edges connecting nodes, and the connection between nodes is represented by an adjacency matrix A, where A ij Represents node v i and v j Using the above method, a cell map can be generated according to the feature map and the cell nucleus centroid position information.

[0065] S5. Generate a symbol vector according to the cell map using the cell map symbolization module.

[0066] Use connection mark M l ={m1, m2, ..., m n} is used to represent the adjacency relationship between n nodes. If two nodes m i , m j are connected by an edge, then [m i ,m j ][m i ,m i ] T =1 and [m i ,m j ][m j ,m j ] T =1, otherwise the dot product is zero.

[0067] The connection markers M are calculated by Laplace eigendecomposition of the adjacency matrix A of the cell graph. l .

[0068] The visual features of each node are calculated based on the feature graph, and the position features and the visual features of each edge are calculated through the position encoding function. v and edge symbol M e , the symbol vector of each node and the symbol vector of each edge can be calculated by the visual features, position features, visual features, connection marks and symbol marks of the nodes. The specific calculation method is as follows:

[0069]

[0070]

[0071] in, is the symbol vector of the ith node, is the visual feature of the i-th node, ρ i is the position feature of the i-th node, m iis the connection mark of the i-th node, M v are node symbol marks, σ1, σ2, σ3 are all linear operations, which convert the dimensions of their input values ​​into values ​​of the same dimension. is the sign vector of the d-th edge, is the visual feature of the d-th edge, M e is the edge symbol, m j is the connection label of the j-th node.

[0072] Finally, we will get n node symbol vectors and D edge symbol vectors, and then convert each symbol vector into a C-dimensional symbol vector through a linear operation. Finally, the cell graph symbolization module outputs n+D symbol vectors to the transformer encoder.

[0073] S6. Using the transformer encoder to obtain a cell classification according to the symbol vector.

[0074] Each layer of the transformer encoder consists of a set of multi-head self-attention layers and a feedforward network. In order to classify the categories of n nodes, this embodiment only selects the first n features from the output of the transformer encoder and feeds these features into the classification layer. The classification layer consists of a fully connected layer and a Softmax function.

[0075] This embodiment extracts a feature map of a cell image through a feature extractor, and then uses a cell map construction module to generate a cell map with global features of a cell nucleus topological relationship based on the feature map and the position information of the cell nucleus centroid. The cell map is then converted into a symbol vector that can be processed by a transformer encoder through a cell map symbolization module. Finally, the symbol vector is processed by the transformer encoder to ultimately obtain a cell classification. That is, this embodiment makes full use of the classification performance of the transformer encoder and the global topological relationship information of the cell nucleus of the cell image to improve the accuracy of cell classification.

[0076] Example 2

[0077] Figure 3 A flow chart of a cell graph transformer model training method according to an embodiment of the present invention is shown. Figure 3 It can be seen that the cell graph transformer model training method includes the following steps:

[0078] S20. Acquire a training sample set, wherein the training sample set includes a cell image and a cell instance pixel label and a cell instance category label of the cell image.

[0079] There are many ways to obtain cell images, such as using a microscope or other tissue slice observation equipment to obtain cell images, or obtaining them through the Internet, or purchasing them from professional institutions.

[0080] Use an existing cell segmentation model or cell nucleus segmentation model, such as Hover-net, EVA, to segment the cell image to obtain cell instance pixel labels, or ask an expert to annotate the cell instance pixels. At the same time, ask an expert to annotate the cell image with cell instance categories.

[0081] The above cell images and the cell instance pixel labels and cell instance category labels of the cell images are used as training sample sets to train subsequent models.

[0082] S26. Use the training sample set to train a cell graph transformer model, wherein the cell graph transformer model includes a feature extractor, a cell graph construction module, a cell graph symbolization module and a transformer encoder.

[0083] During training, cell images are input into the feature extractor to extract visual features containing topological perception information. For each cell image, adjacent cell nucleus nodes are connected together to construct a cell graph, and the connection relationship between cell nucleus nodes is represented by an adjacency matrix.

[0084] This embodiment makes full use of the ability of the feature extractor to perceive topological information, the ability of the cell graph construction module to use the adjacency matrix to represent the connection relationship between cell nucleus nodes, and the powerful cell classification ability of the transformer encoder, and trains a cell graph transformer model with high cell classification accuracy.

[0085] In order to further reduce the sample data required for training and the noise of the cell graph transformer model training method in the above embodiment, so that the training can converge better and the trained cell graph transformer model has higher cell classification accuracy, in another embodiment, the cell graph transformer model training method is used as follows: Figure 4 The cell graph transformer model training method shown in FIG. 1 includes the following steps:

[0086] S20. Acquire a training sample set, wherein the training sample set includes a cell image and a cell instance pixel label and a cell instance category label of the cell image.

[0087] There are many ways to obtain cell images, such as using a microscope or other tissue slice observation equipment to obtain cell images, or obtaining them through the Internet, or purchasing them from professional institutions.

[0088] Use an existing cell segmentation model or cell nucleus segmentation model, such as Hover-net, MCSpatnet, to segment the cell image to obtain cell instance pixel labels, or ask an expert to annotate the cell instance pixels. At the same time, ask an expert to annotate the cell image with cell instance categories.

[0089] The above cell images and the cell instance pixel labels and cell instance category labels of the cell images are used as training sample sets to train subsequent models.

[0090] S22. Use the training sample set to train a graph neural network model.

[0091] The graph neural network model can be a spectrum-based or space-based graph convolutional neural network model, or a graph attention neural network model. The feature extractor of the graph neural network model can be a convolution-based neural network, a transformer-based feature extractor, or a multi-layer perceptron.

[0092] During training, cell images are input into the feature extractor to extract visual features containing topological perception information. For each cell image, adjacent cell nucleus nodes are connected together to construct a cell graph, and the connection relationship between cell nucleus nodes is represented by an adjacency matrix.

[0093] The loss function of training the graph neural network model using the training sample set is:

[0094] L GCN =L dice (P s ,Y s )+L CE (P c ,Y c )

[0095] Among them, P s is the pixel prediction for the cell instance, P c is the cell instance category prediction, Y s is the pixel label of the cell instance, Y c is the cell instance category label, L dice is the Dice loss, L CE is the cross entropy loss.

[0096] This embodiment trains the graph neural network model to achieve feature extractor training. Through technical means similar to knowledge distillation, the trained feature extractor has the ability to perceive the topological relationship of the cell image, and then learns high-quality global features, reducing the noise during the subsequent cell graph Transformer model training, making the subsequent training process more robust. Preferably, the output feature map of the feature extractor will also input a semantic segmentation branch to output pixel-level predictions, so as to better train the feature extractor.

[0097] S24. After the graph neural network model training converges, the parameters of the feature extractor of the graph neural network model are loaded into the feature extractor of the cell graph transformer model.

[0098] The parameters of the feature extractor of the trained graph neural network model are loaded into the feature extractor of the cell graph Transformer model as initialization parameters.

[0099] In this embodiment, the structures of the two feature extractors are preferably the same, which is more conducive to model training and allows the trained cell map transformer model to have higher cell classification accuracy.

[0100] S26. Use the training sample set to train a cell graph transformer model, wherein the cell graph transformer model includes a feature extractor, a cell graph construction module, a cell graph symbolization module and a transformer encoder.

[0101] The loss function of the cell graph transformer model trained using the training sample set is:

[0102]

[0103] Where r is a hyperparameter, the value range is a positive integer, preferably [1, 5], and 2 is the best value, B is the number of cell instance categories, Y b is the category label of the b-th cell instance, P b is the category prediction of the b-th cell instance, T b is the loss weight, which is the ratio of the total number of cell instances to the number of class b cell instances.

[0104] During training, the feature extractor of the cell graph transformer model is trained end-to-end with the transformer encoder and the parameters are updated synchronously. The transformer encoder contains several standard transformer layers. After training, the parameter weights of the entire model are fixed for subsequent applications of the cell graph transformer model.

[0105] This embodiment pre-trains a feature extractor by training the graph neural network model, so that the feature extractor learns the topological logical relationship in the cell image, and then loads the parameters of the feature extractor of the graph neural network model into the feature extractor of the cell graph transformer model, so that the two feature extractors have the same ability to perceive the topological logical relationship of the cell image, thereby reducing the noise during the training of the cell graph transformer model, making the training process more robust and easier to converge, reducing the number of data required for the training sample set, and making the finally trained cell graph transformer model have higher cell classification accuracy.

[0106] Example 3

[0107] Based on the cell graph transformer model obtained by the cell graph transformer model training method in the above embodiment, this embodiment provides a cell classification method. Figure 5 Another cell classification method flow chart of the present invention is shown. Figure 5 It can be seen that the cell classification method includes the following steps:

[0108] S30, obtaining an image of cells to be classified.

[0109] There are many ways to obtain the images of cells to be classified, such as using a microscope or other tissue slice observation equipment to obtain the images of cells to be classified, or obtaining them through the Internet, or purchasing them from professional institutions.

[0110] S32, inputting the cell image to be classified into a cell segmentation model to obtain a cell segmentation map or inputting the cell image into a cell detection model to obtain a cell detection map.

[0111] The cell image to be classified is processed using an existing cell segmentation model to obtain a cell segmentation map, or the cell image to be classified is processed using an existing cell detection model to obtain a cell detection map. In this embodiment, it is better to use a cell nucleus segmentation model to replace the cell segmentation model and to use a cell nucleus detection model to replace the cell detection model, such as Hover-net and MCSpatnet, which can further improve the accuracy of subsequent cell map construction and the accuracy of generated symbol vectors, thereby improving the accuracy of the cell classification method.

[0112] S34. Input the cell image to be classified and the cell segmentation map into a cell map transformer model or input the cell image to be classified and the cell detection map into a cell map transformer model to obtain cell classification, wherein the cell map transformer model is trained by the cell map transformer model training method in the aforementioned embodiment.

[0113] This embodiment makes full use of the classification performance of the transformer encoder and the global topological relationship information of the cell nucleus of the cell image to improve the accuracy of cell classification.

[0114] Example 4

[0115] Figure 6 A schematic diagram of the structure of a cell classification system according to an embodiment of the present invention is shown. Figure 6 It is known that a cell classification system includes: a cell image acquisition module 42, a cell image processing module 44, and a cell classification module 46;

[0116] The cell image acquisition module 42 is used to acquire the cell image to be classified;

[0117] The cell image processing module 44 is used to input the cell image to be classified into a cell segmentation model to obtain a cell segmentation map or input it into a cell detection model to obtain a cell detection map;

[0118] The cell classification module 46 is used to input the cell image to be classified and the cell segmentation map into a cell map transformer model or input the cell image to be classified and the cell detection map into a cell map transformer model to obtain cell classification, wherein the cell map transformer model is trained using the cell map transformer model training method in the aforementioned embodiment.

[0119] This embodiment makes full use of the classification performance of the transformer encoder and the global topological relationship information of the cell nucleus of the cell image to improve the accuracy of cell classification.

[0120] Example 5

[0121] Figure 7 A schematic diagram of the structure of an electronic device according to an embodiment of the present invention is shown. Figure 7 It can be seen that an electronic device 50 includes: a memory 52 and a processor 54;

[0122] The memory 52 may be an independent physical unit, and may be connected to the processor 54 via a bus 56. The memory 52 and the processor 54 may also be integrated together and implemented via hardware.

[0123] The memory 52 is used to store program instructions, and the processor 54 calls the program instructions to execute the cell map transformer model training method provided by any of the above method embodiments.

[0124] Optionally, when part or all of the methods of the above embodiments are implemented by software, the above electronic device 50 may also only include a processor 54. The memory 52 for storing programs is located outside the electronic device 50, and the processor 54 is connected to the memory through circuits / wires to read and execute the programs stored in the memory.

[0125] The processor 54 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and a NP.

[0126] The processor 54 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0127] The memory 52 may include a volatile memory, such as a random-access memory (RAM); the memory may also include a non-volatile memory, such as a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); the memory may also include a combination of the above types of memory.

[0128] Example 6

[0129] An embodiment of the present invention provides a readable storage medium, comprising: computer program instructions, which, when executed by at least one processor of an electronic device, enable the electronic device to implement the cell graph transformer model training method provided in any of the above method embodiments.

[0130] Table 1 shows the test results of different cell classification methods on the PanNuke and CoNSeP datasets. The PanNuke dataset contains 7899 images of size 256*256 obtained from 19 organs, with a total of five cell instance category labels. The CoNSeP dataset contains 41 images of size 1000*1000, with a total of four cell instance category labels.

[0131] Among them, the present invention + Hover-net uses the Hover-net model to perform cell segmentation on the cell image, and then uses the cell graph transformer model to classify the cells; the present invention + EVA uses the EVA model to perform cell segmentation on the cell image, and then uses the cell graph transformer model to classify the cells; the present invention + manual labeling uses manual labeling to perform cell segmentation on the cell image, and then uses the cell graph transformer model to classify the cells; MCSPatnet, Mask2former, EVA, Hover-net, SONNET use the same model to perform cell segmentation on the cell image, and then classify the cells.

[0132] AJI and PQ are performance indicators of the cell segmentation model, and F d is the performance index of cell nucleus detection, F i , F c , F d , F e , F ne is the classification accuracy F-score of the cell instance category. avg is the average classification accuracy.

[0133] Table 1

[0134]

[0135] From the test results in the above table, it can be seen that the use of the cell graph transformer model of the present invention can significantly improve the accuracy of cell classification. The test results of the present invention + manual annotation show that the use of manual annotation to annotate the centroid position of the cell nucleus can further improve the accuracy of cell classification.

[0136] Table 2 shows the test results of cell classification on the CoNSep dataset using models trained using different model training methods.

[0137] The baseline model is an existing graph neural network model, and (1) the baseline model + ImageNet pre-training is to use the existing ImageNet pre-training method to train the baseline model to obtain a cell classification model, and then use the cell classification model to perform cell classification tests on cell images in the CoNSep dataset.

[0138] (2) Baseline model + topology-aware pre-training: The baseline model is trained using the pre-training method of the present invention to obtain a cell classification model, and the cell classification model is used to perform a cell classification test on the cell images in the CoNSep dataset.

[0139] (3) Baseline model + cell map Transformer: The cell map Transformer model is directly trained to obtain the cell map Transformer model, and the trained cell map Transformer model is used to perform cell classification tests on cell images in the CoNSep dataset.

[0140] (4) Baseline model + ImageNet pre-training + cell map transformer: The baseline model is trained using the existing ImageNet pre-training method to obtain a cell classification model. The parameters of the feature extractor of the trained cell classification model are then poured into the feature extractor of the cell map transformer model. The cell map transformer model is then trained again. Finally, the trained cell map transformer model is used to perform a cell classification test on the cell images in the CoNSep dataset.

[0141] (5) Baseline model + topology-aware pre-training + cell map transformer: the baseline model is trained using the pre-training method of the present invention to obtain a cell classification model, and then the parameters of the feature extractor of the trained cell classification model are poured into the feature extractor of the cell map transformer model, and then the cell map transformer model is trained again, and finally the trained cell map transformer model is used to perform a cell classification test on the cell images in the CoNSep dataset.

[0142] Table 2

[0143]

[0144] By comparing (1) and (2) in Table 2, it can be seen that the cell classification method using the pre-training method of the present invention has a higher accuracy in cell classification.

[0145] Comparing (3), (4) and (5), it can be seen that the use of the pre-training method of the present invention makes the model training more robust and improves the cell classification ability.

[0146] Comparing (2) and (5), it can be shown that the cell graph Transformer model proposed in the present invention can further surpass the cell classification method based on the graph neural network model and improve the accuracy of cell classification by learning global features.

[0147] Example 7

[0148] An embodiment of the present invention provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the cell classification method of the above embodiment is implemented.

[0149] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A cell classification method, characterized in that: The method is based on a cell graph transformer model, wherein the cell graph transformer model includes a feature extractor, a cell graph construction module, a cell graph symbolization module, and a transformer encoder; and the method includes the following steps: Acquire cell images; Inputting the cell image into a cell segmentation model to obtain a cell segmentation map or inputting the cell image into a cell detection model to obtain a cell detection map; Inputting the cell image into the feature extractor to obtain a feature map; Using the cell map construction module to generate a cell map according to the feature map and the cell segmentation map, or to generate a cell map according to the feature map and the cell detection map, wherein the cell map is a graph structure, each node in the graph is represented by a cell nucleus, and adjacent cells are connected by edges to construct a channel for information transmission; The generating of a cell map according to the feature map and the cell segmentation map by using the cell map construction module or generating a cell map according to the feature map and the cell detection map comprises: Using the cell map construction module to calculate the cell nucleus centroid position information according to the cell segmentation map or the cell detection map; Using the cell map construction module to generate a cell map according to the feature map and the cell nucleus centroid position information; Generating a cell map according to the feature map and the cell nucleus centroid position information comprises: Construct an undirected cell graph by connecting each cell nucleus centroid to its K nearest cell nucleus centroids; From the feature graph, node features are obtained by bilinear interpolation according to the cell nucleus centroid position information; The connections between the nodes of the undirected cell graph are represented by an adjacency matrix, and then the cell graph is generated by combining the node features; generating a symbol vector according to the cell map using the cell map symbolization module; The generating a symbol vector according to the cell map by using the cell map symbolization module comprises: The connection labels are calculated by Laplace eigendecomposition of the adjacency matrix of the cell graph; The visual features of each node are calculated according to the feature graph, the position features are calculated by the position encoding function, and the visual features of each edge, and the node symbol mark and the edge symbol mark are customized; The symbol vector of each node and the symbol vector of each edge are calculated based on the visual features, position features, visual features, connection marks and symbol marks of the nodes; The transformer encoder is used to obtain a cell classification according to the symbol vector.

2. A cell graph transformer model training method, characterized in that: The steps include: Acquire a training sample set, wherein the training sample set includes a cell image and a cell instance pixel label and a cell instance category label of the cell image; Using the training sample set to train a cell graph transformer model, wherein the cell graph transformer model comprises a feature extractor, a cell graph construction module, a cell graph symbolization module and a transformer encoder; The cell map construction module calculates the cell nucleus centroid position information according to the cell segmentation map or the cell detection map; Construct an undirected cell graph by connecting each cell nucleus centroid to its K nearest cell nucleus centroids; From the feature graph, node features are obtained by bilinear interpolation according to the centroid position information of the cell nucleus; The connection between the nodes of the undirected cell graph is represented by an adjacency matrix, and then the cell graph is generated by combining the node features, wherein the cell graph is a graph structure, each node in the graph is represented by a cell nucleus, and adjacent cells are connected by edges to construct a channel for information transmission; The cell graph symbolization module calculates connection labels by Laplace eigendecomposition of the adjacency matrix of the cell graph; The visual features of each node are calculated according to the feature graph, the position features are calculated by the position encoding function, and the visual features of each edge, and the node symbol mark and the edge symbol mark are customized; The symbol vector of each node and the symbol vector of each edge are calculated based on the visual features, position features of the nodes, the visual features of the edges, the connection labels and the symbol labels.

3. The cell graph transformer model training method according to claim 2, characterized in that: The step of using the training sample set to train the cell graph transformer model comprises: Using the training sample set to train the graph neural network model; After the graph neural network model training converges, the parameters of the feature extractor of the graph neural network model are loaded into the feature extractor of the cell graph transformer model, and then the training sample set is used to train the cell graph transformer model.

4. The cell graph transformer model training method according to claim 3, characterized in that: The loss function of training the graph neural network model using the training sample set is: L GCN =L dice (P s ,Y s )+L CE (P c ,Y c ) Among them, P s is the pixel prediction for the cell instance, P c is the cell instance category prediction, Y s is the pixel label of the cell instance, Y c is the cell instance category label, L dice is the Dice loss, L CE is the cross entropy loss.

5. The cell graph transformer model training method according to claim 3 or 2, characterized in that: The loss function of the cell graph transformer model trained using the training sample set is: Among them, r is a hyperparameter, B is the number of cell instance categories, and Y b is the category label of the b-th cell instance, P b is the category prediction of the b-th cell instance, T b is the loss weight, which is the ratio of the total number of cell instances to the number of class b cell instances.

6. A cell classification system, characterized in that: include: A cell image acquisition module, used for acquiring the cell image to be classified; A cell image processing module, used for inputting the cell image to be classified into a cell segmentation model to obtain a cell segmentation map or inputting it into a cell detection model to obtain a cell detection map; A cell classification module, used for inputting the cell image to be classified and the cell segmentation map into a cell map transformer model or inputting the cell image to be classified and the cell detection map into a cell map transformer model to obtain cell classification, wherein the cell map transformer model is trained using the cell map transformer model training method described in any one of claims 2-5.

7. An electronic device, characterized in that: include: Memory and processor; The memory is configured to store computer program instructions; The processor is configured to execute the computer program instructions so that the electronic device implements the cell graph transformer model training method as described in any one of claims 2-5.

8. A readable storage medium, characterized in that: include: Computer program instructions; The electronic device executes the computer program instructions so that the electronic device implements the cell graph transformer model training method as described in any one of claims 2-5.

Images