Method, device and equipment for classifying cells based on cell classification model

Abstract

The embodiments of the present application provide a method, an apparatus and a device for classifying cells based on a cell classification model. The method for classifying cells based on a cell classification model includes: obtaining a target cell image; inputting the target cell image into a first convolutional neural network for feature extraction, and outputting a plurality of first coarse classification features; inputting the plurality of first coarse classification features and the target cell image into a splicing layer for feature splicing, and outputting a splicing feature; inputting the splicing feature into a second convolutional neural network for feature extraction, and outputting a plurality of fine classification features; inputting the plurality of fine classification features into the first convolutional neural network for feature extraction, and outputting a plurality of second coarse classification features, wherein the plurality of second coarse classification features include a target coarse classification feature, and the target coarse classification feature is marked with a coarse classification result of the target cell image; the embodiments of the present application can solve the problem of low cell classification accuracy of the existing tumor cell classification model.

Description

Technical Field

[0001] The present application belongs to the field of deep learning, and in particular, relates to a method, device and equipment for classifying cells based on a cell classification model. Background Art

[0002] In my country, malignant tumors have long been the leading cause of death for urban and rural residents. The mortality rate of malignant tumors is at a relatively high level in the world and is showing a continuous growth trend. Pathological classification of cell samples is of great significance for cancer treatment.

[0003] In recent years, the development of artificial intelligence and deep learning has brought about rapid and revolutionary changes in the field of medical health. Artificial intelligence for disease diagnosis and pathological analysis has emerged in an endless stream, and deep learning has now been used in cell classification.

[0004] The existing tumor cell classification model usually uses a convolutional neural network (CNN) to train the classifier and only performs binary classification on the cell samples, resulting in low classification accuracy. Summary of the invention

[0005] The embodiments of the present application provide a method, device and equipment for classifying cells based on a cell classification model, which can solve the problem of low cell classification accuracy of existing tumor cell classification models.

[0006] In a first aspect, an embodiment of the present application provides a method for classifying cells based on a cell classification model, wherein the cell classification model includes a cascaded first convolutional neural network, a concatenated layer first convolutional neural network and a second convolutional neural network;

[0007] The method includes:

[0008] Acquire target cell images;

[0009] Inputting the target cell image into a first convolutional neural network for feature extraction, and outputting a plurality of first rough classification features;

[0010] Perform feature splicing on the multiple first coarse classification features and the target cell image input splicing layer, and output splicing features;

[0011] Inputting the spliced ​​features into a second convolutional neural network for feature extraction, and outputting a plurality of refined classification features, wherein the plurality of refined classification features include a target refined classification feature, and the target refined classification feature identifies a refined classification result of a target cell image;

[0012] The multiple fine classification features are input into the first convolutional neural network for feature extraction, and multiple second coarse classification features are output. The multiple second coarse classification features include a target coarse classification feature, and the target coarse classification feature identifies a coarse classification result of the target cell image.

[0013] Further, in one embodiment, the first convolutional neural network includes a plurality of cascaded convolutional layers;

[0014] The target cell image is input into the first convolutional neural network for feature extraction, and multiple first rough classification features are output, including:

[0015] Input the target cell image into the first convolution layer in the first convolutional neural network for feature extraction, and output the first coarse classification feature;

[0016] For multiple convolutional layers except the first convolutional layer in the first convolutional neural network, the following operations are performed:

[0017] The first coarse classification feature output by the previous convolution layer is input into the current convolution layer, and the first coarse classification feature is output, until the last convolution layer outputs the last first coarse classification feature, thereby obtaining multiple first coarse classification features.

[0018] Further, in one embodiment, the second convolutional neural network includes a plurality of cascaded convolutional layers;

[0019] The concatenated features are input into the second convolutional neural network for feature extraction, and multiple refined classification features are output, including:

[0020] Input the concatenated features into the first convolutional layer in the second convolutional neural network for feature extraction, and output the first refined classification feature;

[0021] For multiple convolutional layers except the first convolutional layer in the second convolutional neural network, the following operations are performed:

[0022] The refined classification features output by the previous convolutional layer are input into the current convolutional layer, and the refined classification features are output until the last convolutional layer outputs the last refined classification feature, thereby obtaining multiple refined classification features.

[0023] Furthermore, in one embodiment, the method further comprises:

[0024] The refined classification features output by the last convolutional layer of the second convolutional neural network are used as the target refined classification features.

[0025] Furthermore, in one embodiment, a plurality of fine classification features are input into a first convolutional neural network for feature extraction, and a plurality of second coarse classification features are output, including:

[0026] The multiple fine classification features are respectively input into multiple convolutional layers in the first convolutional neural network, and multiple second coarse classification features are output.

[0027] Furthermore, in one embodiment, the method further comprises:

[0028] The second coarse classification feature output by the last convolutional layer of the first convolutional neural network is used as the target coarse classification feature.

[0029] In a second aspect, an embodiment of the present application provides a method for training a cell classification model, the method comprising:

[0030] Obtaining a training sample set, the training sample set includes multiple training samples, each training sample includes an original cell image and a corresponding category label;

[0031] For each training sample, perform the following operations:

[0032] Inputting the original cell image into the first convolutional neural network of the target model for feature extraction, and outputting a plurality of first coarse classification training features;

[0033] Perform feature splicing on the splicing layer of the target model inputting the plurality of first coarse classification training features and the original cell image to obtain splicing training features;

[0034] Input the concatenated training features into the second convolutional neural network of the target model for feature extraction, and output a plurality of refined classification training features, wherein the plurality of refined classification training features include a target refined classification training feature, and the target refined classification training feature identifies a refined classification result of the training sample;

[0035] Inputting a plurality of fine classification training features into a first convolutional neural network of a target model for feature extraction, and outputting a plurality of second coarse classification training features, wherein the plurality of second coarse classification training features include a target coarse classification training feature, and the target coarse classification training feature identifies a coarse classification result of a training sample;

[0036] Calculate the cross entropy loss function of the target model based on the refined classification training features of each target, its corresponding category label, and the proportion of each category of cells to the total number of cells;

[0037] When the output value of the cross entropy loss function is not less than the preset threshold, the model parameters of the target model are adjusted according to each target fine classification training feature, target coarse classification training feature and their corresponding category labels; the first convolutional neural network that inputs the original cell image into the target model is returned to perform feature extraction until the output value of the cross entropy loss function is less than the preset threshold, thereby obtaining the cell classification model.

[0038] In a third aspect, an embodiment of the present application provides a device for classifying cells based on a cell classification model, wherein the cell classification model includes a cascaded first convolutional neural network, a concatenated layer of a second convolutional neural network;

[0039] The device includes:

[0040] An acquisition module, used for acquiring target cell images;

[0041] A first feature extraction module, used for inputting the target cell image into a first convolutional neural network for feature extraction, and outputting a plurality of first coarse classification features;

[0042] A feature splicing module, used for performing feature splicing on a plurality of first rough classification features and a target cell image input splicing layer, and outputting a splicing feature;

[0043] A second feature extraction module is used to input the spliced ​​features into a second convolutional neural network for feature extraction, and output a plurality of refined classification features, wherein the plurality of refined classification features include a target refined classification feature, and the target refined classification feature identifies a refined classification result of a target cell image;

[0044] The third feature extraction module is used to input multiple fine classification features into the first convolutional neural network for feature extraction, and output multiple second coarse classification features, wherein the multiple second coarse classification features include a target coarse classification feature, and the target coarse classification feature identifies the coarse classification result of the target cell image.

[0045] In a fourth aspect, an embodiment of the present application provides a device for classifying cells based on a cell classification model. The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, a method for classifying cells based on a cell classification model and a method for training a cell classification model are implemented.

[0046] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program for implementing information transmission is stored. When the program is executed by a processor, a method for classifying cells based on a cell classification model and a method for training a cell classification model are implemented.

[0047] The method, device and apparatus for classifying cells based on the cell classification model of the embodiment of the present application can obtain classification results of two accuracies by using two convolutional neural networks to classify the target cell image, and the output features of the first convolutional neural network are spliced ​​with the target cell image as the input of the second convolutional neural network, so that the second convolutional neural network can obtain more local features. The output of the second convolutional neural network is then returned to the first convolutional neural network for input, so that the first convolutional neural network can obtain more global features, the output of the first convolutional neural network is used as the rough classification result of the cell, and the output of the second convolutional neural network is used as the fine classification result, with high classification accuracy and accurate classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0049] Figure 1 It is a flowchart of a method for training a cell classification model provided in one embodiment of the present application;

[0050] Figure 2 It is a flowchart of a method for training a breast tumor cell classification model provided by an embodiment of the present application;

[0051] Figure 3 It is a schematic diagram of a flow chart of using a two-level convolutional neural network to train a breast tumor cell classification model provided by an embodiment of the present application;

[0052] Figure 4 This is a schematic diagram of classification results output by a breast tumor cell classification model provided in one embodiment of the present application;

[0053] Figure 5 It is a schematic flow chart of a method for classifying cells based on a cell classification model provided in one embodiment of the present application;

[0054] Figure 6 It is a schematic diagram of the structure of a training device for a cell classification model provided in one embodiment of the present application;

[0055] Figure 7 This is a schematic diagram of the structure of a device for classifying cells based on a cell classification model provided in one embodiment of the present application;

[0056] Figure 8 It is a schematic diagram of the structure of a device for classifying cells based on a cell classification model provided in one embodiment of the present application. DETAILED DESCRIPTION

[0057] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only configured to explain the present application and are not configured to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0058] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0059] In my country, malignant tumors have long been the leading cause of death for urban and rural residents. The mortality rate of malignant tumors is at a relatively high level in the world and is showing a continuous growth trend. Pathological classification of cell samples is of great significance for cancer treatment.

[0060] In recent years, the development of artificial intelligence and deep learning has brought about rapid and revolutionary changes in the field of medical health. Artificial intelligence for disease diagnosis and pathological analysis has emerged in an endless stream, and deep learning has now been used in cell classification.

[0061] The existing tumor cell classification model usually uses a single convolutional neural network (CNN) to train the classifier. In the problem of multi-classification of tumor cells, it has the following disadvantages: large intra-class differences, small inter-class differences, and non-mutual exclusivity between classes. At the same time, its loss function does not consider the balance of the number of samples in each class, resulting in low classification precision and accuracy.

[0062] In order to solve the problems of the prior art, the embodiments of the present application provide a method, device and equipment for classifying cells based on a cell classification model. It should be noted that in the method for classifying cells based on a cell classification model provided in the embodiments of the present application, it is necessary to use a pre-trained cell classification model to classify the target cell image. Therefore, before applying the cell classification model, it is necessary to complete the training of the cell classification model. Based on this, the specific implementation method of a cell classification model training method provided in the embodiments of the present application is first described in conjunction with the accompanying drawings.

[0063] Figure 1 FIG. 1 is a flow chart of a method for training a cell classification model provided by an embodiment of the present application. Figure 1 As shown, the method may include the following steps:

[0064] S100, obtaining a training sample set.

[0065] The training sample set includes multiple training samples, which can be obtained from the hospital's cell database. Each training sample includes an original cell image and a corresponding category label; the category label refers to the pathological type of the cell corresponding to the original cell image, such as benign, malignant, and normal; another example is fibroadenoma, phyllodes tumor, and tubular tumor.

[0066] For each training sample, perform the following operations:

[0067] S102, inputting the original cell image into the first convolutional neural network of the target model for feature extraction, and outputting a plurality of first coarse classification training features.

[0068] It should be noted that the first convolutional neural network of the present application includes multiple convolutional layers, so multiple first coarse classification training features will be generated when extracting features from the original cell image. Since the first coarse classification training features are obtained only based on the training samples themselves, their classification accuracy is poor, so they are not used as the final coarse classification results.

[0069] S104, performing feature splicing on the plurality of first coarse classification training features and the splicing layer of the cell original image input target model to obtain splicing training features.

[0070] Since the spliced ​​training features are generated based on the original cell image and the first coarse classification training features, they have sufficient global features and local features at the same time. Cell classification based on the spliced ​​training features has a higher classification result accuracy.

[0071] S106, input the concatenated training features into the second convolutional neural network of the target model for feature extraction, and output a plurality of refined classification training features.

[0072] The multiple refined classification training features include a target refined classification training feature, and the target refined classification training feature identifies the refined classification results of the training samples. It should be noted that the second convolutional neural network of the present application includes multiple convolutional layers, so multiple refined classification training features will be generated when feature extraction is performed on the spliced ​​training features, and the refined classification features output by the last convolutional layer in the multiple convolutional layers are used as the target refined classification training features.

[0073] S108, inputting the multiple fine classification training features into the first convolutional neural network of the target model for feature extraction, and outputting multiple second coarse classification training features.

[0074] Among them, the multiple second coarse classification training features include a target coarse classification training feature, and the target coarse classification training feature identifies the coarse classification result of the training sample. Since the second coarse classification training feature is extracted based on the fine classification training feature, and the fine classification training feature is extracted based on the splicing training feature that has sufficient global features and local features at the same time, the fine classification training feature also has sufficient global features and local features, and further, the second coarse classification training feature also has sufficient global features and local features, and can be used as the final coarse classification result.

[0075] S110, calculating the cross entropy loss function of the target model according to each target fine classification training feature, its corresponding category label and the ratio of the number of cells of each category to the total number of cells.

[0076] It should be noted that, since the proportion of each sample type to the total number of samples is different, when calculating the cross entropy loss function of the target model, the contribution of the categories of a large number of samples to the model should be suppressed, and the contribution of the categories of a small number of samples to the model should be enhanced, thereby improving the overall performance of the model. Based on this, the present application introduces the proportion of the number of cells of each type to the total number of cells to calculate the cross entropy loss function of the target model, which ensures the classification performance of the target model without increasing the number of samples of each type and the complexity of the network model.

[0077] S112, when the output value of the cross entropy loss function is not less than the preset threshold, adjust the model parameters of the target model according to each target fine classification training feature, target coarse classification training feature and their corresponding category labels; return to the first convolutional neural network that inputs the original cell image into the target model for feature extraction until the output value of the cross entropy loss function is less than the preset threshold, and obtain the cell classification model.

[0078] The training method of the cell classification model of the embodiment of the present application can obtain two classification results of precision by using two convolutional neural networks to classify the target cell image, and the output features of the first convolutional neural network are spliced ​​with the target cell image as the input of the second convolutional neural network, so that the second convolutional neural network can obtain more local features. Then the output of the second convolutional neural network is returned to the first convolutional neural network for input, so that the first convolutional neural network can obtain more global features, the output of the first convolutional neural network is used as the rough classification result of the cell, and the output of the second convolutional neural network is used as the fine classification result, with high classification precision and accurate classification.

[0079] At this point, the training of the cell classification model has been completed. To help understand, the following will take the training sample set of breast tumor cells as an example to further illustrate the training process of the breast tumor cell classification model. A two-level CNN network is used as the breast tumor cell classification model to classify the coarse and fine granularity of cells.

[0080] When training the CNN model, an improved loss function is used to improve the accuracy of tumor segmentation. The CNN network consists of at least one convolutional layer (conv), at least one global average pooling layer (avg pooling), at least one maximum pooling layer (max pooling), at least one fully connected layer (fc), batch normalization (batch normal) and rectified linear unit (relu). Figure 2 As shown in the figure, the secondary CNN network of breast tumor cells takes the original image and the output of the three-scale convolutional layer of the first-level CNN network as its input together with the original image block; in the second-level CNN network, the output of the three-scale convolutional layer of the network is back-propagated to the first-level CNN network, thus forming an end-to-end multi-cascade neural network. Finally, the trained secondary CNN model combination is constructed into a computer-aided diagnosis system. After reading the pathological sections of breast tumor cells to be diagnosed and inputting them into the system, the probability of the pathological sections belonging to adenosis, fibroadenoma, phyllodes tumor, tubular tumor, carcinoma, lobular carcinoma, mucinous carcinoma and papillary carcinoma is obtained. When establishing the network model, the cross entropy loss function is improved to suppress the contribution of the categories of a large number of samples to the model and enhance the contribution of the categories of a small number of samples to the model, thereby improving the overall performance of the model.

[0081] Figure 2 A flow chart of the training method of the breast tumor cell classification model is shown, as Figure 2 As shown, the method includes:

[0082] For a training cell sample (ie, a cell sample with a labeled cell type), a full-size pathological section corresponding to the training cell sample is obtained.

[0083] This application first processes the training cell samples by wax immersion and hematoxylin-eosin staining to obtain full-size pathological sections. The cell samples can be obtained from the hospital. The cell samples are labeled with the corresponding cell types, for example, whether the cells are normal cells or tumor cells. If they are tumor cells, the tumor cell type of the cells will also be marked accordingly. Hematoxylin stain is alkaline, which mainly makes the chromatin in the cell nucleus and the nucleic acid in the cytoplasm purple-blue; eosin is an acidic dye, which mainly makes the components of the cytoplasm and extracellular matrix red. The full-size pathological section contains a cell tissue part and a cell-free part. The cell-free part has no image for the classification of tumor cells, but it takes up a lot of space. Therefore, it is necessary to process the full-size pathological section to obtain an image with a cell tissue part.

[0084] The full-size pathological slices are processed by image preprocessing to obtain small image patches, which are training sample sets.

[0085] Specifically, the image preprocessing process includes:

[0086] The maximum inter-class variance method is used to extract the cell tissue part in the full-size pathological section, and then the cell tissue part is divided into a large number of 50×50 pixel image blocks, thus completing the acquisition of the training sample set. The training sample set refers to the labeled image blocks, and the training sample set can be input into the secondary CNN network for model training. After training, the neural network model can be obtained.

[0087] For the cell sample to be tested, the same processing process as the training cell sample is used to obtain a small image block, which is input into the neural network model trained above to obtain the breast tumor cell type corresponding to the cell sample to be tested.

[0088] Figure 3 A schematic diagram of the process of training a breast tumor cell classification model using a two-level convolutional neural network is shown, and the process includes:

[0089] The image blocks are input into two cascaded networks: the first-level CNN network (Inception-v3) and the second-level CNN network (ResNet50). The first-level Inception-v3 network is used to identify normal cells, benign cells, and malignant cells; the second-level ResNet50 network uses the original image and the output results of the three convolutional layers of the Inception-v3 network at different scales to be spliced ​​and input into the ResNet50 network. At the same time, the output results of the three convolutional layers of the ResNet50 network at different scales are input into the Inception-v3 network. Using the coarse classification results of the first-level Inception-v3 network, the coarse classification features learned by the first-level network are input into the second-level ResNet50 network to output the coarse classification results. At the same time, the feature map of the second-level ResNet50 network is reversely input into the first-level Inception-v3 network to output the fine classification results, such as adenosis, fibroadenoma, phyllodes tumor, tubular tumor, carcinoma, lobular carcinoma, mucinous carcinoma, and papillary carcinoma, etc., so that the first-level network can obtain more global features and the second-level network can obtain more local features.

[0090] Figure 4 A schematic diagram showing the classification results output by the breast tumor cell classification model.

[0091] At this point, the cross entropy loss function can be calculated. The calculation process is as follows:

[0092] based on Figure 3 The identified classification results, assuming that the percentage vector of the eight conditions of adenosis, fibroadenoma, phyllodes tumor, tubular tumor, carcinoma, lobular carcinoma, mucinous carcinoma and papillary carcinoma is P:

[0093] P=[p1,p2,p3,p4,p5,p6,p7,p8]

[0094] The weight vector is W:

[0095]

[0096] Improved cross entropy loss function:

[0097] Loss=-W T ·Label·P groundtruth ·(1-P pred ) r ·log(P pred ·P groundtruth )

[0098] Where Label represents the category label vector represented by one-hot encoding, P groundtruth Represents the true value of the sample, that is, the actual type corresponding to the cell sample, Ppred Represents the category prediction value output by the model, that is, the output coarse classification result and fine classification result. r is a hyperparameter and its value is greater than 1.

[0099] At this point, the training of the breast tumor cell classification model is completed.

[0100] The present application uses two convolutional neural networks to classify the target cell image, and can obtain two classification results of precision, and the output features of the first convolutional neural network are spliced ​​with the target cell image as the input of the second convolutional neural network, so that the second convolutional neural network can obtain more local features. The output of the second convolutional neural network is then returned to the first convolutional neural network for input, so that the first convolutional neural network can obtain more global features, and the output of the first convolutional neural network is used as the coarse classification result of the cell, and the output of the second convolutional neural network is used as the fine classification result, with high classification accuracy and accurate classification. The present application introduces the ratio of the number of each type of cells to the total number of cells to calculate the cross entropy loss function of the target model, which ensures the classification performance of the target model without increasing the number of samples of various types and the complexity of the network model.

[0101] The above is a specific implementation of the method for training a cell classification model provided in the embodiments of the present application. The cell classification model obtained through the above training can be applied to the method for classifying cells based on the cell classification model provided in the following embodiments.

[0102] The following is combined with Figure 5 The method for classifying cells based on the cell classification model provided in the present application is described in detail. The cell classification model includes a cascaded first convolutional neural network, a concatenated layer of a first convolutional neural network and a second convolutional neural network, wherein the first convolutional neural network and the second convolutional neural network both include a plurality of cascaded convolutional layers.

[0103] Figure 5 A schematic diagram of a process for classifying cells based on a cell classification model provided by an embodiment of the present application is shown, and the method includes:

[0104] S500, acquiring a target cell image.

[0105] That is, the image of the cell that needs to be classified.

[0106] S502, inputting the target cell image into a first convolutional neural network for feature extraction, and outputting a plurality of first coarse classification features.

[0107] It should be noted that the first convolutional neural network of the present application includes multiple convolutional layers, so multiple first rough classification features will be generated when extracting features from the original cell image. Since the first rough classification feature is obtained only based on the sample itself, its classification accuracy is poor, so it is not used as the final rough classification result.

[0108] In one embodiment, S502 may include:

[0109] The target cell image is input into the first convolution layer in the first convolutional neural network for feature extraction, and the first first coarse classification feature is output; for multiple convolutional layers except the first convolutional layer in the first convolutional neural network, the following operations are performed: the first coarse classification feature output by the previous convolutional layer is input into the current convolutional layer, and the first coarse classification feature is output, until the last convolutional layer outputs the last first coarse classification feature, and multiple first coarse classification features are obtained.

[0110] S504, performing feature splicing on the plurality of first coarse classification features and the target cell image input splicing layer, and outputting splicing features.

[0111] Since the splicing feature is generated based on the original cell image and the first rough classification feature, it has sufficient global features and local features at the same time. Cell classification based on the splicing feature has a higher classification result accuracy.

[0112] S506, input the spliced ​​features into the second convolutional neural network for feature extraction, and output a plurality of refined classification features, wherein the plurality of refined classification features include a target refined classification feature, and the target refined classification feature identifies a refined classification result of the target cell image.

[0113] It should be noted that the second convolutional neural network of the present application includes multiple convolutional layers, so multiple refined classification features will be generated when feature extraction is performed on the splicing features, and the refined classification features output by the last convolutional layer in the multiple convolutional layers can be used as target refined classification features.

[0114] In one embodiment, S506 may include:

[0115] The concatenated features are input into the first convolutional layer in the second convolutional neural network for feature extraction, and the first refined classification feature is output; for multiple convolutional layers except the first convolutional layer in the second convolutional neural network, the following operations are performed: the refined classification features output by the previous convolutional layer are input into the current convolutional layer, and the refined classification features are output, until the last convolutional layer outputs the last refined classification feature, thereby obtaining multiple refined classification features.

[0116] In one embodiment, the method may further include:

[0117] The refined classification features output by the last convolutional layer of the second convolutional neural network are used as the target refined classification features.

[0118] S508, inputting the multiple fine classification features into the first convolutional neural network for feature extraction, and outputting multiple second coarse classification features.

[0119] The plurality of second coarse classification features include a target coarse classification feature, and the target coarse classification feature identifies a coarse classification result of the target cell image.

[0120] Since the second coarse classification feature is extracted based on the fine classification feature, and the fine classification feature is extracted based on the splicing feature that has sufficient global features and local features, the fine classification feature also has sufficient global features and local features. Furthermore, the second coarse classification feature also has sufficient global features and local features, and it can be used as the final coarse classification result.

[0121] In one embodiment, the method may further include:

[0122] The second coarse classification feature output by the last convolutional layer of the first convolutional neural network is used as the target coarse classification feature.

[0123] This application uses two convolutional neural networks to classify target cell images, and can obtain classification results with two accuracies. In addition, the output features of the first convolutional neural network are spliced ​​with the target cell image as the input of the second convolutional neural network, so that the second convolutional neural network can obtain more local features. The output of the second convolutional neural network is then returned to the first convolutional neural network for input, so that the first convolutional neural network can obtain more global features, and the output of the first convolutional neural network is used as the rough classification result of the cell, and the output of the second convolutional neural network is used as the fine classification result, with high classification accuracy and accurate classification.

[0124] Figure 1-5 The training method of the cell classification model provided in the embodiment of the present application and the method of classifying cells based on the cell classification model are described below in conjunction with the attached Figure 6-8 Describe the device provided in the embodiment of the present application.

[0125] Figure 6 A schematic diagram of the structure of a training device for a cell classification model provided in one embodiment of the present application is shown. Figure 6 Each module in the device shown has the function of realizing Figure 1 The functions of each step in the process can achieve the corresponding technical effects. Figure 6 As shown, the device may include:

[0126] The training acquisition module 600 is used to acquire a training sample set.

[0127] The training sample set includes multiple training samples, which can be obtained from the hospital's cell database. Each training sample includes an original cell image and a corresponding category label; the category label refers to the pathological type of the cell corresponding to the original cell image, such as benign, malignant, and normal; another example is fibroadenoma, phyllodes tumor, and tubular tumor.

[0128] For each training sample, perform the following operations:

[0129] The first feature extraction module 602 for training is used to input the original cell image into the first convolutional neural network of the target model for feature extraction, and output a plurality of first coarse classification training features.

[0130] It should be noted that the first convolutional neural network of the present application includes multiple convolutional layers, so multiple first coarse classification training features will be generated when extracting features from the original cell image. Since the first coarse classification training features are obtained only based on the training samples themselves, their classification accuracy is poor, so they are not used as the final coarse classification results.

[0131] The training feature splicing module 604 is used to perform feature splicing on the splicing layer of the target model input with the multiple first coarse classification training features and the original cell image to obtain spliced ​​training features.

[0132] Since the spliced ​​training features are generated based on the original cell image and the first coarse classification training features, they have sufficient global features and local features at the same time. Cell classification based on the spliced ​​training features has a higher classification result accuracy.

[0133] The second feature extraction module 606 for training is used to input the spliced ​​training features into the second convolutional neural network of the target model for feature extraction, and output a plurality of refined classification training features.

[0134] The multiple refined classification training features include a target refined classification training feature, and the target refined classification training feature identifies the refined classification results of the training samples. It should be noted that the second convolutional neural network of the present application includes multiple convolutional layers, so multiple refined classification training features will be generated when feature extraction is performed on the spliced ​​training features, and the refined classification features output by the last convolutional layer in the multiple convolutional layers are used as the target refined classification training features.

[0135] The third feature extraction module 608 for training is used to input multiple fine classification training features into the first convolutional neural network of the target model for feature extraction, and output multiple second coarse classification training features.

[0136] Among them, the multiple second coarse classification training features include a target coarse classification training feature, and the target coarse classification training feature identifies the coarse classification result of the training sample. Since the second coarse classification training feature is extracted based on the fine classification training feature, and the fine classification training feature is extracted based on the splicing training feature that has sufficient global features and local features at the same time, the fine classification training feature also has sufficient global features and local features, and further, the second coarse classification training feature also has sufficient global features and local features, and can be used as the final coarse classification result.

[0137] The calculation module 610 is used to calculate the cross entropy loss function of the target model according to each target fine classification training feature, its corresponding category label and the proportion of each category of cells to the total number of cells.

[0138] It should be noted that, since the proportion of each sample type to the total number of samples is different, when calculating the cross entropy loss function of the target model, the contribution of the categories of a large number of samples to the model should be suppressed, and the contribution of the categories of a small number of samples to the model should be enhanced, thereby improving the overall performance of the model. Based on this, the present application introduces the proportion of the number of cells of each type to the total number of cells to calculate the cross entropy loss function of the target model, which ensures the classification performance of the target model without increasing the number of samples of each type and the complexity of the network model.

[0139] The model generation module 612 is used to adjust the model parameters of the target model according to each target fine classification training feature, target coarse classification training feature and their corresponding category labels when the output value of the cross entropy loss function is not less than the preset threshold; return to the first convolutional neural network that inputs the original cell image into the target model for feature extraction until the output value of the cross entropy loss function is less than the preset threshold, thereby obtaining the cell classification model.

[0140] The training device of the cell classification model of the embodiment of the present application can obtain two classification results of precision by using two convolutional neural networks to classify the target cell image, and the output features of the first convolutional neural network are spliced ​​with the target cell image as the input of the second convolutional neural network, so that the second convolutional neural network can obtain more local features. Then the output of the second convolutional neural network is returned to the first convolutional neural network for input, so that the first convolutional neural network can obtain more global features, the output of the first convolutional neural network is used as the rough classification result of the cell, and the output of the second convolutional neural network is used as the fine classification result, with high classification precision and accurate classification.

[0141] Figure 7 A schematic diagram of the structure of a device for classifying cells based on a cell classification model provided in one embodiment of the present application is shown. Figure 7 Each module in the device shown has the function of realizing Figure 5 The functions of each step in the process can achieve the corresponding technical effects. Figure 7 As shown, the device may include:

[0142] The acquisition module 700 is used to acquire the target cell image.

[0143] That is, the image of the cell that needs to be classified.

[0144] The first feature extraction module 702 inputs the target cell image into the first convolutional neural network for feature extraction, and outputs a plurality of first coarse classification features.

[0145] It should be noted that the first convolutional neural network of the present application includes multiple convolutional layers, so multiple first rough classification features will be generated when extracting features from the original cell image. Since the first rough classification feature is obtained only based on the sample itself, its classification accuracy is poor, so it is not used as the final rough classification result.

[0146] In one embodiment, the first feature extraction module 702 may include:

[0147] The first output unit is used to input the target cell image into the first convolution layer in the first convolutional neural network for feature extraction, and output a first first coarse classification feature.

[0148] The second output unit is used to perform the following operations on multiple convolutional layers except the first convolutional layer in the first convolutional neural network: input the first coarse classification feature output by the previous convolutional layer into the current convolutional layer, output the first coarse classification feature, until the last convolutional layer outputs the last first coarse classification feature, to obtain multiple first coarse classification features.

[0149] The feature stitching module 706 is used to stitch the plurality of first rough classification features with the target cell image input stitching layer and output the stitching features.

[0150] Since the splicing feature is generated based on the original cell image and the first rough classification feature, it has sufficient global features and local features at the same time. Cell classification based on the splicing feature has a higher classification result accuracy.

[0151] The second feature extraction module 706 is used to input the spliced ​​features into the second convolutional neural network for feature extraction, and output a plurality of refined classification features, wherein the plurality of refined classification features include a target refined classification feature, and the target refined classification feature identifies the refined classification result of the target cell image.

[0152] It should be noted that the second convolutional neural network of the present application includes multiple convolutional layers, so multiple refined classification features will be generated when feature extraction is performed on the splicing features, and the refined classification features output by the last convolutional layer in the multiple convolutional layers can be used as target refined classification features.

[0153] In one embodiment, the second feature extraction module 706 may include:

[0154] A third output unit is used to input the concatenated features into the first convolution layer in the second convolutional neural network for feature extraction, and output a first refined classification feature;

[0155] The fourth output unit is used to perform the following operations for multiple convolutional layers except the first convolutional layer in the second convolutional neural network: input the refined classification features output by the previous convolutional layer into the current convolutional layer, output the refined classification features, until the last convolutional layer outputs the last refined classification feature, to obtain multiple refined classification features; wherein the multiple second refined classification features include a target refined classification feature, and the target refined classification feature identifies the refined classification result of the target cell image.

[0156] In one embodiment, the fourth output unit may also be used for:

[0157] The refined classification features output by the last convolutional layer of the second convolutional neural network are used as the target refined classification features.

[0158] The third feature extraction module 708 is used to input the multiple fine classification features into the first convolutional neural network for feature extraction, and output multiple second coarse classification features.

[0159] The plurality of second coarse classification features include a target coarse classification feature, and the target coarse classification feature identifies a coarse classification result of the target cell image.

[0160] In one embodiment, the third feature extraction module 708 is further configured to use the second coarse classification feature output by the last convolutional layer of the first convolutional neural network as the target coarse classification feature.

[0161] Since the second coarse classification feature is extracted based on the fine classification feature, and the fine classification feature is extracted based on the splicing feature that has sufficient global features and local features, the fine classification feature also has sufficient global features and local features. Furthermore, the second coarse classification feature also has sufficient global features and local features, and it can be used as the final coarse classification result.

[0162] This application uses two convolutional neural networks to classify target cell images, and can obtain classification results with two accuracies. In addition, the output features of the first convolutional neural network are spliced ​​with the target cell image as the input of the second convolutional neural network, so that the second convolutional neural network can obtain more local features. The output of the second convolutional neural network is then returned to the first convolutional neural network for input, so that the first convolutional neural network can obtain more global features, and the output of the first convolutional neural network is used as the rough classification result of the cell, and the output of the second convolutional neural network is used as the fine classification result, with high classification accuracy and accurate classification.

[0163] Figure 8 FIG. 1 is a schematic diagram showing a structure of a device for classifying cells based on a cell classification model provided by an embodiment of the present application. Figure 8 As shown, the device may include a processor 801 and a memory 802 storing computer program instructions.

[0164] Specifically, the processor 801 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0165] The memory 802 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 802 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In an example, the memory 802 may include a removable or non-removable (or fixed) medium, or the memory 802 is a non-volatile solid-state memory. The memory 802 may be inside or outside the integrated gateway disaster recovery device.

[0166] In one example, the memory 802 may be a read-only memory (ROM). In one example, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0167] The processor 801 reads and executes the computer program instructions stored in the memory 802 to implement Figure 1-5 The method / steps in the embodiment shown in the figure can achieve Figure 1-5 The corresponding technical effects achieved by executing the methods / steps in the example shown are not repeated here for the sake of brevity.

[0168] In one example, the device for classifying cells based on the cell classification model may further include a communication interface 808 and a bus 810. Figure 8 As shown, the processor 801, the memory 802, and the communication interface 808 are connected via a bus 810 and communicate with each other.

[0169] The communication interface 808 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0170] Bus 810 includes hardware, software or both, and the components of online data flow billing equipment are coupled to each other. For example, but not limitation, the bus may include Accelerated Graphics Port (AGP) or other graphics bus, Enhanced Industry Standard Architecture (EISA) bus, Front Side Bus (FSB), Hyper Transport (HT) interconnection, Industry Standard Architecture (ISA) bus, InfiniBand interconnection, Low Pin Count (LPC) bus, Memory bus, Micro Channel Architecture (MCA) bus, Peripheral Component Interconnect (PCI) bus, PCI-Express (PCI-X) bus, Serial Advanced Technology Attachment (SATA) bus, Video Electronics Standards Association Local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 810 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0171] The device for classifying cells based on the cell classification model can perform Figure 1-5 The method / steps in the embodiment shown in the figure are used to achieve Figure 1-5 Describe the method.

[0172] In addition, in combination with the method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the methods in the above embodiment is implemented.

[0173] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0174] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), appropriate firmware, plug-in, function card, etc. When implemented in software, the elements of the present application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or communication link by a data signal carried in a carrier. "Machine-readable medium" may include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (Radio Frequency, RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0175] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0176] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0177] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A method for classifying cells based on a cell classification model, characterized in that: The cell classification model includes a cascaded first convolutional neural network, a splicing layer, and a second convolutional neural network; The method comprises: Acquire target cell images; Inputting the target cell image into the first convolutional neural network for feature extraction, and outputting a plurality of first coarse classification features; Inputting the plurality of first rough classification features and the target cell image into the splicing layer for feature splicing, and outputting splicing features; Inputting the splicing features into the second convolutional neural network for feature extraction, and outputting a plurality of refined classification features, wherein the plurality of refined classification features include a target refined classification feature, and the target refined classification feature identifies a refined classification result of the target cell image; The multiple fine classification features are input into the first convolutional neural network for feature extraction, and multiple second coarse classification features are output, wherein the multiple second coarse classification features include a target coarse classification feature, and the target coarse classification feature identifies a coarse classification result of the target cell image.

2. The method for classifying cells based on a cell classification model according to claim 1, characterized in that: The first convolutional neural network includes a plurality of cascaded convolutional layers; The step of inputting the target cell image into the first convolutional neural network for feature extraction and outputting a plurality of first rough classification features comprises: Inputting the target cell image into the first convolutional layer in the first convolutional neural network for feature extraction, and outputting a first first coarse classification feature; For the multiple convolutional layers except the first convolutional layer in the first convolutional neural network, the following operations are performed: The first coarse classification feature output by the previous convolution layer is input into the current convolution layer, and the first coarse classification feature is output, until the last convolution layer outputs the last first coarse classification feature, thereby obtaining multiple first coarse classification features.

3. The method for classifying cells based on a cell classification model according to claim 1, characterized in that: The second convolutional neural network includes a plurality of cascaded convolutional layers; The step of inputting the concatenated features into the second convolutional neural network for feature extraction and outputting a plurality of refined classification features includes: Inputting the concatenated features into the first convolutional layer in the second convolutional neural network for feature extraction, and outputting a first refined classification feature; For the multiple convolutional layers except the first convolutional layer in the second convolutional neural network, the following operations are performed: The refined classification features output by the previous convolutional layer are input into the current convolutional layer, and the refined classification features are output until the last convolutional layer outputs the last refined classification feature, thereby obtaining multiple refined classification features.

4. The method for classifying cells based on a cell classification model according to claim 3, characterized in that: The method further comprises: The refined classification features output by the last convolutional layer of the second convolutional neural network are used as the target refined classification features.

5. The method for classifying cells based on a cell classification model according to claim 2, characterized in that: The step of inputting the plurality of fine classification features into the first convolutional neural network for feature extraction and outputting a plurality of second coarse classification features comprises: The multiple fine classification features are respectively input into multiple convolutional layers in the first convolutional neural network, and the multiple second coarse classification features are output.

6. The method for classifying cells based on a cell classification model according to claim 5, characterized in that: The method further comprises: The second coarse classification feature output by the last convolutional layer of the first convolutional neural network is used as the target coarse classification feature.

7. A method for training a cell classification model, characterized in that: include: Acquire a training sample set, wherein the training sample set includes a plurality of training samples, and each training sample includes an original cell image and a corresponding category label; For each of the training samples, the following operations are performed respectively: Inputting the original cell image into a first convolutional neural network of a target model for feature extraction, and outputting a plurality of first coarse classification training features; Inputting the plurality of first coarse classification training features and the original cell image into the splicing layer of the target model for feature splicing to obtain splicing training features; Inputting the concatenated training features into the second convolutional neural network of the target model for feature extraction, and outputting a plurality of refined classification training features, wherein the plurality of refined classification training features include a target refined classification training feature, and the target refined classification training feature is identified with a refined classification result of the training sample; Inputting the plurality of fine classification training features into the first convolutional neural network of the target model for feature extraction, and outputting a plurality of second coarse classification training features, wherein the plurality of second coarse classification training features include a target coarse classification training feature, and the target coarse classification training feature is identified with a coarse classification result of the training sample; Calculate the cross entropy loss function of the target model according to each of the target fine classification training features, the corresponding category labels and the proportion of each category of cells to the total number of cells; When the output value of the cross entropy loss function is not less than the preset threshold, the model parameters of the target model are adjusted according to each of the target fine classification training features, the target coarse classification training features and their corresponding category labels; the first convolutional neural network that inputs the original cell image into the target model is returned to perform feature extraction until the output value of the cross entropy loss function is less than the preset threshold, thereby obtaining the cell classification model.

8. A device for classifying cells based on a cell classification model, characterized in that: The cell classification model includes a cascaded first convolutional neural network, a splicing layer, and a second convolutional neural network; The device comprises: An acquisition module, used for acquiring target cell images; A first feature extraction module, used for inputting the target cell image into the first convolutional neural network for feature extraction, and outputting a plurality of first coarse classification features; A feature stitching module, used for inputting the plurality of first coarse classification features and the target cell image into the stitching layer for feature stitching, and outputting a stitching feature; A second feature extraction module is used to input the splicing feature into the second convolutional neural network for feature extraction, and output a plurality of refined classification features, wherein the plurality of refined classification features include a target refined classification feature, and the target refined classification feature is identified with a refined classification result of the target cell image; The third feature extraction module is used to input the multiple fine classification features into the first convolutional neural network for feature extraction, and output multiple second coarse classification features, wherein the multiple second coarse classification features include a target coarse classification feature, and the target coarse classification feature is identified with a coarse classification result of the target cell image.

9. A device for classifying cells based on a cell classification model, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method according to any one of claims 1 to 7 when executed by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an implementation program for information transmission, and when the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images