Pathological cell image classification methods, devices, electronic equipment and storage media

Abstract

This disclosure relates to a method, apparatus, electronic device, and storage medium for classifying pathological cell images. The method involves acquiring images of stained cells, including multiple cell instances, from a target pathological slide and inputting these images into an image classification model. A prediction module within the model predicts location, distance, and a first category. A cell graph structure is constructed by extracting morphological features from each cell instance in the stained cell image. The graph node attributes corresponding to the cell graph structure are determined based on the location, distance, and first category predictions, and a graph convolution module is used to determine a second category prediction. Finally, the cell category corresponding to each pixel location in the stained cell image is determined based on both the first and second category predictions. This disclosure improves the accuracy of the classification by using two prediction methods to jointly predict the classification result of the stained cell image input to the model and combining the two prediction results.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method, apparatus, electronic device, and storage medium for classifying pathological cell images. Background Technology

[0002] The diverse distribution and proportion of tumor cells, B cells, and T cells are of paramount scientific value for a deep understanding of the composition of the tumor microenvironment, its evolutionary mechanisms in neoadjuvant therapy, and treatment response. Currently, precise cell classification on hematoxylin-eosin (HE) images not only reveals the complexity of the tertiary lymphoid microenvironment but also provides crucial reference for optimizing neoadjuvant therapy strategies. However, when using artificial intelligence models for image-based cell classification, due to technical limitations or sample characteristics, automatically generated HE image cell labels often contain noise, meaning that some cell classification labels may be incorrect or inaccurate. This, to some extent, restricts the depth of research and the accuracy of model training. Summary of the Invention

[0003] In view of this, this disclosure proposes a method, apparatus, electronic device and storage medium for classifying pathological cell images, aiming to improve the accuracy of classification results of stained cell images.

[0004] According to a first aspect of this disclosure, a pathological cell imaging method is provided, the method comprising:

[0005] Images of the target pathological section were acquired to obtain stained cell images including multiple cell instances;

[0006] The stained cell image is input into an image classification model, and the prediction module therein predicts the location prediction result, the distance prediction result, and the first category prediction result. The location prediction result is used to characterize whether each pixel location is the location of a cell instance, and the distance prediction result is used to characterize the distance between each pixel location and the centroid of the cell in the cell instance.

[0007] A cell graph structure is constructed by extracting the morphological features of each cell instance in the stained cell image. Each node in the cell graph structure is used to represent a cell instance in the stained cell image, and the edges are used to represent the nearest neighbor relationship between different cell instances.

[0008] By performing feature fusion on the location prediction result, distance prediction result, and first category prediction result, the graph node attributes corresponding to the cell graph structure are obtained;

[0009] The cell graph structure and the graph node attributes are input into the graph convolution module of the image classification model to obtain the second category prediction result;

[0010] Based on the first category prediction result and the second category prediction result, the cell category corresponding to each pixel position in the stained cell image is determined.

[0011] In one possible implementation, the prediction module includes an encoder, a first decoder, a second decoder, and a third decoder. The step of inputting the stained cell image into an image classification model and obtaining location prediction results, distance prediction results, and a first category prediction result through the prediction module includes:

[0012] The stained cell image is input into an image classification model, and the stained cell image is encoded by an encoder to obtain the image encoding result;

[0013] The first decoder, the second decoder, and the third decoder decode the image encoding results respectively to obtain the corresponding location prediction results, distance prediction results, and first category prediction results.

[0014] In one possible implementation, constructing the cell map structure by extracting morphological features of each cell instance in the stained cell image includes:

[0015] The morphological features of each cell instance in the stained cell image are extracted to obtain the corresponding morphological feature vector;

[0016] The degree of dispersion between every two cell instances is calculated based on the morphological feature vector of each cell instance;

[0017] Determine that each cell instance has a nearest neighbor relationship with a preset number of cell instances with the highest degree of dispersion;

[0018] The cell graph structure is determined by using each cell instance as a node and nearest neighbor relationships as edges.

[0019] In one possible implementation,

[0020] The degree of dispersion is expressed by the formula The calculation yielded that, L2(c) is used to characterize all cell instances included in the stained cell image. i c j Let c be any two cell instances. i and c j The degree of dispersion between them, c il and c jl Characterizing cell instance c i and c j The eigenvalue of the l-th dimension in the corresponding morphological feature vector.

[0021] In one possible implementation, the morphological features include morphological features and internal texture features.

[0022] In one possible implementation, the step of obtaining the graph node attributes corresponding to the cell graph structure by performing feature fusion on the location prediction result, distance prediction result, and first category prediction result includes:

[0023] The first feature attribute is obtained by concatenating the location prediction result, distance prediction result, and first category prediction result.

[0024] The second feature attribute is determined based on the morphological feature vector corresponding to each cell instance;

[0025] The graph node attributes are obtained by fusing the first feature attribute and the second feature attribute.

[0026] In one possible implementation, the first category prediction result includes a first category and a corresponding confidence level for each pixel location, and the second category prediction result includes a second category and a corresponding confidence level for each pixel location;

[0027] The step of determining the cell category corresponding to each pixel position in the stained cell image based on the first category prediction result and the second category prediction result includes:

[0028] In response to the first category and the second category being the same at the pixel location, the cell category is determined to be the same as the first category and the second category, and the corresponding confidence level is the average of the confidence levels of the first category and the second category;

[0029] In response to the difference between the first and second categories at the pixel location, the cell category is determined to be the same as the category with the higher confidence level among the first and second categories, with the confidence level being the same category confidence level.

[0030] In one possible implementation, the method further includes:

[0031] The first category prediction result is corrected based on the location prediction result and the distance prediction result using the watershed algorithm.

[0032] In one possible implementation, the training process of the image classification model includes:

[0033] Determine the training and test sets, which include multiple sample cell images and corresponding labeled category information;

[0034] A first test set and a second test set are determined based on the test set, wherein the sample cell images in the first test set and the second test set are mutually exclusive;

[0035] The image classification model is initially trained based on the training set and the first test set, wherein sample cell images from the first test set are input into the image classification model, and the corresponding predicted classification results are output.

[0036] In response to the fact that the target category corresponding to the pixel position is the same in the prediction classification results of a preset number of consecutive times and the confidence level is greater than a preset value, and the target category is different from the labeled category of the pixel position in the labeled category information, the labeled category is modified to the target category;

[0037] The image classification model after initial training is fine-tuned based on the training set and the second test set.

[0038] In one possible implementation, the determination includes a training set and a test set comprising multiple sample cell images and corresponding labeled category information, including:

[0039] Multiple pathological sections were collected, along with corresponding sample cell images and immunofluorescence images.

[0040] Alignment processing is performed on the cell instances in the sample cell image and the immunofluorescence image corresponding to each pathological section, aligning the cell instances in the sample cell image with the cell instances in the corresponding immunofluorescence image;

[0041] The corresponding category is determined based on the color of cell instances in the sample cell image in the immunofluorescence image, thus obtaining the labeled category information;

[0042] The training set and test set are obtained by randomly selecting sample cell images and corresponding labeled category information.

[0043] According to a second aspect of this disclosure, a pathological cell imaging device is provided, the device comprising:

[0044] The image acquisition module is used to acquire images of the target pathological slide and obtain stained cell images including multiple cell instances;

[0045] The first prediction module is used to input the stained cell image into an image classification model, and obtain a location prediction result, a distance prediction result, and a first category prediction result through the prediction module therein. The location prediction result is used to characterize whether each pixel position is the location of a cell instance, and the distance prediction result is used to characterize the distance between each pixel position and the centroid of the cell in the cell instance.

[0046] The graph structure determination module is used to construct a cell graph structure by extracting the morphological features of each cell instance in the stained cell image. Each node in the cell graph structure is used to represent a cell instance in the stained cell image, and the edges are used to represent the nearest neighbor relationship between different cell instances.

[0047] The attribute determination module is used to obtain the graph node attributes corresponding to the cell graph structure by performing feature fusion on the location prediction result, distance prediction result and first category prediction result;

[0048] The second prediction module is used to input the cell graph structure and the graph node attributes into the graph convolution module in the image classification model to obtain the second category prediction result.

[0049] The cell classification module is used to determine the cell category corresponding to each pixel position in the stained cell image based on the first category prediction result and the second category prediction result.

[0050] In one possible implementation, the prediction module includes an encoder, a first decoder, a second decoder, and a third decoder, wherein the first prediction module is further configured to:

[0051] The stained cell image is input into an image classification model, and the stained cell image is encoded by an encoder to obtain the image encoding result;

[0052] The first decoder, the second decoder, and the third decoder decode the image encoding results respectively to obtain the corresponding location prediction results, distance prediction results, and first category prediction results.

[0053] In one possible implementation, the graph structure determination module is further configured to:

[0054] The morphological features of each cell instance in the stained cell image are extracted to obtain the corresponding morphological feature vector;

[0055] The degree of dispersion between every two cell instances is calculated based on the morphological feature vector of each cell instance;

[0056] Determine that each cell instance has a nearest neighbor relationship with a preset number of cell instances with the highest degree of dispersion;

[0057] The cell graph structure is determined by using each cell instance as a node and nearest neighbor relationships as edges.

[0058] In one possible implementation,

[0059] The degree of dispersion is expressed by the formula The calculation yielded that, L2(c) is used to characterize all cell instances included in the stained cell image. i c j Let c be any two cell instances. i and c j The degree of dispersion between them, c il and c jl Characterizing cell instance c i and c j The eigenvalue of the l-th dimension in the corresponding morphological feature vector.

[0060] In one possible implementation, the morphological features include morphological features and internal texture features.

[0061] In one possible implementation, the attribute determination module is further configured to:

[0062] The first feature attribute is obtained by concatenating the location prediction result, distance prediction result, and first category prediction result.

[0063] The second feature attribute is determined based on the morphological feature vector corresponding to each cell instance;

[0064] The graph node attributes are obtained by fusing the first feature attribute and the second feature attribute.

[0065] In one possible implementation, the first category prediction result includes a first category and a corresponding confidence level for each pixel location, and the second category prediction result includes a second category and a corresponding confidence level for each pixel location;

[0066] The cell classification module is further used for:

[0067] In response to the first category and the second category being the same at the pixel location, the cell category is determined to be the same as the first category and the second category, and the corresponding confidence level is the average of the confidence levels of the first category and the second category;

[0068] In response to the difference between the first and second categories at the pixel location, the cell category is determined to be the same as the category with the higher confidence level among the first and second categories, with the confidence level being the same category confidence level.

[0069] In one possible implementation, the device further includes:

[0070] The result correction module is used to correct the first category prediction result based on the location prediction result and the distance prediction result using the watershed algorithm.

[0071] In one possible implementation, the training process of the image classification model includes:

[0072] Determine the training and test sets, which include multiple sample cell images and corresponding labeled category information;

[0073] A first test set and a second test set are determined based on the test set, wherein the sample cell images in the first test set and the second test set are mutually exclusive;

[0074] The image classification model is initially trained based on the training set and the first test set, wherein sample cell images from the first test set are input into the image classification model, and the corresponding predicted classification results are output.

[0075] In response to the fact that the target category corresponding to the pixel position is the same in the prediction classification results of a preset number of consecutive times and the confidence level is greater than a preset value, and the target category is different from the labeled category of the pixel position in the labeled category information, the labeled category is modified to the target category;

[0076] The image classification model after initial training is fine-tuned based on the training set and the second test set.

[0077] In one possible implementation, the determination includes a training set and a test set comprising multiple sample cell images and corresponding labeled category information, including:

[0078] Multiple pathological sections were collected, along with corresponding sample cell images and immunofluorescence images.

[0079] Alignment processing is performed on the cell instances in the sample cell image and the immunofluorescence image corresponding to each pathological section, aligning the cell instances in the sample cell image with the cell instances in the corresponding immunofluorescence image;

[0080] The corresponding category is determined based on the color of cell instances in the sample cell image in the immunofluorescence image, thus obtaining the labeled category information;

[0081] The training set and test set are obtained by randomly selecting sample cell images and corresponding labeled category information.

[0082] According to a third aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described method when executing instructions stored in the memory.

[0083] According to a fourth aspect of this disclosure, a non-volatile computer-readable storage medium is provided that stores computer program instructions thereon, wherein the computer program instructions, when executed by a processor, implement the above-described method.

[0084] According to a fifth aspect of this disclosure, a computer program product is provided, including computer-readable code or a non-volatile computer-readable storage medium carrying the computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0085] In this embodiment, an image of stained cells, comprising multiple cell instances, is acquired from a target pathological slide and input into an image classification model. The model's prediction module predicts location, distance, and a first category. A cell graph structure is constructed by extracting morphological features from each cell instance in the stained cell image. The graph node attributes corresponding to the cell graph structure are determined based on the location, distance, and first category predictions, and a graph convolution module is used to determine a second category prediction. Based on the first and second category predictions, the cell category corresponding to each pixel in the stained cell image is determined. This disclosure uses two prediction methods to jointly predict the classification result of the stained cell image input to the model and combines the two prediction results to obtain the final classification result, thus improving the accuracy of the classification.

[0086] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0087] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0088] Figure 1 A flowchart illustrating a pathological cell image classification method according to an embodiment of the present disclosure is shown.

[0089] Figure 2 A schematic diagram illustrating a modified first category prediction result according to an embodiment of the present disclosure is shown;

[0090] Figure 3 A schematic diagram illustrating the preliminary training process of an image classification model according to an embodiment of the present disclosure is shown.

[0091] Figure 4 A schematic diagram of a pathological cell image classification device according to an embodiment of the present disclosure is shown;

[0092] Figure 5 A schematic diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0093] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0094] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0095] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0096] The pathological cell image classification method of this disclosure can be executed by an electronic device such as a terminal device or a server. The terminal device can be any fixed or mobile terminal, such as a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, vehicle-mounted device, or wearable device. The server can be a single server or a server cluster consisting of multiple servers. Any electronic device can implement the pathological cell image classification method of this disclosure by having its processor call computer-readable instructions stored in its memory.

[0097] Figure 1 A flowchart illustrating a pathological cell image classification method according to an embodiment of the present disclosure is shown. Figure 1 As shown, the pathological cell image classification method of this disclosure embodiment may include the following steps S10-S60.

[0098] Step S10: Acquire images of the target pathological section to obtain stained cell images including multiple cell instances.

[0099] In one possible implementation, in scenarios requiring the classification of pathological cells, an electronic device can acquire images of a target pathological slide obtained after staining the pathological cells, resulting in a stained cell image including multiple cell instances. The multiple cell instances included in the stained cell image are cells stained with cell staining substances such as hematoxylin and eosin (HE).

[0100] Optionally, the electronic device can acquire images of the target pathological slide directly from its built-in or connected image acquisition device. Alternatively, the target pathological slide can be imaged using an image acquisition device from another device, and the acquired stained cell images can be sent to the electronic device for pathological cell image classification. The format of the stained cell images can be any image format, and no limitation is made here.

[0101] Step S20: Input the stained cell image into the image classification model, and obtain the location prediction result, distance prediction result and first category prediction result through the prediction module therein.

[0102] In one possible implementation, after determining the stained cell image, the electronic device inputs the stained cell image into an image classification model. The pre-trained image classification model then classifies each cell instance in the stained cell image to obtain the cell category corresponding to each cell instance. The image classification model may include two parts: a prediction module and a graph convolution module, used to predict a first category and a second category, respectively, to obtain the final cell category based on the two category predictions.

[0103] Optionally, after the stained cell image is input into the image classification model, the prediction module can first predict the stained cell image to obtain a location prediction result, a distance prediction result, and a first category prediction result. The location prediction result characterizes whether each pixel location in the stained cell image is the location of a cell instance; the distance prediction result characterizes the distance between each pixel location in the stained cell image and the centroid of the cell instance; and the first category prediction result characterizes the cell instance category at each pixel location in the stained cell image predicted by the prediction module.

[0104] Furthermore, the prediction module can be a deep learning network comprising an encoder, a first decoder, a second decoder, and a third decoder. During the processing of stained cell images, the stained cell images can be input into an image classification model, and the encoder encodes the stained cell images to obtain image encoding results. Then, the first decoder, second decoder, and third decoder decode based on the image encoding results to obtain the corresponding location prediction results, distance prediction results, and first category prediction results, respectively. Optionally, the encoder can be a RenseNet50 encoder, whose basic units are constructed through short-circuit connections. The input of each layer depends on the output and input of the previous layer, which can be expressed by formula X. l =H l (X l-1 )+X l-1The encoder represents the data processing at each layer, where X is the input to the corresponding layer, H represents the nonlinear change of the corresponding layer, and l represents the layer number. The decoder can be a Dense Unit decoder, where the basic units are densely connected. The input to each layer is obtained by combining the outputs of all layers preceding it; that is, it can be obtained through X. l =H l ([X0, X1, ..., X l-1 [X0, X1, ..., X] represents the input of the l-th layer, the same as the encoder, X is the input of the corresponding layer, H represents the nonlinear change of the corresponding layer, and l represents the corresponding layer number. l-1 ] indicates that the output feature maps from layer 0 to layer (l-1) are combined.

[0105] In one possible implementation, the distance prediction result includes horizontal distance prediction and vertical distance prediction, used to mark the distances of each pixel location in the stained cell image to the centroid of the cell instance in the horizontal and vertical directions, respectively. The first category prediction result may also include the first category of each pixel location and the corresponding confidence level, that is, the possible cell category corresponding to each pixel location in the stained cell image predicted by the prediction module, and the confidence level used to characterize the probability of that cell category.

[0106] Furthermore, in this embodiment of the invention, after the electronic device obtains the location prediction result, distance prediction result, and first category prediction result based on the prediction module, it can correct the first category prediction result to improve the accuracy of the prediction result. Specifically, the first category prediction result can be corrected based on the location prediction result and the distance prediction result using a watershed algorithm.

[0107] Figure 2 A schematic diagram illustrating a modified first-category prediction result according to an embodiment of the present disclosure is shown. Figure 2 As shown, the process of correcting the first category prediction result can be achieved by combining the location prediction result and the distance prediction result, and using a label-controlled watershed algorithm to refine the cell instance segmentation. Specifically, the location prediction result is used as the input to the watershed algorithm, which treats the location prediction result as a topographic map and segments different regions by simulating a water immersion process. Simultaneously, the distance prediction result is used as prior knowledge to control the flow rules of the "water" in the watershed algorithm, determining which regions in the stained cell image should be merged or segmented, resulting in a more accurate location prediction result and avoiding over-segmentation or under-segmentation. Furthermore, the first category prediction result is corrected based on the optimized location prediction result. The principle of this correction method is to correct the first category corresponding to the same region of the same cell instance.

[0108] Step S30: Construct a cell map structure by extracting the morphological features of each cell instance in the stained cell image.

[0109] In one possible implementation, after determining the stained cell image, the electronic device can further extract the morphological features of each cell instance from the dimension of morphological features to construct a cell graph structure for another cell category prediction. Each node in the cell graph structure represents a cell instance in the stained cell image, and each edge represents the nearest neighbor relationship between different cell instances. The nearest neighbor relationship between different cell instances can be determined by the morphological features of each cell instance. The morphological features in this embodiment can include morphological features and internal texture features. Morphological features can include area, perimeter, and polygonal outline, etc., while internal texture features can include LBP features (Local Binary Pattern), GLCM features (Gray-Level Co-occurrence Matrix), and color features, etc.

[0110] Optionally, the electronic device first extracts the morphological features of each cell instance in the stained cell image, and then concatenates the features extracted from each class to obtain the corresponding morphological feature vector. Then, each cell instance can be treated as a node, and P( The node features are represented by n, where n is the number of cell instances in the stained cell image, and k is the cell feature vector extracted for each cell instance. Furthermore, a cell graph structure is constructed based on the spatial relationships between cell instances using the k-nearest neighbor algorithm. Specifically, this graph structure construction process involves calculating the degree of dispersion between every two cell instances based on their morphological feature vectors, determining the nearest neighbor relationships between each cell instance and a predetermined number of cell instances with the highest degree of dispersion, and then using each cell instance as a node and the nearest neighbor relationships as edges to determine the cell graph structure.

[0111] For example, the electronic device can first pass through Characterize all cell instances in the stained cell image, and then use Euclidean distance to measure the dispersion between cell instances L2(c i c j ), that is, it can be done through the formula The degree of dispersion between each pair of cell instances is calculated, where, L2(c) is used to characterize all cell instances included in the stained cell image. i c j Let c be any two cell instances. i and c j The degree of dispersion between them, c il and c jl Characterizing cell instance c i and c iThe eigenvalue of the l-th dimension in the corresponding morphological feature vector. After calculating the discreteness between every two cells, the discreteness of all cell instances is sorted, and the top k cell instances with the closest discreteness for each cell instance are selected as the cell instances with nearest neighbor relationships {c1, c2, ..., ck}. k}, where k is the preset quantity. After obtaining each cell instance c i Let ε represent the set of k nearest neighbors of a cell instance c. i and c i If the relationship is k-nearest neighbor, then add an edge (c) to the cell graph structure. i c j ),

[0112] ε={(c i c j )|c j ∈{c i1 c i2 c ik}ot c i ∈{c j1 c j2 c jk The final cell diagram structure was determined. Represented as

[0113] Step S40: By performing feature fusion on the location prediction result, distance prediction result and first category prediction result, the graph node attributes corresponding to the cell graph structure are obtained.

[0114] In one possible implementation, after determining the cell graph structure, the electronic device can determine the graph node attributes corresponding to the cell graph structure based on the position prediction result, distance prediction result, and first category prediction result output by the prediction module. Optionally, the graph node attributes can be determined by concatenating the above three prediction structures. For example, the position prediction result, distance prediction result, and first category prediction result can be concatenated to obtain a first feature attribute, then a second feature attribute can be determined based on the morphological feature vector corresponding to each cell instance, and finally the first feature attribute and the second feature attribute can be fused to obtain the graph node attributes.

[0115] For example, the location prediction result, distance prediction result, and first category prediction result can all be obtained through feature map R. The feature map R′ is obtained by concatenating the three feature maps on the channel and then using global pooling and broadcasting. As the first feature attribute, it is then combined with the second feature attribute, which is composed of the morphological feature vector corresponding to each cell instance, to obtain the graph node attribute P′. Furthermore, based on the cell graph structure and the corresponding graph node attributes, the representation of the nodes can be learned through graph propagation to obtain the corresponding classification results.

[0116] Step S50: Input the cell graph structure and the graph node attributes into the graph convolution module in the image classification model to obtain the second category prediction result.

[0117] In one possible implementation, after obtaining the cell graph structure and corresponding graph node attributes, the cell graph structure and graph node attributes are input into the graph convolution module of the image classification model for graph propagation to learn the representation of nodes in the cell graph structure, thereby performing category prediction to obtain a second category prediction result. The second category prediction result may include the category predicted by the graph convolution module for each pixel location in the stained cell image as the second category, and a confidence level representing the probability that each pixel location belongs to the corresponding second category.

[0118] Step S60: Determine the cell category corresponding to each pixel position in the stained cell image based on the first category prediction result and the second category prediction result.

[0119] In one possible implementation, after the electronic device performs classification using a prediction module and a graph convolution module across different dimensions to obtain a first category prediction result and a second category prediction result, it can determine the cell category corresponding to each pixel location in the stained cell image based on the first and second category prediction results. Specifically, in embodiments of this disclosure, if the first and second categories at a pixel location are the same, the cell category is determined to be the same as both the first and second categories, with the corresponding confidence level being the average of the confidence levels of the first and second categories. If the first and second categories at a pixel location are different, the cell category is determined to be the same as the category with the higher confidence level among the first and second categories, with the confidence level being the confidence level of the same category.

[0120] Optionally, in this embodiment of the present disclosure, the training process of the image classification model for classifying cell instances in stained cell images can be a label update training process, that is, the labels of the sample images used for training are updated in real time during the training process. Specifically, the electronic device can first determine a training set and a test set, including multiple sample cell images and corresponding labeled category information. Then, a first test set and a second test set are determined based on the test set, and the sample cell images in the first test set and the second test set are mutually exclusive. The image classification model is initially trained based on the training set and the first test set, wherein the sample cell images in the first test set are input into the image classification model, and the corresponding predicted classification results are output. If the target category corresponding to a pixel position is the same in the predicted classification results for a consecutive preset number of times and the confidence level is greater than a preset value, and the target category is different from the labeled category at that pixel position in the labeled category information, the labeled category is modified to the target category. Finally, the image classification model after initial training is fine-tuned based on the training set and the second test set.

[0121] The process of determining the training and test sets may include acquiring multiple pathological slides with corresponding sample cell images and immunofluorescence images. For each pathological slide, cell instances in the corresponding sample cell image and immunofluorescence image are aligned. The cell instances in the sample cell image and the corresponding cell instances in the immunofluorescence image are then classified according to their color in the sample cell image, resulting in labeled category information. Sample cell images and their corresponding labeled categories are then randomly selected to obtain the training and test sets.

[0122] For example, embodiments of this disclosure can collect a large number of stained and fluorescent images corresponding to patient pathological sections, perform image preprocessing processes such as segmentation, and obtain HE-stained images of each pathological section with dimensions meeting preset requirements as sample cell images, and fluorescent images as immunofluorescence images. This segmentation process can be preset, for example, segmenting the two types of images at 20x magnification into 1000*1000 pixel images respectively. Due to washing the staining agent and human operation during pathological section processing, image shifts and misalignments may occur, therefore, HE-stained images and their corresponding immunofluorescence images are not naturally paired. Further, electronic devices can align cell instances in the sample cell images and immunofluorescence images corresponding to the same region in the same pathological section through rotation, scaling, translation, etc., and can segment each cell instance in the sample cell image using an existing automatic segmentation system. Optionally, a corresponding classification label is determined based on the color of the corresponding cell instance in the immunofluorescence image for each cell instance, obtaining corresponding labeling category information. Specifically, different color development thresholds can be set for each color channel corresponding to different categories, and the cell instance is assigned a corresponding classification label when the cell instance channel value is greater than the corresponding color development threshold.

[0123] Furthermore, after obtaining the sample cell images and labeling information corresponding to each pathological section, the existing sample cell images can be randomly divided into training and testing sets according to a preset ratio. Simultaneously, slight rotation and translation data augmentation operations can be performed on the data to be trained. Optionally, a portion of the data in the testing set can be randomly selected to determine the first testing set, with the remaining data serving as the second testing set. During training, the first testing set is used as the test set including noise during label updates, while the second testing set serves as the gold standard test set without any noise. To further improve the training effect, after obtaining the first testing set, the classification labels of each cell instance in the sample cell images can be manually checked and corrected to correct incorrectly classified labeling information caused by varying staining depths.

[0124] During model training, the image classification model is first initially trained using a training set and a first test set. Sample cell images from the first test set are input into the model, which outputs corresponding predicted classification results. If a pixel location corresponds to the same target category in a predetermined number of consecutive predicted classification results, and the confidence level is greater than a predetermined value, while the target category differs from the labeled category at that pixel location in the annotation information, the labeled category is changed to the target category. Finally, the initially trained image classification model is fine-tuned using the training set and the second test set.

[0125] Figure 3 This diagram illustrates the preliminary training process of an image classification model according to an embodiment of the present disclosure. Figure 3 As shown, the electronic device can first perform preliminary training on the image classification model based on the training set and the first test set. When the model loss of the image classification model tends to decrease slowly, and the confidence of the target category in each predicted classification result no longer increases and tends to stabilize, if the target category and the corresponding confidence of a cell instance both satisfy the label update rule, then its corresponding target category is replaced with the labeled category information of the original test set.

[0126] Specifically, the rule for updating the labeled category information is to begin updating the labels only after the image classification model has been trained on the training set and has reached a certain level of maturity. That is, label updates begin when the metrics of the training set and the first test set tend to stabilize. During the label update process, the target category in the predicted classification result for each cell instance in the first test set is recorded during three consecutive iterations. If a cell instance is predicted to have a different target category than the labeled category information three times consecutively, and the incorrect category is the same category, then it is further determined whether its corresponding confidence level meets the threshold requirement. For example, if the confidence threshold is set to 0.9, when the prediction probability is greater than 0.9 in three consecutive iterations, the label in the original labeled category information is considered to be incorrect. The predicted target category is then used to replace the corresponding label in the original labeled category information. In the next training iteration of the model, the latest updated labeled category information is loaded, and training continues for three more iterations, using the label update algorithm until the training set and the first test set stabilize again, thus completing the initial training. After completing the initial training, the image classification model is fine-tuned using the training set and the second test set until the various indicators of the gold standard second test set no longer change and tend to stabilize, thus completing the model training process and obtaining an image classification model that can perform accurate classification.

[0127] Based on the aforementioned technical features, this embodiment of the present disclosure uses both encoding / decoding and graph convolution prediction methods to jointly predict the classification results of the stained cell images input to the model, and combines the two prediction results to obtain the classification result, thereby improving the accuracy of the classification result. Simultaneously, by executing a label update algorithm during model training to correct the training data, the accuracy of the model training process is improved, enabling the acquisition of a more accurate image classification model through training.

[0128] Figure 4 A schematic diagram of a pathological cell image classification device according to an embodiment of the present disclosure is shown. Figure 4 As shown, the pathological cell image classification device in this embodiment may include:

[0129] The image acquisition module 40 is used to acquire images of the target pathological section and obtain stained cell images including multiple cell instances;

[0130] The first prediction module 41 is used to input the stained cell image into an image classification model, and obtain a location prediction result, a distance prediction result and a first category prediction result through the prediction module therein. The location prediction result is used to characterize whether each pixel position is the location of a cell instance, and the distance prediction result is used to characterize the distance between each pixel position and the centroid of the cell in the cell instance.

[0131] The graph structure determination module 42 is used to construct a cell graph structure by extracting the morphological features of each cell instance in the stained cell image. Each node in the cell graph structure is used to represent a cell instance in the stained cell image, and the edges are used to represent the nearest neighbor relationship between different cell instances.

[0132] The attribute determination module 43 is used to obtain the graph node attributes corresponding to the cell graph structure by performing feature fusion on the location prediction result, distance prediction result and first category prediction result;

[0133] The second prediction module 44 is used to input the cell graph structure and the graph node attributes into the graph convolution module in the image classification model to obtain the second category prediction result.

[0134] The cell classification module 45 is used to determine the cell category corresponding to each pixel position in the stained cell image based on the first category prediction result and the second category prediction result.

[0135] In one possible implementation, the prediction module includes an encoder, a first decoder, a second decoder, and a third decoder, wherein the first prediction module 41 is further configured to:

[0136] The stained cell image is input into an image classification model, and the stained cell image is encoded by an encoder to obtain the image encoding result;

[0137] The first decoder, the second decoder, and the third decoder decode the image encoding results respectively to obtain the corresponding location prediction results, distance prediction results, and first category prediction results.

[0138] In one possible implementation, the graph structure determination module 42 is further configured to:

[0139] The morphological features of each cell instance in the stained cell image are extracted to obtain the corresponding morphological feature vector;

[0140] The degree of dispersion between every two cell instances is calculated based on the morphological feature vector of each cell instance;

[0141] Determine that each cell instance has a nearest neighbor relationship with a preset number of cell instances with the highest degree of dispersion;

[0142] The cell graph structure is determined by using each cell instance as a node and nearest neighbor relationships as edges.

[0143] In one possible implementation,

[0144] The degree of dispersion is expressed by the formula The calculation yielded that, L2(c) is used to characterize all cell instances included in the stained cell image. i c j Let c be any two cell instances. i and c j The degree of dispersion between them, c il and c jl Characterizing cell instance c i and c j The eigenvalue of the l-th dimension in the corresponding morphological feature vector.

[0145] In one possible implementation, the morphological features include morphological features and internal texture features.

[0146] In one possible implementation, the attribute determination module 43 is further configured to:

[0147] The first feature attribute is obtained by concatenating the location prediction result, distance prediction result, and first category prediction result.

[0148] The second feature attribute is determined based on the morphological feature vector corresponding to each cell instance;

[0149] The graph node attributes are obtained by fusing the first feature attribute and the second feature attribute.

[0150] In one possible implementation, the first category prediction result includes a first category and a corresponding confidence level for each pixel location, and the second category prediction result includes a second category and a corresponding confidence level for each pixel location;

[0151] The cell classification module 45 is further used for:

[0152] In response to the first category and the second category being the same at the pixel location, the cell category is determined to be the same as the first category and the second category, and the corresponding confidence level is the average of the confidence levels of the first category and the second category;

[0153] In response to the difference between the first and second categories at the pixel location, the cell category is determined to be the same as the category with the higher confidence level among the first and second categories, with the confidence level being the same category confidence level.

[0154] In one possible implementation, the device further includes:

[0155] The result correction module is used to correct the first category prediction result based on the location prediction result and the distance prediction result using the watershed algorithm.

[0156] In one possible implementation, the training process of the image classification model includes:

[0157] Determine the training and test sets, which include multiple sample cell images and corresponding labeled category information;

[0158] A first test set and a second test set are determined based on the test set, wherein the sample cell images in the first test set and the second test set are mutually exclusive;

[0159] The image classification model is initially trained based on the training set and the first test set, wherein sample cell images from the first test set are input into the image classification model, and the corresponding predicted classification results are output.

[0160] In response to the fact that the target category corresponding to the pixel position is the same in the prediction classification results of a preset number of consecutive times and the confidence level is greater than a preset value, and the target category is different from the labeled category of the pixel position in the labeled category information, the labeled category is modified to the target category;

[0161] The image classification model after initial training is fine-tuned based on the training set and the second test set.

[0162] In one possible implementation, the determination includes a training set and a test set comprising multiple sample cell images and corresponding labeled category information, including:

[0163] Multiple pathological sections were collected, along with corresponding sample cell images and immunofluorescence images.

[0164] Alignment processing is performed on the cell instances in the sample cell image and the immunofluorescence image corresponding to each pathological section, aligning the cell instances in the sample cell image with the cell instances in the corresponding immunofluorescence image;

[0165] The corresponding category is determined based on the color of cell instances in the sample cell image in the immunofluorescence image, thus obtaining the labeled category information;

[0166] The training set and test set are obtained by randomly selecting sample cell images and corresponding labeled category information.

[0167] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0168] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium can be volatile or non-volatile.

[0169] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0170] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0171] Figure 5 A schematic diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 may be provided as a server or a terminal device. (Refer to...) Figure 5 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0172] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). Electronic device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM Mac OS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0173] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0174] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0175] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0176] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0177] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0178] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0179] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0180] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0181] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0182] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for classifying pathological cell images, characterized in that, The method includes: Images of the target pathological section were acquired to obtain stained cell images including multiple cell instances; The stained cell image is input into an image classification model, and the prediction module therein predicts the location prediction result, the distance prediction result, and the first category prediction result. The location prediction result is used to characterize whether each pixel position is the location of the cell instance, and the distance prediction result is used to characterize the distance between each pixel position and the centroid of the cell in the cell instance. A cell graph structure is constructed by extracting the morphological features of each cell instance in the stained cell image. Each node in the cell graph structure is used to represent a cell instance in the stained cell image, and the edges are used to represent the nearest neighbor relationship between different cell instances. By performing feature fusion on the location prediction result, distance prediction result, and first category prediction result, the graph node attributes corresponding to the cell graph structure are obtained; The cell graph structure and the graph node attributes are input into the graph convolution module of the image classification model to obtain the second category prediction result; Based on the first category prediction result and the second category prediction result, the cell category corresponding to each pixel position in the stained cell image is determined.

2. The method according to claim 1, characterized in that, The prediction module includes an encoder, a first decoder, a second decoder, and a third decoder. The step of inputting the stained cell image into an image classification model and obtaining location prediction results, distance prediction results, and a first category prediction result through the prediction module includes: The stained cell image is input into an image classification model, and the stained cell image is encoded by an encoder to obtain the image encoding result; The first decoder, the second decoder, and the third decoder decode the image encoding results respectively to obtain the corresponding location prediction results, distance prediction results, and first category prediction results.

3. The method according to claim 1, characterized in that, The step of constructing a cell map structure by extracting morphological features of each cell instance in the stained cell image includes: The morphological features of each cell instance in the stained cell image are extracted to obtain the corresponding morphological feature vector; The degree of dispersion between every two cell instances is calculated based on the morphological feature vector of each cell instance; Determine that each cell instance has a nearest neighbor relationship with a preset number of cell instances with the highest degree of dispersion; The cell graph structure is determined by using each cell instance as a node and nearest neighbor relationships as edges.

4. The method according to claim 3, characterized in that, The degree of dispersion is expressed by the formula The calculation yielded that, Used to characterize all cell instances included in the stained cell image. For any two cell instances and The degree of dispersion between them and Characterizing cell instances respectively and In the corresponding morphological feature vector, the first Eigenvalues ​​of dimension.

5. The method according to claim 3, characterized in that, The morphological features include morphological features and internal texture features.

6. The method according to any one of claims 3-5, characterized in that, The step of fusing features from the location prediction result, distance prediction result, and first category prediction result to obtain the graph node attributes corresponding to the cell graph structure includes: The first feature attribute is obtained by concatenating the location prediction result, distance prediction result, and first category prediction result. The second feature attribute is determined based on the morphological feature vector corresponding to each cell instance; The graph node attributes are obtained by fusing the first feature attribute and the second feature attribute.

7. The method according to any one of claims 1-5, characterized in that, The first category prediction result includes a first category and a corresponding confidence level for each pixel location, and the second category prediction result includes a second category and a corresponding confidence level for each pixel location; The step of determining the cell category corresponding to each pixel position in the stained cell image based on the first category prediction result and the second category prediction result includes: In response to the first category and the second category being the same at the pixel location, the cell category is determined to be the same as the first category and the second category, and the corresponding confidence level is the average of the confidence levels of the first category and the second category; In response to the difference between the first and second categories at the pixel location, the cell category is determined to be the same as the category with the higher confidence level among the first and second categories, with the confidence level being the same category confidence level.

8. The method according to any one of claims 1-5, characterized in that, The method further includes: The first category prediction result is corrected based on the location prediction result and the distance prediction result using the watershed algorithm.

9. The method according to any one of claims 1-5, characterized in that, The training process of the image classification model includes: Determine the training and test sets, which include multiple sample cell images and corresponding labeled category information; A first test set and a second test set are determined based on the test set, wherein the sample cell images in the first test set and the second test set are mutually exclusive; The image classification model is initially trained based on the training set and the first test set, wherein sample cell images from the first test set are input into the image classification model, and the corresponding predicted classification results are output. In response to the fact that the target category corresponding to the pixel position is the same in the prediction classification results of a preset number of consecutive times and the confidence level is greater than a preset value, and the target category is different from the labeled category of the pixel position in the labeled category information, the labeled category is modified to the target category; The image classification model after initial training is fine-tuned based on the training set and the second test set.

10. The method according to claim 9, characterized in that, The determination includes a training set and a test set comprising multiple sample cell images and corresponding labeled category information, including: Multiple pathological sections were collected, along with corresponding sample cell images and immunofluorescence images. Alignment processing is performed on the cell instances in the sample cell image and the immunofluorescence image corresponding to each pathological section, aligning the cell instances in the sample cell image with the cell instances in the corresponding immunofluorescence image; The corresponding category is determined based on the color of cell instances in the sample cell image in the immunofluorescence image, thus obtaining the labeled category information; The training set and test set are obtained by randomly selecting sample cell images and corresponding labeled category information.

11. A pathological cell image classification device, characterized in that, The device includes: The image acquisition module is used to acquire images of the target pathological slide and obtain stained cell images including multiple cell instances; The first prediction module is used to input the stained cell image into an image classification model, and obtain a location prediction result, a distance prediction result, and a first category prediction result through the prediction module therein. The location prediction result is used to characterize whether each pixel position is the location of a cell instance, and the distance prediction result is used to characterize the distance between each pixel position and the centroid of the cell in the cell instance. The graph structure determination module is used to construct a cell graph structure by extracting the morphological features of each cell instance in the stained cell image. Each node in the cell graph structure is used to represent a cell instance in the stained cell image, and the edges are used to represent the nearest neighbor relationship between different cell instances. The attribute determination module is used to obtain the graph node attributes corresponding to the cell graph structure by performing feature fusion on the location prediction result, distance prediction result and first category prediction result; The second prediction module is used to input the cell graph structure and the graph node attributes into the graph convolution module in the image classification model to obtain the second category prediction result. The cell classification module is used to determine the cell category corresponding to each pixel position in the stained cell image based on the first category prediction result and the second category prediction result.

12. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 10 when executing instructions stored in the memory.

13. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 10.

Images