A graph neural network-based method for cell nucleus classification in pathological images

Abstract

The present invention discloses a method for classifying cell nuclei in pathological images based on a graph neural network. The method first obtains a pathological tissue image containing cell nuclei and its instance segmentation results. Then, a shape-guided three-branch fusion network is constructed. The pathological tissue image containing cell nuclei is input into a deep feature branch to extract global features, obtain a pixel-level feature map, and obtain deep features for each cell nucleus. The manual feature branch is used to extract the morphological, color, and texture features of the cell nuclei from the instance segmentation results to obtain manual features, and the dimensionality is increased to obtain a manual feature vector. The shape feature branch is used to extract shape features based on contour points from the instance segmentation results. Finally, the method is fed into a feature gated fusion mechanism for fusion to obtain a fusion vector. The fusion vector is input into a classification output module for classification, and the classification result is output. The present invention fully utilizes the shape information of cell nuclei, enabling the network to better distinguish cell nuclei with complex morphologies, thereby improving classification performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and specifically relates to a method for classifying cell nuclei in pathological images based on graph neural networks. Background Art

[0002] Cell nucleus classification is a core task in computational pathology, crucial for cancer diagnosis, disease classification, and prognosis prediction. Nuclei in pathological tissue sections often exhibit significant morphological, texture, and color variations. Different nuclei have complex appearances, while nuclei of the same type can exhibit high variability in different tissue environments. This poses significant challenges for automated cell nucleus classification.

[0003] Currently, mainstream methods for nucleus classification are mainly divided into CNN-based methods and GNN-based methods. CNN-based methods typically use classic network architectures such as fully convolutional networks (FCN), ResNet, VGG, and DenseNet, and classify nuclei by extracting pixel-level texture information from the nucleus. For example, HoVer-Net (Graham S, Vu QD, Raza SEA, et al. Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images[J]. Medical image analysis, 2019, 58:101563.) is a CNN-based joint segmentation and classification method. However, HoVer-Net relies primarily on pixel-level features and does not explicitly model the shape of nuclei, resulting in limited effectiveness in classifying nuclei with variable morphology. TSFD-Net (Ilyas T, Mannan ZI, Khan A, et al. TSFD-Net: Tissue-specific feature distillation network for nuclei segmentation and classification [J]. Neural Networks, 2022, 151: 1-15.) uses a two-stream feature extraction structure: one part extracts texture information based on RGB channels, and the other part learns features based on segmentation boundary information, combined with an attention mechanism for classification. Although this method utilizes boundary information, it still relies primarily on local features extracted by CNN and cannot model the global shape structure of the cell nucleus. Therefore, the CNN method has some limitations. First, CNN relies primarily on pixel-level information, making it difficult to model the global shape characteristics of the cell nucleus. It is less effective in classifying cell nuclei with similar morphology but different categories. Second, due to the lack of shape information modeling, CNN can only learn pixel texture features and is insufficient in modeling the geometric features of the cell nucleus outline.

[0004] In recent years, methods based on graph neural networks (GNNs) have been increasingly applied to pathological image analysis. By modeling the proximity relationships between cell nuclei, GNNs enhance classification models' ability to leverage information about the spatial distribution of nuclei. For example, MCSpatNet (Abousamra S, Belinsky D, Van Arnam J, et al. Multi-class cell detection using spatial context representation[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021: 4005-4014.) proposes a multi-cell spatial representation learning method. This method learns the distribution patterns between nuclei through a graph structure, but focuses solely on the interactions between nuclei without explicitly modeling the shape characteristics of individual nuclei. SENC (Lou W, Wan X, Li G, et al. Structure embedded nucleus classification for histopathology images [J]. IEEE Transactions on Medical Imaging, 2024.) combines polygonal structural feature learning (PSL) and GNN-based nucleus spatial relationship modeling (GNN) in an end-to-end framework to simultaneously utilize the local shape features of the nucleus and the spatial relationship between the nuclei to improve classification performance. Although it depicts the overall change trend of the nucleus shape to a certain extent, since the RNN in PSL is mainly used to model temporal information, its learning ability of local geometric features is limited and it cannot fully model the shape details of the nucleus. In addition, the edge feature between the nuclei is introduced into the GNN structure, but this feature is mainly based on the background texture information between adjacent nuclei, and does not further model the geometric relationship between the nucleus contour points. As a result, the model has information loss when capturing the changes in the nucleus morphology, affecting the classification performance. Summary of the Invention

[0005] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and provide a pathological image cell nucleus classification method based on graph neural network. Feature extraction is based on three information paths: deep features, manual features and shape features, and fusion is performed through a feature gating fusion mechanism to fully utilize the multi-dimensional information of the cell nucleus and improve the classification performance.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] The first object of the present invention is to provide a method for classifying cell nuclei in pathological images based on a graph neural network, comprising the following steps:

[0008] Obtain pathological tissue images containing cell nuclei and their instance segmentation results;

[0009] Construct a shape-guided three-branch fusion network, including a deep feature branch, a manual feature branch, a shape feature branch, a feature gating mechanism, and a classification output module;

[0010] The pathological tissue image containing the cell nucleus is input into the deep feature branch to extract the global features to obtain the pixel-level feature map and obtain the deep features of each cell nucleus;

[0011] Use the manual feature branch to extract the morphology, color and texture features of the cell nucleus from the instance segmentation results to obtain manual features and then upgrade the dimension to obtain the manual feature vector;

[0012] The shape feature branch is used to extract shape features based on contour points from the instance segmentation results;

[0013] The deep features, manual feature vectors and shape features based on contour points are fed into the feature gating fusion mechanism to fuse them and obtain the fusion vector;

[0014] The fusion vector is input into the classification output module for classification and the classification result is output.

[0015] As a preferred technical solution, the pixel-level feature map is obtained and the depth feature of each cell nucleus is obtained, specifically:

[0016] A pre-trained encoder is used to perform global feature encoding on the input pathological tissue image containing cell nuclei to obtain a pixel-level feature map.

[0017] Combined with the nucleus bounding box information provided by the instance segmentation ground truth, the ROI Align technique is used to extract the regional features corresponding to each nucleus from the pixel-level feature map. The regional features are then unified into a fixed dimension through a pooling operation to obtain the deep features of each nucleus.

[0018] The pre-trained encoder does not contain the final fully connected layer.

[0019] As a preferred technical solution, the manual features are obtained and the dimension is increased to obtain the manual feature vector, specifically:

[0020] Extract m-dimensional manual features from pathological tissue images containing cell nuclei using instance segmentation results, including morphological features, color features, and texture features; the morphological features include cell nucleus area, cell nucleus perimeter, and cell nucleus roundness; the color features include cell nucleus color entropy and cell nucleus color moment; the texture features include cell nucleus edge color contrast and contrast, correlation, energy, homogeneity, and ASM corresponding to the gray-level co-occurrence matrix;

[0021] The morphological features, color features and texture features are combined to form a 1×m vector, which is then subjected to nonlinear mapping and dimensionality increase through several fully connected layers to obtain a manual feature vector.

[0022] As a preferred technical solution, the shape feature branch is used to extract shape features based on contour points from the instance segmentation results, specifically:

[0023] Based on the instance segmentation results, the cell nucleus contour points are sampled, the node features of the contour points are constructed, and the contour point feature matrix of each cell nucleus is obtained;

[0024] Input the contour point feature matrix into the improved graph attention network to self-connect each contour point and obtain the self-connection feature of each contour point;

[0025] An attention pooling operation is performed on the self-connected features of each contour point, and the contour points of each cell nucleus are globally aggregated to obtain the shape features of each cell nucleus.

[0026] As a preferred technical solution, the cell nucleus contour point sampling and the construction of the node features of the contour points are specifically as follows:

[0027] From the nucleus contour of the instance segmentation result, n 360-degree circles are evenly divided outward according to the center of the nucleus. ° / n angular area;

[0028] Select a contour point in each angle region;

[0029] Constructing node features of each contour point, including basic geometric features and local image features; the basic geometric features include normalized coordinates, curvature features and direction vectors; the local image features include RGB color information and texture features;

[0030] The node features of all contour points of each cell nucleus are represented as a matrix to obtain the contour point feature matrix of each cell nucleus.

[0031] As a preferred technical solution, the self-connection feature of each contour point is obtained as follows:

[0032] The contour point feature matrix is ​​increased in dimension through the fully connected layer:

[0033] X ’ = ELU ( W fc · X ),

[0034] in, X ’ is the contour point feature matrix after dimension increase, ELU is the exponential linear unit activation function, W fc ∈ R (n )×(H×h) is the weight matrix of the fully connected layer, H represents the hidden layer dimension of each attention head, h represents the number of attention heads, n Represents the input contour point feature matrix X Dimensions;

[0035] Use the GATConv layer with a self-connection mechanism to perform two self-connections on each contour point to obtain the self-connection features of each contour point:

[0036] ,

[0037] in, x i For the i The node features of the contour points, Indicates the i The set of neighbor nodes of the contour points, α ij is the weight obtained by attention calculation, x j For the j The node features of neighbor nodes, β is the self-connection weight decay factor, FC Represents a linear map.

[0038] As a preferred technical solution, deep features, manual feature vectors, and contour-based shape features are fed into a feature gating fusion mechanism for fusion. During fusion, a gating function is used to dynamically adjust the importance of each feature. The process is described as follows:

[0039] ,

[0040] in, F is the fusion vector, σ () is the activation function, W f is the weight matrix, f 1 is the depth feature, f 2 is a manual feature vector,f 3 is the shape feature based on contour points, b f is the weight bias matrix, For Hadamard, f 1; f 2; f 3] is a joint feature vector composed of deep features, manual feature vectors and shape features based on contour points.

[0041] As a preferred technical solution, the shape-guided three-branch fusion network is obtained through training, and the Focal Loss function is used during training, which is expressed as:

[0042] L Focal = α t (1- p t ) γ (- log ( p t )),

[0043] in, α t is the category weight, p t For the real category t The predicted probability of γ is the regulating factor.

[0044] The second object of the present invention is to provide a pathological image cell nucleus classification system based on graph neural network, which is applied to the pathological image cell nucleus classification method based on graph neural network, including a data acquisition module, a network construction module, a deep feature extraction module, a manual feature extraction module, a shape feature extraction module, a gated fusion module and a classification result module;

[0045] The data acquisition module is used to acquire pathological tissue images containing cell nuclei and instance segmentation results thereof;

[0046] The network construction module is used to construct a shape-guided three-branch fusion network, including a deep feature branch, a manual feature branch, a shape feature branch, a feature gating mechanism and a classification output module;

[0047] The depth feature extraction module is used to input the pathological tissue image containing the cell nucleus into the depth feature branch to extract the global feature to obtain the pixel-level feature map and obtain the depth feature of each cell nucleus;

[0048] The manual feature extraction module is used to use the manual feature branch to extract the morphological, color and texture features of the cell nucleus from the instance segmentation result to obtain manual features and to perform dimension upgrade to obtain a manual feature vector;

[0049] The shape feature module is used to extract shape features based on contour points from the instance segmentation results using the shape feature branch;

[0050] The gated fusion module is used to send the deep features, manual feature vectors and shape features based on contour points into the feature gated fusion mechanism for fusion to obtain a fusion vector;

[0051] The classification result module is used to input the fusion vector into the classification output module for classification and output the classification result.

[0052] The third object of the present invention is to provide a computer-readable storage medium storing a program, which, when executed by a processor, implements the pathological image cell nucleus classification method based on graph neural network.

[0053] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0054] First, the shape-guided three-branch fusion network in the present invention fully utilizes the shape information of the cell nucleus through the shape feature branch, overcoming the limitation of traditional CNN that only relies on local texture features. Compared with the SENC method that uses RNN to process the sequence of contour points, the present invention can directly construct the graph structure between contour points, and learn the geometric relationship between nodes through the graph attention network, so as to more accurately capture the local bending features, direction vectors and contour texture information, so that the network has better discrimination ability for cell nuclei with complex morphology. Secondly, the present invention adopts a feature gated fusion mechanism for multimodal feature fusion. Unlike the traditional method of directly splicing multiple features and inputting them into the fully connected layer, the feature gated fusion mechanism can adaptively adjust the contribution weights of different feature sources, thereby enhancing the nonlinear interaction between deep features, manual features and shape features, and improving the overall classification performance of the network. In addition, in order to address the problem of category imbalance, the present invention adopts a weighted Focal Loss loss function during training, which effectively reduces the misclassification phenomenon of low-frequency categories, making the network more balanced in different categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0056] Figure 1 This is a flow chart of the pathological image cell nucleus classification method based on graph neural network in an embodiment of the present invention.

[0057] Figure 2 2 is a diagram of the training process of the shape-guided three-branch fusion network in an embodiment of the present invention.

[0058] Figure 3 Schematic diagram of the structure of a pathological image cell nucleus classification system based on graph neural network in an embodiment of the present invention.

[0059] Figure 4 Schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0061] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0062] Current cell nucleus classification methods primarily rely on CNNs to extract local texture features or GNNs to model the spatial relationships between cell nuclei, but neither method fully exploits the shape characteristics of the nuclei. Due to the limited local receptive field, CNN methods struggle to capture the overall morphological information of the nuclei, resulting in unstable classification models when dealing with nuclei with complex shapes or blurred boundaries. While GNN methods have made progress in modeling the topological structure between nuclei, they lack explicit modeling of the nuclei's contour morphology and fail to fully express the local geometric characteristics of the nuclei. In particular, the SENC method, while incorporating polygonal structure feature learning (PSL) and nucleus spatial relationship modeling (GSL), struggles with its RNN-based processing of contour point sequences to accurately characterize local geometric variations in the nuclei. Furthermore, the relatively simple edge feature design fails to effectively model the geometric relationships between nucleus contour points, compromising the accuracy of the classification model.

[0063] Therefore, the goal of the present invention is to propose a method for comprehensively modeling the shape, texture and spatial relationship of the cell nucleus, breaking through the limitations of existing CNN and GNN methods in shape information modeling and category imbalance optimization. The present invention adopts a multi-branch structure, and combines deep features, manual features and learnable shape features to enhance the model's understanding of different feature dimensions of the cell nucleus. In particular, the present invention constructs the cell nucleus contour points into a graph structure through a cell nucleus shape modeling graph neural network, and explicitly models the geometry, curvature, direction and texture features of the cell nucleus shape under the GNN framework, so that the classification model can accurately capture the local and overall morphological characteristics of the cell nucleus. In addition, in order to solve the problem of category imbalance, the network constructed by the present invention introduces weighted Focal Loss to optimize the learning of low-frequency categories, so that the classification model performs more balanced on different cell nucleus categories.

[0064] See below. Figure 1 This embodiment describes in detail the steps of a method for classifying cell nuclei in pathological images based on a graph neural network, including:

[0065] S1. Obtain a pathological tissue image containing cell nuclei and its instance segmentation results.

[0066] S2. Construct a shape-guided three-stream shape fusion network (3SF-Net) to improve the accuracy of cell nucleus classification tasks, including a deep feature branch, a manual feature branch (HCFeature branch), a shape feature branch (shape GAT branch), a feature gating mechanism (FG), and a classification output module.

[0067] S3. Input the pathological tissue image containing the cell nucleus into the deep feature branch to extract the global features to obtain the pixel-level feature map and obtain the deep features of each cell nucleus.

[0068] The deep feature branch is mainly used to capture the global context information and deep texture information of the image where the cell nucleus is located. The specific steps are:

[0069] S3.1. First, a pre-trained encoder (excluding the final fully connected layer) is used to perform global feature encoding on the input pathological tissue image containing cell nuclei to obtain a pixel-level feature map.

[0070] S3.2. Then, combined with the nucleus bounding box information provided by the instance segmentation ground truth, the ROI Align (region of interest alignment) technique is used to extract the regional features corresponding to each nucleus from the pixel-level feature map, and the regional features are unified into a fixed dimension through pooling operations to obtain the deep features of each nucleus. f 1.

[0071] In this embodiment, the pre-trained encoder uses a Swin Transformer encoder.

[0072] S4. Use the manual feature branch to extract the morphological, color and texture features of the cell nucleus from the instance segmentation results to obtain manual features and then upgrade the dimension to obtain a manual feature vector.

[0073] The manual feature branch (HC Feature branch) can fully utilize the traditional features of the cell nucleus and help make up for the shortcomings of deep features in fine-grained morphological expression. The specific steps are as follows:

[0074] S4.1. Extract m-dimensional handcrafted features from pathological tissue images containing cell nuclei using instance segmentation results, including morphological, color, and texture features. Morphological features include nuclear area, nuclear perimeter, and nuclear roundness; color features include nuclear color entropy and nuclear color moments (mean, variance, and skewness calculated for the L, B, and P channels, respectively); and texture features include nuclear edge color contrast and gray-level co-occurrence matrix (GLCM)-based contrast, correlation, energy, homogeneity, and ASM.

[0075] S4.2. Combine morphological features, color features, and texture features to form a 1×m vector, which is then subjected to nonlinear mapping and dimensionality increase through several fully connected layers to obtain a handcrafted feature vector. f 2.

[0076] In this embodiment, 19-dimensional manual features are extracted and combined to form a 1×19 vector.

[0077] S5. Use the shape feature branch to extract shape features based on contour points from the instance segmentation results.

[0078] For the shape feature branch, its purpose is to finely depict the local shape information of the cell nucleus through the graph neural network. The specific steps are as follows:

[0079] S5.1. Sample the contour points of the cell nucleus based on the instance segmentation results, construct the node features of the contour points, and obtain the contour point feature matrix of each cell nucleus.

[0080] Specifically, from the outline of the cell nucleus, n 360-degree angles are evenly divided outward from the center of the cell nucleus. ° / n angle area, select a contour point in each area, the node feature of each contour point is a 1×8 vector, and the node feature includes basic geometric features and local image features; among them, the basic geometric features include normalized coordinates, that is, the coordinate offset of the contour point relative to the center of the cell nucleus ( x , y ); the curvature feature represents the rate of change of the angles of the adjacent points before and after the contour point, thus reflecting the curvature of the local contour; the direction vector describes the tangent direction between the contour point and the previous sampled contour point, in the form of a unit vector; the local image features include RGB color information, which is obtained by calculating the mean or standard deviation of the pixels in a small 2×2 area around the contour point; the texture feature is calculated using the local binary pattern (LBP) to enhance the expression of edge details. The features of all contour points can be represented as matrices X ∈ R N×8 ,in N Represents the number of contour points, and each row corresponds to the 1×8-dimensional node feature of a contour point.

[0081] S5.2. Input the contour point feature matrix into the improved graph attention network (GAT) to self-connect each contour point and obtain the self-connection feature of each contour point.

[0082] More specifically, the improved GAT working process is:

[0083] First, the contour point feature matrix is ​​dimensionally increased through the fully connected layer:

[0084] X ’ = ELU ( W fc · X ),

[0085] in, X ’ is the contour point feature matrix after dimension increase, ELU is the exponential linear unit activation function, W fc ∈ R (n )×(H×h) is the weight matrix of the fully connected layer, H represents the hidden layer dimension of each attention head, h represents the number of attention heads, n Represents the input contour point feature matrix X dimension.

[0086] Next, use the GATConv layer that includes a self-connection mechanism to perform two self-connections on each contour point to obtain the self-connection features of each contour point:

[0087] ,

[0088] in, x i For the iThe node features of the contour points, For the i The set of neighbor nodes of the contour points, α ij is the weight obtained by attention calculation, x j For the j The node features of neighbor nodes, β is the self-connection weight decay factor, FC Represents a linear map.

[0089] S5.3. Perform an attention pooling operation on the self-connected features of each contour point, and globally aggregate the contour points of each cell nucleus to obtain the shape features of each cell nucleus. The attention pooling calculation method is:

[0090] α i = softmax ( W att · x i ), ,

[0091] in, W att ∈ R (out_dim)×1 Calculate the matrix for attention weights, z The shape characteristics of the cell nucleus after aggregation f 3.

[0092] S6. Send the deep features, manual feature vectors and shape features based on contour points into the feature gated fusion mechanism for fusion to obtain a fusion vector.

[0093] In order to achieve efficient fusion of three-branch features, this paper introduces a feature gating fusion mechanism to combine deep features f 1. Manual feature vector f 2 and shape features based on contour points f 3. The feature is fed into the gating fusion mechanism for fusion. During fusion, the gating function is used to dynamically adjust the importance of each feature. The fusion formula is:

[0094] ,

[0095] in, F is the fusion vector, σ () is the activation function (such as Sigmoid function), W f is the weight matrix, f 1 is the depth feature, f 2 is a manual feature vector, f3 is the shape feature based on contour points, b f is the weight bias matrix, For Hadamard, f 1; f 2; f 3] is a joint feature vector composed of deep features, manual feature vectors and shape features based on contour points.

[0096] S7. Finally, the fusion vector is input into the classification output module for classification and the classification result is output.

[0097] In this embodiment, the classification output module is composed of several fully connected layers and a Softmax layer, which ultimately outputs the classification probability.

[0098] Furthermore, to address the class imbalance problem in the dataset, the shape-guided three-branch fusion network of the present invention uses the Focal Loss loss function during training. Its purpose is to dynamically scale the loss, reduce the loss contribution of easily classified samples, and enhance the weight of difficult-to-classify samples. The definition formula of Focal Loss is:

[0099] L Focal = α t (1- p t ) γ (- log ( p t )),

[0100] in, α t is the category weight, which is used to balance the category imbalance problem; p t For the real category t The predicted probability of γ The training process of the shape-guided three-branch fusion network in the present invention is as follows: Figure 2 As shown, the conventional training method is used, and only the Focal Loss loss function is used to calculate the loss, so this application does not go into details about the training process.

[0101] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0102] Based on the same concept as the graph neural network-based pathology image cell nucleus classification method in the above-mentioned embodiment, the present invention also provides a graph neural network-based pathology image cell nucleus classification system, which can be used to implement the above-mentioned graph neural network-based pathology image cell nucleus classification method. For ease of explanation, the structural diagram of the embodiment of the graph neural network-based pathology image cell nucleus classification system only shows the parts related to the embodiment of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation of the device, and it may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0103] like Figure 3 As shown, another embodiment of the present invention provides a pathological image cell nucleus classification system based on a graph neural network, including a data acquisition module, a network construction module, a deep feature extraction module, a manual feature extraction module, a shape feature extraction module, a gated fusion module and a classification result module;

[0104] The data acquisition module is used to obtain pathological tissue images containing cell nuclei and their instance segmentation results;

[0105] The network construction module is used to construct a shape-guided three-branch fusion network, including a deep feature branch, a manual feature branch, a shape feature branch, a feature gating mechanism, and a classification output module;

[0106] The deep feature extraction module is used to input the pathological tissue image containing cell nuclei into the deep feature branch to extract global features to obtain pixel-level feature maps and obtain the deep features of each cell nucleus;

[0107] The manual feature extraction module is used to use the manual feature branch to extract the morphological, color and texture features of the cell nucleus from the instance segmentation results to obtain manual features and then perform dimension upgrade to obtain a manual feature vector;

[0108] The shape feature module is used to extract shape features based on contour points from instance segmentation results using the shape feature branch;

[0109] The gated fusion module is used to feed deep features, manual feature vectors, and contour point-based shape features into the feature gated fusion mechanism for fusion to obtain a fusion vector.

[0110] The classification result module is used to input the fusion vector into the classification output module for classification and output the classification result.

[0111] It should be noted that the pathological image cell nucleus classification system based on graph neural network of the present invention corresponds one-to-one to the pathological image cell nucleus classification method based on graph neural network of the present invention. The technical features and beneficial effects described in the above-mentioned embodiment of the pathological image cell nucleus classification method based on graph neural network are applicable to the embodiment of the pathological image cell nucleus classification system based on graph neural network. For specific contents, please refer to the description in the embodiment of the method of the present invention. No further details will be given here. This is hereby declared.

[0112] In addition, in the implementation of the graph neural network-based pathological image cell nucleus classification system in the above embodiment, the logical division of each program module is only an example. In actual application, the above functions can be assigned to different program modules as needed, for example, for the configuration requirements of the corresponding hardware or the convenience of software implementation. That is, the internal structure of the graph neural network-based pathological image cell nucleus classification system is divided into different program modules to complete all or part of the functions described above.

[0113] like Figure 4 As shown, in one embodiment, a computer-readable storage medium is provided, which stores a program in a memory. When the program is executed by a processor, the method for classifying cell nuclei in pathological images based on graph neural networks is implemented, specifically:

[0114] Obtain pathological tissue images containing cell nuclei and their instance segmentation results;

[0115] Construct a shape-guided three-branch fusion network, including a deep feature branch, a manual feature branch, a shape feature branch, a feature gating mechanism, and a classification output module;

[0116] The pathological tissue image containing the cell nucleus is input into the deep feature branch to extract the global features to obtain the pixel-level feature map and obtain the deep features of each cell nucleus;

[0117] Use the manual feature branch to extract the morphology, color and texture features of the cell nucleus from the instance segmentation results to obtain manual features and then upgrade the dimension to obtain the manual feature vector;

[0118] The shape feature branch is used to extract shape features based on contour points from the instance segmentation results;

[0119] The deep features, manual feature vectors and shape features based on contour points are fed into the feature gating fusion mechanism to fuse them and obtain the fusion vector;

[0120] The fusion vector is input into the classification output module for classification and the classification result is output.

[0121] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0122] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0123] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A method for cell nucleus classification in pathological images based on graph neural network, characterized in that: The steps include: Obtain pathological tissue images containing cell nuclei and their instance segmentation results; Construct a shape-guided three-branch fusion network, including a deep feature branch, a manual feature branch, a shape feature branch, a feature gating mechanism, and a classification output module; The pathological tissue image containing the cell nucleus is input into the deep feature branch to extract the global features to obtain the pixel-level feature map and obtain the deep features of each cell nucleus; Use the manual feature branch to extract the morphology, color and texture features of the cell nucleus from the instance segmentation results to obtain manual features and then upgrade the dimension to obtain the manual feature vector; The manual features are obtained and the dimension is increased to obtain the manual feature vector, specifically: Extract m-dimensional manual features from pathological tissue images containing cell nuclei using instance segmentation results, including morphological features, color features, and texture features; the morphological features include cell nucleus area, cell nucleus perimeter, and cell nucleus roundness; the color features include cell nucleus color entropy and cell nucleus color moment; the texture features include cell nucleus edge color contrast and contrast, correlation, energy, homogeneity, and ASM corresponding to the gray-level co-occurrence matrix; The morphological features, color features, and texture features are combined to form a 1×m vector, which is then passed through several fully connected layers for nonlinear mapping and dimension increase to obtain a handcrafted feature vector. The shape feature branch is used to extract shape features based on contour points from the instance segmentation results, specifically: Based on the instance segmentation results, the cell nucleus contour points are sampled, the node features of the contour points are constructed, and the contour point feature matrix of each cell nucleus is obtained; Input the contour point feature matrix into the improved graph attention network to self-connect each contour point and obtain the self-connection feature of each contour point; Perform attention pooling on the self-connected features of each contour point, and globally aggregate the contour points of each cell nucleus to obtain the shape features of each cell nucleus; The deep features, manual feature vectors, and contour point-based shape features are fed into the feature gating fusion mechanism for fusion to obtain a fusion vector. During fusion, the gating function is used to dynamically adjust the importance of each feature. The process is described as follows: ， in, F is the fusion vector, σ () is the activation function, W f is the weight matrix, f 1 is the depth feature, f 2 is a manual feature vector, f 3 is the shape feature based on contour points, b f is the weight bias matrix, For Hadamard, f 1; f 2; f 3] is a joint feature vector composed of deep features, manual feature vectors, and shape features based on contour points; The fusion vector is input into the classification output module for classification and the classification result is output.

2. The method for pathological image cell nucleus classification based on graph neural network according to claim 1, characterized in that: The pixel-level feature map is obtained and the depth feature of each cell nucleus is obtained, specifically: A pre-trained encoder is used to perform global feature encoding on the input pathological tissue image containing cell nuclei to obtain a pixel-level feature map. Combined with the nucleus bounding box information provided by the instance segmentation ground truth, the ROI Align technique is used to extract the regional features corresponding to each nucleus from the pixel-level feature map. The regional features are then unified into a fixed dimension through a pooling operation to obtain the deep features of each nucleus. The pre-trained encoder does not contain the final fully connected layer.

3. The method for pathological image cell nucleus classification based on graph neural network according to claim 1, characterized in that: The cell nucleus contour point sampling and the construction of the node features of the contour points are specifically as follows: From the nucleus contour of the instance segmentation result, n 360-degree circles are evenly divided outward according to the center of the nucleus. ° / n angular area; Select a contour point in each angle region; Construct node features for each contour point, including basic geometric features and local image features; The basic geometric features include normalized coordinates, curvature features and direction vectors; the local image features include RGB color information and texture features; The node features of all contour points of each cell nucleus are represented as a matrix to obtain the contour point feature matrix of each cell nucleus.

4. The method for pathological image cell nucleus classification based on graph neural network according to claim 1, characterized in that: The self-connection feature of each contour point is obtained as follows: The contour point feature matrix is ​​increased in dimension through the fully connected layer: X ’ = ELU ( W fc · X )， in, X ’ is the contour point feature matrix after dimension increase, ELU is the exponential linear unit activation function, W fc ∈ R (n)×(H×h) is the weight matrix of the fully connected layer, H represents the hidden layer dimension of each attention head, h represents the number of attention heads, n Represents the input contour point feature matrix X Dimensions; Use the GATConv layer with a self-connection mechanism to perform two self-connections on each contour point to obtain the self-connection features of each contour point: ， in, x i For the i The node features of the contour points, Indicates the i The set of neighbor nodes of the contour points, α ij is the weight obtained by attention calculation, x j For the j The node characteristics of neighbor nodes, β is the self-connection weight decay factor, FC Represents a linear map.

5. The method for pathological image cell nucleus classification based on graph neural network according to claim 1, characterized in that: The shape-guided three-branch fusion network is obtained through training, and the Focal Loss function is used during training, which is expressed as: L Focal = α t (1 - p t ) γ (- log ( p t ))， in, α t is the category weight, p t For the real category t The predicted probability of γ is the regulating factor.

6. A pathological image cell nucleus classification system based on graph neural network, characterized by: The method for classifying cell nuclei in pathological images based on a graph neural network as claimed in any one of claims 1 to 5 comprises a data acquisition module, a network construction module, a deep feature extraction module, a manual feature extraction module, a shape feature extraction module, a gated fusion module and a classification result module; The data acquisition module is used to acquire pathological tissue images containing cell nuclei and instance segmentation results thereof; The network construction module is used to construct a shape-guided three-branch fusion network, including a deep feature branch, a manual feature branch, a shape feature branch, a feature gating mechanism and a classification output module; The depth feature extraction module is used to input the pathological tissue image containing the cell nucleus into the depth feature branch to extract the global feature to obtain the pixel-level feature map and obtain the depth feature of each cell nucleus; The manual feature extraction module is used to use the manual feature branch to extract the morphological, color and texture features of the cell nucleus from the instance segmentation result to obtain manual features and to perform dimension upgrade to obtain a manual feature vector; The shape feature module is used to extract shape features based on contour points from the instance segmentation results using the shape feature branch; The gated fusion module is used to send the deep features, manual feature vectors and shape features based on contour points into the feature gated fusion mechanism for fusion to obtain a fusion vector; The classification result module is used to input the fusion vector into the classification output module for classification and output the classification result.

7. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the pathological image cell nucleus classification method based on graph neural network according to any one of claims 1 to 5 is implemented.

Images