Pathological image cell nucleus classification method based on graph neural network

Abstract

The invention discloses a pathological image cell nucleus classification method based on a graph neural network. The method comprises the following steps: firstly, acquiring a pathological tissue image containing a cell nucleus and an instance segmentation result of the pathological tissue image; then, constructing a shape-guided three-branch fusion network; inputting the pathological tissue image containing the cell nucleus into the depth feature branch to extract global features to obtain a pixel-level feature map and obtain the depth feature of each cell nucleus; extracting form, color and texture features of a cell nucleus from an instance segmentation result by using a manual feature branch to obtain a manual feature, and raising dimensionality to obtain a manual feature vector; shape feature branches are adopted to extract shape features based on contour points from the instance segmentation result; finally, sending to a feature gating fusion mechanism for fusion to obtain a fusion vector; and inputting the fusion vector into a classification output module for classification, and outputting a classification result. According to the method, the shape information of the cell nucleuses is fully utilized, so that the network has better distinguishing capability on the cell nucleuses with complex forms, and the classification performance is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and particularly relates to a method for classifying pathological image cell nuclei based on a graph neural network. Background Art

[0002] Cell nucleus classification is one of the core tasks in the field of Computational Pathology and is of great significance for cancer diagnosis, disease typing, and prognosis prediction. The cell nuclei in pathological tissue sections usually have significant morphological, texture, and color differences. The appearance of different types of cell nuclei is complex, and the same type of cell nucleus may exhibit high variability in different tissue environments, which poses a great challenge to automated cell nucleus classification.

[0003] Currently, the mainstream methods for nucleus classification are mainly divided into CNN-based methods and GNN-based methods. CNN-based methods usually adopt classical network architectures such as fully convolutional network (FCN), ResNet, VGG, DenseNet, etc., and classify by extracting pixel-level texture information of the nucleus. For example, HoVer-Net (Graham S, Vu Q D, Raza S E A, etal. Hover-net: Simultaneous segmentation and classification of nucleiinmulti-tissue histology images[J]. Medical image analysis, 2019, 58:101563.) is a CNN-based joint segmentation and classification method, but HoVer-Net mainly relies on pixel-level features and does not explicitly model the shape information of the nucleus. Therefore, its classification effect on nuclei with variable shapes is limited. TSFD-Net (Ilyas T,Mannan Z I, Khan A, et al. TSFD-Net: Tissue specificfeature distillationnetwork for nuclei segmentation and classification[J].Neural Networks, 2022,151: 1-15.) adopts a two-stream feature extraction structure. One part extracts texture information based on the RGB channels, and the other part learns features based on the segmentation boundary information and combines the attention mechanism for classification. Although this method utilizes boundary information, it still mainly relies on local features extracted by CNN and cannot model the global shape structure of the nucleus. Therefore, CNN methods have some limitations. On the one hand, CNN mainly relies on pixel-level information and is difficult to model the global shape features of the nucleus, resulting in poor classification effects for nuclei with similar shapes but different categories. On the other hand, lacking shape information modeling, CNN can only learn pixel texture features and is insufficient in modeling the geometric features of the nucleus contour.

[0004] In recent years, methods based on graph neural networks (GNNs) have gradually been applied to pathological image analysis. By modeling the adjacency relationships between cell nuclei, GNNs enhance the classification model's ability to utilize the spatial distribution information of cell nuclei. For example, MCSpatNet (Abousamra S, Belinsky D, Van Arnam J, et al. Multi-class cell detection using spatial context representation[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021: 4005-4014.) proposed a method for learning multi-cell spatial representation, which learns the distribution pattern between cell nuclei through graph structure, but only focuses on the interaction relationships between cell nuclei and does not explicitly model the shape features of individual cell nuclei. SENC (Lou W, Wan X, Li G, et al. Structure embedded nucleus classification for histopathology images[J]. IEEE Transactions on Medical Imaging, 2024.) combines polygon structure feature learning (PSL) and GNN-based spatial relationship modeling of cell nuclei in an end-to-end framework to simultaneously utilize the local shape features of cell nuclei and the spatial relationships between cell nuclei to improve classification performance. Although it depicts the overall change trend of the cell nucleus shape to a certain extent, since the RNN in PSL is mainly used to model temporal information, its ability to learn local geometric features is limited and it cannot fully model the shape details of cell nuclei. In addition, edge features between cell nuclei are introduced in the GNN structure, but this feature is mainly based on the background texture information between adjacent cell nuclei and does not further model the geometric relationships between the contour points of cell nuclei, resulting in information loss when the model captures the morphological changes of cell nuclei and affecting the classification performance. Summary of the Invention

[0005] The main objective of the present invention is to overcome the drawbacks and deficiencies of the prior art, and provide a method for classifying cell nuclei in pathological images based on graph neural networks. Feature extraction is performed based on three information paths: deep features, manual features, and shape features, and fusion is carried out through a feature gating fusion mechanism to fully utilize the multi-dimensional information of cell nuclei and improve the classification performance.

[0006] To achieve the above objective, the present invention adopts the following technical solutions: The first object of the present invention is to provide a method for classifying cell nuclei in pathological images based on graph neural networks, including the following steps: Obtain a pathological tissue image containing cell nuclei and its instance segmentation result; Construct a shape-guided three-branch fusion network, including a deep feature branch, a handcrafted feature branch, a shape feature branch, a feature gating mechanism, and a classification output module; Input the pathological tissue image containing cell nuclei into the deep feature branch to extract global features to obtain a pixel-level feature map and obtain the deep features of each cell nucleus; Use the handcrafted feature branch to extract the morphological, color, and texture features of the cell nuclei from the instance segmentation result to obtain handcrafted features and perform dimensionality elevation to obtain a handcrafted feature vector; Adopt the shape feature branch to extract shape features based on contour points from the instance segmentation result; Send the deep features, the handcrafted feature vector, and the shape features based on contour points into the feature gating fusion mechanism for fusion to obtain a fusion vector; Input the fusion vector into the classification output module for classification and output the classification result.

[0007] As a preferred technical solution, the obtaining of the pixel-level feature map and the obtaining of the deep features of each cell nucleus are specifically as follows: Adopt a pre-trained encoder to perform global feature encoding on the input pathological tissue image containing cell nuclei to obtain a pixel-level feature map; Combine the cell nucleus bounding box information provided by the instance segmentation ground truth, use the ROI Align technique to extract the region features corresponding to each cell nucleus from the pixel-level feature map, and unify the region features into a fixed dimension through a pooling operation to obtain the deep features of each cell nucleus; The pre-trained encoder does not include the last fully connected layer.

[0008] As a preferred technical solution, the obtaining of the handcrafted features and the dimensionality elevation to obtain a handcrafted feature vector are specifically as follows: Extract m-dimensional handcrafted features from the pathological tissue image containing cell nuclei by using the instance segmentation result, including morphological features, color features, and texture features; the morphological features include cell nucleus area, cell nucleus perimeter, and cell nucleus roundness; the color features include cell nucleus color entropy and cell nucleus color moment; the texture features include cell nucleus edge color contrast and contrast, correlation, energy, homogeneity, and ASM corresponding to the gray-level co-occurrence matrix; After combining the morphological features, color features, and texture features to form a 1×m vector, perform non-linear mapping and dimensionality elevation through several fully connected layers to obtain a handcrafted feature vector.

[0009] As a preferred technical solution, the shape feature branch is used to extract the shape features based on the contour points from the instance segmentation result, specifically as follows: Sample the nucleus contour points according to the instance segmentation result, construct the node features of the contour points, and obtain the contour point feature matrix of each nucleus; Input the contour point feature matrix into the improved graph attention network to perform self-connection on each contour point, and obtain the self-connection feature of each contour point; Perform attention pooling operation on the self-connection feature of each contour point, and perform global aggregation on the contour points of each nucleus to obtain the shape feature of each nucleus.

[0010] As a preferred technical solution, the sampling of the nucleus contour points and the construction of the node features of the contour points are specifically as follows: Evenly divide n 360 ° / n angular regions outward from the nucleus center point in the nucleus contour of the instance segmentation result; Select a contour point in each angular region; Construct the node features of each contour point, including basic geometric features and local image features; the basic geometric features include normalized coordinates, curvature features, and direction vectors; the local image features include RGB color information and texture features; Represent the node features of all contour points of each nucleus as a matrix to obtain the contour point feature matrix of each nucleus.

[0011] As a preferred technical solution, the obtaining of the self-connection feature of each contour point is specifically as follows: Perform dimensionality increase on the contour point feature matrix through a fully connected layer: X ’ = ELU ( W fc · X ) where X ’ is the contour point feature matrix after dimensionality increase, ELU is the exponential linear unit activation function, W fc ∈ R (n )×(H×h) is the weight matrix of the fully connected layer, H represents the hidden layer dimension of each attention head, h represents the number of attention heads, n represents the input contour point feature matrix X dimension; Use the GATConv layer with a self-connection mechanism to perform self-connection on each contour point twice to obtain the self-connection features of each contour point: , wherein, x i is the node feature of the i th contour point, represents the set of neighbor nodes of the i th contour point, α ij is the weight obtained through attention calculation, x j is the node feature of the j th neighbor node, β is the self-connection weight decay factor, FC represents a linear mapping.

[0012] As a preferred technical solution, the depth feature, the handcrafted feature vector, and the shape feature based on contour points are fed into a feature gating fusion mechanism for fusion, and a gating function is used to dynamically adjust the importance of each feature during fusion; the process is described as: , wherein, F is the fusion vector, σ () is the activation function, W f is the weight matrix, f 1 is the depth feature, f 2 is the handcrafted feature vector, f 3 is the shape feature based on contour points, b f is the weight bias matrix, is the Hadamard product, f 1; f 2; f 3] is the joint feature vector formed by concatenating the depth feature, the handcrafted feature vector, and the shape feature based on contour points.

[0013] As a preferred technical solution, the shape-guided three-branch fusion network is obtained through training. During training, the Focal Loss function is used, which is expressed as: L Focal = α t (1 - p t ) γ (- log ( p t ))), wherein, αt is the class weight, p t is the true class t predicted probability of, γ is the adjustment factor.

[0014] The second object of the present invention is to provide a pathological image nucleus classification system based on a graph neural network, which is applied to the pathological image nucleus classification method based on a graph neural network, and includes a data acquisition module, a network construction module, a deep feature extraction module, a manual feature extraction module, a shape feature extraction module, a gating fusion module, and a classification result module; The data acquisition module is used to acquire pathological tissue images containing cell nuclei and their instance segmentation results; The network construction module is used to construct a shape-guided three-branch fusion network, including a deep feature branch, a manual feature branch, a shape feature branch, a feature gating mechanism, and a classification output module; The deep feature extraction module is used to input a pathological tissue image containing cell nuclei into the deep feature branch to extract global features to obtain a pixel-level feature map and obtain the deep features of each cell nucleus; The manual feature extraction module is used to use the manual feature branch to extract the morphological, color, and texture features of the cell nuclei from the instance segmentation results to obtain manual features and perform dimension elevation to obtain manual feature vectors; The shape feature module is used to extract shape features based on contour points from the instance segmentation results using the shape feature branch; The gating fusion module is used to send the deep features, manual feature vectors, and shape features based on contour points into the feature gating fusion mechanism for fusion to obtain a fusion vector; The classification result module is used to input the fusion vector into the classification output module for classification and output the classification result.

[0015] The third object of the present invention is to provide a computer-readable storage medium storing a program, which, when executed by a processor, implements the pathological image nucleus classification method based on a graph neural network.

[0016] Compared with the prior art, the present invention has the following advantages and beneficial effects: First, the shape-guided three-branch fusion network in the present invention makes full use of the shape information of the cell nucleus through the shape feature branch, overcoming the limitation that traditional CNNs only rely on local texture features. Compared with the SENC method that uses RNN to process the contour point sequence, the present invention can directly construct the graph structure between contour points and learn the geometric relationship between nodes through the graph attention network, so as to more accurately capture local bending features, direction vectors, and contour texture information, enabling the network to have better discrimination ability for cell nuclei with complex morphologies. Secondly, the present invention adopts a feature gating fusion mechanism for multi-modal feature fusion. Different from the traditional method of directly splicing various features and inputting them into a fully connected layer, the feature gating fusion mechanism can adaptively adjust the contribution weights of different feature sources, thereby enhancing the non-linear interaction between deep features, manual features, and shape features and improving the overall classification performance of the network. In addition, aiming at the problem of class imbalance, the weighted Focal Loss function is adopted during training, effectively reducing the misclassification phenomenon of low-frequency classes and making the network perform more evenly on different classes. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0018] Figure 1 It is a flow framework diagram of the method for classifying cell nuclei in pathological images based on a graph neural network in an embodiment of the present invention.

[0019] Figure 2 It is a training process diagram of the shape-guided three-branch fusion network in an embodiment of the present invention.

[0020] Figure 3 It is a structural schematic diagram of the system for classifying cell nuclei in pathological images based on a graph neural network in an embodiment of the present invention.

[0021] Figure 4 It is a structural schematic diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] To enable those skilled in the art to better understand the solutions of the present application, the following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts fall within the scope of protection of the present application.

[0023] Reference to "embodiment" in this application means that the specific features, structures, or characteristics described in connection with the embodiment may be included in at least one embodiment of this application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0024] Current methods for classifying cell nuclei mainly rely on CNNs to extract local texture features or GNNs to model the spatial relationships between cell nuclei, but neither fully utilizes the shape features of cell nuclei. Due to the limitation of the local receptive field, CNN methods are difficult to capture the overall morphological information of cell nuclei, resulting in unstable performance of the classification model when dealing with cell nuclei with complex shapes or blurred boundaries. Although GNN methods have made progress in modeling the topological structure between cell nuclei, they lack explicit modeling of the contour morphology of cell nuclei themselves and fail to fully express the local geometric features of cell nuclei. In particular, for the SENC method, although it introduces polygon structure feature learning (PSL) + cell nucleus spatial relationship modeling (GSL), the way its RNN processes the sequence of contour points is difficult to accurately depict the local geometric changes of cell nuclei, and at the same time, the edge feature design is relatively simple and fails to effectively model the geometric relationship between contour points of cell nuclei, affecting the accuracy of the classification model.

[0025] Therefore, the objective of the present invention is to propose a method for comprehensively modeling the shape, texture, and spatial relationships of cell nuclei, breaking through the limitations of existing CNN and GNN methods in shape information modeling and class imbalance optimization. The present invention adopts a multi-branch structure and combines deep features, manual features, and learnable shape features simultaneously to enhance the model's ability to understand different feature dimensions of cell nuclei. In particular, through the graph neural network for cell nucleus shape modeling, the present invention constructs the contour points of cell nuclei into a graph structure and explicitly models the geometric, curvature, direction, and texture features of the cell nucleus shape under the GNN framework, enabling the classification model to accurately capture the local and overall morphological features of cell nuclei. In addition, to solve the problem of class imbalance, the network constructed by the present invention introduces weighted Focal Loss to optimize the learning of low-frequency classes, making the performance of the classification model more balanced across different cell nucleus classes.

[0026] Next, please refer to Figure 1 , and this embodiment elaborates in detail the steps of a method for classifying cell nuclei in pathological images based on a graph neural network, including: S1. Obtain a pathological tissue image containing cell nuclei and its instance segmentation result.

[0027] S2. Construct a shape-guided three-stream fusion network (Three-Stream Shape Fusion Network, 3SF-Net) aiming to improve the accuracy of nucleus classification tasks, including a deep feature branch, a handcrafted feature branch (HCFeature branch), a shape feature branch (Shape GAT branch), a feature gating mechanism (FG), and a classification output module.

[0028] S3. Input the pathological tissue image containing nuclei into the deep feature branch to extract global features, obtain a pixel-level feature map, and acquire the deep features of each nucleus.

[0029] For the deep feature branch, it is mainly used to capture the global context information and deep texture information of the image where the nuclei are located. The specific steps are as follows: S3.1. First, use a pre-trained encoder (excluding the last fully connected layer) to perform global feature encoding on the input pathological tissue image containing nuclei to obtain a pixel-level feature map. S3.2. Subsequently, combine the nucleus bounding box information (bounding box) provided by the instance segmentation ground truth, use the ROI Align (Region of Interest Align) technique to extract the region features corresponding to each nucleus from the pixel-level feature map, and unify the region features into a fixed dimension through a pooling operation to obtain the deep features of each nucleus. f 1.

[0030] In this embodiment, the pre-trained encoder uses a Swin Transformer encoder.

[0031] S4. Use the handcrafted feature branch to extract the morphological, color, and texture features of the nuclei from the instance segmentation results, obtain handcrafted features, and perform dimensionality increase to obtain handcrafted feature vectors.

[0032] For the handcrafted feature branch (HC Feature branch), this branch can make full use of the traditional features of the nuclei, which helps to make up for the deficiency of deep features in fine-grained morphological expression. The specific steps are as follows: S4.1. Use the instance segmentation results to extract m-dimensional handcrafted features from the pathological tissue image containing nuclei, including morphological features, color features, and texture features. Among them, the morphological features include nucleus area, nucleus perimeter, nucleus roundness, etc.; the color features include nucleus color entropy and nucleus color moments (calculate the mean, variance, and skewness for each of the L, B, and P channels respectively); the texture features include nucleus edge color contrast and contrast, correlation, energy, homogeneity, and ASM corresponding to the gray-level co-occurrence matrix (GLCM). S4.2. Combine the morphological features, color features, and texture features to form a 1×m vector, which undergoes non-linear mapping and dimensionality increase through several fully connected layers to obtain a handcrafted feature vector. f 2.

[0033] In this embodiment, 19-dimensional handcrafted features are extracted and combined to form a 1×19 vector.

[0034] S5. Use the shape feature branch to extract shape features based on contour points from the instance segmentation results.

[0035] For the shape feature branch, its purpose is to finely depict the local shape information of the cell nucleus through a graph neural network. The specific steps are as follows: S5.1. Sample the contour points of the cell nucleus according to the instance segmentation results, construct the node features of the contour points, and obtain the contour point feature matrix of each cell nucleus.

[0036] Specifically, from the cell nucleus contour, n 360 ° / n angular regions are evenly divided outward from the cell nucleus center point. One contour point is selected in each region. The node feature of each contour point is a 1×8 vector, and the node feature includes basic geometric features and local image features. Among them, the basic geometric features include normalized coordinates, that is, the coordinate offset of the contour point relative to the cell nucleus center ( x , y ); the curvature feature represents the angular change rate of the adjacent points before and after this contour point, thus reflecting the bending degree of the local contour; the direction vector describes the tangent direction between this contour point and the previous sampled contour point, in the form of a unit vector; the local image features include RGB color information, and the color features are obtained by calculating the mean or standard deviation of the pixels in a 2×2 small area around this contour point; the texture feature is calculated using local binary pattern (LBP) to enhance the expression ability of edge details. The features of all contour points can be represented as a matrix X ∈ R N×8 , where N represents the number of contour points, and each row corresponds to the 1×8-dimensional node feature of a contour point.

[0037] S5.2. Input the contour point feature matrix into the improved graph attention network (GAT) to perform self-connection on each contour point and obtain the self-connection feature of each contour point.

[0038] More specifically, the working process of the improved GAT is as follows: First, perform dimensionality increase on the contour point feature matrix through a fully connected layer: X ’ =ELU ( W fc · X ), Among them, X ’ is the contour point feature matrix after dimension elevation, ELU is the exponential linear unit activation function, W fc ∈ R (n )×(H×h) is the weight matrix of the fully connected layer, H represents the hidden layer dimension of each attention head, h represents the number of attention heads, n represents the input contour point feature matrix X dimension.

[0039] Next, use the GATConv layer with self-connection mechanism to perform self-connection on each contour point twice to obtain the self-connection feature of each contour point: , Among them, x i is the node feature of the i th contour point, is the set of neighbor nodes of the i th contour point, α ij is the weight obtained through attention calculation, x j is the node feature of the j th neighbor node, β is the self-connection weight decay factor, FC represents the linear mapping.

[0040] S5.3. Perform attention pooling operation on the self-connection feature of each contour point, and perform global aggregation on the contour points of each cell nucleus to obtain the shape feature of each cell nucleus. The attention pooling calculation method is: α i = softmax ( W att · x i ), , Among them, W att ∈ R (out_dim)×1 is the attention weight calculation matrix, z is the aggregated cell nucleus shape feature f 3.

[0041] S6. Feed the depth features, the handcrafted feature vectors, and the shape features based on contour points into the feature gating fusion mechanism for fusion to obtain a fused vector.

[0042] To achieve efficient fusion of the three-branch features, the present invention introduces a feature gating fusion mechanism, which feeds the depth features f 1. Handcrafted feature vectors f 2 and the shape features based on contour points f 3 into the feature gating fusion mechanism for fusion. When fusing, a gating function is used to dynamically adjust the importance of each feature; the fusion formula is: , where, F is the fused vector, σ () is an activation function (such as the Sigmoid function), W f is the weight matrix, f 1 is the depth feature, f 2 is the handcrafted feature vector, f 3 is the shape feature based on contour points, b f is the weight bias matrix, is the Hadamard product, f 1; f 2; f 3] is the joint feature vector concatenated by the depth feature, the handcrafted feature vector, and the shape feature based on contour points.

[0043] S7. Finally, input the fused vector into the classification output module for classification and output the classification result.

[0044] In this embodiment, the classification output module is several fully connected layers and a Softmax layer, and finally the classification probability is output.

[0045] Furthermore, to solve the problem of class imbalance in the dataset, the shape-guided three-branch fusion network of the present invention adopts the Focal Loss function during training, and its purpose is to dynamically scale the loss, reduce the loss contribution of easily classified samples, and at the same time enhance the weight of difficult-to-classify samples. The definition formula of Focal Loss is: L Focal = α t (1 - p t ) γ (- log ( p t ))), where, αt is the class weight, used to balance the class imbalance problem; p t is the true class t of the predicted probability, γ is the adjustment factor. The training process of the shape-guided three-branch fusion network in the present invention is as Figure 2 shown. Using the conventional training method, only the Focal Loss function is used to calculate the loss during calculation. Therefore, the training process is not elaborated in this application.

[0046] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be carried out in other sequences or simultaneously.

[0047] Based on the same idea as the method for classifying pathological image cell nuclei based on a graph neural network in the above embodiment, the present invention also provides a system for classifying pathological image cell nuclei based on a graph neural network. This system can be used to execute the above method for classifying pathological image cell nuclei based on a graph neural network. For the sake of convenience of description, in the structural schematic diagram of the embodiment of the system for classifying pathological image cell nuclei based on a graph neural network, only the parts related to the embodiment of the present invention are shown. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and it may include more or fewer components than those illustrated, or combine certain components, or arrange different components.

[0048] As Figure 3 shown, another embodiment of the present invention provides a system for classifying pathological image cell nuclei based on a graph neural network, including a data acquisition module, a network construction module, a deep feature extraction module, a manual feature extraction module, a shape feature extraction module, a gated fusion module, and a classification result module; Among them, the data acquisition module is used to acquire pathological tissue images containing cell nuclei and their instance segmentation results; The network construction module is used to construct a shape-guided three-branch fusion network, including a deep feature branch, a manual feature branch, a shape feature branch, a feature gating mechanism, and a classification output module; The deep feature extraction module is used to input the pathological tissue image containing cell nuclei into the deep feature branch to extract global features to obtain a pixel-level feature map and obtain the deep features of each cell nucleus; The manual feature extraction module is used to use the manual feature branch to extract the morphological, color, and texture features of the cell nuclei from the instance segmentation results to obtain manual features and dimension up to obtain a manual feature vector; The shape feature module is used to extract shape features based on contour points from the instance segmentation results using the shape feature branch; The gated fusion module is used to send the depth features, the handcrafted feature vectors, and the shape features based on contour points into the feature gated fusion mechanism for fusion to obtain a fusion vector; The classification result module is used to input the fusion vector into the classification output module for classification and output the classification result.

[0049] It should be noted that the pathological image nucleus classification system based on graph neural network of the present invention corresponds one-to-one with the pathological image nucleus classification method based on graph neural network of the present invention. The technical features and their beneficial effects described in the embodiments of the above-mentioned pathological image nucleus classification method based on graph neural network are applicable to the embodiments of the pathological image nucleus classification system based on graph neural network. For the specific content, reference can be made to the description in the method embodiments of the present invention, which will not be elaborated here, and this is hereby declared.

[0050] In addition, in the implementation manner of the pathological image nucleus classification system based on graph neural network in the above embodiments, the logical division of each program module is only an example. In practical applications, according to needs, for example, considering the configuration requirements of the corresponding hardware or the convenience of software implementation, the above functions can be assigned to different program modules to complete, that is, the internal structure of the pathological image nucleus classification system based on graph neural network is divided into different program modules to complete all or part of the functions described above.

[0051] As Figure 4 shown, in one embodiment, a computer-readable storage medium is provided, which stores a program in a memory. When the program is executed by a processor, the pathological image nucleus classification method based on graph neural network described above is implemented, specifically: Obtain a pathological tissue image containing cell nuclei and its instance segmentation result; Construct a shape-guided three-branch fusion network, including a depth feature branch, a handcrafted feature branch, a shape feature branch, a feature gating mechanism, and a classification output module; Input the pathological tissue image containing cell nuclei into the depth feature branch to extract global features to obtain a pixel-level feature map and obtain the depth features of each cell nucleus; Use the handcrafted feature branch to extract the morphological, color, and texture features of the cell nuclei from the instance segmentation result to obtain handcrafted features and perform dimension elevation to obtain handcrafted feature vectors; Adopt the shape feature branch to extract the shape features based on contour points from the instance segmentation result; Send the depth features, the handcrafted feature vectors, and the shape features based on contour points into the feature gating fusion mechanism for fusion to obtain a fusion vector; Input the fusion vector into the classification output module for classification and output the classification result.

[0052] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in this application can include non-volatile and / or volatile memories. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0053] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0054] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and are all included in the protection scope of the present invention.

Claims

1. A method for classifying cell nuclei in pathological images based on graph neural networks, characterized in that, It includes the following steps: Obtain a pathological tissue image containing cell nuclei and its instance segmentation result; Construct a shape-guided three-branch fusion network, including a depth feature branch, a manual feature branch, a shape feature branch, a feature gating mechanism, and a classification output module; Input the pathological tissue image containing cell nuclei into the depth feature branch to extract global features to obtain a pixel-level feature map and obtain the depth features of each cell nucleus; Use the manual feature branch to extract the morphological, color, and texture features of the cell nuclei from the instance segmentation result to obtain manual features and perform dimensionality increase to obtain a manual feature vector; Adopt the shape feature branch to extract the shape features based on contour points from the instance segmentation result; Send the depth features, manual feature vector, and shape features based on contour points into the feature gating fusion mechanism for fusion to obtain a fusion vector; Input the fusion vector into the classification output module for classification and output the classification result.

2. The method for classifying cell nuclei in pathological images based on a graph neural network according to claim 1, wherein The specific process of obtaining the pixel-level feature map and obtaining the depth features of each cell nucleus is as follows: Use a pre-trained encoder to perform global feature encoding on the input pathological tissue image containing cell nuclei to obtain a pixel-level feature map; Combine the cell nucleus bounding box information provided by the instance segmentation ground truth, use the ROI Align technique to extract the region features corresponding to each cell nucleus from the pixel-level feature map, and unify the region features into a fixed dimension through a pooling operation to obtain the depth features of each cell nucleus; The pre-trained encoder does not include the last fully connected layer.

3. The method for classifying cell nuclei in pathological images based on a graph neural network according to claim 1, wherein The specific process of obtaining the manual features and performing dimensionality increase to obtain a manual feature vector is as follows: Use the instance segmentation result to extract m-dimensional manual features from the pathological tissue image containing cell nuclei, including morphological features, color features, and texture features; the morphological features include cell nucleus area, cell nucleus perimeter, and cell nucleus roundness; the color features include cell nucleus color entropy and cell nucleus color moment; the texture features include cell nucleus edge color contrast and contrast, correlation, energy, homogeneity, and ASM corresponding to the gray-level co-occurrence matrix; Combine the morphological features, color features, and texture features to form a 1×m vector, and perform non-linear mapping and dimensionality increase through several fully connected layers to obtain a manual feature vector.

4. The method for classifying cell nuclei in pathological images based on a graph neural network according to claim 1, wherein The specific process of adopting the shape feature branch to extract the shape features based on contour points from the instance segmentation result is as follows: Sample the cell nucleus contour points according to the instance segmentation result, construct the node features of the contour points, and obtain the contour point feature matrix of each cell nucleus; Input the contour point feature matrix into an improved graph attention network to perform self-connection on each contour point to obtain the self-connection feature of each contour point; Perform an attention pooling operation on the self-connection feature of each contour point, and perform global aggregation on the contour points of each cell nucleus to obtain the shape feature of each cell nucleus.

5. The method for classifying cell nuclei in pathological images based on a graph neural network according to claim 4, wherein, The specific process of sampling the cell nucleus contour points and constructing the node features of the contour points is as follows: Evenly divide n angular regions of 360 / n from the center points of the cell nuclei in the cell nucleus contours of the instance segmentation results ° / n outward; Select a contour point in each angular region; Construct the node features of each contour point, including basic geometric features and local image features; The basic geometric features include normalized coordinates, curvature features, and direction vectors; the local image features include RGB color information and texture features; Represent the node features of all contour points of each cell nucleus as a matrix to obtain the contour point feature matrix of each cell nucleus.

6. The method for classifying cell nuclei in pathological images based on a graph neural network according to claim 4, wherein The specific method for obtaining the self-connection feature of each contour point is as follows: Increase the dimension of the contour point feature matrix through a fully connected layer: X ’ = ELU ( W fc · X )， Among them, X ’ is the contour point feature matrix after dimensionality elevation, ELU is the exponential linear unit activation function, W fc ∈ R (n)×(H×h) is the weight matrix of the fully connected layer, H represents the hidden layer dimension of each attention head, h represents the number of attention heads, n represents the input contour point feature matrix X dimension; Use the GATConv layer with a self-connection mechanism to perform self-connection on each contour point twice to obtain the self-connection feature of each contour point: ， Among them, x i is the node feature of the i th contour point, represents the set of neighbor nodes of the i th contour point, α ij is the weight obtained by attention calculation, x j is the node feature of the j th neighbor node, β is the self-connection weight decay factor, FC represents a linear mapping.

7. The method for classifying cell nuclei in pathological images based on a graph neural network according to claim 1, characterized in that, Send the depth feature, the manual feature vector, and the shape feature based on the contour points into the feature gating fusion mechanism for fusion, and use a gating function to dynamically adjust the importance of each feature during fusion; the process is described as: ， Among them, F is the fusion vector, σ () is the activation function, W f is the weight matrix, f 1 is the depth feature, f 2 is the handcrafted feature vector, f 3 is the shape feature based on contour points, b f is the weight bias matrix, is the Hadamard product, f 1; f 2; f 3] is the joint feature vector formed by concatenating the depth feature, the handcrafted feature vector, and the shape feature based on contour points.

8. The method for classifying cell nuclei in pathological images based on a graph neural network according to claim 1, wherein The shape-guided three-branch fusion network is obtained through training. During training, the Focal Loss function is used, which is expressed as: L Focal = α t (1 - p t ) γ (- log ( p t ))， Among them, α t is the category weight, p t is the true category t of the predicted probability, γ is the adjustment factor.

9. A pathological image nucleus classification system based on a graph neural network, characterized in that, Applied to the graph neural network-based pathological image cell nucleus classification method described in any one of claims 1-8, including a data acquisition module, a network construction module, a depth feature extraction module, a manual feature extraction module, a shape feature extraction module, a gating fusion module, and a classification result module; The data acquisition module is used to acquire pathological tissue images containing cell nuclei and their instance segmentation results; The network construction module is used to construct a shape-guided three-branch fusion network, including a depth feature branch, a manual feature branch, a shape feature branch, a feature gating mechanism, and a classification output module; The depth feature extraction module is used to input the pathological tissue image containing cell nuclei into the depth feature branch to extract global features to obtain a pixel-level feature map and obtain the depth feature of each cell nucleus; The manual feature extraction module is used to use the manual feature branch to extract the morphological, color, and texture features of the cell nucleus from the instance segmentation result to obtain manual features and increase the dimension to obtain a manual feature vector; The shape feature module is used to use the shape feature branch to extract the shape feature based on the contour points from the instance segmentation result; The gating fusion module is used to send the depth feature, the manual feature vector, and the shape feature based on the contour points into the feature gating fusion mechanism for fusion to obtain a fusion vector; The classification result module is used to input the fusion vector into the classification output module for classification and output the classification result.

10. A computer-readable storage medium storing a program, characterized in that, When the program is executed by a processor, it implements the graph neural network-based pathological image cell nucleus classification method described in any one of claims 1-8.

Images