Pathological image segmentation method and system based on graph neural network and cross-region fusion

Abstract

The application provides a pathological image segmentation method and system based on a graph neural network and cross-region fusion, and relates to the technical field of medical image processing.The method comprises the following steps: acquiring a pathological image and performing image region division to obtain a plurality of image regions; using a cross-attention mechanism to perform local cross-fusion on the image regions to obtain local cross-fusion features; using a graph attention network to model the local cross-fusion features to obtain inter-region graph attention features; extracting convolution features of the image through an encoder, and combining a self-attention mechanism and a graph isomorphism network to perform global structure learning to obtain graph isomorphism network features; fusing the above features to obtain global and local interaction features; and finally inputting the global and local interaction features and the convolution features into a decoder to output a segmentation result.The application effectively solves the segmentation problem of HE-stained pathological images caused by staining heterogeneity and morphological diversity, and significantly improves the segmentation accuracy and robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, and in particular to a pathological image segmentation method and system based on a graph neural network and cross-region fusion. BACKGROUND

[0002] Histopathological examination is an important link in clinical diagnosis, especially in the qualitative, grading and prognosis evaluation of diseases such as cancer. Hematoxylin and Eosin (HE) staining is one of the most commonly used pathological staining techniques, which can clearly show the organization structure of cell nucleus, cytoplasm and interstitial. However, due to the dependence of HE staining process on manual operation, it is affected by many factors such as staining time, reagent concentration, and washing degree, resulting in significant staining heterogeneity in different regions of the same tissue, different sections, and even the same section. This heterogeneity is manifested as over-dyeing or under-dyeing, which makes it difficult to distinguish normal cells from cancer cells, blood vessel structures from background noise at the pixel level, and seriously affects the accuracy of automatic segmentation based on traditional image processing methods.

[0003] In addition, biological tissues themselves have high morphological diversity and structural complexity. Even the same type of lesion (such as lung cancer) may exhibit different cell arrangement, blood vessel morphology and interstitial distribution in different patients. Necrotic, inflammatory and normal regions often coexist in tissues, with blurred boundaries between these regions and lack of clear pixel-level boundaries, further increasing the difficulty of structure segmentation. Traditional segmentation methods based on pixel intensity or fixed shape templates (such as circular, tubular structures) are difficult to adapt to such unstructured image data, resulting in low accuracy and poor generalization ability of the segmentation results.

[0004] In recent years, with the development of deep learning technology, especially the application of convolutional neural networks (CNN) and graph neural networks (GNN) in image segmentation, some studies have attempted to improve the understanding of complex pathological images by introducing attention mechanisms, graph structure modeling, etc. However, existing methods still have the following limitations:

[0005] 1. Most methods rely only on convolution operations to extract local features, making it difficult to effectively model long-distance pixel dependencies and inter-regional semantic relationships;

[0006] 2. Although graph neural networks can model structural information, they have limitations in graph isomorphism discrimination and graph structure expression ability;

[0007] 3. Existing fusion mechanisms are mostly limited to local or global single scale, lacking effective cross-scale feature interaction mechanisms;

[0008] 4. When facing staining heterogeneity and morphological diversity, the robustness and generalization ability of the model still need to be improved.

[0009] Therefore, there is an urgent need for an automated pathological image segmentation system that can effectively fuse local details and global context, has strong structural discrimination ability, and can adapt to staining and morphological changes, in order to improve the accuracy and efficiency of pathological detection. SUMMARY

[0010] To this end, the embodiments of the present application provide a pathological image segmentation method and system based on graph neural network and cross-region fusion, to solve the problems of inaccurate HE staining pathological image structure segmentation and poor generalization ability caused by staining heterogeneity and tissue morphological complexity in the prior art.

[0011] To solve the above technical problems, the embodiments of the present application provide a pathological image segmentation method based on graph neural network and cross-region fusion, which comprises:

[0012] Obtaining a pathological image to be segmented;

[0013] Dividing the pathological image into image regions to obtain a plurality of image regions;

[0014] Performing local cross-fusion on the plurality of image regions to obtain local cross-fusion features of each image region;

[0015] Modeling the inter-region relationship of the local cross-fusion features based on a graph attention mechanism to obtain inter-region graph attention features;

[0016] Encoding the pathological image to extract convolutional features;

[0017] Modeling the inter-pixel dependency relationship of the convolutional features through a self-attention mechanism, and learning the global structure of the modeled features based on a graph isomorphism network to obtain graph isomorphism network features;

[0018] Fusing the inter-region graph attention features and the graph isomorphism network features to obtain global and local interaction features;

[0019] Decoding the global and local interaction features and the convolutional features to output a final segmentation result.

[0020] Preferably, the dividing the pathological image into image regions to obtain a plurality of image regions specifically comprises:

[0021] Performing superpixel segmentation on the pathological image to obtain a superpixel segmentation map, wherein each superpixel region is an image region.

[0022] Preferably, the performing local cross-fusion on the plurality of image regions to obtain local cross-fusion features of each image region specifically comprises:

[0023] For each image region, feature interaction is performed with at least one other image region through cross-attention mechanism to calculate the inter-region dependency;

[0024] The features after interaction are merged in the channel and calculated through a forward neural network to obtain the local cross-fusion feature.

[0025] Preferably, the local cross-fusion feature is modeled based on a graph attention mechanism, specifically including:

[0026] The local cross-fusion feature of each image region is taken as a node in the graph;

[0027] The attention weight between nodes is dynamically calculated by using a graph attention network, and the neighbor node features are aggregated to update the feature representation of each node to obtain the inter-region graph attention feature.

[0028] Preferably, the inter-pixel dependency of the convolutional feature is modeled through a self-attention mechanism, specifically including:

[0029] The convolutional feature is flattened into a matrix form;

[0030] The self-attention weight between pixels is calculated by using a learnable query matrix, key matrix and value matrix;

[0031] The pixel features are weighted and fused according to the self-attention weight to obtain the inter-pixel self-attention feature.

[0032] Preferably, the global structure learning of the modeled feature is performed based on a graph isomorphism network, specifically including:

[0033] The self-attention feature of each pixel is taken as the initial feature of a node in the graph;

[0034] For each node, the features of its neighbor nodes are aggregated layer by layer, and the aggregation result is merged with the feature of itself;

[0035] The merged feature is transformed by using a multi-layer perceptron to update the node feature to obtain the graph isomorphism network feature.

[0036] Preferably, the inter-region graph attention feature and the graph isomorphism network feature are fused to obtain the global and local interaction feature, specifically including:

[0037] The interaction relationship between the inter-region graph attention feature and the graph isomorphism network feature is calculated through an attention mechanism;

[0038] The features are weighted and fused according to the interaction relationship to obtain the global and local interaction feature.

[0039] The embodiment of the present application also provides a pathological image segmentation system based on a graph neural network and cross-region fusion, which is used for implementing the pathological image segmentation method based on the graph neural network and the cross-region fusion.

[0040] An image acquisition module is configured to acquire a pathological image to be segmented.

[0041] A region division module is configured to divide the pathological image into a plurality of image regions.

[0042] A cross-region pixel-level fusion module is configured to perform local cross-fusion on the plurality of image regions to obtain local cross-fusion features of each image region.

[0043] An inter-region attention module is configured to model an inter-region relationship of the local cross-fusion features based on a graph attention mechanism to obtain inter-region graph attention features.

[0044] An encoding module is configured to encode the pathological image to extract convolutional features.

[0045] An inter-pixel isomorphism module is configured to model an inter-pixel dependency relationship of the convolutional features through a self-attention mechanism and perform global structure learning on the modeled features based on a graph isomorphism network to obtain graph isomorphism network features.

[0046] A global and local fusion module is configured to fuse the inter-region graph attention features and the graph isomorphism network features to obtain global and local interaction features.

[0047] A decoding module is configured to decode the global and local interaction features and the convolutional features to output a final segmentation result.

[0048] The embodiment of the present application also provides an electronic device, which comprises a processor, a memory and a bus system, the processor and the memory are connected through the bus system, the memory is used for storing instructions, and the processor is used for executing the instructions stored in the memory to implement the pathological image segmentation method based on the graph neural network and the cross-region fusion.

[0049] The embodiment of the present application also provides a computer storage medium, which stores a computer software product, the computer software product comprises a plurality of instructions, and is used for enabling a computer device to execute the pathological image segmentation method based on the graph neural network and the cross-region fusion.

[0050] From the above technical solutions, the present application has the following beneficial effects:

[0051] (1) Effectively cope with staining heterogeneity and improve local feature consistency: Through superpixel segmentation and cross-attention mechanism, local feature interaction and fusion of different staining areas are realized, which enhances the robustness of the model to staining differences and avoids missegmentation caused by uneven staining.

[0052] (2) Model the structure and global context dependence between complex regions: Use graph attention network (GAT) to model the semantic relationship between superpixel regions, and use graph isomorphism network (GIN) to learn pixel-level global structure representation, which significantly improves the model's ability to distinguish morphological diversity and structural complexity.

[0053] (3) Realize multi-scale feature fusion, significantly improve segmentation accuracy and generalization ability: Through global and local fusion modules, local region features and global structure features are attention weighted and fused, realizing cross-scale information complementation, which significantly outperforms existing methods in multiple evaluation indicators (mDSC, mIoU) and has stronger generalization performance. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly described below. The features and advantages of the present application can be more clearly understood by referring to the drawings, which are schematic and should not be understood as any limitation on the present application. Those skilled in the art can obtain other drawings from these drawings without creative labor. Among them:

[0055] Figure 1 is a flowchart of a pathological image segmentation method based on graph neural network and cross-region fusion provided by the present application;

[0056] Figure 2 is a block diagram of a pathological image segmentation system based on graph neural network and cross-region fusion provided by the present application;

[0057] Figure 3 is a comparison diagram of the segmentation results of the present application method and the prior art. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.

[0059] To solve the problems of inaccurate structure segmentation and poor generalization ability of HE staining pathological image caused by staining heterogeneity and complex tissue morphology in the prior art, as shown in Figure 1 The present application provides a pathological image segmentation method based on graph neural network and cross-region fusion, which comprises the following steps:

[0060] S1: obtaining a pathological image to be segmented;

[0061] S2: dividing the pathological image into image regions to obtain a plurality of image regions;

[0062] S3: performing local cross-fusion on the plurality of image regions to obtain local cross-fusion features of each image region;

[0063] S4: modeling the inter-region relationship of the local cross-fusion features based on a graph attention mechanism to obtain inter-region graph attention features;

[0064] S5: encoding the pathological image to extract convolutional features;

[0065] S6: modeling the inter-pixel dependency relationship of the convolutional features through a self-attention mechanism, and learning the global structure of the modeled features based on a graph isomorphism network to obtain graph isomorphism network features;

[0066] S7: fusing the inter-region graph attention features and the graph isomorphism network features to obtain global and local interaction features;

[0067] S8: decoding the global and local interaction features and the convolutional features to output the final segmentation result.

[0068] As can be seen from the above technical solution, the present application provides a pathological image segmentation method based on graph neural network and cross-region fusion. The method first obtains a pathological image to be segmented, and then divides the image into superpixel regions through image region division to provide a structural basis for subsequent local fusion and adapt to staining differences. Then, the superpixel regions are locally cross-fused using a cross-attention mechanism to enhance inter-region feature interaction and improve local consistency. The inter-region semantic relationship is modeled by a graph attention network (GAT) to strengthen the feature expression of important regions. At the same time, the convolutional features of the image are extracted by a CNN encoder to capture local details, and the global dependency relationship between pixels is learned by a self-attention mechanism and a graph isomorphism network (GIN) to enhance the ability to distinguish complex morphologies. Then, the inter-region graph attention features and the graph isomorphism network features are fused to realize multi-scale global and local information complementation. Finally, all high-level features are fused by a decoder to reconstruct an accurate segmentation result, realizing high-precision and high-robustness automatic segmentation. The present application effectively solves the problem of inaccurate segmentation of HE staining images caused by staining heterogeneity and complex tissue morphology.

[0069] In step S1, the pathological image to be segmented is first acquired. .

[0070] In step S2, the pathological images are... The image is divided into multiple regions, specifically including:

[0071] Pathological images Perform superpixel segmentation to obtain a superpixel segmentation map. Each superpixel region is considered as an image region. Superpixel segmentation map. Contains multiple superpixel segmentation regions , Represents superpixel segmentation map The first in Each segmented region.

[0072] In step S3, local cross-fusion is performed on multiple image regions to obtain the local cross-fusion features of each image region, specifically including:

[0073] For each image region, it is interacted with at least one other image region through a cross-attention mechanism to calculate the dependencies between regions.

[0074] The interacting features are channel-merged and then calculated using a feedforward neural network to obtain local cross-fusion features.

[0075] Specifically, this embodiment first starts with the superpixel segmentation map Take out the first one All pixels of a segmented region, represented as Then fill or discard Construct a matrix ,in Indicates the number of rows in the matrix. Indicates the number of columns in the matrix; from the superpixel segmentation map Take out the first one All pixels in a superpixel segmented region are represented as Then, by filling or discarding Construct a matrix Then, through a cross-attention mechanism, the dependencies between multiple superpixel segmentation regions are captured:

[0076] , , ;

[0077] in, Represents the query matrix. Represents the key matrix. Represents a value matrix, 、 、 are learnable parameters, denotes a position encoding matrix, , wherein denotes a row index, denotes a column index.

[0078] Then the fusion feature of the i-th superpixel segmentation region and the rest of the random superpixel segmentation regions is calculated:

[0079] ; ;

[0080] wherein, denotes channel merging, denotes cross attention of the i-th superpixel segmentation region and the j-th superpixel segmentation region, denotes a padding matrix, wherein denotes the i-th row, denotes the j-th row, denotes learnable parameters. Finally, the local cross fusion feature is sent into the forward neural network to calculate the final local cross fusion feature :

[0081] ; wherein,

[0082] denotes layer normalization operation, denotes a forward neural network.

[0083] In step S4, based on the graph attention mechanism, the inter-region relationship of the local cross fusion feature is modeled to obtain the inter-region graph attention feature, specifically including: The local cross fusion feature of each image region is taken as a node in the graph;

[0084] The attention weight between nodes is dynamically calculated by using the graph attention network, and the neighbor node features are aggregated to update the feature representation of each node to obtain the inter-region graph attention feature.

[0085]

[0086]

[0087] ​​​​​Specifically, in order to effectively capture the interaction of complex different superpixel segmentation regions and the importance region, the graph attention network is adopted. The graph attention network introduces a dynamically adjusted attention mechanism based on the relationship between the interactions of superpixel segmentation regions.

[0088] Further, the graph attention network of the present application comprises nodes, for the first node of the first layer , the weight of the edge between the node and the node in the neighborhood node set of the node is:

[0089] ;

[0090] wherein, is an activation function, , denotes a learnable weight, denotes the graph attention feature of the first node of the first layer, when , , denotes the graph attention feature of the first node of the first layer, when , , denotes the local cross-fusion feature of the first superpixel segmentation region.

[0091] for the first node of the first layer , the attention between the node and the node in the neighborhood node set of the node is represented as:

[0092] .

[0093] for the first node of the first layer graph attention feature is represented as:

[0094] .

[0095] In step S5, the pathological image is encoded to extract convolutional features. Specifically, the pathological image is input into a convolutional network encoder such as a UNet encoder to extract convolutional features ,​​ wherein, represents the number of convolution modules in the convolution network encoder, represents the feature output by the i-th convolution module in the convolution network encoder.

[0096] In step S6, the inter-pixel dependency of the convolution feature is modeled by a self-attention mechanism, and global structure learning is performed on the modeled feature based on a graph isomorphism network to obtain a graph isomorphism network feature, which specifically includes:

[0097] The convolution feature is flattened into a matrix form;

[0098] The self-attention weight between pixels is calculated through a learnable query matrix, a key matrix, and a value matrix;

[0099] The pixel features are weighted and fused according to the self-attention weight to obtain the self-attention feature between pixels;

[0100] The self-attention feature of each pixel is taken as the initial feature of a node in the graph;

[0101] For each node, the features of its neighbor nodes are aggregated layer by layer, and the aggregation result is combined with the feature of the node itself;

[0102] The combined feature is transformed by a multi-layer perceptron to update the node feature, and the graph isomorphism network feature is obtained.

[0103] Specifically, the embodiment first flattens the feature of all pixels to form a matrix Through a self-attention mechanism, the dependency between pixels is captured:

[0104] ;

[0105] ;

[0106] ;

[0107] ;

[0108] ;

[0109] wherein, , , represent learnable parameters, represents a convolution query matrix, represents a convolution key matrix, represents a convolution value matrix, represents a convolution position encoding matrix, represents​​ Row index, represents Column index.

[0110] Then calculate the self-attention feature between pixels :

[0111] .

[0112] Further, a graph isomorphism network (GIN) is constructed for the self-attention feature between pixels, and the core design goal is to realize the discrimination ability of the "graph isomorphism" feature, that is, the same representation is output for the graph with the same structure (isomorphism), and the distinguishable representation is output for the graph with different structure (non-isomorphism), which solves the limitation of early graph neural networks (such as graph convolution network, graph attention network) in graph structure discrimination ability. The representation learning is realized by "layer-by-layer aggregation of neighbor features + updating own features", and the specific steps are as follows:

[0113] Initial node feature: the input feature of each node in the graph is taken as the initial feature , wherein represents the first row feature of .

[0114] Intra-layer feature aggregation: for the first layer, for each node : aggregate the first layer features of all neighbor nodes of , to get the neighbor feature aggregation result: , wherein represents the neighbor node set of ; the first layer feature of the node itself is combined with the neighbor aggregation result , and is transformed through a parameterized mapping function (such as MLP, multi-layer perceptron), to get the first layer feature of the node : , wherein is a learnable parameter (used to adjust the weight of the own feature and the neighbor feature), represents the first layer multi-layer perceptron, which is responsible for mapping the combined feature to a new dimension. The final output feature of the node in the graph isomorphism network is:

[0115] ; ​

[0116] in, This indicates the number of layers in a graph isomorphic network.

[0117] In step S7, the inter-region graph attention features and graph isomorphic network features are fused to obtain global and local interaction features, specifically including:

[0118] The interaction relationship between inter-region graph attention features and graph isomorphic network features is calculated using an attention mechanism;

[0119] The features are weighted and fused based on the interaction relationship to obtain global and local interaction features.

[0120] Specifically, in this embodiment, the graph attention network's first... Layer node graph attention features ( ) constitute the local feature matrix Nodes in a graph isomorphic network Final output features Constructing the global feature matrix Then calculate the attention between the global and local attention:

[0121] ;

[0122] in, , , This represents the learnable parameters. Represents a local region of a superpixel With high-level feature pixels The interaction relationships between them, and the interaction characteristics between local and global aspects are as follows: ,in, yes The OK, yes The OK.

[0123] In step S8, the global and local interaction features and convolutional features are decoded to output the final segmentation result. Specifically, the local and global interaction features... As the highest-level feature, it is related to convolutional features. ( The data are fed together into a segmentation decoder, such as the UNet decoder, to output the segmentation result. Finally, the cross-entropy loss function is used to optimize the entire segmentation network, completing the network training.

[0124] like Figure 2As shown, the present application provides a pathological image segmentation system based on graph neural network and cross-region fusion, which is used to realize the pathological image segmentation method based on graph neural network and cross-region fusion as described above, and specifically comprises:

[0125] An image acquisition module 100 is configured to acquire a pathological image to be segmented.

[0126] A region division module 200 is configured to perform image region division on the pathological image to obtain a plurality of image regions.

[0127] A cross-region pixel-level fusion module 300 is configured to perform local cross-fusion on the plurality of image regions to obtain local cross-fusion features of each image region.

[0128] An inter-region attention module 400 is configured to model inter-region relationships of the local cross-fusion features based on a graph attention mechanism to obtain inter-region graph attention features.

[0129] An encoding module 500 is configured to encode the pathological image to extract convolutional features.

[0130] An inter-pixel isomorphism module 600 is configured to model inter-pixel dependency relationships of the convolutional features through a self-attention mechanism, and perform global structure learning on the modeled features based on a graph isomorphism network to obtain graph isomorphism network features.

[0131] A global and local fusion module 700 is configured to fuse the inter-region graph attention features and the graph isomorphism network features to obtain global and local interaction features.

[0132] A decoding module 800 is configured to decode the global and local interaction features and the convolutional features to output a final segmentation result.

[0133] Specifically, the image acquisition module 100 acquires a pathological image to be segmented The pathological image is input into the region division module 200 to perform superpixel segmentation to obtain a superpixel segmentation map The pathological image and the superpixel segmentation map are input into the cross-region pixel-level fusion module 300 to calculate local cross-fusion features The local cross-fusion features are input into the inter-region attention module 400 to obtain inter-region graph attention features The pathological image is input into the encoding module 500, such as a UNet encoder, to extract convolutional features The graph isomorphism network features are calculated through the inter-pixel isomorphism module 600. The inter-region graph attention features Graph isomorphism network features The input global and local fusion module 700 generates the interaction features of the local and global as , As the highest level features, the convolution features are sent into the decoding module 800, such as UNet decoder, together with the global and local fusion features, to output the segmentation result.

[0134] The pathological image segmentation system based on the graph neural network and cross-region fusion of the embodiment is used to realize the pathological image segmentation method based on the graph neural network and cross-region fusion described above, and therefore the specific embodiments of the pathological image segmentation system based on the graph neural network and cross-region fusion can be seen from the foregoing embodiment part of the pathological image segmentation method based on the graph neural network and cross-region fusion, for example, the image acquisition module 100, the region division module 200, the cross-region pixel-level fusion module 300, the inter-region attention module 400, the encoding module 500, the pixel inter-isomorphism module 600, the global and local fusion module 700, and the decoding module 800, which are respectively used to realize steps S1, S2, S3, S4, S5, S6, S7, and S8 in the foregoing pathological image segmentation method based on the graph neural network and cross-region fusion, so the specific embodiments can refer to the description of the respective embodiment part, and for the sake of avoiding redundancy, the description is not repeated here.

[0135] In order to further verify the advantages of the present application, the following will be described in combination with specific experiments.

[0136] In the pathological image data set collected in the experiment, there are 26183 images, and the images include regions of five types of tissue components, i.e., Tumor (tumor), Stroma (stroma), Lymphocytic infiltrate (lymphocyte infiltration), Necrosis (necrosis), and Others (other). The training set, the verification set, and the test set are randomly divided according to the ratio of 7:1:2. All experiments are carried out on a single NVIDIA GeForce RTX4090 GPU with a resolution of 24 GB memory card in Python3.8 using Pytorch1.8. The experimental settings of the comparative method are modified from the publicly available implementation. All models are trained using the Adam optimizer with a learning rate of , and the training period is 200.

[0137] To quantitatively evaluate the performance of the method of the present application, two mainstream and representative evaluation indexes in segmentation tasks are selected to comprehensively measure the accuracy and integrity of the model in pathological image segmentation. Mean Dice Similarity Coefficient (mDSC), mean Intersection over Union (mIoU).

[0138] In addition, the segmentation results of the method of the present application are compared with three representative existing methods published in recent years in the same task (such as tongue segmentation), and the specific comparison results are shown in Table 1. All methods are evaluated based on the test set of the same data set, and the data set division, data enhancement strategy and hardware test environment are consistent to ensure the fairness and reliability of the comparison results. Table 1 clearly lists the mDSC and mIoU index values of the method of the present application and each existing method on the test set. From the table, it can be seen that the method of the present application is significantly better than the classic basic method and the field optimization method in both indexes, and compared with the current SOTA method, the mDSC is improved by 3.4%~9.7%, and the mIoU is improved by 6.3%~12.1%, which fully proves the advantage of the method of the present application in segmentation precision.

[0139] Table 1 Comparison of segmentation performance of the method of the present application and existing methods

[0140]

[0141] Among them, the existing method 1 is a domain-incremental weakly supervised state space model for pathology image segmentation (Adomain-incremental weakly supervised state space model for pathology image segmentation); the existing method 2 is a breast cancer pathological image segmentation method based on template matching (Pathological image segmentation of breast cancer via template matching); and the existing method 3 is HADiff: hierarchy aggregated diffusion model for pathology image segmentation (HADiff: hierarchy aggregated diffusion model for pathology image segmentation).

[0142] As Figure 3The image shows three comparative cases. Column 1 is the pathological image, column 2 is the manual segmentation result, column 3 is the segmentation result of the method of this invention, column 4 is the segmentation result of method 1, column 5 is the segmentation result of method 2, and column 6 is the segmentation result of method 3. Column 3 shows that the segmentation result of the method of this invention is very close to the manual segmentation result. In row 1, column 4, method 1 classifies lymphocyte infiltration and other types of regions as stroma, with some stroma incorrectly classified as blue lymphocyte infiltration. In row 1, column 5, method 2 incorrectly classifies the stroma as blue lymphocyte infiltration, and other types of regions as stroma, with some stroma incorrectly classified as blue lymphocyte infiltration. In row 1, column 6, method 3 classifies lymphocyte infiltration and other types of regions as stroma, with some stroma incorrectly classified as blue lymphocyte infiltration. In row 2, columns 4-6, methods 1-3 fail to correctly segment the boundary between the tumor and the stroma. In row 3, columns 4-5, methods 1 and 2 classify other types of regions as stroma, and some lymphocyte infiltration is also classified as stroma. In row 3, column 6, other types of regions are not classified as stroma, and some stroma is incorrectly classified as tumor.

[0143] Furthermore, to verify the effectiveness of each module proposed in the architecture of this invention, ablation experiments were conducted. As shown in Table 2, compared with the baseline model, the mDSC and mIoU observed in the cross-region pixel-level fusion module and the inter-region attention module are relatively improved, which proves the effectiveness of the features of this invention in the segmentation task. The mDSC and mIoU of the cross-region pixel-level fusion are 3.5 and 6.1 percentage points higher than the baseline, respectively. In addition, the mDSC and mIoU of the inter-region attention module are further improved by 4.5 and 2.5 percentage points, respectively. The mDSC and mIoU of the inter-pixel isomorphic module are further improved by 2.5 and 2.6 percentage points, respectively. In addition, the mDSC and mIoU of the global and local fusion modules are further improved by 2.9 and 2.3 percentage points, respectively. This shows that all models of this invention are effective.

[0144] Table 2 Ablation experiments of the present invention

[0145]

[0146] In summary, the application designs a cross-regional pixel-level fusion module 300 to perform local cross-fusion on superpixel segmentation regions, strengthens the fusion between local pixels, and thus can improve the interaction between local pixels. The application designs an inter-region attention module 400, adopts a graph attention network, effectively captures the interaction and importance region of complex different superpixel segmentation regions, and strengthens the fusion between superpixel segmentation regions. The application designs an inter-pixel isomorphism module 600, captures the dependency relationship between pixels through a self-attention mechanism, constructs a graph isomorphism network for the self-attention features between pixels, and realizes global feature learning through “layer-by-layer aggregation of neighbor features + updating of own features”. The application designs a global and local fusion module 700, the cross-regional pixel-level fusion module 300 and the inter-region attention module 400 calculate the local information interaction, the inter-pixel isomorphism module 600 calculates the global information of high-level features of the pathological image The global and local fusion module 700 fuses the local and global features, and improves the segmentation performance.

[0147] The embodiment of the application provides an electronic device, the electronic device includes a processor, a memory and a bus system, the processor and the memory are connected through the bus system, the memory is used for storing instructions, and the processor is used for executing the instructions stored in the memory to realize the pathological image segmentation method based on the graph neural network and the cross-regional fusion.

[0148] The embodiment of the application provides a computer storage medium, the computer storage medium stores a computer software product, the computer software product includes a plurality of instructions, and the instructions are used to make a computer equipment execute the pathological image segmentation method based on the graph neural network and the cross-regional fusion.

[0149] Those skilled in the art should understand that the embodiments of the application can be provided as a method, a system or a computer program product. Therefore, the application can adopt a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0150] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0151] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks Figure 1 The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks

[0152] Obviously, the above-mentioned embodiments are only examples for clearly illustrating the present application and are not intended to limit the implementation modes. Based on the above description, other different forms of changes or variations can be made by those skilled in the art. Here, all the implementation modes are not required or can not be exhausted. The obvious changes or variations derived therefrom are still within the protection scope of the present application.

Claims

1. A pathological image segmentation method based on graph neural networks and cross-region fusion, characterized in that, include: Obtain the pathological image to be segmented; The pathological image is divided into multiple image regions. Local cross-fusion is performed on the multiple image regions to obtain the local cross-fusion features of each image region; Based on the graph attention mechanism, the inter-regional relationship model is performed on the local cross-fusion features to obtain inter-regional graph attention features. Specifically, this includes: taking the local cross-fusion features of each image region as a node in the graph; dynamically calculating the attention weights between nodes using a graph attention network, aggregating the features of neighboring nodes, and updating the feature representation of each node to obtain the inter-regional graph attention features. The pathological images are encoded, and convolutional features are extracted. The convolutional features are modeled using a self-attention mechanism to model inter-pixel dependencies, and the modeled features are then subjected to global structure learning based on a graph isomorphic network to obtain graph isomorphic network features. By fusing the inter-region graph attention features with the graph isomorphic network features, global and local interaction features are obtained. The global and local interaction features are decoded along with the convolutional features to output the final segmentation result.

2. The pathological image segmentation method based on graph neural networks and cross-regional fusion according to claim 1, characterized in that, The process of dividing the pathological image into multiple image regions specifically includes: The pathological image is segmented into superpixels to obtain a superpixel segmentation map, where each superpixel region is considered as an image region.

3. The pathological image segmentation method based on graph neural networks and cross-regional fusion according to claim 2, characterized in that, The step of performing local cross-fusion on the multiple image regions to obtain local cross-fusion features for each image region specifically includes: For each image region, it is interacted with at least one other image region through a cross-attention mechanism to calculate the dependencies between regions. The interacting features are channel-merged and then calculated using a feedforward neural network to obtain the local cross-fusion features.

4. The pathological image segmentation method based on graph neural networks and cross-regional fusion according to claim 1, characterized in that, The step of modeling the inter-pixel dependencies of the convolutional features using a self-attention mechanism specifically includes: Flatten the convolutional features into a matrix form; Self-attention weights between pixels are calculated using a learnable query matrix, key matrix, and value matrix. The pixel features are weighted and fused according to the self-attention weights to obtain the self-attention features between pixels.

5. The pathological image segmentation method based on graph neural networks and cross-regional fusion according to claim 4, characterized in that, The global structure learning of the modeled features based on graph isomorphic networks specifically includes: Use the self-attention features of each pixel as the initial features of a node in the graph; For each node, the features of its neighboring nodes are aggregated layer by layer, and the aggregation result is merged with its own features. The merged features are transformed using a multilayer perceptron to update the node features, thus obtaining the graph isomorphic network features.

6. The pathological image segmentation method based on graph neural networks and cross-regional fusion according to claim 1, characterized in that, The fusion of the inter-region graph attention features and the graph isomorphic network features to obtain global and local interaction features specifically includes: The interaction relationship between the inter-region graph attention features and the graph isomorphic network features is calculated using an attention mechanism; The features are weighted and fused according to the interaction relationship to obtain the global and local interaction features.

7. A pathological image segmentation system based on graph neural networks and cross-regional fusion, characterized in that, The system is used to implement the pathological image segmentation method based on graph neural networks and cross-region fusion as described in any one of claims 1 to 6, specifically including: The image acquisition module is used to acquire pathological images to be segmented; The region segmentation module is used to segment the pathological image into multiple image regions. A cross-region pixel-level fusion module is used to perform local cross-fusion of the multiple image regions to obtain the local cross-fusion features of each image region; The inter-regional attention module is used to model the inter-regional relationships of the local cross-fusion features based on the graph attention mechanism, so as to obtain inter-regional graph attention features. The encoding module is used to encode the pathological images and extract convolutional features; The pixel isomorphic module is used to model the pixel dependencies of the convolutional features through a self-attention mechanism, and to perform global structure learning on the modeled features based on the graph isomorphic network to obtain graph isomorphic network features. The global and local fusion module is used to fuse the inter-region graph attention features and the graph isomorphic network features to obtain global and local interaction features. The decoding module is used to decode the global and local interaction features and the convolutional features, and output the final segmentation result.

8. An electronic device, characterized in that, The electronic device includes a processor, a memory, and a bus system. The processor and the memory are connected through the bus system. The memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the pathological image segmentation method based on graph neural networks and cross-regional fusion as described in any one of claims 1 to 6.

9. A computer storage medium, characterized in that, The computer storage medium stores a computer software product, which includes several instructions to cause a computer device to execute the pathological image segmentation method based on graph neural networks and cross-regional fusion as described in any one of claims 1 to 6.

Images