Pathological image segmentation method and system based on graph neural network and cross-region fusion

Abstract

The invention provides a pathological image segmentation method and system based on a graph neural network and cross-region fusion, and relates to the technical field of medical image processing, and the method comprises the steps: obtaining a pathological image, and carrying out the image region division, and obtaining a plurality of image regions; performing local cross fusion on the image region by using a cross attention mechanism to obtain local cross fusion features; carrying out modeling on the local cross fusion features by adopting a graph attention network to obtain inter-region graph attention features; extracting convolution features of the image through an encoder, and performing global structure learning in combination with a self-attention mechanism and a graph isomorphic network to obtain graph isomorphic network features; fusing the features to obtain global and local interaction features; and finally, inputting the global and local interaction features and the convolution features into a decoder, and outputting a segmentation result. According to the method, the segmentation problem of the HE staining pathological image caused by staining heterogeneity and morphological diversity is effectively solved, and the segmentation precision and robustness are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to a method and system for pathological image segmentation based on graph neural networks and cross-regional fusion. Background Technology

[0002] Histopathological examination is a crucial part of clinical diagnosis, playing an irreplaceable role, especially in the qualitative, grading, and prognostic assessment of diseases such as cancer. Hematoxylin and eosin (HE) staining is one of the most commonly used pathological staining techniques, capable of clearly displaying tissue structures such as cell nuclei, cytoplasm, and stroma. However, because HE staining relies on manual operation, it is affected by various factors such as staining time, reagent concentration, and washing degree, leading to significant staining heterogeneity in different regions, sections, and even within the same section of the same tissue. This heterogeneity manifests as staining that is too dark or too light, making it difficult to distinguish normal cells from cancer cells, vascular structures from background noise, etc., at the pixel level, severely impacting the accuracy of automated segmentation based on traditional image processing methods.

[0003] Furthermore, biological tissues themselves exhibit high morphological diversity and structural complexity. Even the same type of lesion (such as lung cancer) may show different cell arrangements, vascular morphology, and stroma distribution in different patients. Necrotic areas, inflammatory areas, and normal areas often coexist in tissues, with blurred boundaries and a lack of clear pixel-level demarcation, further increasing the difficulty of structural segmentation. Traditional segmentation methods based on pixel intensity or fixed morphological templates (such as circular or tubular structures) are ill-suited to this type of unstructured image data, resulting in low segmentation accuracy and poor generalization ability.

[0004] In recent years, with the development of deep learning technology, especially the application of convolutional neural networks (CNNs) and graph neural networks (GNNs) in image segmentation, some studies have attempted to improve the understanding of complex pathological images by introducing attention mechanisms and graph structure modeling. However, existing methods still have the following limitations: 1. Most methods rely solely on convolution operations to extract local features, making it difficult to effectively model long-distance pixel dependencies and semantic relationships between regions; 2. Although graph neural networks can model structural information, they have shortcomings in graph isomorphism discrimination and graph structure representation. 3. Existing fusion mechanisms are mostly limited to a single local or global scale, lacking effective cross-scale feature interaction mechanisms; 4. The robustness and generalization ability of the model still need to be improved when faced with staining heterogeneity and morphological diversity.

[0005] Therefore, there is an urgent need for an automated pathological image segmentation system that can effectively integrate local details with global context, has strong structural discrimination capabilities, and can adapt to staining and morphological changes, in order to improve the accuracy and efficiency of pathological examination. Summary of the Invention

[0006] To address this, embodiments of the present invention provide a pathological image segmentation method and system based on graph neural networks and cross-regional fusion, which solves the problems of inaccurate structural segmentation and poor generalization ability of HE-stained pathological images caused by staining heterogeneity and tissue morphology complexity in the prior art.

[0007] To address the aforementioned technical problems, embodiments of the present invention provide a pathological image segmentation method based on graph neural networks and cross-region fusion, the method comprising: Obtain the pathological image to be segmented; The pathological image is divided into multiple image regions. Local cross-fusion is performed on the multiple image regions to obtain the local cross-fusion features of each image region; Based on the graph attention mechanism, the inter-regional relationship model is performed on the local cross-fusion features to obtain inter-regional graph attention features; The pathological images are encoded, and convolutional features are extracted. The convolutional features are modeled using a self-attention mechanism to model inter-pixel dependencies, and the modeled features are then subjected to global structure learning based on a graph isomorphic network to obtain graph isomorphic network features. By fusing the inter-region graph attention features with the graph isomorphic network features, global and local interaction features are obtained. The global and local interaction features are decoded along with the convolutional features to output the final segmentation result.

[0008] Preferably, the step of dividing the pathological image into multiple image regions specifically includes: The pathological image is segmented into superpixels to obtain a superpixel segmentation map, where each superpixel region is considered as an image region.

[0009] Preferably, the step of performing local cross-fusion on the multiple image regions to obtain local cross-fusion features for each image region specifically includes: For each image region, it is interacted with at least one other image region through a cross-attention mechanism to calculate the dependencies between regions. The interacting features are channel-merged and then calculated using a feedforward neural network to obtain the local cross-fusion features.

[0010] Preferably, the step of modeling the inter-regional relationships of the local cross-fusion features based on the graph attention mechanism specifically includes: The local cross-fusion features of each image region are used as a node in the graph; The attention weights between nodes are dynamically calculated using a graph attention network, and the features of neighboring nodes are aggregated to update the feature representation of each node, thereby obtaining the inter-region graph attention features.

[0011] Preferably, the step of modeling the inter-pixel dependencies of the convolutional features using a self-attention mechanism specifically includes: Flatten the convolutional features into a matrix form; Self-attention weights between pixels are calculated using a learnable query matrix, key matrix, and value matrix. The pixel features are weighted and fused according to the self-attention weights to obtain the self-attention features between pixels.

[0012] Preferably, the global structure learning of the modeled features based on graph isomorphic networks specifically includes: Use the self-attention features of each pixel as the initial features of a node in the graph; For each node, the features of its neighboring nodes are aggregated layer by layer, and the aggregation result is merged with its own features. The merged features are transformed using a multilayer perceptron to update the node features, thus obtaining the graph isomorphic network features.

[0013] Preferably, the fusion of the inter-region graph attention features and the graph isomorphic network features to obtain global and local interaction features specifically includes: The interaction relationship between the inter-region graph attention features and the graph isomorphic network features is calculated using an attention mechanism; The features are weighted and fused according to the interaction relationship to obtain the global and local interaction features.

[0014] This invention also provides a pathological image segmentation system based on graph neural networks and cross-region fusion. This system is used to implement the aforementioned pathological image segmentation method based on graph neural networks and cross-region fusion, specifically including: The image acquisition module is used to acquire pathological images to be segmented; The region segmentation module is used to segment the pathological image into multiple image regions. A cross-region pixel-level fusion module is used to perform local cross-fusion of the multiple image regions to obtain the local cross-fusion features of each image region; The inter-regional attention module is used to model the inter-regional relationships of the local cross-fusion features based on the graph attention mechanism, so as to obtain inter-regional graph attention features. The encoding module is used to encode the pathological images and extract convolutional features; The pixel isomorphic module is used to model the pixel dependencies of the convolutional features through a self-attention mechanism, and to perform global structure learning on the modeled features based on the graph isomorphic network to obtain graph isomorphic network features. The global and local fusion module is used to fuse the inter-region graph attention features and the graph isomorphic network features to obtain global and local interaction features. The decoding module is used to decode the global and local interaction features and the convolutional features, and output the final segmentation result.

[0015] This invention also provides an electronic device, which includes a processor, a memory, and a bus system. The processor and the memory are connected through the bus system. The memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the pathological image segmentation method based on graph neural networks and cross-regional fusion described above.

[0016] This invention also provides a computer storage medium storing a computer software product, the computer software product including several instructions to cause a computer device to execute the above-described pathological image segmentation method based on graph neural networks and cross-regional fusion.

[0017] As can be seen from the above technical solutions, this invention application has the following beneficial effects: (1) Effectively address staining heterogeneity and improve local feature consistency: Through superpixel segmentation and cross attention mechanism, local feature interaction and fusion of different staining regions are realized, which enhances the robustness of the model to staining differences and avoids missegmentation caused by uneven staining.

[0018] (2) Modeling complex regional structure and global context dependency: Graph Attention Network (GAT) is used to model the semantic relationship between superpixel regions, and Graph Isomorphic Network (GIN) is combined to learn pixel-level global structure representation, which significantly improves the model's ability to distinguish morphological diversity and structural complexity.

[0019] (3) Achieve multi-scale feature fusion and significantly improve segmentation accuracy and generalization ability: Through the global and local fusion module, the local regional features and global structural features are fused with attention weighting, which realizes cross-scale information complementarity. It is significantly better than existing methods in multiple evaluation indicators (mDSC, mIoU) and has stronger generalization performance. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Referring to the drawings will make the features and advantages of the present invention clearer. The drawings are illustrative and should not be construed as limiting the present invention in any way. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart of a pathological image segmentation method based on graph neural networks and cross-regional fusion provided by the present invention; Figure 2 This is a block diagram of a pathological image segmentation system based on graph neural networks and cross-regional fusion provided by the present invention; Figure 3 This is a schematic diagram comparing the segmentation results of the method of this invention with those of existing methods. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] To address the problems of inaccurate structural segmentation and poor generalization ability in HE-stained pathological images caused by staining heterogeneity and tissue morphology complexity in existing technologies, such as... Figure 1 As shown, this invention proposes a pathological image segmentation method based on graph neural networks and cross-region fusion, which includes: S1: Obtain the pathological image to be segmented; S2: Divide the pathological image into multiple image regions; S3: Perform local cross-fusion on multiple image regions to obtain the local cross-fusion features of each image region; S4: Based on the graph attention mechanism, the inter-regional relationship is modeled on the local cross-fusion features to obtain the inter-regional graph attention features; S5: Encode pathological images and extract convolutional features; S6: The inter-pixel dependencies of convolutional features are modeled through a self-attention mechanism, and the global structure of the modeled features is learned based on a graph isomorphic network to obtain graph isomorphic network features. S7: Integrate inter-region graph attention features and graph isomorphic network features to obtain global and local interaction features; S8: Decode the global and local interactive features and convolutional features to output the final segmentation result.

[0023] As can be seen from the above technical solution, this invention proposes a pathological image segmentation method based on graph neural networks and cross-region fusion. This method first acquires the pathological image to be segmented, then divides the image into superpixel regions through image region segmentation, providing a structural basis for subsequent local fusion and adapting to staining differences. Next, a cross-attention mechanism is used to perform local cross-fusion of the superpixel regions to enhance feature interaction between regions and improve local consistency. Then, a graph attention network (GAT) is used to model the semantic relationships between regions, strengthening the feature representation of important regions. Simultaneously, a CNN encoder is used to extract convolutional features of the image to capture local details, and a self-attention mechanism and a graph isomorphic network (GIN) are used to learn global dependencies between pixels, enhancing the ability to discriminate complex morphologies. Afterwards, the graph attention features between regions and the graph isomorphic network features are fused to achieve multi-scale global and local information complementarity. Finally, a decoder fuses all high-level features to reconstruct accurate segmentation results, achieving high-precision and highly robust automated segmentation. This invention effectively solves the problem of inaccurate segmentation of HE-stained images due to staining heterogeneity and tissue morphological complexity.

[0024] In step S1, the pathological image to be segmented is first acquired. .

[0025] In step S2, the pathological images are... The image is divided into multiple regions, specifically including: Pathological images Perform superpixel segmentation to obtain a superpixel segmentation map. Each superpixel region is considered as an image region. Superpixel segmentation map. Contains multiple superpixel segmentation regions , Represents superpixel segmentation map The first in Each segmented region.

[0026] In step S3, local cross-fusion is performed on multiple image regions to obtain the local cross-fusion features of each image region, specifically including: For each image region, it is interacted with at least one other image region through a cross-attention mechanism to calculate the dependencies between regions. The interacting features are channel-merged and then calculated using a feedforward neural network to obtain local cross-fusion features.

[0027] Specifically, this embodiment first starts with the superpixel segmentation map Take out the first one All pixels of a segmented region, represented as Then fill or discard Construct a matrix ,in Indicates the number of rows in the matrix. Indicates the number of columns in the matrix; from the superpixel segmentation map Take out the first one All pixels in a superpixel segmented region are represented as Then, by filling or discarding Construct a matrix Then, through a cross-attention mechanism, the dependencies between multiple superpixel segmentation regions are captured: , , ; in, Represents the query matrix. Represents the key matrix. Represents a value matrix, , , All are learnable parameters. Represents the position encoding matrix, , ,in express Row index, express Column index.

[0028] Next, calculate the first... Each superpixel segmentation region and the remaining random Fusion features of superpixel segmentation regions: ; ; in, Indicates channel merging. Indicates the first The superpixel segmentation region and the first Cross-attention of superpixel segmentation regions Indicates a filling matrix. ,in express No. OK, express No. OK, This represents the learnable parameters.

[0029] Finally, the data is fed into a feedforward neural network to calculate the final local cross-fusion features. : ; in, Representation layer normalization operation, This represents a feedforward neural network.

[0030] In step S4, based on the graph attention mechanism, inter-regional relationship modeling is performed on the local cross-fusion features to obtain inter-regional graph attention features, specifically including: The local cross-fusion features of each image region are used as a node in the graph; By dynamically calculating the attention weights between nodes using a graph attention network and aggregating the features of neighboring nodes, the feature representation of each node is updated to obtain the inter-region graph attention features.

[0031] Specifically, to effectively capture the interactions and importance regions of complex and different superpixel segmentation regions, this invention employs a graph attention network. The graph attention network introduces a dynamically adjusted attention mechanism to measure the relationships based on the interactions between superpixel segmentation regions.

[0032] Furthermore, the graph attention network of the present invention includes For the nth node, The first layer Node and its neighboring node set nodes in The weights of the edges between them are: ; in, For activation function, , Indicates learnable weights, Indicates for the first The first layer Node graph attention features, when hour, , Indicates for the first The first layer Node graph attention features, when hour, , Indicates the first Local cross-fusion features of superpixel segmentation regions.

[0033] For the The first layer Node and its neighboring node set nodes in attention between Represented as: .

[0034] For the The first layer Node graph attention features Represented as: .

[0035] In step S5, the pathological images are... Encoding is performed, and convolutional features are extracted. Specifically, this invention encodes pathological images. The data is fed into a convolutional network encoder, such as the UNet encoder, to extract convolutional features. , ,in, This indicates the number of convolutional modules in the convolutional network encoder. This represents the first convolutional network encoder. Features output by each convolutional module.

[0036] In step S6, the inter-pixel dependencies of the convolutional features are modeled using a self-attention mechanism, and the global structure of the modeled features is learned based on a graph isomorphic network to obtain graph isomorphic network features, specifically including: Flatten the convolutional features into a matrix form; Self-attention weights between pixels are calculated using a learnable query matrix, key matrix, and value matrix. The self-attention features between pixels are obtained by weighted fusion of pixel features based on self-attention weights. Use the self-attention features of each pixel as the initial features of a node in the graph; For each node, the features of its neighboring nodes are aggregated layer by layer, and the aggregation result is merged with its own features. The merged features are transformed using a multilayer perceptron to update the node features, resulting in graph isomorphic network features.

[0037] Specifically, this embodiment first defines the features All pixels flattened form a matrix By employing a self-attention mechanism, the dependencies between pixels are captured: ; ; ; ; ; in, , , Indicates learnable parameters, Represents the convolution query matrix. Represents the convolutional bond matrix. Represents the convolution value matrix, This represents the convolution position encoding matrix. express Row index, express Column index.

[0038] Next, the self-attention features between pixels are calculated. : .

[0039] Furthermore, a Graph Isomorphism Network (GIN) is constructed to address the self-attention features between pixels. The core design goal is to achieve the ability to discriminate graph isomorphism—that is, to output the same representation for graphs with identical structures (isomorphic) and to output distinguishable representations for graphs with different structures (non-isomorphic), thus overcoming the limitations of early graph neural networks (such as graph convolutional networks and graph attention networks) in graph structure discrimination. Representation learning is achieved through "layer-by-layer aggregation of neighbor features + updating its own features," with the specific steps as follows: Initial node features: The input features of each node in the graph are used as initial features. ,in, express The Line characteristics.

[0040] Intra-layer feature aggregation: for the first layer Layers, for each node Aggregate all its neighbor nodes The Layer features The neighbor feature aggregation results are obtained: ,in, express The set of neighboring nodes; "nodes" Its own first Layer features Aggregation results with neighbors Merging is performed by transforming the data using a parameterized mapping function (such as an MLP or multilayer perceptron) to obtain the nodes. The Layer features: ,in, These are learnable parameters (used to adjust the weights of the user's own features and those of its neighbors). Indicates the first A multilayer perceptron is responsible for mapping the merged features to a new dimension. Nodes in a graph isomorphic network... The final output features are: ; in, This indicates the number of layers in a graph isomorphic network.

[0041] In step S7, the inter-region graph attention features and graph isomorphic network features are fused to obtain global and local interaction features, specifically including: The interaction relationship between inter-region graph attention features and graph isomorphic network features is calculated using an attention mechanism; The features are weighted and fused based on the interaction relationship to obtain global and local interaction features.

[0042] Specifically, in this embodiment, the graph attention network's first... Layer node graph attention features ( ) constitute the local feature matrix Nodes in a graph isomorphic network Final output features Constructing the global feature matrix Then calculate the attention between the global and local attention: ; in, , , This represents the learnable parameters. Represents a local region of a superpixel With high-level feature pixels The interaction relationships between them, and the interaction characteristics between local and global aspects are as follows: ,in, yes The OK, yes The OK.

[0043] In step S8, the global and local interaction features and convolutional features are decoded to output the final segmentation result. Specifically, the local and global interaction features... As the highest-level feature, it is related to convolutional features. ( The data are fed together into a segmentation decoder, such as the UNet decoder, to output the segmentation result. Finally, the cross-entropy loss function is used to optimize the entire segmentation network, completing the network training.

[0044] like Figure 2 As shown, this invention provides a pathological image segmentation system based on graph neural networks and cross-region fusion. This system is used to implement the aforementioned pathological image segmentation method based on graph neural networks and cross-region fusion, specifically including: Image acquisition module 100 is used to acquire pathological images to be segmented; The region division module 200 is used to divide the pathological image into multiple image regions. The cross-region pixel-level fusion module 300 is used to perform local cross-fusion of multiple image regions to obtain the local cross-fusion features of each image region. The inter-regional attention module 400 is used to model the inter-regional relationship of local cross-fusion features based on the graph attention mechanism, and obtain inter-regional graph attention features. The encoding module 500 is used to encode pathological images and extract convolutional features; The pixel isomorphic module 600 is used to model the pixel dependencies of convolutional features through a self-attention mechanism, and to perform global structure learning on the modeled features based on the graph isomorphic network to obtain graph isomorphic network features. The global and local fusion module 700 is used to fuse inter-region graph attention features and graph isomorphic network features to obtain global and local interaction features. The decoding module 800 is used to decode global and local interactive features and convolutional features, and output the final segmentation result.

[0045] Specifically, the image acquisition module 100 acquires the pathological image to be segmented. Pathological images The input region segmentation module 200 performs superpixel segmentation to obtain a superpixel segmentation map. Pathological images With superpixel segmentation map The local cross-fusion features are obtained by feeding the cross-region pixel-level fusion module 300 into the module. Local cross-fusion characteristics The inter-region attention module 400 is used to obtain the inter-region graph attention features. Pathological images The data is fed into an encoding module 500, such as a UNet encoder, to extract convolutional features. The graph isomorphic network features are calculated using the pixel isomorphic module 600. Attention features of the inter-region graph Features of graph isomorphic networks The input global and local fusion module 700 generates the interaction features between local and global data. , As the highest-level feature, it is related to convolutional features. Together, they are fed into the decoding module 800, such as the UNet decoder, to output the segmentation result.

[0046] This embodiment provides a pathological image segmentation system based on graph neural networks and cross-regional fusion, used to implement the aforementioned pathological image segmentation method based on graph neural networks and cross-regional fusion. Therefore, the specific implementation of the pathological image segmentation system based on graph neural networks and cross-regional fusion can be found in the previous embodiment section of the pathological image segmentation method based on graph neural networks and cross-regional fusion. For example, the image acquisition module 100, region division module 200, cross-region pixel-level fusion module 300, inter-regional attention module 400, encoding module 500, inter-pixel isomorphism module 600, global and local fusion module 700, and decoding module 800 are respectively used to implement steps S1, S2, S3, S4, S5, S6, S7, and S8 in the aforementioned pathological image segmentation method based on graph neural networks and cross-regional fusion. Therefore, its specific implementation can be referred to the description of the corresponding embodiments. To avoid redundancy, it will not be repeated here.

[0047] To further verify the advantages of the present invention, specific experiments are described below.

[0048] This experiment collected 26,183 pathological image datasets, including regions of five tissue components: Tumor, Stroma, Lymphocytic infiltrate, Necrosis, and Others. The training, validation, and test sets were randomly partitioned in a 7:1:2 ratio. All experiments were conducted using Python 3.8 with PyTorch 1.8 on a single NVIDIA GeForce RTX 4090 GPU with 24GB of RAM. The code for the comparison method was adapted from a publicly available implementation of the experimental setup. The Adam optimizer was used, with a learning rate of [missing value]. To train all models, the training period is 200.

[0049] To quantitatively evaluate the performance of the method of this invention, two mainstream and representative evaluation metrics from segmentation tasks were selected to comprehensively measure the accuracy and completeness of the model in pathological image segmentation. These metrics are: mean Dice Similarity Coefficient (mDSC) and mean Intersection over Union (mIoU).

[0050] Furthermore, the segmentation results of the method of this invention were compared with three representative existing methods published in recent years for the same task (such as tongue segmentation). The specific comparison results are shown in Table 1. All methods were evaluated based on the same test set, and the dataset partitioning, data augmentation strategies, and hardware testing environment were kept consistent to ensure the fairness and reliability of the comparison results. Table 1 clearly lists the mDSC and mIoU values ​​of the method of this invention and each existing method on the test set. It can be clearly seen from the table that the method of this invention significantly outperforms the classical basic method and the domain optimization method in both metrics. Compared with the current state-of-the-art methods, the mDSC is improved by 3.4% to 9.7%, and the mIoU is improved by 6.3% to 12.1%, which fully demonstrates the advantage of the method of this invention in segmentation accuracy.

[0051] Table 1 Comparison of segmentation performance between the method of the present invention and existing methods

[0052] Among them, Existing Method 1 is a domain-incremental weakly supervised state space model for pathology imagesegmentation; Existing Method 2 is a pathological image segmentation of breast cancer via template matching; Existing Method 3 is HADiff: hierarchy aggregated diffusion model for pathology image segmentation.

[0053] like Figure 3The image shows three comparative cases. Column 1 is the pathological image, column 2 is the manual segmentation result, column 3 is the segmentation result of the method of this invention, column 4 is the segmentation result of method 1, column 5 is the segmentation result of method 2, and column 6 is the segmentation result of method 3. Column 3 shows that the segmentation result of the method of this invention is very close to the manual segmentation result. In row 1, column 4, method 1 classifies lymphocyte infiltration and other types of regions as stroma, with some stroma incorrectly classified as blue lymphocyte infiltration. In row 1, column 5, method 2 incorrectly classifies the stroma as blue lymphocyte infiltration, and other types of regions as stroma, with some stroma incorrectly classified as blue lymphocyte infiltration. In row 1, column 6, method 3 classifies lymphocyte infiltration and other types of regions as stroma, with some stroma incorrectly classified as blue lymphocyte infiltration. In row 2, columns 4-6, methods 1-3 fail to correctly segment the boundary between the tumor and the stroma. In row 3, columns 4-5, methods 1 and 2 classify other types of regions as stroma, and some lymphocyte infiltration is also classified as stroma. In row 3, column 6, other types of regions are not classified as stroma, and some stroma is incorrectly classified as tumor.

[0054] Furthermore, to verify the effectiveness of each module proposed in the architecture of this invention, ablation experiments were conducted. As shown in Table 2, compared with the baseline model, the mDSC and mIoU observed in the cross-region pixel-level fusion module and the inter-region attention module are relatively improved, which proves the effectiveness of the features of this invention in the segmentation task. The mDSC and mIoU of the cross-region pixel-level fusion are 3.5 and 6.1 percentage points higher than the baseline, respectively. In addition, the mDSC and mIoU of the inter-region attention module are further improved by 4.5 and 2.5 percentage points, respectively. The mDSC and mIoU of the inter-pixel isomorphic module are further improved by 2.5 and 2.6 percentage points, respectively. In addition, the mDSC and mIoU of the global and local fusion modules are further improved by 2.9 and 2.3 percentage points, respectively. This shows that all models of this invention are effective.

[0055] Table 2 Ablation experiments of the present invention

[0056] In summary, this invention designs a cross-region pixel-level fusion module 300 to perform local cross-fusion of superpixel segmentation regions, enhancing the fusion between local pixels and thus improving the interaction between them. This invention also designs an inter-regional attention module 400, employing a graph attention network to effectively capture the interactions and importance regions of complex and different superpixel segmentation regions, strengthening the fusion between these regions. Furthermore, this invention designs an inter-pixel isomorphism module 600, which captures the dependencies between pixels through a self-attention mechanism, constructs a graph isomorphism network based on the self-attention features between pixels, and achieves global feature learning through "layer-by-layer aggregation of neighbor features + updating its own features." Finally, this invention designs a global and local fusion module 700. The cross-region pixel-level fusion module 300 and the inter-regional attention module 400 calculate the local information interaction, while the inter-pixel isomorphism module 600 calculates the pathological image. The high-level features, which integrate local and global features, improve segmentation performance.

[0057] This invention provides an electronic device, which includes a processor, a memory, and a bus system. The processor and the memory are connected through the bus system. The memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the above-mentioned pathological image segmentation method based on graph neural networks and cross-regional fusion.

[0058] This invention provides a computer storage medium storing a computer software product, which includes several instructions to cause a computer device to execute the aforementioned pathological image segmentation method based on graph neural networks and cross-regional fusion.

[0059] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0060] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0061] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0062] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A pathological image segmentation method based on graph neural networks and cross-region fusion, characterized in that, include: Obtain the pathological image to be segmented; The pathological image is divided into multiple image regions. Local cross-fusion is performed on the multiple image regions to obtain the local cross-fusion features of each image region; Based on the graph attention mechanism, the inter-regional relationship model is performed on the local cross-fusion features to obtain inter-regional graph attention features. Specifically, this includes: taking the local cross-fusion features of each image region as a node in the graph; dynamically calculating the attention weights between nodes using a graph attention network, aggregating the features of neighboring nodes, and updating the feature representation of each node to obtain the inter-regional graph attention features. The pathological images are encoded, and convolutional features are extracted. The convolutional features are modeled using a self-attention mechanism to model inter-pixel dependencies, and the modeled features are then subjected to global structure learning based on a graph isomorphic network to obtain graph isomorphic network features. By fusing the inter-region graph attention features with the graph isomorphic network features, global and local interaction features are obtained. The global and local interaction features are decoded along with the convolutional features to output the final segmentation result.

2. The pathological image segmentation method based on graph neural networks and cross-regional fusion according to claim 1, characterized in that, The process of dividing the pathological image into multiple image regions specifically includes: The pathological image is segmented into superpixels to obtain a superpixel segmentation map, where each superpixel region is considered as an image region.

3. The pathological image segmentation method based on graph neural networks and cross-regional fusion according to claim 2, characterized in that, The step of performing local cross-fusion on the multiple image regions to obtain local cross-fusion features for each image region specifically includes: For each image region, it is interacted with at least one other image region through a cross-attention mechanism to calculate the dependencies between regions. The interacting features are channel-merged and then calculated using a feedforward neural network to obtain the local cross-fusion features.

4. The pathological image segmentation method based on graph neural networks and cross-regional fusion according to claim 1, characterized in that, The step of modeling the inter-pixel dependencies of the convolutional features using a self-attention mechanism specifically includes: Flatten the convolutional features into a matrix form; Self-attention weights between pixels are calculated using a learnable query matrix, key matrix, and value matrix. The pixel features are weighted and fused according to the self-attention weights to obtain the self-attention features between pixels.

5. The pathological image segmentation method based on graph neural networks and cross-regional fusion according to claim 4, characterized in that, The global structure learning of the modeled features based on graph isomorphic networks specifically includes: Use the self-attention features of each pixel as the initial features of a node in the graph; For each node, the features of its neighboring nodes are aggregated layer by layer, and the aggregation result is merged with its own features. The merged features are transformed using a multilayer perceptron to update the node features, thus obtaining the graph isomorphic network features.

6. The pathological image segmentation method based on graph neural networks and cross-regional fusion according to claim 1, characterized in that, The fusion of the inter-region graph attention features and the graph isomorphic network features to obtain global and local interaction features specifically includes: The interaction relationship between the inter-region graph attention features and the graph isomorphic network features is calculated using an attention mechanism; The features are weighted and fused according to the interaction relationship to obtain the global and local interaction features.

7. A pathological image segmentation system based on graph neural networks and cross-regional fusion, characterized in that, The system is used to implement the pathological image segmentation method based on graph neural networks and cross-region fusion as described in any one of claims 1 to 6, specifically including: The image acquisition module is used to acquire pathological images to be segmented; The region segmentation module is used to segment the pathological image into multiple image regions. A cross-region pixel-level fusion module is used to perform local cross-fusion of the multiple image regions to obtain the local cross-fusion features of each image region; The inter-regional attention module is used to model the inter-regional relationships of the local cross-fusion features based on the graph attention mechanism, so as to obtain inter-regional graph attention features. The encoding module is used to encode the pathological images and extract convolutional features; The pixel isomorphic module is used to model the pixel dependencies of the convolutional features through a self-attention mechanism, and to perform global structure learning on the modeled features based on the graph isomorphic network to obtain graph isomorphic network features. The global and local fusion module is used to fuse the inter-region graph attention features and the graph isomorphic network features to obtain global and local interaction features. The decoding module is used to decode the global and local interaction features and the convolutional features, and output the final segmentation result.

8. An electronic device, characterized in that, The electronic device includes a processor, a memory, and a bus system. The processor and the memory are connected through the bus system. The memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the pathological image segmentation method based on graph neural networks and cross-regional fusion as described in any one of claims 1 to 6.

9. A computer storage medium, characterized in that, The computer storage medium stores a computer software product, which includes several instructions to cause a computer device to execute the pathological image segmentation method based on graph neural networks and cross-regional fusion as described in any one of claims 1 to 6.

Images